{"title": "Spatial Organization of Neural Networks: A Probabilistic Modeling Approach", "book": "Neural Information Processing Systems", "page_first": 740, "page_last": 749, "abstract": null, "full_text": "740 \n\nSPATIAL ORGANIZATION OF NEURAL NEn~ORKS: \n\nA PROBABILISTIC MODELING APPROACH \n\nA. Stafylopatis \nM. Dikaiakos \nD. Kontoravdis \n\nNational Technical University of Athens, Department of Electri(cid:173)\ncal Engineering, Computer Science Division, 15773 Zographos, \nAthens, Greece. \n\nABSTRACT \n\nThe aim of this paper is to explore the spatial organization of \nneural networks under Markovian assumptions, in what concerns the be(cid:173)\nhaviour of individual cells and the interconnection mechanism. Space(cid:173)\norganizational properties of neural nets are very relevant in image \nmodeling and pattern analysis, where spatial computations on stocha(cid:173)\nstic two-dimensional image fields are involved. As a first approach \nwe develop a random neural network model, based upon simple probabi(cid:173)\nlistic assumptions, whose organization is studied by means of dis(cid:173)\ncrete-event simulation. We then investigate the possibility of ap(cid:173)\nproXimating the random network's behaviour by using an analytical ap(cid:173)\nproach originating from the theory of general product-form queueing \nnetworks. The neural network is described by an open network of no(cid:173)\ndes, in which customers moving from node to node represent stimula(cid:173)\ntions and connections between nodes are expressed in terms of sui(cid:173)\ntably selected routing probabilities. We obtain the solution of the \nmodel under different disciplines affecting the time spent by a sti(cid:173)\nmulation at each node visited. Results concerning the distribution \nof excitation in the network as a function of network topology and \nexternal stimulation arrival pattern are compared with measures ob(cid:173)\ntained from the simulation and validate the approach followed. \n\nINTRODUCTION \n\nNeural net models have been studied for many years in an attempt \nto achieve brain-like performance in computing systems. These models \nare composed of a large number of interconnected computational ele(cid:173)\nments and their structure reflects our present understanding of the \norganizing principles of biological nervous systems. \nIn the begin(cid:173)\ning, neural nets, or other equivalent models, were rather intended \nto represent the logic arising in certain situations than to provide \nan accurate description in a realistic context. However, in the last \ndecade or so the knowledge of what goes on in the brain has increased \ntremendously. New discoveries in natural systems, make it now rea(cid:173)\nsonable to examine the possibilities of using modern technology in \norder to synthesige systems that have some of the properties of real \nneural systems 8,9,10,11. \n\nIn the original neural net model developed in 1943 by McCulloch \n\nand Pitts 1,2 the network is made of many interacting components, \nknown as the \"McCulloch-Pitts cells\" or \"formal neurons II , which are \nsimple logical units with two possible states changing state accord-\n\n\u00ae American Institute of Physics 1988 \n\n\f741 \n\ning to a threshold function of their inputs. Related automata models \nhave been used later for gene control systems (genetic networks) 3, \nin which genes are represented as binary automata changing state ac(cid:173)\ncording to boolean functions of their inputs. Boolean networks con(cid:173)\nstitute a more general model, whose dynamical behaviour has been stu(cid:173)\ndied extensively. Due to the large number of elements, the exact \nstructure of the connections and the functions of individual compo(cid:173)\nnents are generally unknown and assumed to be distributed at random. \nSeveral studies on these random boolean networks 5,6 have shown that \nthey exhibit a surprisingly stable behaviour in what concerns their \ntemporal and spatial organization. However, very few formal analyti(cid:173)\ncal results are available, since most studies concern statistical \ndescriptions and computer simulations. \n\nThe temporal and spatial organization of random boolean networks \nis of particular interest in the attempt of understanding the proper(cid:173)\nties of such systems, and models originating from the theory of sto(cid:173)\nchastic processes 13 seem to be very useful. Spatial properties of \nneural nets are most important in the field of image recognition 12. \nIn the biological eye, a level-normalization computation is performed \nby the layer of horizontal cells, which are fed by the immediately \npreceding layer of photoreceptors. The horizontal cells take the \noutputs of the receptors and average them spatially, this average \nbeing weighted on a nearest-neighbor basis. This procedure corres(cid:173)\nponds to a mechanism for determining the brightness level of pixels \nin an image field by using an array of processing elements. The \nprinciple of local computation is usually adopted in models used for \nrepresenting and generating textured images. Among the stochastic \nmodels applied to analyzing the parameters of image fields, the ran(cid:173)\ndom Markov field model 7,14 seems to give a suitably structured re(cid:173)\npresentation, which is mainly due to the application of the marko(cid:173)\nvian property in space. This type of modeling constitutes a promi(cid:173)\nsing alternative in the study of spatial organization phenomena in \nneura 1 nets. \n\nThe approach taken in this paper aims to investigate some as(cid:173)\npects of spatial organization under simple stochastic assumptions. \nIn the next section we develop a model for random neural networks \nassuming boolean operation of individual cells. The behaviour of \nthis model, obtained through simulation experiments, is then appro(cid:173)\nximated by using techniques from the theory of queueing networks. \nThe approximation yields quite interesting results as illustrated by \nvarious examples. \n\nTHE RANDOM NETWORK MODEL \n\nWe define a random neural network as a set of elements or cells, \n\neach one of which can be in one of two different states: firing or \nquiet. Cells are interconnected to form an NxN grid, where each grid \npoint is occupied by a cell. We shall consider only connections be(cid:173)\ntween neighbors, so that each cell is connected to 4 among the other \ncells: two input and two output cells \n(the output of a cell is equal \n'to its internal state and it is sent to its output cells which use \n;it as one of their inputs). The network topology is thus specified \n\n\f742 \n\nby its incidence matrix A of dimension MxM, where M=N2. This matrix \ntakes a simple form in the case of neighbor-connection considered \nhere. We further assume a periodic structure of connections in what \nconcerns inputs and outputs; we will be interested in the following \ntwo types of networks depending upon the period of reproduction for \nelementary square modules 5, as shown in Fig.l: \n- Propagative nets (Period 1) \n- Looping nets \n(Period 2) \n\n.... \n\n-\n\n\"' \n\n\"';> \n\n, \n\n\\ \nI \n\n(a) \n\n(b) \n\n-\n\n--\n\n- '\"' \n\nFig.1. (a) Propagative connections, (b) Looping connections \n\nAt the edges of the grid, circular connections are established (mo(cid:173)\ndulo N), so that the network can be viewed as supported by a torus. \nThe operation of tile network is non-autonomous: changes of sta(cid:173)\nte are determined by both the interaction among cells and the influ(cid:173)\nence of external stimulations. We assume that stimulations arrive \nfrom the outside world according to a Poisson process with parameter \nA. Each arriving stimulation is associated with exactly one cell of \nthe network; the cell concerned is determined by means of a given \ndiscrete probability distribution qi (l~i~M), considering an one-di(cid:173)\nmensional labeling of the M cells. \n\nThe operation of each individual cell is asynchronous and can be \n\ndescribed in terms of the following rules: \n- A quiet cell moves to the firing state if it receives an arriving \nstimulation or if a boolean function of its inputs becomes true. \n\n- A firing cell moves to the quiet state if a boolean function of its \n\ninputs becomes false. \n\n- Changes of state imply a reaction delay of the cell concerned; the(cid:173)\nse delays are independent identically distributed random variables \nfollowing a negative exponential distribution with parameter y. \n\nAccording to these rules, the operation of a cell can be viewed as il(cid:173)\nlustrated by Fig.2, where the horizontal axis represents time and the \nnumbers 0,1,2 and 3 represent phases of an operation cycle. Phases 1 \nand 3, as indicated in Fig.2, correspond to reaction delays. In this \nsense, the qui et and fi ri ng s ta tes, as defi ned in the begi ni ng of thi s \nsection, represent the aggregates of phases 0,1 and 2,3 respectively. \nExternal stimulations affect the receiving cell only when it is in pha(cid:173)\nse 0; otherwise we consider that the stimulation is lost. \nIn the sa(cid:173)\nme way, we assume tha t changes of the value of the input boo 1 ean func(cid:173)\ntion do not affect the operation of the cell during phases land 3. The \nconditions are checked only at the end of the respective reaction delay. \n\n\f743 \n\nstate \n\nquiet I firing state \n\n0 \n\n~ /r 2 ~ / \n\n0 \n\nFig.2. Changes of state for individual cells \n\nThe above defi ned model i ncl udes some fea tures of the ori gi na 1 \nMcCulloch-Pitts cells 1,2. \nIn fact, it represents an asynchronous \ncounterpart of the latter, in which boolean functions are considered \ninstead of threshold functions. However, it can be shown that any \nMcCulloch and Pitts' neural network can be implemented by a boolean \nnetwork designed in an appropriate fashion 5. In what follows, we will \nconsider that the firing condition for each individual cell is de(cid:173)\ntermined by an \"or\" function of its inputs. \n\nUnder the assumptions adopted, the evolution of the network in \ntime can be described by a conti nuous-parameter Markov process. How(cid:173)\never, the size of the state-space and the complexity of the system \nare such that no analytical solution is tractable. The spatial orga(cid:173)\nnization of the network could be expressed in terms of the steady(cid:173)\nstate probability distribution for the Markov process. A more useful \nrepresentation is provided by the marginal probability distributions \nfor all cells in the network, or equivalently by the probability of \nbeing in the firing state for each cell. This measure expresses the \nlevel of excitation for each point in the grid. \n\nWe have studied the behaviour of the above model by means of si(cid:173)\nmulation experiments for various cases depending upon the network si(cid:173)\nze, the connection type, the distribution of external stimulation ar(cid:173)\nrivals on the grid and the parameters A and V. Some examples are il(cid:173)\nlustrated in the last section, in comparison with results obtained \nusing the approach discussed in the next section. The estimations ob(cid:173)\nta i ned concern the probabil i ty of bei ng in the fi ri ng s ta te for all \ncells in the network. The simulation was implemented according to \nthe \"batched means\" method; each run was carried out unti 1 the width \nof the 95% confidence interval was less that 10% of the estimated \nmean value for each cell, or until a maximum number of events had \nbeen simulated depending upon the size of the network. \n\nTHE ANALYTICAL APPROACH \n\nThe neural network model considered in the previous section exhi(cid:173)\nbited the markovian property in both time and space. Markovianity in \nspace, expressed by the principle of \"neighbor-connections\", is the \nbasic feature of Markov random fields 7,14, as already discussed. Our \nidea is to attempt an approximation of the random neural network mo(cid:173)\ndel by usi ng a well-known model, wlli ch is markovi an in time, and ap(cid:173)\nplying the constraint of markovianity in space. The model considered \nis an open queueing network, which belongs to the general class of \nqueueing networks admitting a product-form solution 4. \nIn fact, one \ncould distinguish several common features in the two network models. \n\n\f744 \n\nA neural network, in general, receives information in the form of ex(cid:173)\nternal stimulation signals and performs some computation on this in(cid:173)\nformation, which is represented by changes of its state. The opera(cid:173)\ntion of the network can be viewed as a flow of excitement among the \ncells and the spatial distribution of this excitement represents the \nresponse of the network to the information received. This kind of ope(cid:173)\nration is particularly relevant in the processing of image fields. On \nthe other hand, in queueing networks, composed of a number of service \nstation nodes, customers arrive from the outside world and spend some \ntime in the network, during which they more from node to node, wait(cid:173)\ning and receiving service at each node visited. Following the exter(cid:173)\nnal arrival pattern, the interconnection of nodes and the other net(cid:173)\nwork parameters, the operation of the network is characterized by a \ndistribution of activity among the nodes. \n\nLet us now consider a queueing network, where nodes represent \ncells and customers represent stimulations moving from cell to cell \nfollowing the topology of the network. Our aim is to define the net(cid:173)\nwork's characteristics in a way to match those of the neural net mo(cid:173)\ndel as much as possible. Our queueing network model is completely \nspecified by the following assumptions: \n- The network is composed of M=N2 nodes arranged on an NxN rectangu(cid:173)\nlar grid, as in the previous case. Interconnections are expressed by \nmeans of a matrix R of routing probabilities: rij (l~i,j~) repre(cid:173)\nsents the probability that a stimulation (customer) leaving node i \nwill next visit node j. Since it is an open network, after visiting an \narbitrary number of cells, stimulations may eventually leave the net(cid:173)\nwork. Let riO denote the probability of leaving the network upon lea(cid:173)\nving node i. In what follows, we will assume that riO=s for all node's. \nIn what concerns the routing probauilities rij, they are determined \nby the two interconnection schemata considered in the previous sec(cid:173)\ntion (propagative and looping connections): each node i has two out(cid:173)\nput nodes j, for which the routing probabilities are equally distri(cid:173)\nbuted. Thus, rij=(1-s)/2 for the two output nodes of i and equal to \nzero for a 11 0 ther nodes in the network. \n- External stimulation arrivals follow a Poisson process with parame(cid:173)\nter A and are routed to the nodes according to the probability dis(cid:173)\ntribution qi (l~iO}=1-p.(O) = \n\n1 \n\n1 \n\n1 Pi \n1-e \n\n_po \n1 \n\n(Type 1) \n\n(Type 2) \n\n(5 ) \n\nThe variation in space of the above quantity will be studied with res(cid:173)\npect to the corresponding measure obtained from simulation experi-\nments for the neural network model. \n\n. \n\nNUMERICAL AND SIMULATION EXAMPLES \n\nSimulations and numerical solutions of the queueing network mo(cid:173)\ntiel were run for different values of the parameters. The network si(cid:173)\nzes considered are relatively small but can provide useful informa(cid:173)\ntion on the spatial organization of the networks. For both types of \nservice discipline discussed in the previous section, the approach \nfollowed yields a very good approximation of the network's organiza(cid:173)\ntion in most regions of the rectangular grid. The choice of the pro(cid:173)\nbability s of leaving the network plays a critical role in the beha-\n\n\f746 \n\n(a) \n\n(b) \n\nFig.3. A 10xiO network with A=l, V=l and propagative connections. \nExternal stimulations are uniformly distributed over a 3x3 square \non the upper left corner of the grid. (a) simulation (b) Queueing \nnetwork approach with s=0.05 and type 2 nodes. \n\n(a) \n\n(b) \n\nFig.4. The network of Fig.3 with A=2 (a) Simulation (b) Queueing \nnetwork approach with s=0.08 and type 2 nodes. \n\nviour of the queueing model,and must have a non-zero value in order \nfor the network to be stable. Good results are obtained for very \nsmall values of s; in fact, this parameter represents the phenomenon \nof excitation being \"lost\" somewhere in the network. Graphical re(cid:173)\npresentations for various cases are shown in Figures 3-7. We have \nused a coloring of five \"grey levels\", defined by dividing into five \nsegments the interval between the smallest and the largest value of \nthe probability on the grid; the normalization is performed with res(cid:173)\npect to simulation results. This type of representation is less ac(cid:173)\ncurate than directly providing numerical values, but is more clear \nfor describing the organization of the system. \nIn each case, the \nresults shown for the queueing model concern only one type of nodes, \nthe one that best fits the simulation results, which is type 2 in \nthe majority of cases examined. The graphical representation illu(cid:173)\nstrates the structuring of the distribution of excitation on the \ngrid in terms of functionally connected regions of high and low \n\n\f747 \n\n(a) \n\n(b) \n\nFig.5. A 10xl0 network with A=l, V=l and looping connections. \nExternal stimulations are uniformly distributed over a 4x4 \nsquare on the center of the grid. \nnetwork approach wi th s= 0.07 and type 2 nodes. \n\n(a) Simulation (b) Queueing \n\n(a) \n\n(b) \n\nFig.6. The network of Fig.5 with A=0.5 (a) Simulation (b) Queue(cid:173)\ni ng network approach wi th s= 0.03 and type 2 nodes. \n\nexcitation. We notice that clustering of nodes mainly follows the \nspatial distribution of external stimulations and is more sharply \nstructured in the case of looping connections. \n\nCONCLUSION \n\nWe have developed in this paper a simple continuous-time pro(cid:173)\n\nbabilistic model of neural nets in an attempt to investigate their \nspatial organization. The model incorporates some of the main fea(cid:173)\ntures of the McCulloch-Pitts \"formal neurons\" model and assumes boo(cid:173)\nlean operation of the elementary cells. The steady-state behaviour \nof the model was approximated by means of a queueing network model \nwith suitably chosen parameters. Results obtained from the solution \nof the above approximation were compared with simulation results of \nthe initial model, which validate the approximation. This simpli(cid:173)\nfied approach is a first step in an attempt to study the organiza-\n\n\f748 \n\n(a) \n\n(b) \n\nFig.7. A 16x16 network with A=1, V=1 and looping connections. \nExternal stimulations are uniformly distributed over two 4x4 \nsquares on the upper left and lower right corners of the grid. \n(a) Simulation (b) Queueing network approach with s=0.05 and \ntype 1 nodes. \n\n\ftional properties of neural nets by means of markovian modeling te(cid:173)\nchn; ques. \n\nREFERENCES \n\n1. W. S. McCulloch, W. Pitts, \"A Logical Calculus of the Ideas Im(cid:173)\nmanent in Nervous Activity\", Bull. of Math. Biophysics 5, 115-\n133 (1943). \n\n2. M. L. Minsky, Computation: Finite and Infinite Machines (Pren(cid:173)\n\ntice Hall, 1967). \n\n13. E. Gelenbe, A. Stafylopatis, \"Temporal Behaviour of Neural Net(cid:173)\nworks\", IEEE First Intern. Conf. on Neural Networks, San Diego, \nCA (June 1987). \n\n14. L. Onural, \"A Systematic Procedure to Generate Connected Binary \nFractal Patterns with Resolution-varying Texture\", Sec. Intern. \nSympt. on Compo and Inform. Sciences, Istanbul, Turkey (Oct. \n1987) . \n\n3. S. Kauffman, \"Behaviour of Randomly Constructed Genetic Nets\", \n\nin Towards a Theoretical Biology, Ed. C. H. Waddington (Edin(cid:173)\nburgh University Press, 1970). \n\n4. F. Baskett, K. M. Chandy, R. R. Muntz, F. G. Palacios, \"Open, \nClosed and Mixed Networks of Queues with Different Classes of \nCustomers\", J. ACM, 22 (1975). \n\n5. H. Atlan, F. Fogelman-Soulie, J. Salomon, G. Weisbuch, \"Random \n\nBoolean Networks\", Cyb. and Syst. 12 (1981). \n\n6. F. Folgeman-Soulie, E. Goles-Chacc, G. Weisbuch, \"Specific Roles \nof the Different Boolean Mappings in Random Networks\", Bull. of \nMath. Biology, Vol.44, No 5 (1982). \n\n7. G. R. Cross, A. K. Jain, \"Markov Random Field Texture Models\", \n\nIEEE Trans. on PAMI, Vol. PAMI-5, No 1 (1983). \n\n8. E. R. Kandel, J. H. Schwartz, Principles of Neural Science, \n\n(Elsevier, N.Y., 1985). \n\n9. J. J. Hopfield, D. W. Tank, \"Computing with Neural Circuits: \n\nA Model\", Sc i ence, Vol. 233, 625-633 (1986). \n\n10. Y. S. Abu-Mostafa, D. Psaltis, \"Optical Neural Computers\", \n\nScient. Amer., 256, 88-95 (1987). \n\n11. R. P. Lippmann, \"An Introduction to Computing with Neural Nets\", \n\nIEEE ASSP Mag. (Apr. 1987). \n\n12. C. A. Mead, \"Neural Hardware for Vision\", Eng. and Scie. (June \n\n1987) . \n\n\f", "award": [], "sourceid": 69, "authors": [{"given_name": "Andreas", "family_name": "Stafylopatis", "institution": null}, {"given_name": "Marios", "family_name": "Dikaiakos", "institution": null}, {"given_name": "D.", "family_name": "Kontoravdis", "institution": null}]}