{"title": "A Mean Field Theory of Layer IV of Visual Cortex and Its Application to Artificial Neural Networks", "book": "Neural Information Processing Systems", "page_first": 683, "page_last": 692, "abstract": null, "full_text": "683 \n\nA MEAN FIELD THEORY OF LAYER IV OF VISUAL CORTEX \n\nAND ITS APPLICATION TO ARTIFICIAL NEURAL NETWORKS* \n\nChristopher L. Scofield \n\nCenter for Neural Science and Physics Department \n\nBrown University \n\nProvidence, Rhode Island 02912 \n\nand \n\nNestor, Inc., 1 Richmond Square, Providence, Rhode Island, \n\n02906. \n\nABSTRACT \n\nA single cell theory for the development of selectivity and \nocular dominance in visual cortex has been presented previously \nby Bienenstock, Cooper and Munrol. This has been extended to a \nnetwork applicable to layer IV of visual cortex2 . \nIn this paper \nwe present a mean field approximation that captures in a fairly \ntransparent manner \nthe \nquantitative, results of the network theory. Finally, we consider \nthe application of this theory to artificial neural networks and \nshow that a significant reduction in architectural complexity is \npossible. \n\nthe qualitative, \n\nand many of \n\nA SINGLE LAYER NETWORK AND THE MEAN FIELD \n\nAPPROXIMATION \n\nWe consider a \nreceive signals from \nthe layer (Figure 1). \n\nsingle layer network of ideal neurons which \noutside of the layer and from cells within \nThe activity of the ith cell in the network is \n\nc' - m' d + ~ T .. c' \n\"\"' ~J J' \n1 -\nJ \n\n1 \n\n(1) \n\nthen \n\nto \n\nthe network. \n\nEach cell \noutside of the cortical network \nmi' \nIntra-layer input to each cell \nthe matrix of cortico-cortical \n\nis a vector of afferent signals \n\nd \nreceives \ninput from n fibers \nthrough the matrix of synapses \nis \nthrough \nsynapses L. \n\ntransmitted \n\n\u00a9 American Institute of Physics 1988 \n\n\f-~ \n\nr;. \n\n\",...-\n\n684 \n\nAfferent > \n\nSignals \n\nd \n\nm 1 m2 \n\n... . . \n\nmn \n\n. \n: L \n\n1 \n\n2 \n\n... .. ~ \n\n,~ , ... .. , ~ \n\nc \n\nFigure 1: The general single layer recurrent \nLight circles are the LGN -cortical \nnetwork. \nsynapses. \n(non-\nmodifiable) cortico-cortical synapses. \n\nDark circles \n\nthe \n\nare \n\nWe now expand the response of the ith cell into individual \nterms describing the number of cortical synapses traversed by \nat cell i. \nthe signal d before arriving \nExpanding Cj in (1), the response of cell i becomes \nci = mi d + l: ~j mj d + l: ~jL Ljk mk d + 2: ~j 2Ljk L Lkn mn d + ... (2) \n\nthrough synapses Lij \n\nJ K' n \n\nJ \n\nJ K \n\nNote that each term contains a factor of the form \n\nThis factor describes the first order effect, on cell q, of the \ncortical \nfield \napproximation consists of estimating this factor to be a constant, \nindependant of cell location \n\ntransformation of \n\nthe signal d. \n\nThe mean \n\n(3) \n\n\fThis assumption does not imply that each cell in the network is \nselective to the same pattern, (and thus that mi = mj). Rather, \nthe assumption is that the vector sum is a constant \n\n685 \n\nto assuming \n\nThis amounts \nthe network is \nsurrounded by a population of cells which represent, on average, \nall possible pattern preferences. \nThus the vector sum of the \nafferent synaptic states describing these pattern preferences is a \nconstant independent of location. \n\nthat each cell \n\nin \n\nFinally, if we assume that the lateral connection strengths are \n\na function only of i-j then Lij becomes a circular matrix so that \n\nr. Lij ::: ~ Lji = Lo = constan t. \n\n1 \n\nJ \n\nThen the response of the cell i becomes \n\n(4) \n\nfor I ~ I < 1 \n\nwhere we define the spatial average of cortical cell activity C = in \nd, and N is the average number of intracortical synapses. \n\nHere, in a manner similar to that in the theory of magnetism, \nwe have replaced the effect of individual cortical cells by their \naverage effect (as though all other cortical cells can be replaced \nby an 'effective' cell, figure 2). Note that we have retained all \norders of synaptic traversal of the signal d. \n\nThus, we now focus on \n\nthe activity of the layer after \nIn the mean field approximation we \n\n'relaxation' to equilibrium. \ncan therefore write \n\nwhere the mean field \n\nwith \n\na =am \n\n(5) \n\n\f686 \n\nand we asume \ninhibitory). \n\nthat \n\nLo < 0 \n\n(the network \n\nis, on average, \n\nAfferent > \n\nSignals \n\nd \n\nFigure 2: The single layer mean field network. \nDetailed connectivity between all cells of the \nnetwork \nmodifiable) synapse from an 'effective' cell. \n\nreplaced with a \n\nis \n\nsingle \n\n(non(cid:173)\n\nLEARNING IN THE CORTICAL NETWORK \n\nWe will first consider evolution of the network according to a \nthat has been studied in detail, for \nsynaptic modification rule \nsingle cells, elsewhere!\u00b7 3. We consider the LGN - cortical \nsynapses to be the site of plasticity and assume for maximum \nsimplicity \nis no modification of cortico-cortical \nthat \nsynapses. Then \n\nthere \n\n. \nLij = O. \n\n(6) \n\nIn what follows c denotes the spatial average over cortical cells, \nwhile Cj denotes the time averaged activity of the ith cortical cell. \nThe function cj> has been discussed extensively elsewhere. Here \nwe note that cj> describes a function of the cell response that has \nboth hebbian and anti-hebbian regions. \n\n\f687 \n\nthat have been analyzed partially elsewhere2 . \n\nThis leads to a very complex set of non-linear stochastic \nIn \nthat are \n\nequations \ngeneral, \nstable and selective and unstable fixed points \nselective!, 2. These arguments may now be generalized for the \nnetwork. \n\nIn the mean field approximation \n\nthe afferent synaptic state has fixed points \n\nthat are non(cid:173)\n\n(7) \n\ntime dependent component m. This \nThe mean field, a has a \nvaries as \nthe network modifiable \nsynapses and, in most environmental situations, should change \nslowly compared to the change of the modifiable synapses to a \nsingle cell. Then in this approximation we can write \n\nthe average over all of \n\n\u2022 \n\n(mi(a)-a) = cj>[mi(a) - a] d. \n\nWe see that there is a mapping \n\nmi' <-> mica) - a \n\n(8) \n\n(9) \n\nsuch that for every mj(a) there exists a corresponding (mapped) \npoint mj' which satisfies \n\nthe original equation for the mean field zero theory. \nIt can be \nshown 2, 4 that for every fixed point of mj( a = 0), there exists a \ncorresponding fixed point mj( a) with the same selectivity and \nstability properties. \nthe \nthe network (ILo I is \nneurons if there is sufficient inhibition in \nsufficiently large). \n\nThe fixed points are available \n\nto \n\nAPPLICATION OF THE MEAN FIELD NETWORK TO \n\nLAYER IV OF VISUAL CORTEX \n\nNeurons in the primary visual cortex of normal adult cats are \nsharply tuned for the orientation of an elongated slit of light and \nmost are activated by stimulation of either eye. Both of these \nproperties--orientation selectivity and binocularity--depend on \ntype of visual environment experienced during a critical \nthe \n\n\f688 \n\nperiod of early postnatal development. For example, deprivation \nof patterned input during this critical period leads to \nloss of \norientation selectivity while monocular deprivation (MD) results \nin a dramatic shift in the ocular dominance of cortical neurons \nsuch that most will be responsive exclusively to the open eye. \nThe ocular dominance shift after MD is the best known and most \nintensively studied type of visual cortical plasticity. \n\nthat some cells respond more rapidly \n\nThe behavior of visual cortical cells in various rearing \nto \nconditions suggests \nIn monocular deprivation, \nenvironmental changes than others. \nfor example, some cells remain responsive to the closed eye in \nspite of the very large shift of most cells to the open eye- Singer \net. al.5 found, using intracellular recording, that geniculo-cortical \nsynapses on \nto \nmonocular deprivation \nthan are synapses on pyramidal cell \ndendrites. Recent work suggests that the density of inhibitory \nGABAergic synapses in kitten striate cortex is also unaffected by \nMD during the cortical period 6, 7. \n\ninterneurons are more \n\ninhibitory \n\nresistant \n\nThese results suggest that some LGN -cortical synapses modify \nslow \n\nrapidly, while others modify \nmodification of \ncortical synapses into excitatory cells may be those that modify \nprimarily. To embody these facts we introduce two types of \nLGN -cortical synapses: \nthose (mj) that modify and those (Zk) \nthat remain relatively constant. \n\nsome cortico-cortical synapses. Excitatory LGN(cid:173)\n\nIn a simple limit we have \n\nrelatively slowly, with \n\nand \n\n(10) \n\nWe assume for simplicity and consistent with the above \nphysiological interpretation that these two types of synapses are \nconfined to two different classes of cells and that both left and \nright eye have similar synapses (both m i or both Zk) on a given \ncell. Then, for binocular cells, in the mean field approximation \n(where binocular terms are in italics) \n\n\f689 \n\nwhere dl(r) are the explicit left (right) eye time averaged signals \narriving form \nterms from \nmodifiable and non-modifiable synapses: \n\nthe LGN. Note that a1(r) contain \n\nal(r) = a (ml(r) + zl(r\u00bb). \n\nUnder conditions of monocular deprivation, the animal is reared \nwith one eye closed. For the sake of analysis assume that the \nright eye is closed and \nthat only noise-like signals arrive at \ncortex from the right eye. Then the environment of the cortical \ncells is: \n\nd = (di, n) \n\n(12) \n\nFurther, assume \nthat the left eye synapses have reached their \nselective fixed point, selective to pattern d 1 \u2022 Then (mi' mi ) = \n\n1 \n\nr \n\n(m:*, xi) with IXil \u00ablm!*1. Following the methods of BCM, a local \n\nlinear analysis of the * -\nthe closed eye \n\nfunction is employed to show that for \n\nXi = a (1 - }..a)-li.r. \n\n(13) \nwhere A. = NmIN is the ratio of the number modifiable cells to the \ntotal number of cells in the network. That is, the asymptotic \nstate of the closed eye synapses is a scaled function of the mean(cid:173)\nfield due to non-modifiable (inhibitory) \ncortical cells. The scale \nof this state is set not only by the proportion of non-modifiable \ncells, but \nintracortical synaptic \nstrength Lo. \n\nin addition, by \n\nthe averaged \n\nto zero. \n\nThus contrasted with the mean field zero theory the deprived \neye LGN-cortical synapses do not go \nRather they \napproach the constant value dependent on the average inhibition \nproduced by \nthe non-modifiable cells in such a way that the \nasymptotic output of the cortical cell is zero (it cannot be driven \nby the deprived eye). However lessening the effect of inhibitory \nsynapses (e.g. by application of an inhibitory blocking agent such \nas bicuculine) reduces the magnitude of a so that one could once \nmore obtain a response from the deprived eye. \n\n\f690 \n\nWe find, consistent with previous theory and experiment, \nthe LGN-cortical synapse, for \nSome \n\nthat most learning can occur in \ninhibitory (cortico-cortical) synapses need not modify. \nnon-modifiable LGN-cortical synapses are required. \n\nTHE MEAN FIELD APPROXIMATION AND \n\nARTIFICIAL NEURAL NETWORKS \n\nThe mean field approximation may be applied to networks in \nwhich the cortico-cortical feedback is a general function of cell \nactivity. \nIn particular, the feedback may measure the difference \nbetween the network activity and memories of network activity. \nIn this way, a network may be used as a content addressable \nmemory. We have been discussing the properties of a mean \nfield network after equilibrium has been reached. We now focus \non the detailed time dependence of the relaxation of the cell \nactivity to a state of equilibrium. \n\n'relaxes' \n\nHopfield8 introduced a simple formalism for the analysis of \ntime dependence of network activity. \nthis model, \nthe \nnetwork activity is mapped onto a physical system in which the \nstate of neuron activity is considered as a 'particle' on a potential \nenergy surface. \nIdentification of the pattern occurs when the \nactivity \nThus \nto a nearby minima of the energy. \nmlmma are employed as the sites of memories. For a Hopfield \nnetwork of N neurons, the intra-layer connectivity required is of \norder N2. This connectivity is a significant constraint on the \npractical \nscale \nproblems. Further, the Hopfield model allows a storage capacity \nwhich is limited to m < N memories8, 9. This is a result of the \nproliferation of unwanted local minima in the 'energy' surface. \n\nimplementation of such \n\nsystems \n\nlarge \n\nIn \n\nfor \n\nthe sites of negative \n\nRecently, Bachmann et al. l 0, have proposed a model for the \nin which memories of activity \nrelaxation of network activity \npatterns are \nthe activity \ncaused by a test pattern is a positive test 'charge'. Then in this \nmodel, \nthe electrostatic energy of the \n(unit) test charge with the collection of charges at the memory \nsites \n\nthe energy function \n\n'charges', and \n\nis \n\nE = -IlL ~ Qj I J-l- Xj I - L, \n\nJ \n\n(14) \n\n\f691 \n\nwhere Jl (0) is a vector describing the initial network activity \ncaused by a test pattern, and Xj' the site of the jth memory. L is \na parameter related to the network size. \n\nThis model has the advantage that storage density is not \nrestricted by the the network size as it is in the Hopfield model, \nand in addition, the architecture employs a connectivity of order \nm x N. Note that at each stage in the settling of Jl (t) to a memory \n(of network activity) Xj' the only feedback from the network to \neach cell is the scalar \n\n~ Q. I Jl- X\u00b7 I - L \nJ \n\nJ \n\nJ \n\n(15) \n\nthe distance \n\nThis quantity is an integrated measure of the distance of the \ncurrent network state from stored memories. \nImportantly, this \nmeasure is the same for all cells; it is as if a single virtual cell \nwas computing \nthe \ncurrent state and stored states. The result of the computation is \nthen broadcast to all of the cells in the network. \nThis is a \ngeneralization of the idea that the detailed activity of each cell in \nthe network need not be fed back to each cell. Rather some \nglobal measure, performed by a single 'effective' cell is all that is \nsufficient in the feedback. \n\nin activity space between \n\nDISCUSSION \n\nformalism for \n\nWe have been discussing a \nthe analysis of \nnetworks of ideal neurons based on a mean field approximation \nof the detailed activity of the cells in the network. We find that \na simple assumption concerning the spatial distribution of the \npattern preferences of the cells allows a great simplification of \nthe analysis. \nIn particular, the detailed activity of the cells of \nthe network may be replaced with a mean field that in effect is \ncomputed by a single 'effective' cell. \n\nFurther, the application of this formalism to the cortical layer \nIV of visual cortex allows the prediction that much of learning in \ncortex may be localized to the LGN-cortical synaptic states, and \nthat cortico-cortical plasticity is relatively unimportant. We find, \nin agreement with experiment, \nthat monocular deprivation of \nthe cortical cells will drive closed-eye responses to zero, but \nchemical blockage of \ninhibitory pathways would \nreveal non-zero closed-eye synaptic states. \n\nthe cortical \n\n\f692 \n\nFinally, the mean field approximation allows the development \nof single layer models of memory storage that are unrestricted \nin storage density, but require a connectivity of order mxN. This \nis significant for the fabrication of practical content addressable \nmemories. \n\nACKNOWLEOOEMENTS \n\nI would like to thank Leon Cooper for many helpful discussions \nand the contributions he made to this work. \n\n*This work was supported by the Office of Naval Research and \nthe Army Research Office under contracts #NOOOI4-86-K-0041 \nand #DAAG-29-84-K-0202. \n\nREFERENCES \n\nSelectivity and Nervous \n\n[1] Bienenstock, E. L., Cooper, L. N & Munro, P. W. (1982) 1. \nNeuroscience 2, 32-48. \n[2] Scofield, C. L. (I984) Unpublished Dissertation. \n[3] Cooper, L. N, Munro, P. W. & Scofield, C. L. (1985) in Synaptic \nModification, Neuron \nSystem \nOrganization, ed. C. Levy, J. A. Anderson & S. Lehmkuhle, \n(Erlbaum Assoc., N. J.). \n[4] Cooper, L. N & Scofield, C. L. (to be published) Proc. Natl. Acad. \nSci. USA .. \n[5] Singer, W. (1977) Brain Res. 134, 508-000. \n[6] Bear, M. F., Schmechel D. M., & Ebner, F. F. (1985) 1. Neurosci. \n5, 1262-0000. \n[7] Mower, G. D., White, W. F., & Rustad, R. (1986) Brain Res. 380, \n253-000. \n[8] Hopfield, J. J. (1982) Proc. Natl. A cad. Sci. USA 79, 2554-2558. \n[9] Hopfield, J. J., Feinstein, D. 1., & Palmer, R. O. (1983) Nature \n304, 158-159. \n[10] Bachmann, C. M., Cooper, L. N, Dembo, A. & Zeitouni, O. (to be \npublished) Proc. Natl. Acad. Sci. USA. \n\n\f", "award": [], "sourceid": 10, "authors": [{"given_name": "Christopher", "family_name": "Scofield", "institution": null}]}*