{"title": "Associative Decorrelation Dynamics: A Theory of Self-Organization and Optimization in Feedback Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 925, "page_last": 932, "abstract": null, "full_text": "Associative Decorrelation Dynamics: \nA Theory of Self-Organization and \nOptimization in Feedback Networks \n\nDawei W. Dong* \n\nLawrence Berkeley Laboratory \n\nUniversity of California \n\nBerkeley, CA 94720 \n\nAbstract \n\nThis paper outlines a dynamic theory of development and adap(cid:173)\ntation in neural networks with feedback connections. Given in(cid:173)\nput ensemble, the connections change in strength according to an \nassociative learning rule and approach a stable state where the \nneuronal outputs are decorrelated . We apply this theory to pri(cid:173)\nmary visual cortex and examine the implications of the dynamical \ndecorrelation of the activities of orientation selective cells by the \nintracortical connections. The theory gives a unified and quantita(cid:173)\ntive explanation of the psychophysical experiments on orientation \ncontrast and orientation adaptation. Using only one parameter, we \nachieve good agreements between the theoretical predictions and \nthe experimental data. \n\n1 \n\nIntroduction \n\nThe mammalian visual system is very effective in detecting the orientations of lines \nand most neurons in primary visual cortex selectively respond to oriented lines and \nform orientation columns [1) . Why is the visual system organized as such? We \n\n*Present address: Rockefeller University, B272, 1230 York Avenue, NY, NY 10021-6399. \n\n\f926 \n\nDawei W Dong \n\nbelieve that the visual system is self-organized, in both long term development and \nshort term adaptation, to ensure the optimal information processing. \n\nLinsker applied Hebbian learning to model the development of orientation selectiv(cid:173)\nity and later proposed a principle of maximum information preservation in early \nvisual pathways [2]. The focus of his work has been on the feedforward connections \nand in his model the feedback connections are isotropic and unchanged during the \ndevelopment of orientation columns; but the actual circuitry of visual cortex in(cid:173)\nvolves extensive, columnar specified feedback connections which exist even before \nfunctional columns appear in cat striate cortex [3]. \n\nOur earlier research emphasized the important role of the feedback connections in \nthe development of the columnar structure in visual cortex. We developed a the(cid:173)\noretical framework to help understand the dynamics of Hebbian learning in feed(cid:173)\nback networks and showed how the columnar structure originates from symmetry \nbreaking in the development of the feedback connections (intracortical, or lateral \nconnections within visual cortex) [4]. \n\nFigure 1 illustrates our theoretical predictions. The intracortical connections break \nsymmetry and develop strip-like patterns with a characteristic wave length which \nis comparable to the developed intracortical inhibitory range and the LGN-cortex \nafferent range (left). The feedforward (LGN-cortex) connections develop under the \ninfluence of the symmetry breaking development of the intracortical connections. \nThe developed feedforward connections for each cell form a receptive field which \nis orientation selective and nearby cells have similar orientation preference (right) . \nTheir orientations change in about the same period as the strip-like pattern of the \nintracortical connections. \n\nFigure 1: The results of the development of visual cortex with feedback connections. The \nsimulated cortex consists of 48 X 48 neurons, each of which connects to 5 X 5 other cortical \nneurons (left) and receives inputs from 7 X 7 LGN neurons (right). In this figure, white \ninclicates positive connections and black inclicates negative connections. One can see that \nthe change of receptive field's orientation (right) is highly correlated with the strip-like \npattern of intracortical connections (left). \n\nMany aspects of our theoretical predictions agree qualitatively with neurobiologi(cid:173)\ncal observations in primary visual cortex. Another way to test the idea of optimal \n\n\fAssociative Correlation Dynamics \n\n927 \n\ninformation processing or any self-organization theory is through quantitative psy(cid:173)\nchophysical studies. The idea is to look for changes in perception following changes \nin input environments. The psychophysical experiments on orientation illusions \noffer some opportunities to test our theory on orientation selectivity. \n\nOrientation illusions are the effects that the perceived orientations of lines are af(cid:173)\nfected by the neighboring (in time or space) oriented stimuli, which have been \nobserved in many psychophysical experiments and were attributed to the inhibitory \ninteractions between channels tuned to different orientations [5]. But there is no uni(cid:173)\nfied and quantitative explanation. Neurophysiological evidences support our earlier \ncomputational model in which intracortical inhibition plays the role of gain-control \nin orientation selectivity [6]. But in order for the gain-control mechanism to be \neffective to signals of different statistics, the system has to develop and adapt in \ndifferent environments. \n\nIn this paper we examine the implication of the hypothesis that the intracortical \nconnections dynamically decorrelate the activities of orientation selective cells, i.e., \nthe intracortical connections are actively adapted to the visual environment, such \nthat the output activities of orientation selective cells are decorrelated. The dynam(cid:173)\nics which ensures such decorrelation through associative learning is outlined in the \nnext section as the theoretical framework for the development and the adaptation \nof intracortical connections. We only emphasize the feedback connections in the \nfollowing sections and assume that the feedforward connections developed orienta(cid:173)\ntion selectivities based on our earlier works. The quantitative comparisons of the \ntheory and the experiments are presented in section 3. \n\n2 Associative Decorrelation Dynamics \n\nThere are two different kinds of variables in neural networks. One class of variables \nrepresents the activity of the nerve cells, or neurons. The other class of variables \ndescribes the synapses, or connections, between the nerve cells. A complete model \nof an adaptive neural system requires two sets of dynamical equations, one for each \nclass of variables, to specify the evolution and behavior of the neural system. \n\nThe set of equations describing the change of the state of activity of the neurons is \n\ndVi \na-I = -Vi + ~T. .. v.. + 1-\ndt \n\nI \n\nI \n\nL..J I}} \nj \n\n(1) \n\nin which a is a time constant, Tij is the strength of the synaptic connection from \nneuron j to neuron i, and Ii is the additional feedforward input to the neuron besides \nthose described by the feedback connection matrix nj . A second set of equations \ndescribes the way the synapses change with time due to neuronal activity. The \nlearning rule proposed here is \n\nB dnj = (V,. - V.')!, \n\n(2) \nin which B is a time constant and Vi' is the feedback learning signal as described \nin the following. \nThe feedback learning signal Vi' is generated by a Hopfield type associative memory \nnetwork: Vi' = Lj T/j Vi , in which T/j is the strength of the associative connection \n\nI } \n\ndt \n\nI \n\n\f928 \n\nDawei W Dong \n\nfrom neuron j to neuron i, which is the recent correlation between the neuronal \nactivities Vi and Vj determined by Hebbian learning with a decay term [4] \n\n,dTfj \nB dt = -Iij + ViVj \n\n, \n\n(3) \n\nin which B' is a time constant. The Vi' and T[j are only involved in learning and \ndo not directly affect the network outputs. \nIt is straight forward to show that when the time constants B > > B' > > a, the \ndynamics reduces to \n\ndT \n\nB dt = (1- < VVT \u00bb < VIT > \n\n(4) \n\nwhere bold-faced quantities are matrices and vectors and <> denotes ensemble \naverage. It is not difficult to show that this equation has a Lyapunov or \"energy\" \nfunction \n\nL = Tr(1- < VVT \u00bb(1- < VVT >f \n\n(5) \n\n(7) \n\nwhich is lower bounded and satisfies \n\ndL < 0 \ndt -\n\nand \n\ndL =0 -+(cid:173)\ndt \n\ndTij \ndt = \n\n0 \n\n11\" \n\nI' \nlor at,) \n\n(6) \n\nThus the dynamics is stable. When it is stable, the output activities are decorre(cid:173)\nlated, \n\n= 1 \n\nThe above equation shows that this dynamics always leads to a stable state where \nthe neuronal activities are decorrelated and their correlation matrix is orthonormal. \nequation (2) and (3) are \nYet the connections change in an associative fashion -\nalmost Hebbian. That is why we call it associative decorrelation dynamics. From in(cid:173)\nformation processing point of view, a network, self-organized to satisfy equation (7), \nis optimized for Gaussian input ensembles and white output noises [7]. \n\nLinear First Order Analysis \n\nIn applying our theory of associative decorrelation dynamics to visual cortex to \ncompare with the psychophysical experiments on orientation illusions, the linear \nfirst-order approximation is used, which is \n\nT = TO + 6T, \nV = Va +6V, \n\nTO = 0, 6T ex - < I IT > \nVa = I, 6V = TI \n\n(8) \n\nwhere it is assumed that the input correlations are small. It is interesting to notice \nthat the linear first-order approximation leads to anti-Hebbian feedback connec(cid:173)\ntions: Iij ex - < /i/j > which is guarantteed to be stable around T = 0 [8]. \n\n3 Quantitative Predictions of Orientation Illusions \n\nThe basic phenomena of orientation illusions are demonstrated in figure 2 (left). \nOn the top, is the effect of orientation contrast (also called tilt illusion): within the \ntwo surrounding circles there are tilted lines; the orientation of a center rectangle \n\n\fAssociative Correlation Dynamics \n\n929 \n\nappears rotated to the opposite side of its surrounding tilt. Both the two rectan(cid:173)\ngles and the one without surround (at the left-center of this figure) are, in fact, \nexactly same. On the bottom, is the effect of orientation adaptation (also called \ntilt aftereffect): if one fixates at the small circle in one of the two big circles with \ntilted lines for 20 seconds or so and then look at the rectangle without surround, \nthe orientation of the lines of the rectangle appears tilted to the opposite side. \n\nThese two effects of orientation illusions are both in the direction of repulsion: the \napparent orientation of a line is changed to increase its difference from the inducing \nline. Careful experimental measurements also revealed that the angle with the \ninducing line is <\"V 100 for maximum orientation adaptation effect [9] but <\"V 20 0 for \norientation contrast [10]. \n\n1 \n\nOl..---~-~-\"\"';:\"'''''''''---' \n-90 \n90 \nStimulus orientation (J (degree) \n\n-45 \n\no \n\n45 \n\nFigure 2: The effects of orientation contrast (upper-left) and orientation adaptation (lower(cid:173)\nleft) are attributed to feedback connections between cells tuned to different orientations \n(upper-right, network; lower-right, tuning curve). \n\nOrientation illusions are attributed to the feedback connections between orienta(cid:173)\ntion selective cells. This is illustrated in figure 2 (right). On the top is the network \nof orientation selective cells with feedback connections. Only four cells are shown. \nFrom the left, they receive orientation selective feedforward inputs optimal at -45 0 , \n00 ,450 , and 90 0 , respectively. The dotted lines represent the feedback connections \n(only the connections from the second cell are drawn). On the bottom is the orien(cid:173)\ntation tuning curve of the feedforward input for the second cell, optimally tuned to \nstimulus of 00 (vertical), which is assumed to be Gaussian of width (T = 200 \u2022 Be(cid:173)\ncause of the feedback connections, the output of the second cell will have different \ntuning curves from its feedforward input, depending on the activities of other cells. \n\nFor primary visual cortex, we suppose that there are orientation selective neurons \ntuned to all orientations. It is more convenient to use the continuous variable e \ninstead of the index i to represent neuron which is optimally tuned to the orientation \nof angle e. The neuronal activity is represented by V(e) and the feedforward input \nto each neuron is represented by I(e). The feedforward input itself is orientation \n\n\f930 \n\nDawei W. Dong \n\nselective: given a visual stimulus of orientation eo, the input is \n\nJ(e) = e-(9-9o)2/ q 2 \n\n(9) \n\nThis kind of the orientation tuning has been measured by experiments (for refer(cid:173)\nences, see [6]). Various experiments give a reasonable tuning width around 20\u00b0 \n\u00ab(7\" = 20\u00b0 is used for all the predictions). \nPredicted Orientation Adaptation \nFor the orientation adaptation to stimulus of angle eo, substituting equation (9) \ninto equation (8), it is not difficult to derive that the network response to stimulus \nof angle 0 (vertical) is changed to \n\nV(e) = e_92 / q2 _ ae-(9-9o )2/ q 2 e-9~/2q2 \n\n(10) \n\nin which (7\" is the feedforward tuning width chosen to be 20\u00b0 and a is the parameter \nof the strength of decorrelation feedback. \nThe theoretical curve of perceived orientation \u00a2(eo) is derived by assuming the \nmaximum likelihood of the the neural population, i.e., the perceived angle \u00a2 is the \nangle at which Vee) is maximized. It is shown in figure 3 (right). The solid line is \nthe theoretical curve and the experimental data come from [9] (they did not give \nthe errors, the error bars are of our estimation,...., 0.2\u00b0). The parameter obtained \nthrough X2 fit is the strength of decorrelation feedback: a = 0.42. \n\n2.0 ;--'\"T\"\"---,--...,.....-----.------, \n\n-~ 1.5 \n\n~ \nII) \n\nCD 1.0 \n~ \n.\" \n~ 0.5 \n'il \n~ Q., 0.0 f - - - - - - - - - - - - - J \n\n4.0 -~ 3.0 \n\n~ \n\nII) \n\n} 2.0 \n\n.\" \nII) 1.0 \n> \n'il \ni:! II) 0.0 \n\nQ., \n\no \n\n10 \n\n20 \n\n30 \n\n40 \n\n50 \n\n0 \n\n10 \n\n20 \n\n30 \n\n40 \n\n50 \n\nSurround angle 80 (degree) \n\nAdaptation angle 80 (degree) \n\nFigure 3: Quantitative comparison of the theoretical predictions with the experimental \ndata of orientation contrast (left) and orientation adaptation (right). \n\nIt is very interesting that we can derive a relationship which is independent of the \nparameter of the strength of decorrelation feedback a, \n(eo - \u00a2m)(3eo - 2\u00a2m) = (7\"2 \n\n(11) \nin which eo is the adaptation angle at which the tilt aftereffect is most significant \nand \u00a2m is the perceived angle. \n\nPredicted Orientation Contrast \n\nFor orientation contrast, there is no specific adaptation angle, i.e., the network has \ndeveloped in an environment of all possible angles. In this case, when the surround \nis of angle eo, the network response to a stimulus of angle e1 is \n\nVee) = e-(9-91)2/ q 2 _ ae-(9-9o)2/3q2 \n\n(12) \n\n\fAssociative Correlation Dynamics \n\n931 \n\nin which fr and a has the same meaning as for orientation adaptation. Again assum(cid:173)\ning the maximum likelihood, \u00a2(eo), the stimulus angle e1 at which it is perceived \nas angle 0, is derived and shown in figure 3 (left). The solid line is the theoretical \ncurve and the experimental data come from [10] and their estimated error is \"\" 0.20. \nThe parameter obtained through X 2 fit is the strength of decorrelation feedback: \na = 0.32. \nWe can derive the peak position eo, i.e., the surrounding angle eo at which the \norientation contrast is most significant, \n\n(13) \nFor fr = 200 , one immediately gets eo = 240 \u2022 This is in good agreement with \nexperiments, most people experience the maximum effect of orientation contrast \naround this angle. \n\n~e~ = fr2 \n3 \n\nOur theory predicts that the peak position of the surround angle for orientation \ncontrast should be constant since the orientation tuning width fr is roughly the \nsame for different human observers and is not going to change much for different \nexperimental setups. But the peak value of the perceived angle is not constant since \nthe decorrelation feedback parameter a is not necessarily same, indeed, it could be \nquite different for different human observers and different experimental setups. \n\n4 Discussion \n\nFirst, we want to emphasis that in all the comparisons, the same tuning width fr is \nused and the strength of decorrelation feedback a is the only fit parameter. It does \nnot take much imagination to see that the quantitative agreements between the \ntheory and the experiments are good. Further more, we derived the relationships \nfor the maximum effects, which are independent of the parameter a and have been \npartially confirmed by the experiments. \n\nRecent neurophysiological experiments revealed that the surrounding lines did in(cid:173)\nfluence the orientation selectivity of cells in primary visual cortex of the cat [11]. \nThose single cell experiments land further support to our theory. But one should \nbe cautioned that the cells in our theory should be considered as the average over \na large population of cells in cortex. \n\nThe theory not only explains the first order effects which are dominant in angle \nrange of 00 to 500 , as shown here, but also accounts for the second order effects \nwhich can be seen in 500 to 90 0 range, where the sign of the effects is reversed. \nThe theory also makes some predictions for which not much experiment has been \ndone yet, for example, the prediction about how orientation contrast depends on \nthe distance of surrounding stimuli from the test stimulus [7]. \n\nFinally, this is not merely a theory for the development and the adaptation of \norientation selective cells, it can account for effect such as human vision adaptation \nto colors as well [7]. We can derive the same equation as Atick etal [12] which agrees \nwith the experiment on the appearance of color hue after adaptation. We believe \nthat future psychophysical experiments could give us more quantitative results to \nfurther test our theory and help our understanding of neural systems in general. \n\n\f932 \n\nDawei W. Dong \n\nAcknowledgements \nThis work was supported in part by the Director, Office of Energy Research, Di(cid:173)\nvision of Nuclear Physics of the Office of High Energy and Nuclear Physics of the \nU.S. Department of Energy under Contract No. DE-AC03-76SF00098. \n\nReferences \n\n[1] Hubel DH, Wiesel TN, 1962 Receptive fields, binocular interactions, and functional \narchitecture in the cat's visual cortex J Physiol (London) 160, 106- 54. -\n1963 \nShape and arrangement of columns in cat's striate cortex J Physiol (London) 165, \n559-68. \n\n[2] Linsker R, 1986 From basic network principles to neural architecture ... Proc Natl \n\nAcad Sci USA 83, 7508 8390 8779. -, 1989 An application of the principle of max(cid:173)\nimum information preservation to linear systems Advances in Neural Information \nProcessing Systems 1, Touretzky DS, ed, Morgan Kaufman, San Mateo, CA 186-94. \n\n[3] Gilbert C, Wiesel T, 1989 Columnar Specificity of intrinsic horizontal and cortic(cid:173)\n\nocortical connections in cat visual cortex J Neurosci 9(7), 2432-42. Luhmann HJ, \nMartinez L, Singer W, 1986 Development of horizontal intrinsic connections in cat \nstriate cortex Exp Brain Res 63, 443-8. \n\n[4] Dong DW, 1991 Dynamic properties of neural network with adapting synapses Proc \nInternational Joint Conference on Neural Networks, Seattle, 2, 255- 260. -, 1991 \nDynamic Properties of Neural Networks Ph D thesis, University Microfilms Interna(cid:173)\ntional, Ann Arbor, ML Dong DW, Hopfield JJ, 1992 Dynamic properties of neural \nnetworks with adapting synapses Network: Computation in Neural Systems, 3(3), \n267- 83. \n\n[5] Gibson J J, Radner M, 1937 Adaptation, after-effect and contrast in the perception \n\nof tilted lines J of Exp Psy 20, 453-67. Carpenter RHS, Blakemore C, 1973 Inter(cid:173)\nactions between orientations in human vision Exp Brain Res 18, 287-303. Tolhurst \nDJ, Thompson PG, 1975 Orientation illusions and after-effects: Inhibition between \nchannels Vis Res 15,967-72. Barlow HB, Foldiak P, 1989 Adaptation and decorre(cid:173)\nlation in the cortex The Computing Neuron, Durbin R, Miall C, Mitchison G, eds, \nAddison- Wesley, New York, NY. \n\n[6] Wehmeier U, Dong DW, Koch C, Van Essen DC, 1989 Modeling the mammalian \nvisual system Methods in Neuronal Modeling: From Synapses to Networks, Koch C, \nSegev I, eds, MIT Press, Cambridge, MA 335-60. \n\n[7] Dong DW, 1993 Associative Decorrelation Dynamics in Visual Cortex Lawrence \n\nBerkeley Laboratory Technical Report LBL-34491. \n\n[8] Dong DW, 1993 Anti-Hebbian dynamics and total recall of associative memory Proc \n\nWorld Congress on Neural Networks, Portland, 2, 275-9. \n\n[9] Campbell FW, Maffei L, 1971 The tilt after-effect: a fresh look Vis Res 11, 833-40. \n[10] Westheimer G, 1990 Simultaneous orientation contrast for lines in the human fovea \n\nVis Res 30, 1913-21. \n\n[11] Gilbert CD, Wiesel TN, 1990 The influence of contextual stimuli on the orientation \n\nselectivity of cells in primary visual cortex of the cat Vis Res 30,1689-701. \n\n[12] Atick JJ, Li Z, Redlich AN, 1993 What does post-adaptation color appearance reveal \n\nabout cortical color representation Vis Res 33, 123-9. \n\n\f", "award": [], "sourceid": 1014, "authors": [{"given_name": "Dawei", "family_name": "Dong", "institution": null}]}