{"title": "Analog Implementation of Shunting Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 695, "page_last": 702, "abstract": null, "full_text": "ANALOG IMPLEMENTATION OF SHUNTING \n\nNEURAL NETWORKS \n\n695 \n\nBahram Nabet, Robert B. Darling, and Robert B. Pinter \n\nDepartment of Electrical Engineering, FT-lO \n\nUniversity of Washington \n\nSeattle, WA 98195 \n\nABSTRACT \n\nAn extremely compact, all analog and fully parallel implementa(cid:173)\ntion of a class of shunting recurrent neural networks that is ap(cid:173)\nplicable to a wide variety of FET-based integration technologies is \nproposed. While the contrast enhancement, data compression, and \nadaptation to mean input intensity capabilities of the network are \nwell suited for processing of sensory information or feature extrac(cid:173)\ntion for a content addressable memory (CAM) system, the network \nalso admits a global Liapunov function and can thus achieve stable \nCAM storage itself. In addition the model can readily function as \na front-end processor to an analog adaptive resonance circuit. \n\nINTRODUCTION \n\nShunting neural networks are networks in which multiplicative, or shunting, terms \nof the form Xi Lj f;(Xj) or Xi Lj Ij appear in the short term memory equations, \nwhere Xi is activity of a cell or a cell population or an iso-potential portion of a \ncell and Ii are external inputs arriving at each site. The first case shows recurrent \nactivity, while the second case is non-recurrent or feed forward. The polarity of \nthese terms signify excitatory or inhibitory interactions. \n\nShunting network equations can be derived from various sources such as the passive \nmembrane equation with synaptic interaction (Grossberg 1973, Pinter 1983), models \nof dendritic interaction (RaIl 1977), or experiments on motoneurons (Ellias and \nGrossberg 1975). \n\nWhile the exact mechanisms of synaptic interactions are not known in every in(cid:173)\ndividual case, neurobiological evidence of shunting interactions appear in several \n\n\f696 \n\nNabet, Darling and Pinter \n\nareas such as sensory systems, cerebellum, neocortex, and hippocampus (Grossberg \n1973, Pinter 1987). In addition to neurobiology, these networks have been used to \nsuccessfully explain data from disciplines ranging from population biology (Lotka \n1956) to psychophysics and behavioral psychology (Grossberg 1983). \n\nShunting nets have important advantages over additive models which lack the ex(cid:173)\ntra nonlinearity introduced by the multiplicative terms. For example, the total \nactivity of the network, shown by Li Xi, approaches a constant even as the input \nstrength grows without bound. This normalization in addition to being computa(cid:173)\ntionally desirable has interesting ramifications in visual psychophysics (Grossberg \n1983). Introduction of multiplicative terms also provides a negative feedback loop \nwhich automatically controls the gain of each cell, contributes to the stability of the \nnetwork, and allows for large dynamic range of the input to be processed by the \nnetwork. The automatic gain control property in conjunction with properly chosen \nnonlinearities in the feedback loop makes the network sensitive to small input values \nby suppressing noise while not saturating at high input values (Grossberg 1973). \nFinally, shunting nets have been shown to account for short term adaptation to \ninput properties, such as adaptation level tuning and the shift of sensitivity with \nbackground strength (Grossberg 1983), dependence of visual size preference and \nlatency of response on contrast and mean luminance, and dependence of temporal \nand spatial frequency tuning on contrast and mean luminance (Pinter 1985). \n\nIMPLEMENTATION \n\nThe advantages, generality, and applicability of shunting nets as cited in the previ(cid:173)\nous section make their implementation very desirable, but digital implementation \nof these networks is very inefficient due to the need for analog to digital conver(cid:173)\nsion, multiplication and addition instructions, and implementation of iterative al(cid:173)\ngorithms. A linear feedback class of these networks (Xi Lj !; (Xj) = Xi Li J{ijXj), \nhowever, can be implemented very efficiently with simple, completely parallel and \nall analog circuits. \n\nFRAMEWORK \n\nFigure 1 shows the design framework for analog implementation of a class of shunt(cid:173)\ning nets. In this design addition (subtraction) is achieved, via Kirchoff's current \nlaw by placing transistors in upper (lower) rails, and through the choice of deple(cid:173)\ntion or enhancement mode devices. Multiplicative, or shunting, interconnections \nare done by one transistor per interconnect, using a field-effect transistor (FET) in \nthe voltage-variable conductance region. Temporal properties are characterized by \ncell membrane capacitance C, which can be removed, or in effect replaced by the \nparasitic device capacitances, if higher speed is desired. A buffer stage is necessary \nfor correct polarity of interconnections and the large fan-out associated with high \nconnectivity of neural networks. \n\n\fAnalog Implementation of Shunting Neural Networks \n\n697 \n\nI. , \n\n+ \nx. , \n\nc \n\n-t\" '.J \n\n. \n\nx\u00b7 \nJ \n\nVdd \n\nVss \n\nFigure 1. Design framework for implementation of one cell in a \nshunting network. Voltage output of other cells is connected to the \ngate of transistors Qi,i' \n\nSuch a circuit is capable of implementing the general network equation: \n\n(1) \n\nExcitatory and inhibitory input current sources can also be shunted, with extra \ncircuitry, to implement non-recurrent shunting networks. \n\nNMOS, CMOS and GALLIUM ARSENIDE \nSince the basic cell of Fig. 1 is very similar to a standard logic gate inverter, but with \nthe transistors sized by gate width-to-Iength ratio to operate in the nonsaturated \ncurrent region, this design is applicable to a variety of FET technologies including \nNMOS, CMOS, and gallium arsenide (GaAs). \nA circuit made of all depletion-mode devices such as GaAs MESFET buffered FET \nlogic, can implement all the terms of Eq. (1) except shunting excitatory terms and \nrequires a level shifter in the buffer stage. A design with all enhancement mode \ndevices such as silicon NMOS can do the same but without a level shifter. With \nthe addition of p-channel devices, e.g. Si CMOS, all polarities and all terms of Eq. \n(1) can be realized. As mentioned previously a buffer stage is necessary for correct \npolarity of interconnections and fan out/fan in capacity. \nFigure 2 shows a GaAs MESFET implementation with only depletion mode devices \nwhich employs a level shifter as the buffer stage. \n\n\f698 \n\nNabet, Darling and Pinter \n\nVDD-------------r------~--------------~--\n\nINPUTS: \n\nEXTERNAL OR FROM \n\nPREVJOUS LAYER \n\nEXCITATORY \nCONNECTIONS \n\nINHIBITORY \nCONNECTIONS \n\nGN~-L--~------~------~--\n\nTUNABLE \n\nSELF-RELAXATION \n\nCONNECTION \n\nOUTPUT TO \nNEXT LAYER \n\nVSS--~--....L....\u00ad\n\nLEVEL SHIFT AND BUFFER STAGE \n\nFigure 2. Gallium arsenide MESFET implementation with level \nshifter and depletion mode devices. Lower rail transistors produce \nshunting off-surround terms. Upper transistors can produce addi(cid:173)\ntive excitatory connections. \n\nSPECIFIC IMPLEMENTATION \n\nThe simplest shunting network that can be implemented by the general framework \nof Fig.1 is Fig. 2 with only inhibitory connections (lower rail transistors). This \ncircuit implements the network model \n\nd/ = Ii - a,X, + Xi(J(iXi) - Xi(L....J J(ijXj) \ndX\u00b7 \n\n\" \nj#i \n\n(2), \n\nThe simplicity of the implementation is notable; a linear array with nearest neighbor \ninterconnects consists of only 5 transistors, 1-3 diodes, and if required 1 capacitor \nper cell. \nA discrete element version of this implementation has been constructed and shows \ngood agreement with expected properties. Steady state output is proportional to \nthe square root of a uniform input thereby compressing the input data and showing \nadaptation to mean input intensity (figure 3). The network exhibits contrast en(cid:173)\nhancement of spatial edges which increases with higher mean input strength (figure \n4). A point source input elicits an on-center off-surround response, similar to the \ndifference-of-Gaussians receptive field of many excitable cells. This 'receptive field' \nbecomes more pronounced as the input intensity increases, showing the dependence \nof spatial frequency tuning on mean input level (figure 5). The temporal response \nof the network is also input dependent since the time constant of the exponential \n\n\fAnalog Implementation of Shunting Neural Networks \n\n699 \n\ndecay of the impulse response decreases with input intensity. Finally, the depen(cid:173)\ndence of the above properties on mean input strength can be tuned by varying the \nconductance of the central FET. \n\n700.0 \n\n- - -\n\n> \nE \n11.1 \nII \n!:i \n~ \n5 \n~ o \n\n600.0 \n\n500.0 \u00b7 \n\n400.0 \n\n300.0 \n\n200.0 \n\n100.0 \n\n0 . 1 \n\n0.3 \n\n0.5 \n\n0 .7 \n\n0.9 \n\n1. 1 \n\n1.3 \n\n1.5 \n\n1.7 \n\n1.11 \n\n2. 1 \n\nINPUT CURRENT rnA \n\nFigure 3. Response of network to uniform input. Output is pro(cid:173)\nportional to the square root of the input. \n\nDEPENDENCE OF ENHANCEMENT ON MEAN INPUT \n\nII: \n\n~ :J \n0 ... \na \nN i \n0 z \n5 \n~ 0 \na \nf ... \n\n::l \nIL \n~ \n\n1.5 \n\n1.4 \n\n1.3 \n\n1.2 \n\n1.1 \n\n1.0 \n\n0 .11 \n\n0.8 \n\n0 .7 \n\n0.11 \n\n2 \n\n3 \n\n5 \n\n7 \n\nCELL NUMBER \n\nFigure 4. Response of network to spatial edge patterns with the \nsame contrast but increasing mean input level. \n\n\f700 \n\nNabet, Darling and Pinter \n\n~ g \n\n:> \n0 \nQ \n~ \n!5 II. \nI \niii \ni \n\nII: \n0 \nZ \n\nImox ~ 2 .0J rnA, llmox -\n\nJ75.74 mil \n\n1.0 \n\n0 .11 \n\n0.11 \n\n0 .7 \n\n0.11 \n\n0.5 \n\n0.4 \n\nO.J \n\n2 \n\nJ \n\n4 \n\n5 \n\nII \n\n7 \n\nINPUT \n\ncEll NUMBER \n\n-x- OUTPUT \n\nFigure 5. Response of network to a point source input. Inset \nshows the receptive field of fly's lamina monopolar cells (LMC of \nLucilia sericata). Horizontal axis of inset in visual angle, vertical \na.xis relative voltage units of hyperpolarization. Inset from Pinter \net al. (in preparation) \n\nCONTENT ADDRESSABILITY AND RELATION TO ART \n\nUsing a theorem by Cohen and Grossberg (1983), it can be shown that the network \nequa.tion (2) a.dmits the global Liapunov function \n\nn \n\nV -\n\n- \"'(l\u00b7ln(xi) - a'x ' + K'x~) +.! '\" K\u00b7\u00b7x 'XL \n\n2 ~ IJ \n\n- ~ 1 \n\n1 \n\n1 \n\n1 \n\nJ \n\n.. , \n\n1 \n\n>. \n\nn \n\n(3) \n\n;=1 \n\nj,k=l \n\nwhere>. is a constant, under the constraints Kij = Kji and Xi > O. This shows that \nin response to an arbitrary input the network always approaches an equilibrium \npoint. The equilibria represent stored patterns and this is Content Addressable \nMemory (CAM) property. \nIn addition, Eq. (2) is a special case of the feature representation field of an analog \nadaptive resonance theory ART-2 circuit, (Carpenter and Grossberg 1987), and \nhence this design can operate as a module in a learning multilayer ART architecture. \n\n\fAnalog Implementation of Shunting Neural Networks \n\n701 \n\nFUTURE PLANS \n\nDue to the very small number of circuit components required to construct a cell, \nthis implementation is quite adaptable to very high integration densities. A solid \nstate implementation of the circuit of figure (2) on a gallium arsenide substrate, \nchosen for its superiority for opto-electronics applications, is in progress. The chip \nincludes monolithically fabricated photosensors for processing of visual information. \nAll of the basic components of the circuit have been fabricated and tested. With \nstandard 2 micron GaAs BFL design rules, a chip could contain over 1000 cells per \ncm2 , assuming an average of 20 inputs per cell. \n\nCONCLUSIONS \n\nThe present work has the following distinguishing features: \n\n\u2022 Implements a mathematically well described and stable model. \n\u2022 Proposes a framework for implementation of shunting nets which are biologically \nfeasible, explain variety of psychophysical and psychological data and have many \ndesirable computational properties. \n\n\u2022 Has self-sufficient computational capabilities; especially suited for processing of sen(cid:173)\n\nsory information in general and visual information in particular (N abet and Darling \n1988). \n\n\u2022 Produces a 'good representation' of the input data which is also compatible with \n\nthe self-organizing multilayer neural network architecture ART-2. \n\n\u2022 Is suitable for implementation in variety of technologies. \n\u2022 Is parallel, analog, and has very little overhead circuitry . \n\n.. - . \n\n\f702 \n\nN abet, Darling and Pinter \n\nREFERENCES \n\nCarpenter, G.A. and Grossberg, S. (1987) \"ART 2: self organization of stable cat(cid:173)\negory recognition codes for analog input patterns,\". Applied Optics 26, pp. 4919-\n4930. \nCohen,M.A. and Grossberg, S. (1983) \"Absolute stability of global pattern forma(cid:173)\ntion and parallel memory storage by competitive neural networks\" , IEEE Transac(cid:173)\ntions on Systems Man and Cybernetics SMC-13, pp. 815-826. \nEllias, S.A. and Grossberg, S. (1975) \"Pattern formation, contrast control, and \noscillations in the short term memory of shunting on-center off-surround networks\" \nBiological Cybernetics, 20, pp. 69-98. \nGrossberg, S. (1973), \"Contour enhancement, Short term memory and constancies \nin reverberating neural networks,\" Studies in Applied Mathematics, 52, pp. 217-\n257. \nGrossberg, S. (1983), \"The quantized geometry of visual space: the coherent com(cid:173)\nputation of depth, form, and lightness.\" The behavioral and brain sciences, 6, pp. \n625-692. \nLotka, A.J. (1956). Elements of mathematical biology. New York: Dover. \nNabet, B. and Darling, R.B. (1988). \"Implementation of optical sensory neural net(cid:173)\nworks with simple discrete and monolithic circuits,\" (Abstract) Neural Networks, \nVol.l, Suppl. 1, 1988, pp. 396. \nPinter, R.B., (1983). \"The electrophysiological bases for linear and nonlinear prod(cid:173)\nuct term lateral inhibition and the consequences for wide-field textured stimuli\" \nJ. Theor. Bioi. 105 pp. 233-243. \nPinter, R.B. (1985) \" Adaptation of spatial modulation transfer functions via non(cid:173)\nlinear lateral inhibition\" Bioi. Cybernetics 51, pp. 285-291. \nPinter, R.B. (1987) \"Visual system neural networks: Feedback and feedforward lat(cid:173)\neral inhibition\" Systems and Control Encyclopedia (ed.M.G. Singh) Oxford: Perg(cid:173)\namon Press. pp. 5060-5065. \nPinter, R.B., Osorio, D., and Srinivasan, M.V., (in preperation) \"Shift of edge \npreference to scototaxis depends on mean luminance and is predicted by a matched \nfilter hypothesis in fly lamina cells\" \nRaIl, W. (1977). \"Core conductor theory and cable properties of neurons\" in Hand(cid:173)\nbook of Physiology: The Nervous System vol. I, part I, Ed. E.R. Kandel pp. 39-97. \nBethesda, MD: American Physiological Society. \n\n\f", "award": [], "sourceid": 99, "authors": [{"given_name": "Bahram", "family_name": "Nabet", "institution": null}, {"given_name": "Robert", "family_name": "Darling", "institution": null}, {"given_name": "Robert", "family_name": "Pinter", "institution": null}]}