{"title": "Winner-Take-All Networks of O(N) Complexity", "book": "Advances in Neural Information Processing Systems", "page_first": 703, "page_last": 711, "abstract": "", "full_text": "703 \n\nWINNER-TAKE-ALL \n\nNETWORKS OF O(N) COMPLEXITY \n\nJ. Lazzaro, S. Ryckebusch, M.A. Mahowald, and C. A. Mead \n\nCalifornia Institute of Technology \n\nPasadena, CA 91125 \n\nABSTRACT \n\nWe have designed, fabricated, and tested a series of compact CMOS \nintegrated circuits that realize the winner-take-all function. These \nanalog, continuous-time circuits use only O(n) of interconnect to \nperform this function. We have also modified the winner-take-all \ncircuit, realizing a circuit that computes local nonlinear inhibition. \n\nTwo general types of inhibition mediate activity in neural systems: subtractive in(cid:173)\nhibition, which sets a zero level for the computation, and multiplicative (nonlinear) \ninhibition, which regulates the gain of the computation. We report a physical real(cid:173)\nization of general nonlinear inhibition in its extreme form, known as winner-take-all. \n\nWe have designed and fabricated a series of compact, completely functional CMOS \nintegrated circuits that realize the winner-take-all function, using the full analog \nnature of the medium. This circuit has been used successfully as a component \nin several VLSI sensory systems that perform auditory localization (Lazzaro and \nMead, in press) and visual stereopsis (Mahowald and Delbruck, 1988). Winner(cid:173)\ntake-all circuits with over 170 inputs function correctly in these sensory systems. \n\nWe have also modified this global winner-take-all circuit, realizing a circuit that \ncomputes local nonlinear inhibition. The circuit allows multiple winners in the net(cid:173)\nwork, and is well suited for use in systems that represent a feature space topograph(cid:173)\nically and that process several features in parallel. We have designed, fabricated, \nand tested a CMOS integrated circuit that computes locally the winner-take-all \nfunction of spatially ordered input. \n\n\f704 \n\nLazzaro, Ryckebusch, Mahowald and Mead \n\nTHE WINNER-TAKE-ALL CmCUIT \n\nFigure 1 is a schematic diagram of the winner-take-all circuit. A single wire, asso(cid:173)\nciated with the potential Vc, computes the inhibition for the entire circuit; for an \nn neuron circuit, this wire is O(n) long. To compute the global inhibition, each \nneuron k contributes a current onto this common wire, using transistor T2 a.' To \napply this global inhibition locally, each neuron responds to the common wire volt(cid:173)\nage Vc, using transistor Tla.' This computation is continuous in time; no clocks \nare used. The circuit exhibits no hysteresis, and operates with a time constant \nrelated to the size of the largest input. The output representation of the circuit \nis not binary; the winning output encodes the logarithm of its associated input. \n\nFigure 1. Schematic diagram of the winner-take-all circuit. Each neuron receives \na unidirectional current input 11;; the output voltages VI \u2022.. VB represent the result \nof the winner-take-all computation. If II; = max(II \u2022\u2022\u2022 IB ), then VI; is a logarithmic \nfunction of 11;; if Ii <: 11;, then Vi ~ O. \n\nA static and dynamic ana.lysis of the two-neuron circuit illustrates these system \nproperties. Figure 2 shows a schematic diagram of a two-neuron winner-take-all \ncircuit. To understand the beha.vior of the circuit, we first consider the input \ncondition II = 12 = 1m. Transistors TIl ~d T12 have identical potentials at gate \nand source, and are both sinking 1m; thus, the drain potentials VI and V2 must be \nequal. Transistors T21 and T22 have identical source, drain, and gate potentials, \nand therefore must sink the identical current ICI = IC2 = Ic/2. In the subthreshold \nregion of operation, the equation 1m = 10 exp(Vc/Vo) describes transistors Til and \nT12 , where 10 is a fabrication parameter, and Vo = kT/qlt. Likewise, the equation \nIc/2 = 10 exp((Vm - Vel/Volt where Vm = VI = V2, describes transistors T21 and \nT22 . Solving for Vm(Im, Ie) yields \n\nVm = Voln(~:) + Voln(:;). \n\n(1) \n\n\fWinner-Take-All Networks ofO(N) Complexity \n\n705 \n\nThus, for equal input currents, the circuit produces equal output voltages; this \nbehavior is desirable for a winner-take-all circuit. In addition, the output voltage \nV m logarithmically encodes the magnitude of the input current 1m. \n\nFigure 2. Schematic diagram of a two-neuron winner-take-all circuit. \n\nThe input condition II = 1m + Oi, 12 = 1m illustrates the inhibitory action of the \ncircuit. Transistor Til must sink 0, more current than in the previous example; as a \nresult, the gate voltage of Til rises. Transistors Tit and TI2 share a common gate, \nhoweverj thus, TI2 must also sink 1m + 0,. But only 1m is present at the drain of \nT12 \u2022 To compensate, the drain voltage of T12 , V2, must decrease. For small OiS, the \nEarly effect serves to decrease the current through Th , decreasing V2 linearly with \n0,. For large o's, TI2 must leave saturation, driving V2 to approximately 0 volts. \nAs desired, the output associated with the smaller input diminishes. For large OiS, \nIe2 $!:::f 0, and Iel $!:::f Ie. The equation 1m + 0, = 10 exp(Ve/Vo) describes transistor \nTil' and the equation Ie = 10 exp((VI - Vel/Yo) describes transistor T21 \u2022 Solving \nfor VI yields \n\n(2) \n\nThe winning output encodes the logarithm of the associated input. The symmetrical \ncircuit topology ensures similar behavior for increases in 12 relative to II. \n\nEquation 2 predicts the winning response of the circuit; a more complex expression, \nderived in (Lazzaro et.al., 1989), predicts the losing and crossover response of the \ncircuit. Figure 3 is a plot of this analysis, fit to experimental data. Figure 4 shows \nthe wide dynamic range and logarithmic properties of the circuitj the experiment in \nFigure 3 is repeated for several values of 12 , ranging over four orders of magnitude. \nThe conductance of transistors Til and T1:a determines the losing response of the \ncircuit. Variants of the winner-take-all circuit shown in (Lazzaro et. aI., 1988) \nachieve losing responses wider and narrower than Figure 3, using circuit and mask \nlayout techniques. \n\n\f706 \n\nLazzaro, Ryckebusch, Mahowald and Mead \n\nWINNER-TAKE-ALL TIME RESPONSE \n\nA good winner-take-all circuit should be stable, and should not exhibit damped \noscillations (\"ringing\") in response to input changes. This section explores these \ndynamic properties of our winner-take-all circuit, and predicts the temporal re(cid:173)\nsponse of the circuit. Figure 8 shows the two-neuron winner-take-all circuit, with \ncapacitances added to model dynamic behavior. \n\no T \n\nVo \n\n102 \n\nIe \n\nFigure 8. Schematic diagram of a two-neuron winner-take-all circuit, with capac(cid:173)\nis a large MOS capacitor added to each \nitances added for dynamic analysis. 0 \nneuron for smoothingj 0., models the parasitic capacitance contributed by the gates \nof Tu and T12 , the drains of T21 and T22, and the interconnect. \n\n(Lazzaro et. al., 1988) shows a small-signal analysis of this circuit. The transfer \nfunction for the circuit has real poles, and thus the circuit is stable and does not ring, \nif 10 > 41(Oe/O), where 11 RlI2 Rl 1. Figure 9 compares this bound with experimental \ndata. \n\nH Ie > 41(0 0 /0), the circuit exhibits first-order behavior. The time constant OVo/I \nsets the dynamics of the winning neuron, where Vo = A:T /qK. Rl 40 mV. The time \nconstant OVE/I sets the dynamics of the losing neuron, where VE Rl 50 v. Figure 10 \ncompares these predictions with experimental data. \n\n\fWinner-Take-All Networks ofO(N) Complexity \n\n707 \n\nVl,V, \n(V) \n\n2.6 \n\n2.4 \n\n2.2 \n\n2.0 \n\nI.S \n\n1.6 \n\n1.4 \n\n1.2 \n\n1.0+--+--+--+--..... ~I----t~--t---f \n0.92 0.94 0.96 0.98 1.00 1.02 1.04 1.06 1.0S \n\nII/I, \n\nFigure 8. Experimental data (circles) and theory (solid lines) for a two-neuron \nwinner-take-all circuit. II, the input current of the first neuron, is swept about the \nvalue of 12, the input current of the second neuron; neuron voltage outputs VI and \nV2 are plotted versus normalized input current. \n\n2.6 \n\nI.S \n\n1.6 \n\n1.4 \n\n1.2 \n\n10- 11 \n\n10- 10 \n\n10-0 \n\n10- 8 \n\nIdA) \n\nFigure 4. The experiment of Figure 3 is repeated for several values of 12; experi(cid:173)\nmental data of output voltage response are plotted versus absolute input current on \na log scale. The output voltage VI = V2 is highlighted with a circle for each experi(cid:173)\nment. The dashed line is a theoretical expression confirming logarithmic behavior \nover four orders of magnitude (Equation 1). \n\n\f708 \n\nLazzaro, Ryckebusch, Mahowald and Mead \n\nFigure 9. Experimental data (circles) and theoretical statements (solid line) for a \ntwo-neuron winner-take-all circuit, showing the smallest 10 , for a given I, necessary \nfor a first-order response to a small-signal step input. \n\n1 \n\nFigure 10. Experimental data (symbols) and theoretical statements (solid line) for \na two-neuron winner-take-all circuit, showing the time constant of the first-order \nresponse to a small-signal step input. The winning response (filled circles) and losing \nresponse (triangles) of a winner-take-a.ll circuit are shownj the time constants differ \nby several orders of magnit ude. \n\n\fWinner~Take~AlI Networks ofO(N) Complexity \n\n709 \n\nTHE LOCAL NONLINEAR INHIBITION CIRCUIT \n\nThe winner-take-all circuit in Figure 1, as previously explained, locates the largest \ninput to the circuit. Certain applications require a gentler form of nonlinear inhibi(cid:173)\ntion. Sometimes, a circuit that can represent multiple intensity scales is necessary. \nWithout circuit modification, the winner-take-all circuit in Figure 1 can perform \nthis task. (Lazzaro et. al., 1988) explains this mode of operation. \n\nOther applications require a local winner-take-all computation, with each winner \nhaving inHuence over only a limited spatial area. Figure 12 shows a circuit that \ncomputes the local winner-taite-all function. The circuit is identical to the original \nwinner-take-all circuit, except that each neuron connects to its nearest neighbors \nwith a nonlinear resistor circuit (Mead, in press). Each resistor conducts a current \nIr in response to a voltage ~V across it, where Ir = I.tanh(~V/(2Vo)). 1., the \nsaturating current of the resistor, is a controllable parameter. The current source, \n10, present in the original winner-take-all circuit, is distributed between the resistors \nin the local winner-take-all circuit. \n\nFigure 11. Schematic diagram of a section of the local winner-take-all circuit. \nEach neuron i receives a unidirectional current input Iii the output voltages Vi \nrepresent the result of the local winner-take-all computation. \n\nTo understand the operation of the local winner-take-all circuit, we consider the \ncircuit response to a spatial impulse, defined as 1\" :> 1, where 1 == h~\". 1,,:> 1\"-1 \nand 1,,:> 1\"+1, so Ve:,. is much larger than Ve:,._l and Ve:lI+l' and the resistor circuits \nconnecting neuron 1: with neuron 1: - 1 and neuron 1: + 1 saturate. Each resistor \nsinks 1. current when saturatedj transistor T2,. thus conducts 21. + Ie: current. In \nthe subthreshold region of operation, the equation 1\" = 10 exp(Ve:,. /Vo) describes \ntransistor TI ,., and the equation 21. + Ie = Ioexp((V\" - Ve:,.)/Vo) describes transistor \n\n\f710 \n\nLazzaro, Ryckebusch, Mahowald and Mead \n\nT2,.. Solving for VA: yields \n\nVA: = voln((2I. + 10 )/10 ) + voln(IA:/lo). \n\n(4) \n\nAs in the original winner-take-all circuit, the output of a winning neuron encodes \nthe logarithm of that neuron's associated input. \n\nAs mentioned, the resistor circuit connecting neuron Ie with neuron Ie - 1 sinks 1. \nCUlTent. The current sources 10 associated with neurons Ie -1, Ie - 2, ... must supply \nthis current. If the current source 10 for neuron Ie - 1 supplies part of this current, \nthe transistor T2,._1 carries no current, and the neuron output VA:-l approaches zero. \nIn this way, a winning neuron inhibits its neighboring neurons. \n\nThis inhibitory action does not extend throughout the network. Neuron Ie needs \nonly 1. current from neurons Ie - 1, Ie - 2, .... Thus, neurons sufficiently distant \nfrom neuron Ie maintain the service of their current source 10, and the outputs of \nthese distant neurons can be active. Since, for a spatial impulse, all neurons Ie - 1, \nIe - 2, ... have an equal input current I, all distant neurons have the equal output \n\nSimilar reasoning applies for neurons Ie + 1, Ie + 2, .... \nThe relative values of 1. and 10 determine the spatial extent of the inhibitory action. \nFigure 12 shows the spatial impulse response of the local winner-take-all circuit, for \ndifferent settings of 1./10 , \n\n(5) \n\no \n\n2 \n\n4 \n\nI \n8 \n\nI \n10 \n6 \nIe (Pollition) \n\nI \n12 \n\nI \n14 \n\nI \n16 \n\nFigure 12. Experimental data showing the spatial impulse response of the local \nwinner-take-all circuit, for values of 1./10 ranging over a factor of 12.7. Wider \ninhibitory responses correspond to larger ratios. For clarity, the plots are vertically \ndisplaced in 0.25 volt increments. \n\n\fWinner-Take-All Networks ofO(N) Complexity \n\n711 \n\nCONCLUSIONS \n\nThe circuits described in this paper use the full analog nature of MOS devices to \nrealize an interesting class of neural computations efficiently. The circuits exploit \nthe physics of the medium in many ways. The winner-take-all circuit uses a single \nwire to compute and communicate inhibition for the entire circuit. Transistor TI,. \nin the winner-take-all circuit uses two physical phenomena in its computation: its \nexponential current function encodes the logarithm of the input, and the finite \nconductance of the transistor defines the losing output response. As evolution \nexploits all the physical properties of neural devices to optimize system performance, \ndesigners of synthetic neural systems should strive to harness the full potential of \nthe physics of their media. \n\nAcknow ledgments \n\nJohn Platt, John Wyatt, David Feinstein, Mark Bell, and Dave Gillespie provided \nmathematical insights in the analysis of the circuit. Lyn Dupre proofread the docu(cid:173)\nment. We thank Hewlett-Packard for computing support, and DARPA and MOSIS \nfor chip fabrication. This work was sponsored by the Office of Naval Research and \nthe System Development Foundation. \n\nReferences \n\nLazzaro, J. P., Ryckebusch, S., Mahowald, M.A., and Mead, C.A. (1989). Winner(cid:173)\nTake-All Networks of O(N) Oomplexity, Caltech Computer Science Department \nTechnical Report Caltech-CS-TR-21-88. \n\nLazzaro, J. P., and Mead, C.A. {in press}. Silicon Models of Auditory Localization, \nNeural Oomputation. \n\nMahowald, M.A., and Delbruck, T.I. (1988). An Analog VLSI Implementation of \nthe Marr-Poggio Stereo Correspondence Algorithm, Abstracts of the First Annual \nINNS Meeting, Boston, 1988, Vol. I, Supplement I, p. 392. \n\nMead, C. A. (in press). Analog VLSI and Neural Systems. Reading, MA: Addison(cid:173)\nWesley. \n\n\f", "award": [], "sourceid": 151, "authors": [{"given_name": "J.", "family_name": "Lazzaro", "institution": null}, {"given_name": "S.", "family_name": "Ryckebusch", "institution": null}, {"given_name": "M.A.", "family_name": "Mahowald", "institution": null}, {"given_name": "C. A.", "family_name": "Mead", "institution": null}]}