{"title": "Dynamically Adaptable CMOS Winner-Take-All Neural Network", "book": "Advances in Neural Information Processing Systems", "page_first": 713, "page_last": 719, "abstract": null, "full_text": "Dynamically Adaptable CMOS \n\nWinner-Take-AII Neural Network \n\nKunihiko Iizuka, Masayuki Miyamoto and Hirofumi Matsui \n\nInformation Technology Research Laboratories \n\nSharp \n\nTenri, Nara, lAP AN \n\nAbstract \n\nThe major  problem  that  has  prevented  practical  application  of analog \nneuro-LSIs  has  been  poor  accuracy  due  to  fluctuating  analog  device \ncharacteristics  inherent  in  each  device  as  a  result  of  manufacturing. \nThis paper proposes a dynamic control architecture that  allows analog \nsilicon  neural  networks  to  compensate  for  the  fluctuating  device \ncharacteristics  and  adapt  to  a  change  in  input  DC  level.  We  have \napplied  this  architecture to  compensate  for  input  offset  voltages of an \nanalog  CMOS  WTA (Winner-Take-AlI)  chip  that we  have  fabricated. \nExperimental data show the effectiveness of the architecture. \n\nINTRODUCTION \n\n1 \nAnalog  VLSI  implementation  of neural  networks,  such  as  silicon  retinas  and  adaptive \nfilters,  has  been  the  focus  of much  active  research.  Since  it  utilizes  physical  laws  that \nelectric devices obey for neural operation, circuit scale can be much smaller than that of a \ndigital counterpart and  massively parallel implementation is possible. The major problem \nthat has prevented practical applications of these LSIs has  been fluctuating  analog device \ncharacteristics inherent  in  each device  as a result of manufacturing.  Historically,  this  has \nbeen  the  main  reason  most  analog  devices  have  been  superseded  by  digital  devices. \nAnalog neuro VLSI is expected to conquer this problem by making use of its adaptability. \nThis  optimistic  view  comes  from  the  fact  that  in  spite  of  the  unevenness  of  their \ncomponents, biological neural networks show excellent competence. \n\nThis  paper  proposes  a  CMOS  circuit  architecture  that  dynamically  compensates  for \nfluctuating  component  characteristics  and  at  the  same  time  adapts  device  state  to \nincoming  signal  levels.  There  are  some  engineering  techniques  available  to  compensate \n\n\f714 \n\nK.  lizuka, M.  Miyamoto and H. Matsui \n\nfor  MOS  threshold fluctuation,  e.g.,  the  chopper comparator, but  they  need a  periodical \nchange of mode to achieve the desired effect. This is because there are two modes one for \nthe adaptation and one for the signal processing. This is quite inconvenient because extra \nclock signals are needed and a break of signal processing takes place. \n\nIncoming  signals  usually  consist  of  a  rapidly  changing  foreground  component  and  a \nslowly  varying  background  component.  To  process  these  signals  incessantly,  biological \nneural  networks  make use of multiple  channels  having  different  temporaVspatial  scales. \nWhile  a  relatively  slow/large  channel  is  used  to  suppress  background  floating,  a \nfaster/smaller channel  is  devoted to  process the foreground signal. The proposed method \ninspired by this biological consideration  utilizes different frequency  bands for  adaptation \nand signal  processing (Figure 1), where  negative feedback  is  applied through  a low pass \nfilter so that the feedback will not affect the foreground signal processing. \n\nCOMI'ARATOR \n\nGain \n\nLOW PASS \n\nFILTER \n\nInput Signal \n\nOutput SignAl \n\nBACKGROUND \n\nnAND \n\nFOREGROUND \n\nBAND \n\nFrequency \n\n(a) \n\n(b) \n\nFigure  1:  Dynamic  adaptation  by  frequency  divided  control.  (a)  model  diagram,  (b) \nfrequency division. \n\nIn  the  first  part  of this  paper,  a  working  analog  CMOS  WTA  chip  that  we  have  test \nfabricated  is  introduced. Then, dynamical adaptation for this WT A chip is described and \nexperimental results are presented. \n\n2  ANALOG CMOS WTA CHIP \n\n2.1  ARCHITECTURE AND SPECIFICATION \n\nv,\" I \n\n2nd  LAYER \n\nCM \n\nFEEDBACK \n\nCONTROLLER \n\n\u2022 \n\nFigure 2: Analog CMOS WT A chip architecture \n\n\fDynamically Adaptable CMOS Wmner-Take-All Neural Network \n\n715 \n\nM5 \nVhl--f \neM \nIn l)U~ \n\nMI \n\n114 2 \n\nVdd \n\nM4 \nI--Vb2 \n\nM \nOlltput \n\nV!!IS \n\nVII\\~ \n\nI \nI \nI \n\n(ll) \n\n(b) \n\nFigure 3: Circuit diagrams for (a) the competitive cell and (b) the feedback controller. \n\nAs  a  basic  building  block  to  construct  neuro-chips,  analog  Wf A  circuits  have  been \ninvestigated  by  researchers  such  as  [Lazzaro,  1989]  and  [Pedroni,  1994].  All  CMOS \nanalog  WfA circuits  are  based  on  voltage  follower  circuits  [Pedroni,  1995]  to  realize \ncompetition through inhibitory interaction, and they use feedback mechanisms to enhance \nresolution gain. The architecture of the chip that we  have fabricated  is shown  in  Figure 2 \nand the circuit diagram is  in  Figure 3.  This Wf A  chip indicates the  lowest  input  voltage \nby  making the output  voltage corresponds to  the  lowest input  voltage near Vss (winner), \nand others  nearly  the  power supply  voltage  Vdd  (loser).  The circuit is  similar to  [Sheu, \n1993], but represents two advances. \n\n1.  The  steering  current  that  the  feedback  controller  absorbs  from  the  line  CM  is \nenlarged,  allowing  the  winner  cell  can  compete  with  others  in  the  region  where \nresolution gain is the largest. \n\n2  The  feedback  controller  originally  placed  after  the  second  competitive  layer  is \nremoved in order to guarantee the existence of at least one output node whose voltage \nis nearly zero. \n\nTable 1 shows the specifications of the fabricated chip. \n\nTable 1: Specifications of the fabricated WfA chip \n\n2.2 \n\nINPUT OFFSET VOLTAGE \n\nInput offset voltages of a Wf A chip may greatly deteriorate chip performance. Examples \nof input  offset  voltage  distribution  of the  fabricated  chips  are  shown  in  Figure  4.  Each \ninput offset voltage  is  measured  relative  to  the  first  input  node. The input  offset  voltage \n\n\f716 \nK.  lizuJea,  M.  Miyamoto and H.  Matsui \n~Vj of the  j-th  input  node  is  defined  as  ~Vj = Vinj  - Vin1  when  the  voltages  of output \nnodes Outj and Out1 are equal;  Vin1  is  fixed to  a certain voltage and  the  voltage of other \ninput nodes are fixed at a relatively high voltage. \n\nFigure 4: Examples of measured input offset voltage distribution. \n\nThe  primary  factor  of the  input  offset  voltage  is  considered  to  be  fluctuation  of MOS \ntransistor  threshold  voltages  in  the  first  layer  competitive  cell.  Then,  the  input  offset \nvoltage  ~ Vj  of this  cell  yielded  by  the  small  fluctuation  ~ Vth i  of Vthi  is  calculated  as \nfollows: \n\n_ -~Vtht + gd1 + gd 2 + gm2  (~Vth2 _ ~Vth3) + gm4(gd1 + gd2 + gm2)  ~Vth4 \n\ngmt \n\ngm1gm3 \n\nwhere  gmi  and  gdl  are  the  transconductance  and  the  drain  conductance  of  MOS  Mi, \nrespectively.  Using  design  and  process  parameters,  we  can  estimate  the  input  offset \nvoltage to be \n\nAVj. -AVthl + (AVth2 -AVth3 )+O.l5AVth4 \n\n, \n\nBased on  our experiences,  the  maximum  fluctuation  of Vth i  in  a  chip  is  usually  smaller \nthan  20  mY,  and  it  is  reasonable  to  consider  that  the  difference  I~Vth2 - ~VtJrI is  even \nsmaller;  perhaps  less  than 5  m V,  because  M2  and  M3  compose  a current  mirror and  are \nclosely placed. This  implies that the  maximum  of ~Vj is  about  28  mY,  which  is  in  rough \nagreement with the measured data. \n\n3  DYNAMICAL ADAPTATION ARCmTECTURE \nIn Figure 5, we show circuit implementation of the dynamically adaptable wr A function. \nIn  each  feedback  channel,  the  difference  between  each  output  and the  reference  Vref is \nfed  back  to  the  input  node  through  a  low  pass  filter  consisting of Rand C.  The  charge \nstored in capacitor C is controlled by this feedback signal. \nLet the linear approximation of the wr A chip DC characteristic be \n\nVouti  = A  ( Vin, - VOJ, \n\nwhere Vini  and Voutt  are the  voltages at the  nodes In/ and Out, respectively, andA and VOL \nare functions of Vinj (j '\" i ). The input offset voltage relative to  the node In] is considered \nto be the  difference  between  VOL  and V01. On the other hand, the DC characteristic of the \ni-th feedback path can be approximated as \n\n\fDynamically Adaptable CMOS Winner- Take-All Neural Network \n\n717 \n\nIn2' \nInl' \nCl..  Cl.. \n\n, \nIn32 \ncl.. \n\nR \n\n\u2022 \n\n\u2022 \n\n\u2022 \n\nR' \n\nInl \n\nOut l \n\nIn2 \n\nOut2 \n\n\u2022 \n\n\u2022 \n\u2022 \nWTA  Chip \n\u2022 \n\u2022 \n\n\u2022 \n\n\u2022  \u2022  \u2022 \n\nVref \n\nVref \n\n\u2022 \n\n\u2022 \n\n\u2022 \n\nOutl \n\nOut2 \n\nOutn \n\nFigure 5: wrA chip equipped with adaptation circuit where R=10MQ and C=0.33JAF. \n\nIt follows from the above two equations that \n\nYin;  = B (Yout; - Vref). \n\nAB \n\nB \n\nI \n\nVin \u00b7 - - - - Vo\u00b7  - - - Vrer \u2022  Vo \u00b7 \nI \n\nI  1- AB \n\n1- AB \n\n'.J \n\nThe last  term  is  derived using the  assumptions A  \u00bb  1 and B \u00ab  -1. This  means  that  the \nvoltage difference between the DC level of the  input and YO;  is clamped on the capacitor \nC. This in turn implies that the input offset voltage will be successfully compensated for. \n\nThe role of the low pass filters  is twofold. \n\n1.  They  guarantee  stable  dynamics  of the  feedback  loop;  we  can  make  the  cutoff \nfrequency of the low pass filters  small enough so that the gain of the feedback path is \nattenuated before the phase of the feedback signal is delayed by more than 1800  \u2022 \n\n2. They  prevent  the feed-forward  wr A  operation from  being  affected,  as  shown  in \nFigure 1, the adaptive control is carried out on a different, non-overlapped frequency \nband than wr A operation. \n\n4  EXPERIMENTAL RESULTS \nExperiments concerning the adaptable wr A function  were carried out by applying pulses \nof 90%  duty  to  the  input  nodes In',  and  In'l,  while  other  input  nodes  were  fixed  to  a \ncertain  voltage.  In  Figures 6 (a) and 6 (b),  the  output waveforms of Outl ,  Out2,  Out] and \nthe waveform of the pulse applied to  the node In', are shown. Figure 6(a) shows the  result \nwhen  the same  pulse was applied to both In',  and In'z.  Figure 6(b) shows the  result when \nthe amplitude of the  pulse to In', was greater than  that of the  pulse to In'z  by  10 mY.  The \nschematic explanation  of this  behavior is  in  Figure 7. The outputs remained  at  the  same \nlevels for a while after the inputs were shut off, since there was no  strong inducement.  As \na  result  of adaptation,  the  winning  frequencies  of every output nodes  become equal  in  a \nlong time scale. This explains the unstable output during the period of quiescent inputs. \n\n\f718 \n\nK.  Iizuka. M.  Miyamoto and H.  Matsui \n\nThe chip used  in  this  measurement had  a  relative  input offset voltage  of 15  mV  between \nnodes  In1  and In2\u2022  We  can  see  in  Figure  6  (a)  that  this  offset  voltage  was  completely \ncompensated for because the output waveforms of corresponding nodes were the same. \n\n(a) \n\n(b) \n\nFigure  6:  The  output  waveforms  ot  the  dynamically  adaptable  CMOS  WT A  neural \nnetwork. Pulse waves were applied to nodes In'] and In'2;  other nodes voltages were fixed. \nWhen the amplitude of each pulse was the same (a),  the corresponding output waveforms \nwere the same. When the amplitude of the pulse fed to In'I was greater than that to In'2  by \n10 mV (b),  the output voltage at Out]  was low (winner) and that at Out2  was high (loser) \nduring the period the pulse was low (on). \n\nInputs \n\nquiescent \n\nOutputs \n\nOut 1 i;i  Q, L..\" \n\nmii~iii~~iii~!iim \n\nOut2 \n\n~ Out3  iiii .......... Lo- s-er-\n\n~mmmmmm~mm \n\n\u2022 \u2022 \u2022 \n\n\u2022 \u2022 \u2022 \n\nroser \n\n. ~mmmmm~mmm \n~'---v---' \nHysteresis  Unstable \n\nFigure 7: The schematic explanation of the dynamically adaptable WT A behavior. \n\n5  CONCLUSION \nWe  have  proposed a dynamic adaptation architecture that  uses frequency  divided control \nand  applied  this  to  a  CMOS  WT A  chip  that  we  have  fabricated.  Experimental  results \nshow that the architecture successfully compensated for input offset voltages of the WT A \n\n\fDynamically Adaptable CMOS Winner- Take-All Neural Network \n\n719 \n\nchip  due  to  inherent  device characteristic fluctuations.  Moreover,  this  architecture  gives \nanalog  neuro-chips  the  ability  to  adapt  to  incoming  signal  background  levels.  This \nadaptability has a lot of applications. For example, in  vision chips,  the adaptation may be \nused to compensate for the fluctuation of photo sensor characteristics, to adapt the gain of \nphoto sensors to background illumination level and to automatically control color balance. \nAs  another  application,  Figure  8  describes  an  analog  neuron  with  weighted  synapses, \nwhere the time constant RC is much larger than the time constant of input signals. \n\nInputs \n\n11  11 \n\n\u2022\u2022\u2022 \n\nc \n\nOutput \n\nFigure  8:  Analog  neuron  with  weighted  synapses  where  the  time  constant  RC  is  much \nlarger than that of input signals. \n\nThe key  to  this  architecture  is  use  of non-overlapping frequency  bands for  adaptation  to \nbackground  and \nrequires \nimplementing  circuits  with  completely  different  time  scale  constants.  In  modern  VLSI \ntechnology,  however,  this  is  not  a  difficult  problem  because  processes  for  very  high \nresistances, i.e., teraohms, are available. \n\nforeground  signal  processing.  For  neuro-VLSIs, \n\nthis \n\nAcknowledgment \n\nThe authors would like to  thank Morio Osaka for  his help in  chip fabrication  and Kazuo \nHashiguchi for his support in experimental work. \n\nReferences \n\nChoi,  J.  &  Sheu,  B.J.  (1993)  A  high-precision  VLSI  winner-take-all  circuit  for  self(cid:173)\norganizing neural networks. IEEE J.  Solid-State Circuits, vo1.28,  no.5, pp.576-584. \n\nLazzaro,  J.,  Ryckebush,  S.,  Mahowald,  M.A.,  &  Mead,  C.  (1989)  Winner-take-all \nnetworks of O(N) complexity.  In D.S. Touretzky (eds.), Advances in Neural Information \nProcessing Systems 1, pp. 703-711. Cambridge, MA: MIT Press. \n\nPedroni,  V.A.  (1994)  Neural  n-port  voltage comparator network,  Electron.  Lett.,  vo1.30, \nno.21, pp1774-1775. \n\nPedroni, V.A. (1995) Inhibitory Mechanism Analysis of Complexity O(N) MOS Winner(cid:173)\nTake-All Networks. IEEE Trans.  Circuits Syst. I,  vo1.42,  no.3, pp.172-175. \n\n\f", "award": [], "sourceid": 1281, "authors": [{"given_name": "Kunihiko", "family_name": "Iizuka", "institution": null}, {"given_name": "Masayuki", "family_name": "Miyamoto", "institution": null}, {"given_name": "Hirofumi", "family_name": "Matsui", "institution": null}]}