{"title": "A Hybrid Radial Basis Function Neurocomputer and Its Applications", "book": "Advances in Neural Information Processing Systems", "page_first": 850, "page_last": 857, "abstract": null, "full_text": "A Hybrid Radial Basis Function Neurocomputer \n\nand Its Applications \n\nSteven S. Watkins \nECE Department \n\nUCSD \n\nLa Jolla. CA. 92093 \n\nPaul M. Chau \nECE Department \n\nUCSD \n\nLa Jolla, CA. 92093 \n\nRaoul Tawel \n\nJPL \n\nCaltech \n\nPasadena. CA. 91109 \n\nBjorn Lambrigtsen \n\nJPL \n\nCaltech \n\nPasadena. CA. 91109 \n\nMark Plutowski \nCSE Department \n\nUCSD \n\nLa Jolla. CA. 92093 \n\nAbstract \n\nA neurocomputer was  implemented using radial basis functions  and a \ncombination  of  analog  and  digital  VLSI  circuits.  The  hybrid  system \nuses  custom  analog  circuits  for  the  input  layer  and  a  digital  signal \nprocessing board for the hidden and output layers. The system combines \nthe  advantages of both analog and digital circuits. featuring low power \nconsumption while minimizing overall system error. The analog circuits \nhave been fabricated and tested, the  system has been built,  and several \napplications  have  been  executed  on  the  system.  One  application \nprovides significantly better results for  a remote sensing problem than \nhave been previously obtained using conventional methods. \n\n1.0 Introduction \n\nThis  paper  describes  a  neurocomputer  development  system  that  uses  a  radial  basis \nfunction as the transfer function of a neuron rather than the traditional sigmoid function. \nThis neurocOOlputer is a hybrid system which has been implemented with a combination \nof analog  and  digital  VLSI  technologies.  It offers  the  low-power  advantage  of analog \ncircuits operating in the  subthreshold region and the high-precision advantage of digital \ncircuits. The system is targeted for applications that require low-power operation and use \ninput  data  in  analog  form,  particularly  remote  sensing  and  portable  computing \napplications.  It has  already  provided  significantly  better  results  for  a  remote  sensing \n\n850 \n\n\fA Hybrid Radial Basis Function Neurocomputer and Its Applications \n\n851 \n\n,-------- - ----- - -\n\nNEURON \n\n-\n\nYo \n\n2 \n(c I k  - 'k) \n\n:E \n\nEXPONENTlAL \n\n'0 \n\n'(cid:173)\n\n'NPUTS \n\nMUL TlPL Y AND ACCUMULATE \n\nFigure I: Radial Basis Function Network \n\nNEURoN \n\n[~ \n\nFigure 2: Mapping of RBF Network to Hardware \n\nAnalog Board \n\n= \n= \n\nPC \n\nFigure 3: The RBF Neurocomputer Development System \n\n\f852 \n\nWatkins, Chau, Tawel, Lambrigsten, and Plutowski \n\nclimate problem than have been previously obtained using conventional methods. \nFigure  1 illustrates  a radial  basis functioo  (RBF)  network.  Radial  basis  functions  have \nbeen  used  to  solve  mapping  and  function  estimation  problems  with  positive  results \n(Moody  and  Darken.  1989;  Lippman,  1991).  When  coupled  with  a  dynamic  neuron \nallocation  algorithm  such  as  Platt's RANN  (platt.  1991).  RBF networks can usually  be \ntrained much more quickly than a traditional sigmoidal. back-propagation network. \nRBF networlcs have been implemented with completely-analog (platt, Anderson and Kirk. \n1993), c<mpletely-digital (Watkins. Chau and Tawel, Nov .\u2022  1992). and with hybrid analogi \ndigital approaches (Watkins. Chau and Tawel, Oct., 1992). The hybrid approach is optimal \nfor applications which require low power consumption and use input data that is naturally \nin the analog domain while also requiring the high precision of the digital domain. \n\n2.0 System Architecture and Benefits \nFigure  2  shows  the  mapping  of  the  RBF  network  to  hardware.  Figure  3  shows  the \nneurocomputer development system. The system consists of a PC controller, a DSP board \nwith a Motorola 56000 DSP chip and a board with analog multipliers. The benefits of the \nhybrid  approach  are  lower-cost  parallelism  than  is  possible  with  a  completely-digital \nsystem, and more precise computation than is possible with a completely-analog system. \nThe parallelism is available for low cost in terms of area and power, when the inputs are in \nthe  analog  domain.  When comparing  a single  analog  multiplier to  a  100bit  fixed  point \ndigital multiplier,  the  analog cell uses  less  than one-quarter the  area  and  approximately \nfive orders of magnitude less power. When comparing an array of analog multipliers to a \nMotorola 56000 DSP chip, 1000 Gilbert multipliers can fit in an area about half the size of \nthe DSP chip, while consuming .003% of the power. \nThe restriction of requiring  analog inputs is placed on the  system. because if the  inputs \nwere digital, the high cost of D to A conversion would remove the low cost benefit of the \nsystem.  lbis restriction causes  the  neurocomputer  to  be  taIgeted for  applications  using \ninputs  that  are  in  the  analog  domain,  such  as  remote  sensing  applications  that  use \nmicrowave or infrared sensors and speech recognitioo applications that use analog filters. \nThe hybrid system reduces the  overall system error when compared  with a completely(cid:173)\nanalog solution. The digital circuits compute the hidden and output layers with 24 bits of \nprecision while analog circuits  are  limited to  about 8 bits  of precision.  Also  the RANN \nalgorithm requires  a large range  of width variatioo for  the Gaussian function  and  this is \nmore  easily  achieved  with  digital  computation.  Completely  analog  solutions  to  this \nproblem are severely limited by the voltage rails of the chip. \n\n3.0 Circuits \nSeveral different analog circuit approaches were explored as possible implementations of \nthe network. Mter the dust settled, we chose to implement only the input layer with analog \ncircuits  because  it  offers  the  greatest  opportunity  for  parallelism,  providing  parallel \nperformance benefits  at  a low cost in terms of area and power.  The input layer requires \nmore than  0 UP) computations (where N is the number of neurons). while the hidden and \noutput  layers  require  only  0  (N) computations  (because  there  is  one  hidden  layer \ncomputatioo per neuron and the number of outputs is either one or very small). \n\n\fA Hybrid Radial Basis Function Neurocomputer and Its Applications \n\n853 \n\nThe  analog  circuits  used  in  the  input  layer  are  Gilbert  multipliers  (Mead.  1989). 'The \ncircuits  were  fabricated  with  2.0 micron.  double-poly,  P-well.  CMOS  technology.  The \nGilbert cell performs the operation of multiplying two voltage differences:  (Vi-V2)x(V3-\nV4). In this system. Vi =V3 and V2=V4. which causes the circuit to compute the square of \nthe  difference between a stored weight and  the input. The current outputs of the Gilbert \ncells in a row are wired together to sum their currents. giving a sum of squared errors. This \ncurrent is converted to a voltage. fed to an A to D converter and then passed to the DSP \nboard  where  the  hidden  and  output  layers  are  computed.  The  radial  basis  function \n(Gaussian) of the hidden layer is computed by using a lookup table. The system uses the \nfast multiply/accumulate operation of the DSP chip to compute the output layer. \n\n4.0 Applications \nThe  low-power feature  of the  hybrid  system  makes  it  attractive  for  applications  where \npower  consumption  is  a  prime  consideration,  such  as  satellite-based  applications  and \nportable computing  (using battery power). The neurocomputer has been applied to three \nproblems:  a  remote  sensing  climate  problem.  the  Mackey-Glass  chaotic  time  series \nestimation and speech phoneme recognitim. The remote sensing application falls into the \nsatellite category. The Mackey-Glass  and  speech recognition applications  are potentially \nportable.  Systems  fa these  applications  are  likely  to have  inputs  in  the  analog domain \n(eliminating the need for D to A conversion.  as  already discussed) making it feasible  to \nexecute them on the hybrid neurocomputer. \n\n4.1 The Remote Sensing Application \n\nThe remote sensing problem is an  inverse mapping problem that uses microwave energy \nin different bands as input to predict the water vapor content of the atmosphere at different \naltitudes. Water vapor content is a key parameter for predicting weather in the tropics and \nmid-latitudes (Kakar and Lambrigtsen. 1984). The application uses 12 inputs and 1 output. \nThe system input is naturally in analog form. the result of amplified microwave signals, so \nno D  to A conversion of input data is required.  Others have  used neural networks  with \nsuccess  to perform  a similar inverse mapping to predict the  temperature gradient of the \natmosphere  CMotteler  et al ..  1993).  Section  5  details  the  improved  results  of the  RBF \nnetwork  over  conventional  methods.  Since  water  vapor  content  is  a  very  important \ncompment of climate models.  improved results  in predicted water vapor content means \nimproved climate models. \nRemote sensing problems require satellite hardware where power consumptim is always a \nmajor constraint.The low-power nature of the hybrid network would allow the network to \nbe  placed  on  board  a  satellite.  With  future  EOS  missions  requiring  several  thousand \nsensors. the on-board network would reduce the bandwidth requirements of the data being \nsent back to earth. allowing the reduced water vapor content data to be transmitted rather \nthan the raw sensor data. This data bandwidth reduction could be used either to send back \nmore meaningful data to further improve climate models. or to reduce the amount of data \ntransmitted. saving energy. \n\n4.2 The Mackey-Glass Application \n\nThe  Mackey-Glass  chaotic  time  series  application  uses  several  previous  time  sample \nvalues to predict the current value of a time series which was generated by the  Mackey(cid:173)\nGlass  delay-difference  equation.  It  was  used  because  it  has  proved  to be  difficult  for \n\n\f854 \n\nWatkins, Chau, Tawel, Lambrigsten, and Plutowski \n\nsigmoidal neural networks (platt. 1991). The applicatioo uses 4 inputs and  1 output. The \nMackey-Glass  time  series  is  representative of time  series found  in medical applications \nsuch as  detecting  arrhythmias in heartbeats.  It could be  advantageous to implement this \napplication with portable hardware. \n\n4.3 The Speech Phoneme Recognition Application \n\nThe speech phoneme recognition problem  used the  same data as  Waibel  (Waibel  et ai .\u2022 \n1989)  to  learn  to  recognize  the  acoustically  similar  phonemes  of  b.  d  and  g.  The \napplication  uses  240  inputs  and  3  outputs.  The  speech  phoneme  recognition  problem \nrepresents  a sub  problem  of the  more  difficult continuous  speech recognition problem. \nSpeech recognition applications also represent opportunities for portable computing. \n\n5.0 Results \n\n5.1 The Remote Sensing Application \n\nUsing  the  RBF  neural  network  00  the  remote  sensing  climate  problem  produced \nsignificantly better results than had been previously obtained using conventional statistical \nmethods  (Kakar  and  Lambrigtsen.  1984).  The  input  layer  of  the  RBF  network  was \nimplemented in two different ways:  1) it was simulated with 32-bit floating point precision \nto  represent  a  digital  input  layer.  and  2)  it was  implemented  with  the  analog  Gilbert \nmultipliers as the input layer. Both implementations produced similar results. \nAt an  altitude corresponding to 570 mb pressure, the RBF neural network with a digital \ninput  layer  produced  results  with  .33  absolute  rms  error vs.  .42  rms  error for  the  best \nresults using conventional methods. This is an improvement of 21 %. Figure 4 shows the \nplot  of  retrieved  vs.  actual  water  vapor  content  for  both  the  RBF  network  and  the \nconventional method. Using the hybrid neurocomputer with the analog input layer for the \ndata at 570 mb pressure produced results with .338 rms error. This is an improvement of \n19.5%  over  the  conventional method.  Using  the  analog  input layer  produced  nearly  as \nmuch  improvement  as  a  completely-digital  system.  demonstrating  the  feasibility  of \nplacing the network on board a satellite. Similar results were obtained for other altitudes. \nThe  RBF  network  also  was  compared  to  a  sigmoidal  network  using  back  propagation \nlearning  enhanced  with  line-search  capability  (to  automatically  set  step-size).  Both \nnetworks  used eight neurons in the hidden layer.  As Figure 5 shows.  the RBF network \nlearned much faster than the sigmoidal network. \n\n6 \n\n, \nKey ' \no - neural network \n+ _ startatical method \n\n--~-.~ --/ \n\n/' \n\n0  + \n\nf .. ::c \nJ!I !  ] \n\n2 \n\n] \n.;! \n\n+ \n\no \n\nfit-\n\n+ \n\n0 \n\n+ \n0 . - 0 0 \n\nFigure 4:  Comparison of Retrieved vs. Actual Water Vapor Content for 570 mb Pressure \n\nfor RBF Network and Conventional Statistical Method \n\nAct \u2022\u2022 1 Specific l/...,td,1y \n\n6 \n\n\fA Hybrid Radial Basis Function Neurocomputer and Its Applications \n\n855 \n\n------~-solid  _ r bl software \n\ndaahed - rbl analog hardware \ndotted - sigmoid b.IiCkprop \n\n1 1 \n\n09 \n\n~08 \n~0.7 \n\n06 \n\n05 \n\n'_ \n\n-----=-.::..::.~-:.~ ~-::...:..~_- ------____ . __ _ \n\no 4 \n03  ---;-\u00b7-2--3\"4  5  e \n\n8 \nnumber 01 passes through training patter!'!s \n\n-7 \n\n..L...-\n9 \n\nFigure 5: Comparison of Learning Curves for RBF and Sigmoidal Networks for Water \n\nVaptt Application \n\n03  -\n\n-- - , - - , - - - - - , . - - , ---,-, - - , - --\n\nKey' \nsolid  _ rbl software \ndashed - rbl analog hardware \ndotted - Sigmoid b.Iickprop \n\n.... .. .. ..... .  ...... ......... .... . .. . ,  .. .. ..... ..... .... ..... . \n\n025 . \n\n02 \n\n~O 15 \n\u00a7 \n\n01 \n\n\\ ... -\n\n....  -..... ~ ........ --- ....  .,-- \"  .. ....  ,. ........ \n\n%~-~0'5~~; \u00b7--~1~5--~2~~2.5~-73--~3~.5--~4 \nx 10. \n\nnumber 01  paaaes through training patterns \n\nFigure 6: Comparison of Learning Curves for RBF and Sigmoidal Networks for Mackey(cid:173)\n\nGlass Application \n\n5.2 The Mackey-Glass Application \n\nThe RBF network was not compared to any non-neural network method for the Mackey(cid:173)\nGlass time series estimation. It was only compared to a traditional sigmoidal networlc \nusing back propagation learning enhanced with line search. Both networks used four neu(cid:173)\nrons. As Figure 6 shows. applying the RBF neural network to the Mackey-Glass chaotic \ntime series estimation produced much faster learning than the sigmoidal network. The \nRBF network with a digital input layer and the RBF hybrid network with an analog input \nlayer both produced similar results in dropping to an rms error of about .025 after only 5 \nminutes of training on a PC using a 486 CPU. \nUsing the digital input layer. the RBF network reached a minimum absolute rms error of \n.017. while the sigmoidal network reached a minimum absolute rms error of .025. This is \nan  improvement of 32%  over  the  sigmoidal  network.  Using  the  hybrid  neurocomputer \nwith  the  analog  input layer produced a minimum absolute rms error of .022. This  is  an \nimprovement of 12% over the sigmoidal network \n\n\f856 \n\nWatkins, Chau, Tawel, Lambrigsten, and Plutowski \n\n5.3 The Speech Phoneme Recognition Application \n\nThe RBF network was  not compared to  any non-neural network method for  the speech \nphooeme  recognition  problem.  It was  only  compared  to  Waibel's  Tme Delay  Neural \nNetwork (IDNN) (Waibel et al .. 1989). The IDNN uses a topology matched to the time(cid:173)\nvarying nature of speech with two hidden layers of eight and three neurons respectively. \nThe RBF network used a single hidden layer with the number of neurons varying between \neight and one hundred. \nThe IDNN achieved a 98% accuracy on the test set discriminating between the phooemes \nb. d and g. The RBF network achieved over 99% accuracy in training. but was only able to \nachieve  an  86%  accuracy  on  the  test  set.  To  obtain  better results.  it  is  clear  that  the \ntopology of the RBF network needs to be altered to more closely match Waibel's IDNN. \nHowever. this topology will complicate the VLSI implementation. \n\n5.4 The Feasibility of Using the Analog Input Layer \n\nOne potential problem  with  using  an  analog  input layer is  that every  individual  hybrid \nRBF neurocomputer might need to be trained on a problem. rather than being able to use a \ncommon  set  of weights  obtained  from  another  RBF  neurocomputer  (which  had  been \npreviously trained). This potential problem exists because every analog circuit is unique \ndue to variation in the fabrication process. A set of experiments was designed to test this \npossibility. \nThe remote sensing application and the Mackey-Glass application were trained and tested \ntwo  different  ways:  1)  hardware-trainedlhardware-tested.  that  is.  the  analog  input layer \nwas  used  for  both  training  and  testing;  2)  software-trainedlhardware-tested.  that  is  the \nanalog input layer was simulated with 32-bit floating point precision for training and then \nthe  analog  hardware  was  used  for  testing . . The  hardwarelhardware  results  provided  a \nbenchmark.  The  softwarelhardware  results  demonstrated  the  feasibility  of  having  a \nstandard set of weights that are not particular to a given set of analog hardware. For both \nthe  remote  sensing  and  the  Mackey-Glass  applications.  the  rms  error performance was \nonly slightly degraded by using weights  learned during software simulation. The remote \nsensing results  degraded by  only  .Oll  in  terms  of absolute rms  error.  and  the  Mackey(cid:173)\nGlass  results  degraded  by  only  .002  in  terms  of absolute  rms  error.  The  results  of the \nexperiment  indicate  that each individual  hybrid  RBF  neurocomputer  only  needs  to  be \ncalibrated. not trained. \n\n6.0 Conclusions \nA low-power. hybrid analog/digital neurocomputer development system was constructed \nusing  custom hardware.  The system  implements a radial  basis  function  (RBF) network \nand is targeted for applications that require low power consumption and use analog data as \ntheir  input.  particularly  remote  sensing  and  portable  applications.  Several  applications \nwere executed and results were obtained for a remote sensing application that are superior \nto any previous results. Comparison of the results of a completely-digital simulation of the \nRBF network and  the hybrid analog/digital RBF network demonstrated the feasibility of \nthe hybrid approach. \n\n\fA Hybrid Radial Basis Function Neurocomputer and Its Applications \n\n857 \n\nAcknowledgments \nThe  research  described  in  this  paper  was  performed  at  the  Center  for  Space \nMicroelectronics  Technology.  Jet  Propulsion  Laboratory.  California  Institute  of \nTechnology, and  was  sponsored by  the National Aerooautics  and  Space  Admjnjstration. \nOne of the  authors.  Steven S.  Watkins.  acknowledges the receipt of a Graduate  Student \nResearcher's Center Fellowship from the Natiooal Aeronautics and Space Administration. \nUseful  discussions  with  Silvio  Eberhardt,  Roo  Fellman.  Eric  Fossum.  Doug  Kerns. \nFernando Pineda, John Platt, and Anil Thakoor are also gratefully acknowledged. \n\nReferences \nRamesh Kakar and Bjorn Lambrigtsen. \"A Statistical Correlation Method for the Retrieval \nof Atmospheric  Moisture Profiles  by  Microwave Radiometry,\"  Journal  of Climate  and \nApplied Meteorology. vol. 23, no. 7. July 1984, pp.  1110-1114. \nR. P. Lippman. \"A Critical Overview of Neural Network Pattern Oassifiers.\" Proceedings \nof the IEEE Neural Networks for Signal Processing Workshop,  1991, Princeton. NJ .\u2022 pp. \n266-275. \nCarver Mead, Analog VLSI and Neural Systems. Addison-Wesley. 1989, pp. 90-94. \nJ.  Moody  and  C.  Darken.  \"Fast  Learning  in  Networks  of Locally-Tuned  Processing \nUnits,\" Neural Computation, vol.  1. no. 2, Summer 1989. pp. 281-294. \nHoward Motteler, lA. Gualtieri. LL. Strow and Larry McMillin. \"Neural Networks for \nAtmospheric Retrievals,\" NASA Goddard Conference  on Space Applications of Artificial \nIntelligence. 1993, pp.  155-167. \nJohn Platt, \"A Resource-Allocating Neural Network for Function Interpolation,\" Neural \nComputation, vol. 3. no. 2, Summer 1991, pp. 213-225. \nJohn Platt. Janeen Anderson and David B. Kirk. \"An Analog VLSI Qrip for Radial Basis \nFunctions,\" NIPS 5.  1993, pp. 765-772. \nAlexander  Waibel.  T.  Hanazawa.  G.  Hinton.  K.  Shikano  and  K.  Lang.  \"Phoneme \nRecognition  Using  Tune-Delay  Neural  Networks.\"  IEEE  International  Conference  on \nAcoustics, Speech and Signal Processing, May 1989, pp. 393-404. \nSteve Watkins, Paul  Chau  and  Raoul Tawel.  \"A Radial Basis Functioo Neurocomputer \nwith an Analog Input Layer.\" Proceedings of the IJCNN, Beijing. China. November 1992. \npp. ill 225-230. \nSteve Watkins.  Paul Chau  and Raoul Tawel, \"Different Approaches  to Implementing A \nRadial  Basis  Function  Neurocomputer.\"  RNNSIlEEE  Symposium  on  Neuroinformatics \nand Neurocomputing. Rostov-on-Don. Russia. October 1992, pp. 1149-1155. \n\n\f", "award": [], "sourceid": 810, "authors": [{"given_name": "Steven", "family_name": "Watkins", "institution": null}, {"given_name": "Paul", "family_name": "Chau", "institution": null}, {"given_name": "Raoul", "family_name": "Tawel", "institution": null}, {"given_name": "Bjorn", "family_name": "Lambrigtsen", "institution": null}, {"given_name": "Mark", "family_name": "Plutowski", "institution": null}]}