{"title": "Temporal Patterns of Activity in Neural Networks", "book": "Neural Information Processing Systems", "page_first": 297, "page_last": 300, "abstract": null, "full_text": "297 \n\nTEMPORAL PATTERNS OF ACTIVITY IN \n\nNEURAL NETWORKS \n\nPaolo Gaudiano \n\nDept.  of Aerospace  Engineering Sciences, \n\nUniversity of Colorado,  Boulder CO  80309,  USA \n\nJanuary 5,  1988 \n\nAbstract \n\nPatterns  of activity  over  real  neural  structures  are  known  to  exhibit  time(cid:173)\n\ndependent  behavior.  It  would  seem  that  the  brain  may  be  capable  of utilizing \ntemporal  behavior of activity in neural  networks as  a  way  of performing functions \nwhich cannot otherwise be easily implemented.  These might include the origination \nof sequential behavior and  the  recognition  of time-dependent  stimuli.  A  model  is \npresented  here  which  uses  neuronal  populations  with  recurrent  feedback  connec(cid:173)\ntions in an attempt to observe and describe  the resulting time-dependent behavior. \nShortcomings and problems inherent  to  this model are  discussed.  Current models \nby other  researchers  are  reviewed  and their similarities and differences  discussed. \n\nMETHODS  /  PRELIMINARY RESULTS \n\nIn previous  papers,[2,3]  computer models  were  presented  that  simulate a  net  con(cid:173)\n\nsisting of two spatially organized populations of realistic neurons.  The populations are \nrichly  interconnected  and  are  shown  to  exhibit  internally  sustained  activity.  It  was \nshown that if the neurons have response times significantly shorter than the typical unit \ntime characteristic of the input  patterns  (usually  1 msec),  the populations will exhibit \ntime-dependent behavior.  This will typically result in the net falling into a  limit cycle. \nBy a limit cycle, it is meant that the population falls into activity patterns during which \nall of the active cells  fire  in  a  cyclic,  periodic  fashion.  Although  the period of firing  of \nthe individual  cells  may be  different,  after  a  fixed  time  the  overall population activity \nwill  repeat  in  a  cyclic,  periodic  fashion.  For  populations  organized  in  7x7  grids,  the \nlimit  cycle  will  usually  start  20~200 msec  after  the input  is  turned off,  and  its period \nwill be in  the order of 20-100  msec. \n\nThe point ofinterest is that ifthe net is allowed to undergo synaptic modifications by \nmeans  of a  modified  hebbian learning rule  while being presented with a  specific  spatial \npattern (i.e., cells at  specific  spatial locations within the net are externally stimulated), \nsubsequent  presentations  of the  same  pattern  with  different  temporal  characteristics \nwill cause the population to recall patterns which are spatially identical (the same cells \nwill be active)  but which have different  temporal qualities.  In other words, the net can \nfall  into a  different  limit  cycle.  These limit cycles  seem  to behave as  attractors in that \nsimilar input  patterns will result  in  the  same limit cycle,  and hence  each distinct limit \ncycle  appears  to have a  basin of attraction.  Hence a  net  which can  only learn a  small \n\n\u00a9 American Institute of Physics 1988 \n\n\f298 \n\nnumber  of spatially  distinct  patterns  can recall  the  patterns in a  number  of temporal \nmodes.  If it were possible to quantitatively discriminate between such temporal modes, \nit  would  seem  reasonable  to  speculate  that  different  limit  cycles  could  correspond  to \ndifferent  memory traces.  This would  significantly increase estimates on the  capacity of \nmemory  storage in the net. \n\nIt has  also been shown that a net being presented with a  given pattern will fall and \nstay  into  a  limit  cycle  until another  pattern is  presented  which  will  cause  the  system \nto fall  into a  different  basin of attraction.  If no other  patterns  are presented,  the  net \nwill remain in  the  same limit  cycle  indefinitely.  Furthermore, the net  will fall  into the \nsame  limit  cycle  independently  of the  duration  of the  input  stimulus,  so  long  as  the \ninput stimulus is  presented for  a  long enough time to raise the population activity level \nbeyond  a  minimum necessary  to  achieve  self-sustained  activity.  Hence,  if we  suppose \nthat  the  net  \"recognizes\"  the input  when  it  falls  into the  corresponding  limit  cycle,  it \nfollows  that the net will recognize a string of input patterns regardless of the duration of \neach input pattern, so long as each input is presented long enough for the net  to fall into \nthe  appropriate limit  cycle.  In particular,  our  system is  capable of falling  into  a  limit \ncycle within some tens of milliseconds.  This can be fast enough to encode, for example, a \nstring of phonemes as would typically be found in continuous speech.  It may be possible, \nfor  instance,  to create a  model  similar  to Rumelhart  and McClelland's  1981  model  on \nword recognition by appropriately connecting multiple layers  of these networks.  If the \nresponse  time  of the  cells  were  increased  in  higher  layers,  it may  be  possible  to have \nthe  lowest  level  respond  to  stimuli  quickly  enough  to  distinguish  phonemes  (or  some \nsub-phonemic basic linguistic unit), then have populations from this first  level feed  into \na  slower,  word-recognizing population layer,  and  so  On.  Such a  model  may  be  able  to \nperform word  recognition from an input  consisting of continuous phoneme strings even \nwhen  the phonemes  may vary in duration of presentation. \n\nSHORTCOMINGS \n\nUnfortunately, it was  noticed a  short  time ago  that a  consistent  mistake had been \nmade in  the process  of obtaining the  above-mentioned results.  Namely,  in  the  process \nof decreasing the response time of the cells I accidentally reached a  response time below \nthe  time  step  used  in the  numerical approximation that  updates  the  state of each cell \nduring  a  simulation.  The  equations  that  describe  the  state of each cell  depend  on the \nstate of the  cell  at  the  previous  time step  as  well  as  on the  input  at  the  present  time. \nThese  equations  are  of first  order  in  time,  and  an  explicit  discrete  approximation  is \nused in the model.  Unfortunately it is a known fact  that care must be taken in selecting \nthe  size  of the  time  step  in  order to  obtain  reliable  results.  It is  infact  the  case  that \nby reducing  the  time step to  a  level  below  the response  time of the cells  the dynamics \nof the  system  varied  significantly.  It is  questionable  whether  it  would  be  possible  to \nadjust  some of the population parameters within reson to obtain the same results with \na  smaller  step  size,  but  the  following  points  should  be  taken  into  account:  1)  other \nresearchers have created similar models that show such cyclic behavior (see for  example \nSilverman,  Shaw  and  Pearson[7]).  2)  biological  data exists  which  would  indicate  the \nexistance of cyclic  or periodic bahvior in real neural systems  (see for  instance Baird[1]). \nAs  I just recently completed a  series  of studies  at  this  university,  I  will  not  be able \nto perform a  detailed examination of the system described here, but instead I  will more \n\n\f299 \n\nthan likely create new models on different research equipment which will be geared more \nspecifically  towards the study of temporal behavior in neural networks. \n\nOTHER  MODELS \n\nIt  should  be  noted  that  in  the  past  few  years  some  researchers  have  begun  inves(cid:173)\n\ntigating  the  possibility  of neural  networks  that  can exhibit  time-dependent  behavior, \nand I  would like to report on some of the available results as they relate to the topic of \ntemporal patterns.  Baird[l] reports findings  from the rabbit's olfctory bulb which indi(cid:173)\ncate the existance of phase-locked oscillatory  states  corresponding to olfactory stimuli \npresented to the subjects.  He outlines an elegant model which attributes pattern recog(cid:173)\nnition abilities  to  competing instabilities in  the  dynamic  activity of neural  structures. \nHe  further  speculates  that  inhomogeneous  connectivity  in  the  bulb  can  be  selectively \nmodified  to achieve input-sensitive oscillatory states. \nSilverman, Shaw and Pearson[7] have developed a model based on a biologically-inspired \nidealized  neural  structure,  which  they  call  the  trion.  This  unit  represents  a  localized \ngroup of neurons with a  discrete firing  period.  It  was found  that small ensembles of tri(cid:173)\nons  with symmetric connections can exhibit  quasi-stable periodic firing  patterns which \ndo  not  require  pacemakers  or  external  driving.  Their  results  are  inspired  by  existing \nphysiological data and  are consistent  with other works. \nKleinfeld[6],  and  Sompolinsky  and  Kanter[8]  independently  developed  neural network \nmodels that  can generate and recognize sequential or cyclic patterns.  Both models rely \non  what could be summarized as the recirculation of information through time-delayed \nchannels. \nVery  similar results  are  presented by Jordan[4]  who  extends  a  typical  connectionist  or \nPDP  model  to  include  state  and  plan  units  with  recurrent  connections  and  feedback \nfrom  output  units  through  hidden  units.  He  employs  supervised  learning  with  fuzzy \nconstraints to induce learning of sequences  in  the  system. \nFrom  a  slightly  different  approach,  Tank  and  Hopfield[9]  make USe  of patterned  sets \nof delays  which effectively compress  information in  time.  They  develop  a  model which \nrecognizes patterns by falling into local minima of a  state-space energy function.  They \nsuggest  that  a  systematic  selection  of delay  functions  can  be  done  which  will  allow for \ntime distortions that  would be likely  to occur  in the input. \nFinally,  a  somewhat  different  approach is  taken  by  Homma,  Atlas and  Marks[5],  who \ngeneralize a network for spatial pattern recognition to one that performs spatio-temporal \npatterns by  extending  classical principles  from  spatial networks  to  dynamic  networks. \nIn  particular, they  replace multiplication with convolution,  weights  with transfer func(cid:173)\ntions,  and  thresholding  with  non linear  transforms.  Hebbian  and  Delta learning rules \nare  similarly  generalized.  The  resulting models  are  able  to perform  temporal pattern \nrecognition. \n\nThe above is  only a  partial list  of some  of the  relevant  work in this field,  and there \n\nare  probably various other results I  am not aware of. \n\nDISCUSSION \n\nAll of the above results indicate the importance of temporal patterns in  neural net(cid:173)\n\nworks.  The need  is  apparent for  further  formal  models  which can successfully  quantify \ntemporal behavior in  neural  networks.  Several  questions  must  be  answered  to further \n\n\f300 \n\nclarify  the  role  and meaning  of temporal patterns  in neural  nets.  For  instance,  there \nis  an apparent difference between a  model that performs sequential tasks and one that \nperforms recognition of dynamic  patterns.  It  seems  that appropriate selection of delay \nmechanisms  will  be necessary  to account for  many types of temporal pattern recogni(cid:173)\ntion.  The  question of scaling  must also be explored:  mechanism are known to exist in \nthe  brain  which  can  cause  delays  ranging  from  the  millisecond-range  (e.g.  variations \nin synaptic  cleft  size)  to the  tenth of a  second  range  (e.g.  axonal transmission times). \nOn the other hand,  the brain is  capable of rec\"Ignizing sequences of stimuli that can be \nmuch longer than the typical neural event, such as for  instance being able to remember \na  song in its entirety.  These  and other questions could lead to interesting new  aspects \nof brain function which are presently unclear. \n\nReferences \n\n[1]  Baird, B., \"Nonlinear Dynamics of Pattern Formation and Pattern Recognition in \n\nthe Rabbit  Olfactory Bulb\". Physica 22D,  150-175.  1986. \n\n[2]  Gaudiano, P., \"Computer Models of Neural Networks\". Unpublished Master's The(cid:173)\n\nsis.  University of Colorado. 1987. \n\n[3]  Gaudiano,  P.,  MacGregor,  R.J.,  \"Dynamic  Activity  and  Memory  Traces  in \nComputer-Simulated Recurrently-Connected Neural Networks\". Proceedings of the \nFirst International Conference  on Neural  Networks. 2:177-185.  1987. \n\n[4]  Jordan,  M.I.,  \"Attractor Dynamics and  Parallelism in a  Connectionist  Sequential \nMachine\". Proceedings  of the Eighth Annual Conference of the Cognitive Sciences \nSociety. 1986. \n\n[5]  Homma, T., Atlas, L.E., Marks, R.J.II,  \"An Artificial Neural Network for  Spatio(cid:173)\n\nTemporal Bipolar Patterns:  Application to Phoneme Classification\". To appear in \nproceedings of Neural Information Processing Systems Conference  (AlP).  1987. \n\n[6]  Kleinfeld,  D.,  \"Sequential  State  Generation  by  Model  Neural  Networks\".  Proc. \n\nNatl.  Acad.  Sci.  USA.  83:  9469-9473.  1986. \n\n[7]  Silverman,  D.l.,  Shaw,  G.L.,  Pearson,  l.C.  \"Associative  Recall  Properties  of the \n\nTrion Model of Cortical Organization\". Biol.  Cybern.  53:259-271.  1986. \n\n[8]  Sompolinsky,  H.,  Kanter,  I.  \"Temporal  Association  in  Asymmetric  Neural  Net(cid:173)\n\nworks\".  Phys.  Rev.  Let.  57:2861-2864.  1986. \n\n[9]  Tank, D.W., Hopfield, l.l. \"Neural Computation by Concentrating Information in \n\nTime\". Proc.  Natl.  Acad.  Sci.  USA.  84:1896-1900.  1987. \n\n\f", "award": [], "sourceid": 13, "authors": [{"given_name": "Paolo", "family_name": "Gaudiano", "institution": null}]}