{"title": "On the Computational Power of Noisy Spiking Neurons", "book": "Advances in Neural Information Processing Systems", "page_first": 211, "page_last": 217, "abstract": null, "full_text": "On the Computational Power of Noisy \n\nSpiking  Neurons \n\nWolfgang Maass \n\nInstitute for  Theoretical  Computer Science,  Technische  Universitaet  Graz \n\nKlosterwiesgasse 32/2, A-8010  Graz,  Austria, e-mail:  maass@igi.tu-graz.ac.at \n\nAbstract \n\nIt  has  remained  unknown  whether  one  can  in  principle  carry out \nreliable digital  computations with networks of biologically realistic \nmodels  for  neurons.  This  article  presents  rigorous  constructions \nfor  simulating in  real-time arbitrary given  boolean circuits  and fi(cid:173)\nnite automata with arbitrarily high reliability by networks of noisy \nspiking neurons. \nIn  addition  we  show  that  with  the  help  of  \"shunting  inhibition\" \neven  networks  of very  unreliable  spiking  neurons  can  simulate  in \nreal-time  any  McCulloch-Pitts  neuron  (or  \"threshold  gate\"),  and \ntherefore  any  multilayer  perceptron  (or  \"threshold  circuit\")  in  a \nreliable  manner.  These  constructions  provide  a  possible  explana(cid:173)\ntion for  the fact  that biological  neural systems can carry out quite \ncomplex  computations within  100  msec. \nIt  turns  out that the assumption  that these  constructions require \nabout the shape of the  EPSP's and  the behaviour of the  noise  are \nsurprisingly weak. \n\n1 \n\nIntroduction \n\nWe  consider networks that consist  of a  finite  set V  of neurons, a  set  E  ~ V  x  V  of \nsynapses, a weightwu,v ~ 0 and a response junctioncu,v  : R+  -+ R  for each synapse \n\n\f212 \n\nW.MAASS \n\n(u,v)  E E  (where R+ := {x E R: x  ~ O}),  and a  threshold/unction Sv : R+  --t R+ \nfor  each  neuron v  E V. \nIf F u  ~ R + is the set of firing  times of a neuron u, then the potential at the trigger \nzone  of neuron v  at time t  is  given  by  Pv(t)  := \nwu,v' \n\nL \n\nL \n\nu  : (u, v)  E  EsE Fu  : s  < t \n\neu,v(t - s).  The  threshold  function  Sv(t - t')  quantifies  the  \"reluctance\"  of v  to \nfire  again  at  time  t,  if  its  last  previous  firing  was  at  time  t'.  We  assume  that \nSv(O)  E  (0,00),  Sv(x)  =  00  for  x  E  (0, 'TreJ]  (for  some  constant  'TreJ  >  0,  the \n\"absolute refractory period\"), and sup{Sv(x)  : X  ~ 'T}  < 00  for  any'T > 'TreJ. \nIn a  deterministic model for  a spiking neuron (Maass, 1995a, 1996)  one can assume \nthat a  neuron v  fires exactly at those time points t  when  Pv(t)  reaches (from below) \nthe value  Sv(t - t').  We  consider in this article a  biologically more realistic model, \nwhere as in (Gerstner, van Hemmen, 1994) the size of the difference Pv(t)-Sv(t-t') \njust governs the probability that neuron v  fires.  The choice of the exact firing  times \nis left up to some unknown stochastic processes, and it may for example occur that \nv  does  not fire in a time intervall during which  Pv (t) - Sv(t - t') > 0, or that v fires \n\"spontaneously\" at a time t when  Pv(t) -Sv(t-t') < O.  We assume that (apart from \ntheir  communication  via  potential  changes)  the  stochastic  processes  for  different \nneurons v  are independent.  It turns out that the assumptions that one has to make \nabout this stochastic firing mechanism in  order to prove our results are surprisingly \nweak.  We  assume that there exist two arbitrary functions  L, U : R  X  R+  ----1  [0,1]  so \nthat L(~, i) provides a  lower bound  (and U(~, i) provides an  upper bound)  for  the \nprobability that neuron v fires  during a  time intervall of length e with the property \nthat Pv(t)-Sv(t-t') ~ ~ (respectively Pv(t)-Sv(t-t') ~ ~) for all tEl up to the \nnext firing  of v  (t'  denotes the last firing time of v  be/ore  I).  We just assume about \nlim  U(~, i)  = \u00b0 for  any  fixed \nthese functions Land U that they are non-decreasing in each of their two arguments \n(for  any  fixed  value  of the other argument),  that \ni  > 0,  and  that  lim  L(~, e)  > 0  for  allY  fixed  e ~ R/6 (where  R  is  the  assumed \nlength  of the  rising  segment  of an  EPSP,  see  below).  The  neurons  are  allowed  to \nbe  \"arbitrarily  noisy\"  in the sense that the difference  lim  L(~, i) -\nlim  U(~, i) \ncan be arbitrarily small.  Hence our constructions also apply to neurons that exhibit \npersistent firing failures,  and they also allow for synapses that fail  with a rather high \nprobability.  Furthermore a detailed analysis of our constructions shows that we  can \nrelax  the  somewhat  dubious  assumption  that  the  noise-distributions  for  different \nneurons are independent.  Thus we  are also  able to deal  with  \"systematic noise\"  in \nthe distribution of firing  times  of neurons in  a  pool  (e.g.  caused  by  changes in the \nbiochemical environment that simultaneously affect  many neurons in a  pool). \n\n~~-oo \n\n~~-oo \n\n~~OO \n\n~~OO \n\nIt turns out  that it suffices  to assume  only  the following  rather  weak  properties of \nthe other functions  involved  in  our  model: \n\n1)  Each  response  function  CU , I ) \n\n:  R+  ----1  R  is  either  excitatory  or  inhibitory \n(and  for  the sake of biological  realism one may assume that each  neuron u  induces \nonly one type of response).  All  excitatory response functions eu,v(x)  have the value \n\n\fOn the Computational Power of Noisy  Spiking  Neurons \n\n213 \n\no for  x  E  [O,~u,v), and  the  value eE(X - ~u,v)  for  x  ~ ~u,v, where  ~u,v ~ 0  is \nthe  delay for  this  synapse between  neurons u  and v,  and  e E  is  the  common shape \nof all  excitatory  response  functions  (\"EPSP's))).  Corresponding  assumptions  are \nmade about the inhibitory response functions  (\"IPSP's))), whose  common shape is \ndescribed by some function eI \n\n: R+ -+  {x E R  : x  ~ O}. \n\n2) eE is continuous, eE(O)  = 0,  eE(X)  = 0 for  all  sufficiently large x, and there \nexists  some  parameter  R  > 0  such  that  e E  is  non-decreasing in  [0, R],  and  some \nparameter p > 0 such that eE(X + R/6)  ~ p + eE (x)  for  all  x  E  [O,2R/3]. \n\n3)  _eI  satisfies the same conditions as e E . \n\n4) There exists a source BN- of negative  \"background noise\", that contributes \n\nto the potential  Pv(t)  of each neuron v  an additive term  that deviates for  an arbi(cid:173)\ntrarily long time interval  by  an arbitrarily small  percentage from  its average value \nw;  ~ 0 (which we  can choose).  One can delete this assumption if one assumes that \nthe firing  threshold of neurons can  be shifted  by  some other mechanism. \n\nIn section 3  we  will  assume in  addition  the availability of a  corresponding positive \nbackground noise BN+ with average value wt  ~ O. \nIn a  biological  neuron  tI  one  can interpret  BN- and  BN+  as  the  combined  effect \nof a  continuous  bombardment with  a  very  large number  of IPSP's  (EPSP's)  from \nrandomly firing  neurons that arrive at remote synapses on the dendritic tree of v. \nWe  assume that we can choose  the values  of delays  ~u, v  and weights Wu,v, wt ,w; . \nWe  refer  to  all  assumptions  specified  in  this  section  as  our  \"weak  assumptions\" \nabout  noisy  spiking  neurons.  It is  easy  to  see  that  the  most  frequently  studied \nconcrete  model  for  noisy  spiking  neurons,  the  spike  response  model (Gerstner  and \nvan  Hemmen,  1994)  satisfies  these  weak  assumptions,  and  is  hence  a  special  case. \nHowever not even  for  the more  concrete spike response model  (or any other model \nfor  noisy  spiking  neurons)  there  exist  any  rigorous  results  about  computations  in \nthese  models.  In  fact,  one  may  view  this  article  as  being  the  first  that  provides \nresults about the computational complexity of neural  networks for  a  neuron model \nthat is  acceptable to many neurobiologistis as being reasonably realistic. \n\nIn this article we  only address the problem  of reliable  digital computing with noisy \nspiking neurons .  For details of the proofs we refer to the forthcoming journal-version \nof this extended abstract.  For results about analog computations with noisy spiking \nneurons we  refer  to Maass,  1995b. \n\n2  Simulation of Boolean Circuits and Finite Automata with \n\nNoisy Spiking Neurons \n\nTheorem  1:  For  any  deterministic  finite  automaton  D  one  can  construct  a  net(cid:173)\nwork  N(D)  consisting  of any  type  of noisy  spiking  neurons  that  satisfy  our  weak \nassumptions,  so  that  N(D)  can  simulate  computations  of D  of any  given  length \nwith  arbitrarily  high  probability  of correctness. \n\n\f214 \n\nW.MAASS \n\nIdea of the proof: Since the behaviour of a single noisy spiking neuron is completely \nunreliable, we  use  instead  pools  A, B, ... of neurons as  the basic building blocks in \nour construction,  where  all  neurons v  in the same pool  receive  approximately the \nsame  \"input potential\"  Pv(t).  The intricacies of our stochastic neuron model  allow \nus only to employ a  \"weak  coding\" of bits,  where a  \"1\"  is  represented by  a  pool  A \nduring a  time interval  I, if at least PI \u00b7IAI  neurons in A fire  (at least once)  during I \n(where PI  > 0 is a suitable constant), and \"0\"  is represented if at most Po \u00b7IAI firings \nof neurons occur in  A during I,  where Po  with 0 < Po  < PI  is another constant (that \ncan be chosen arbitrarily small  in  our construction). \n\nThe described  coding scheme is  weak since  it  provides  no  useful  upper bound  (e.g. \n1.5\u00b7Pl \u00b7IAI) on the number of neurons that fire  during I  if A represents a  \"1\"  (nor on \nthe number of firings of a  single neuron in  A).  It also  does  not impose  constraints \non the exact timing of firings in A  within I.  However a  \"0\"  can be represented more \nprecisely in our model,  by choosing po  sufficiently small. \n\nThe proof of Theorem  1  shows  that  this  weak  coding  of bits  suffices  for  reliable \ndigital  computations.  The idea of these simulations is  to introduce artificial  nega(cid:173)\ntions into  the computation,  which  allow  us  to exploit  that  \"0\"  has  a  more precise \nrepresentation than  \"1\".  It is apparently impossible to simulate an AND-gate in  a \nstraightforward fashion for  a  weak coding of bits, but one can simulate a  NOR-gate \nin a  reliable manner. \n\u2022 \n\nCorollary 2:  Any boolean function  can  be  computed by  a sufficiently large  network \nof noisy  spiking  neurons  (that  satisfy  our weak  assumptions)  with  arbitrarily  high \nprobability  of correctness. \n\n3  Fast Simulation of Threshold  Circuits via Shunting \n\nInhibition \n\nFor  biologically realistic  parameters,  each  computation step in  the previously  con(cid:173)\nstructed  network  takes  around  25  msec  (see  point  b)  in  section  4}.  However  it \nis  well-known  that biological  neural  systems  can  carry out  complex  computations \nwithin just 100 msec (Churchland, Sejnowski,  1992).  A closer inspection of the pre(cid:173)\nceding construction shows, that one can simulate with the same speed also OR- and \nNOR-gates with a  much  larger fan-in  than just 2.  However wellknown  results from \ntheoretical computer science (see the results about the complexity class ACo in the \nsurvey article by Johnson in  (van  Leeuwen,  1990))  imply that for  any fixed number \nof layers the computational power of circuits with gates for  OR,  NOR,  AND,  NOT \nremains very  weak,  even  if one allows  any polynomial size fan-in  for  such  gates. \n\nIn contrast to that, the construction in this section will  show that by using a  biolog(cid:173)\nically more realistic model  for  a  noisy spiking neuron, one can in principle simulate \nwithin  100  msec  3  or  more  layers  of a  boolean  circuit  that  employs  substantially \nmore powerful  boolean gates:  threshold gates (Le.  \"Mc Culloch-Pitts neurons\", also \ncalled \"perceptrons\").  The use of these gates provides a giant leap in  computational \n\n\fOn the Computational Power of Noisy  Spiking Neurons \n\n215 \n\npower for  boolean circuits with a  small  number of layers:  In spite of many years of \nintensive research, one has not been able to exhibit a  single  concrete  computational \nproblem in the  complexity  classes  P  or NP that can be shown to be not computable \nby  a  polynomial  size  threshold  circuit  with  3  layers  (for  threshold  circuits  with \ninteger weights of unbounded size the same holds  already for  just 2 layers). \n\nIn the neuron model  that we  have employed so far in this  article, we  have assumed \n(as it is common in the spike response model)  that the potential  Pv (t)  at the trigger \nzone  of neuron v  depends  linearly  on  all  the  terms  Wu ,v  . cu,v(t - s).  There exists \nhowever  ample biological  evidence  that this  assumption is  not appropriate for  cer(cid:173)\ntain types of synapses.  An  example are synapses that carry out shunting inhibition \n(see.  e.g.  (Abeles,  1991)  and  (Shepherd,  1990)).  When  a  synapse of this  type  (lo(cid:173)\ncated on the dendritic  tree of a  neuron v)  is  activated, it basically erases  (through \na  short  circuit  mechanism)  for  a  short  time  all  EPSP's  that  pass  the  location  of \nthis  synapse on  their  way  to  the  trigger  zone  of v.  However  in  contrast  to  those \nIPSP's that occur  linearly  in  the  formula for  Pv(t) , the activation  of such  synapse \nfor  shunting inhibition  has  no  impact on  those  EPSP's  that  travel  to  the  trigger \nZOne  of v  through  another part of its  dendritic tree.  We  model  shunting inhibition \nin  our framework  as follows .  We  write r  for  the  subset of all  neurons  'Y  in  V  that \ncan  \"veto\"  other synapses  (u, v)  via shunting inhibition  (we  assume  that the  neu(cid:173)\nrons  in r  have no other role  apart from  that).  We  allow in  our formal  model  that \ncertain 'Y  in r  are assigned as label to certain synapses (u, v)  that have an excitatory \nresponse function  cu,v.  If'Y  is  a  label  of (u, v),  then  this  models  the situation  that \n'Y  can intercept  EPSP's from  u  on  their way  to the  trigger  zone  of v  via shunting \ninhibition.  We  then define \n\nPv(t)  =  L  (L  wtt ,tJ  . Ett,v(t  - s) . \n\nu  E V  : (u, v)  E  E sE  Ftt  : s  < t \n\nII \n\ns...,(t))  , \n\n'Y  is  label  of (u, v) \n\nwhere we  assume that S...,(t)  E  [0,1]  is  arbitrarily close to 0 for  a short time interval \nafter  neuron  'Y  has  fired ,  and  else  equal  to  1.  The  firing  mechanism  for  neurons \n'Y  E r  is  defined  like  for  all  other neurons. \n\nTheorem 3:  One  can  simulate  any  threshold  circuit T  by  a  sufficiently  large  net(cid:173)\nwork  N(T)  of noisy spiking  neurons  with  shunting inhibition  (with  arbitrarily  high \nprobability  of correctness) .  The  computation  time  of N(T)  does  not depend  on  the \nnumber  of gates  in  each  layer,  and  is  proportional  to  the  number  of layers  in  the \nthreshold  circuit T. \n\nIdea  of the proof of Theorem  3:  It is  already impossible to simulate in  a  straight(cid:173)\nforward  manner an  AND-gate with  weak  coding of bits.  The same difficulties  arise \nin  an  even  more  drastic way  if one  wants  to simulate a  threshold  gate  with  large \nfan-in. \n\nThe left part of Figure 1 indicates that with the help  of shunting inhibition one can \ntransform via an intermediate pool of neurons Bl the bit that is  weakly encoded by \n\n\f216 \n\nW.MAASS \n\nAl into a contribution to Pv(t) for  neurons v  E  C  that is  throughout a  time interval \nJ  arbitrarily close  to 0 if Al encodes a  \"0\", and  arbitrarily  close  to  some  constant \nP* > 0 if Al encodes a  \"I\"  (we  will  call  this a  \"strong  coding\"  of a  bit).  Obviously \nit is rather easy to realize a threshold gate if one can make use of such strong coding \nof bits. \n\nE  ) IAII \n\nI \n\nE \n)IB11  SI \n\nE \n:)  ) \n,- , \n'--\n\n8 \n11 \n\n-----+ C \n-----+ \n\nH'~ \n\nI \n\nr---------------------\n: \n, , , , \n: , \n\nI , ,  \u00ae ~ \n\nI )~-4[!]  I  E \n\nlE \n\n: \n, \n: \n: : I \n\n) \n\nFigure  1:  Realization  of a threshold gate G  via  shunting inhibition  (SI). \n\n----------------------~ \n\nn \n\ni=I \n\nThe  task  of  the  module  in  Figure  1  is  to  simulate  with  noisy  spiking  neurons  a \ngiven  boolean  threshold  gate  G  that  outputs  1  if  L:  Q:iXi  ~ e,  and  0  else.  For \nsimplicity  Figure  1 shows  only  the  pool  Al  whose  firing  activity encodes  (in  weak \ncoding)  the first input bit Xl.  The other input bits are represented (in weak coding) \nsimultaneously in pools A:l> ... , An  parallel to AI.  If Xl  =  0,  then the firing  activity \nin  pool  Al  is  low,  hence  the  shunting  inhibition  from  pool  Bl  intercepts  those \nEPSP's  that  are  sent  from  BN+  to  each  neuron  v  in  pool  C.  More  precisely, \nwe  assume  that  each  pool  Bi  associated  with  a  different  input  bit  Xi  carries  out \nshunting  inhibition  on  a  different  subtree  of  the  dendritic  tree  of such  neurOn  v \n(where each such subtree receives  EPSP's from  BN+).  If Xl  =  1,  the higher firing \nactivity in  pool  Al  inhibits the neurons  in  BI  for  some time period.  Hence during \nthe relevant time interval  BN+  contributes an almost constant positive summand \nto the potential  Pv(t)  of neurons v  in  C.  By  choosing wt  and w;  appropriately, \none can achieve  that  during this  time  interval  the  potential  Pv(t)  of neurons v  in \nC  is  arbitrarily  much  positive  if  L:  Q:iXi  ~ e,  and  arbitrarily  much  negative  if \nn L:  Q:iXi  < e. Hence the activity level  of C  encodes the output bit of the threshold \ni=l \ngate  G  (in  weak  coding).  The  purpose  of  the  subsequent  pools  D  and  F  is  to \nsynchronize  (with  the  help  of  \"double-negation\")  the output  of this module  via a \npacemaker or synfire chain PM. In this way one can achieve that all  input  \"bits\"  to \nanother module that simulates a threshold gate On  the next layer of circuit T  arrive \n\u2022 \nsimultaneously. \n\ni=1 \n\n11 \n\n\fOn the Computational Power of Noisy Spiking Neurons \n\n217 \n\n4  ConcI usion \n\nOur  constructions  throw  new  light  on  various  experimental  data,  and  on  our  at(cid:173)\ntempts to understand neural  computation and  coding: \n\na)  If One  would  record  all  firing  times  of a  few  arbitrarily  chosen  neurons  in \nour  networks  during  many  repetitions  of the  same  computation,  one  is  likely  to \nsee  that each  run  yields  quite  different  seemingly  random  firing  sequences,  where \nhowever a few  firing patterns will  occur more frequently than could be explained by \nmere chance.  This is  consistent with the experimental results reported in  (Abeles, \n1991), and one should also note that the synfire  chains of (Abeles,  1991) have many \nfeatures in  common with the here  constructed networks. \n\nb) If one plugs  in  biologically realistic values  (see  (Shepherd,  1990),  (Church(cid:173)\n\nland,  Sejnowski,  1992))  for  the length of transmission  delays  (around  5 msec)  and \nthe duration of EPSP's and IPSP's (around 15  msec for  fast  PSP's), then the com(cid:173)\nputation  time  of our modules  for  NOR- and  threshold  gates  comes  out  to  be not \nmore than 25  msec.  Hence  in  principle a  multi-layer perceptron with up to 4 layers \ncan be simulated within  100 msec. \n\nc)  Our  constructions  provide  new  hypotheses  about  the  computational  roles \nof regular and shunting inh'ibition,  that go far  beyond  their usually assumed  roles. \n\nd)  We  provide new  hypotheses regarding the  computational  role  of randomly \nfiring  neurons,  and  of  EPSP's  and  IPSP's  that  arrive  through  synapses  at  distal \nparts of biological  neurons  (see  the use of BN+  and  BN- in  our constructions). \n\nReferences: \n\nM.  Abeles.  (1991)  Corticonics:  Neural  Circuits of the Cerebral  Cortex.  Cambridge  Uni(cid:173)\n\nversity  Press. \n\nP.  S.  Churchland,  T . J . Sejnowski.  (1992)  The Computational  Brain.  MIT-Press. \n\nW.  Gerstner,  J.  L.  van  Hemmen.  (1994)  How  to  describe neuronal  activity:  spikes,  rates, \nor  assemblies?  Advances  in  Neural  Information  Processing  Systems,  vol.  6,  Morgan \nKaufmann:  463-470. \n\nW.  Maass. \n\n(1995a)  On  the  computational  complexity  of  networks  of  spiking  neuronS \n(extended  abstract).  Advances  in  Neural  Information  Processing  Systems,  vol.  7 \n(Proceedings  of NIPS  '94),  MIT-Press,  183-190. \n\nW. Maass.  (1995b)  An efficient implementation of sigmoidal neural nets in temporal coding \nIGI-Report  422  der  Technischen  Universitiit  Graz, \n\nwith  noisy  spiking  neurons. \nsubmitted for  publication. \n\nW.  Maass. \n\n(1996)  Lower  bounds  for  the  computational  power  of  networks  of spiking \n\nneurons.  N eu.ral  Computation 8: 1,  to appear. \n\nG.  M.  Shepherd.  (1990)  The Synaptic Organization of the Brain.  Oxford  University  Press. \n\nJ.  van  Leeuwen,  ed.  (1990)  Handbook  of Theoretical  Computer  Science,  vol.  A:  Algo(cid:173)\n\nrithms and  Complexity.  MIT-Press. \n\n\f", "award": [], "sourceid": 1158, "authors": [{"given_name": "Wolfgang", "family_name": "Maass", "institution": null}]}