{"title": "On the Computational Power of Noisy Spiking Neurons", "book": "Advances in Neural Information Processing Systems", "page_first": 211, "page_last": 217, "abstract": null, "full_text": "On the Computational Power of Noisy \n\nSpiking Neurons \n\nWolfgang Maass \n\nInstitute for Theoretical Computer Science, Technische Universitaet Graz \n\nKlosterwiesgasse 32/2, A-8010 Graz, Austria, e-mail: maass@igi.tu-graz.ac.at \n\nAbstract \n\nIt has remained unknown whether one can in principle carry out \nreliable digital computations with networks of biologically realistic \nmodels for neurons. This article presents rigorous constructions \nfor simulating in real-time arbitrary given boolean circuits and fi(cid:173)\nnite automata with arbitrarily high reliability by networks of noisy \nspiking neurons. \nIn addition we show that with the help of \"shunting inhibition\" \neven networks of very unreliable spiking neurons can simulate in \nreal-time any McCulloch-Pitts neuron (or \"threshold gate\"), and \ntherefore any multilayer perceptron (or \"threshold circuit\") in a \nreliable manner. These constructions provide a possible explana(cid:173)\ntion for the fact that biological neural systems can carry out quite \ncomplex computations within 100 msec. \nIt turns out that the assumption that these constructions require \nabout the shape of the EPSP's and the behaviour of the noise are \nsurprisingly weak. \n\n1 \n\nIntroduction \n\nWe consider networks that consist of a finite set V of neurons, a set E ~ V x V of \nsynapses, a weightwu,v ~ 0 and a response junctioncu,v : R+ -+ R for each synapse \n\n\f212 \n\nW.MAASS \n\n(u,v) E E (where R+ := {x E R: x ~ O}), and a threshold/unction Sv : R+ --t R+ \nfor each neuron v E V. \nIf F u ~ R + is the set of firing times of a neuron u, then the potential at the trigger \nzone of neuron v at time t is given by Pv(t) := \nwu,v' \n\nL \n\nL \n\nu : (u, v) E EsE Fu : s < t \n\neu,v(t - s). The threshold function Sv(t - t') quantifies the \"reluctance\" of v to \nfire again at time t, if its last previous firing was at time t'. We assume that \nSv(O) E (0,00), Sv(x) = 00 for x E (0, 'TreJ] (for some constant 'TreJ > 0, the \n\"absolute refractory period\"), and sup{Sv(x) : X ~ 'T} < 00 for any'T > 'TreJ. \nIn a deterministic model for a spiking neuron (Maass, 1995a, 1996) one can assume \nthat a neuron v fires exactly at those time points t when Pv(t) reaches (from below) \nthe value Sv(t - t'). We consider in this article a biologically more realistic model, \nwhere as in (Gerstner, van Hemmen, 1994) the size of the difference Pv(t)-Sv(t-t') \njust governs the probability that neuron v fires. The choice of the exact firing times \nis left up to some unknown stochastic processes, and it may for example occur that \nv does not fire in a time intervall during which Pv (t) - Sv(t - t') > 0, or that v fires \n\"spontaneously\" at a time t when Pv(t) -Sv(t-t') < O. We assume that (apart from \ntheir communication via potential changes) the stochastic processes for different \nneurons v are independent. It turns out that the assumptions that one has to make \nabout this stochastic firing mechanism in order to prove our results are surprisingly \nweak. We assume that there exist two arbitrary functions L, U : R X R+ ----1 [0,1] so \nthat L(~, i) provides a lower bound (and U(~, i) provides an upper bound) for the \nprobability that neuron v fires during a time intervall of length e with the property \nthat Pv(t)-Sv(t-t') ~ ~ (respectively Pv(t)-Sv(t-t') ~ ~) for all tEl up to the \nnext firing of v (t' denotes the last firing time of v be/ore I). We just assume about \nlim U(~, i) = \u00b0 for any fixed \nthese functions Land U that they are non-decreasing in each of their two arguments \n(for any fixed value of the other argument), that \ni > 0, and that lim L(~, e) > 0 for allY fixed e ~ R/6 (where R is the assumed \nlength of the rising segment of an EPSP, see below). The neurons are allowed to \nbe \"arbitrarily noisy\" in the sense that the difference lim L(~, i) -\nlim U(~, i) \ncan be arbitrarily small. Hence our constructions also apply to neurons that exhibit \npersistent firing failures, and they also allow for synapses that fail with a rather high \nprobability. Furthermore a detailed analysis of our constructions shows that we can \nrelax the somewhat dubious assumption that the noise-distributions for different \nneurons are independent. Thus we are also able to deal with \"systematic noise\" in \nthe distribution of firing times of neurons in a pool (e.g. caused by changes in the \nbiochemical environment that simultaneously affect many neurons in a pool). \n\n~~-oo \n\n~~-oo \n\n~~OO \n\n~~OO \n\nIt turns out that it suffices to assume only the following rather weak properties of \nthe other functions involved in our model: \n\n1) Each response function CU , I ) \n\n: R+ ----1 R is either excitatory or inhibitory \n(and for the sake of biological realism one may assume that each neuron u induces \nonly one type of response). All excitatory response functions eu,v(x) have the value \n\n\fOn the Computational Power of Noisy Spiking Neurons \n\n213 \n\no for x E [O,~u,v), and the value eE(X - ~u,v) for x ~ ~u,v, where ~u,v ~ 0 is \nthe delay for this synapse between neurons u and v, and e E is the common shape \nof all excitatory response functions (\"EPSP's))). Corresponding assumptions are \nmade about the inhibitory response functions (\"IPSP's))), whose common shape is \ndescribed by some function eI \n\n: R+ -+ {x E R : x ~ O}. \n\n2) eE is continuous, eE(O) = 0, eE(X) = 0 for all sufficiently large x, and there \nexists some parameter R > 0 such that e E is non-decreasing in [0, R], and some \nparameter p > 0 such that eE(X + R/6) ~ p + eE (x) for all x E [O,2R/3]. \n\n3) _eI satisfies the same conditions as e E . \n\n4) There exists a source BN- of negative \"background noise\", that contributes \n\nto the potential Pv(t) of each neuron v an additive term that deviates for an arbi(cid:173)\ntrarily long time interval by an arbitrarily small percentage from its average value \nw; ~ 0 (which we can choose). One can delete this assumption if one assumes that \nthe firing threshold of neurons can be shifted by some other mechanism. \n\nIn section 3 we will assume in addition the availability of a corresponding positive \nbackground noise BN+ with average value wt ~ O. \nIn a biological neuron tI one can interpret BN- and BN+ as the combined effect \nof a continuous bombardment with a very large number of IPSP's (EPSP's) from \nrandomly firing neurons that arrive at remote synapses on the dendritic tree of v. \nWe assume that we can choose the values of delays ~u, v and weights Wu,v, wt ,w; . \nWe refer to all assumptions specified in this section as our \"weak assumptions\" \nabout noisy spiking neurons. It is easy to see that the most frequently studied \nconcrete model for noisy spiking neurons, the spike response model (Gerstner and \nvan Hemmen, 1994) satisfies these weak assumptions, and is hence a special case. \nHowever not even for the more concrete spike response model (or any other model \nfor noisy spiking neurons) there exist any rigorous results about computations in \nthese models. In fact, one may view this article as being the first that provides \nresults about the computational complexity of neural networks for a neuron model \nthat is acceptable to many neurobiologistis as being reasonably realistic. \n\nIn this article we only address the problem of reliable digital computing with noisy \nspiking neurons . For details of the proofs we refer to the forthcoming journal-version \nof this extended abstract. For results about analog computations with noisy spiking \nneurons we refer to Maass, 1995b. \n\n2 Simulation of Boolean Circuits and Finite Automata with \n\nNoisy Spiking Neurons \n\nTheorem 1: For any deterministic finite automaton D one can construct a net(cid:173)\nwork N(D) consisting of any type of noisy spiking neurons that satisfy our weak \nassumptions, so that N(D) can simulate computations of D of any given length \nwith arbitrarily high probability of correctness. \n\n\f214 \n\nW.MAASS \n\nIdea of the proof: Since the behaviour of a single noisy spiking neuron is completely \nunreliable, we use instead pools A, B, ... of neurons as the basic building blocks in \nour construction, where all neurons v in the same pool receive approximately the \nsame \"input potential\" Pv(t). The intricacies of our stochastic neuron model allow \nus only to employ a \"weak coding\" of bits, where a \"1\" is represented by a pool A \nduring a time interval I, if at least PI \u00b7IAI neurons in A fire (at least once) during I \n(where PI > 0 is a suitable constant), and \"0\" is represented if at most Po \u00b7IAI firings \nof neurons occur in A during I, where Po with 0 < Po < PI is another constant (that \ncan be chosen arbitrarily small in our construction). \n\nThe described coding scheme is weak since it provides no useful upper bound (e.g. \n1.5\u00b7Pl \u00b7IAI) on the number of neurons that fire during I if A represents a \"1\" (nor on \nthe number of firings of a single neuron in A). It also does not impose constraints \non the exact timing of firings in A within I. However a \"0\" can be represented more \nprecisely in our model, by choosing po sufficiently small. \n\nThe proof of Theorem 1 shows that this weak coding of bits suffices for reliable \ndigital computations. The idea of these simulations is to introduce artificial nega(cid:173)\ntions into the computation, which allow us to exploit that \"0\" has a more precise \nrepresentation than \"1\". It is apparently impossible to simulate an AND-gate in a \nstraightforward fashion for a weak coding of bits, but one can simulate a NOR-gate \nin a reliable manner. \n\u2022 \n\nCorollary 2: Any boolean function can be computed by a sufficiently large network \nof noisy spiking neurons (that satisfy our weak assumptions) with arbitrarily high \nprobability of correctness. \n\n3 Fast Simulation of Threshold Circuits via Shunting \n\nInhibition \n\nFor biologically realistic parameters, each computation step in the previously con(cid:173)\nstructed network takes around 25 msec (see point b) in section 4}. However it \nis well-known that biological neural systems can carry out complex computations \nwithin just 100 msec (Churchland, Sejnowski, 1992). A closer inspection of the pre(cid:173)\nceding construction shows, that one can simulate with the same speed also OR- and \nNOR-gates with a much larger fan-in than just 2. However wellknown results from \ntheoretical computer science (see the results about the complexity class ACo in the \nsurvey article by Johnson in (van Leeuwen, 1990)) imply that for any fixed number \nof layers the computational power of circuits with gates for OR, NOR, AND, NOT \nremains very weak, even if one allows any polynomial size fan-in for such gates. \n\nIn contrast to that, the construction in this section will show that by using a biolog(cid:173)\nically more realistic model for a noisy spiking neuron, one can in principle simulate \nwithin 100 msec 3 or more layers of a boolean circuit that employs substantially \nmore powerful boolean gates: threshold gates (Le. \"Mc Culloch-Pitts neurons\", also \ncalled \"perceptrons\"). The use of these gates provides a giant leap in computational \n\n\fOn the Computational Power of Noisy Spiking Neurons \n\n215 \n\npower for boolean circuits with a small number of layers: In spite of many years of \nintensive research, one has not been able to exhibit a single concrete computational \nproblem in the complexity classes P or NP that can be shown to be not computable \nby a polynomial size threshold circuit with 3 layers (for threshold circuits with \ninteger weights of unbounded size the same holds already for just 2 layers). \n\nIn the neuron model that we have employed so far in this article, we have assumed \n(as it is common in the spike response model) that the potential Pv (t) at the trigger \nzone of neuron v depends linearly on all the terms Wu ,v . cu,v(t - s). There exists \nhowever ample biological evidence that this assumption is not appropriate for cer(cid:173)\ntain types of synapses. An example are synapses that carry out shunting inhibition \n(see. e.g. (Abeles, 1991) and (Shepherd, 1990)). When a synapse of this type (lo(cid:173)\ncated on the dendritic tree of a neuron v) is activated, it basically erases (through \na short circuit mechanism) for a short time all EPSP's that pass the location of \nthis synapse on their way to the trigger zone of v. However in contrast to those \nIPSP's that occur linearly in the formula for Pv(t) , the activation of such synapse \nfor shunting inhibition has no impact on those EPSP's that travel to the trigger \nZOne of v through another part of its dendritic tree. We model shunting inhibition \nin our framework as follows . We write r for the subset of all neurons 'Y in V that \ncan \"veto\" other synapses (u, v) via shunting inhibition (we assume that the neu(cid:173)\nrons in r have no other role apart from that). We allow in our formal model that \ncertain 'Y in r are assigned as label to certain synapses (u, v) that have an excitatory \nresponse function cu,v. If'Y is a label of (u, v), then this models the situation that \n'Y can intercept EPSP's from u on their way to the trigger zone of v via shunting \ninhibition. We then define \n\nPv(t) = L (L wtt ,tJ . Ett,v(t - s) . \n\nu E V : (u, v) E E sE Ftt : s < t \n\nII \n\ns...,(t)) , \n\n'Y is label of (u, v) \n\nwhere we assume that S...,(t) E [0,1] is arbitrarily close to 0 for a short time interval \nafter neuron 'Y has fired , and else equal to 1. The firing mechanism for neurons \n'Y E r is defined like for all other neurons. \n\nTheorem 3: One can simulate any threshold circuit T by a sufficiently large net(cid:173)\nwork N(T) of noisy spiking neurons with shunting inhibition (with arbitrarily high \nprobability of correctness) . The computation time of N(T) does not depend on the \nnumber of gates in each layer, and is proportional to the number of layers in the \nthreshold circuit T. \n\nIdea of the proof of Theorem 3: It is already impossible to simulate in a straight(cid:173)\nforward manner an AND-gate with weak coding of bits. The same difficulties arise \nin an even more drastic way if one wants to simulate a threshold gate with large \nfan-in. \n\nThe left part of Figure 1 indicates that with the help of shunting inhibition one can \ntransform via an intermediate pool of neurons Bl the bit that is weakly encoded by \n\n\f216 \n\nW.MAASS \n\nAl into a contribution to Pv(t) for neurons v E C that is throughout a time interval \nJ arbitrarily close to 0 if Al encodes a \"0\", and arbitrarily close to some constant \nP* > 0 if Al encodes a \"I\" (we will call this a \"strong coding\" of a bit). Obviously \nit is rather easy to realize a threshold gate if one can make use of such strong coding \nof bits. \n\nE ) IAII \n\nI \n\nE \n)IB11 SI \n\nE \n:) ) \n,- , \n'--\n\n8 \n11 \n\n-----+ C \n-----+ \n\nH'~ \n\nI \n\nr---------------------\n: \n, , , , \n: , \n\nI , , \u00ae ~ \n\nI )~-4[!] I E \n\nlE \n\n: \n, \n: \n: : I \n\n) \n\nFigure 1: Realization of a threshold gate G via shunting inhibition (SI). \n\n----------------------~ \n\nn \n\ni=I \n\nThe task of the module in Figure 1 is to simulate with noisy spiking neurons a \ngiven boolean threshold gate G that outputs 1 if L: Q:iXi ~ e, and 0 else. For \nsimplicity Figure 1 shows only the pool Al whose firing activity encodes (in weak \ncoding) the first input bit Xl. The other input bits are represented (in weak coding) \nsimultaneously in pools A:l> ... , An parallel to AI. If Xl = 0, then the firing activity \nin pool Al is low, hence the shunting inhibition from pool Bl intercepts those \nEPSP's that are sent from BN+ to each neuron v in pool C. More precisely, \nwe assume that each pool Bi associated with a different input bit Xi carries out \nshunting inhibition on a different subtree of the dendritic tree of such neurOn v \n(where each such subtree receives EPSP's from BN+). If Xl = 1, the higher firing \nactivity in pool Al inhibits the neurons in BI for some time period. Hence during \nthe relevant time interval BN+ contributes an almost constant positive summand \nto the potential Pv(t) of neurons v in C. By choosing wt and w; appropriately, \none can achieve that during this time interval the potential Pv(t) of neurons v in \nC is arbitrarily much positive if L: Q:iXi ~ e, and arbitrarily much negative if \nn L: Q:iXi < e. Hence the activity level of C encodes the output bit of the threshold \ni=l \ngate G (in weak coding). The purpose of the subsequent pools D and F is to \nsynchronize (with the help of \"double-negation\") the output of this module via a \npacemaker or synfire chain PM. In this way one can achieve that all input \"bits\" to \nanother module that simulates a threshold gate On the next layer of circuit T arrive \n\u2022 \nsimultaneously. \n\ni=1 \n\n11 \n\n\fOn the Computational Power of Noisy Spiking Neurons \n\n217 \n\n4 ConcI usion \n\nOur constructions throw new light on various experimental data, and on our at(cid:173)\ntempts to understand neural computation and coding: \n\na) If One would record all firing times of a few arbitrarily chosen neurons in \nour networks during many repetitions of the same computation, one is likely to \nsee that each run yields quite different seemingly random firing sequences, where \nhowever a few firing patterns will occur more frequently than could be explained by \nmere chance. This is consistent with the experimental results reported in (Abeles, \n1991), and one should also note that the synfire chains of (Abeles, 1991) have many \nfeatures in common with the here constructed networks. \n\nb) If one plugs in biologically realistic values (see (Shepherd, 1990), (Church(cid:173)\n\nland, Sejnowski, 1992)) for the length of transmission delays (around 5 msec) and \nthe duration of EPSP's and IPSP's (around 15 msec for fast PSP's), then the com(cid:173)\nputation time of our modules for NOR- and threshold gates comes out to be not \nmore than 25 msec. Hence in principle a multi-layer perceptron with up to 4 layers \ncan be simulated within 100 msec. \n\nc) Our constructions provide new hypotheses about the computational roles \nof regular and shunting inh'ibition, that go far beyond their usually assumed roles. \n\nd) We provide new hypotheses regarding the computational role of randomly \nfiring neurons, and of EPSP's and IPSP's that arrive through synapses at distal \nparts of biological neurons (see the use of BN+ and BN- in our constructions). \n\nReferences: \n\nM. Abeles. (1991) Corticonics: Neural Circuits of the Cerebral Cortex. Cambridge Uni(cid:173)\n\nversity Press. \n\nP. S. Churchland, T . J . Sejnowski. (1992) The Computational Brain. MIT-Press. \n\nW. Gerstner, J. L. van Hemmen. (1994) How to describe neuronal activity: spikes, rates, \nor assemblies? Advances in Neural Information Processing Systems, vol. 6, Morgan \nKaufmann: 463-470. \n\nW. Maass. \n\n(1995a) On the computational complexity of networks of spiking neuronS \n(extended abstract). Advances in Neural Information Processing Systems, vol. 7 \n(Proceedings of NIPS '94), MIT-Press, 183-190. \n\nW. Maass. (1995b) An efficient implementation of sigmoidal neural nets in temporal coding \nIGI-Report 422 der Technischen Universitiit Graz, \n\nwith noisy spiking neurons. \nsubmitted for publication. \n\nW. Maass. \n\n(1996) Lower bounds for the computational power of networks of spiking \n\nneurons. N eu.ral Computation 8: 1, to appear. \n\nG. M. Shepherd. (1990) The Synaptic Organization of the Brain. Oxford University Press. \n\nJ. van Leeuwen, ed. (1990) Handbook of Theoretical Computer Science, vol. A: Algo(cid:173)\n\nrithms and Complexity. MIT-Press. \n\n\f", "award": [], "sourceid": 1158, "authors": [{"given_name": "Wolfgang", "family_name": "Maass", "institution": null}]}