{"title": "At the Edge of Chaos: Real-time Computations and Self-Organized Criticality in Recurrent Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 145, "page_last": 152, "abstract": null, "full_text": " At the Edge of Chaos: Real-time Computations and\n Self-Organized Criticality in Recurrent Neural Networks\n\n\n\n\n Thomas Natschl ager Nils Bertschinger Robert Legenstein\n Software Competence Max Planck Institute for Institute for Theoretical\n Center Hagenberg Mathematics in the Sciences Computer Science, TU Graz\n A-4232 Hagenberg, Austria D-04103 Leipzig, Germany A-8010 Graz, Austria\n Thomas.Natschlaeger@scch.at bertschi@mis.mpg.de legi@igi.tu-graz.ac.at\n\n\n\n\n Abstract\n\n In this paper we analyze the relationship between the computational ca-\n pabilities of randomly connected networks of threshold gates in the time-\n series domain and their dynamical properties. In particular we propose\n a complexity measure which we find to assume its highest values near\n the edge of chaos, i.e. the transition from ordered to chaotic dynamics.\n Furthermore we show that the proposed complexity measure predicts the\n computational capabilities very well: only near the edge of chaos are\n such networks able to perform complex computations on time series. Ad-\n ditionally a simple synaptic scaling rule for self-organized criticality is\n presented and analyzed.\n\n\n1 Introduction\nIt has been proposed that extensive computational capabilities are achieved by systems\nwhose dynamics is neither chaotic nor ordered but somewhere in between order and chaos.\nThis has led to the idea of \"computation at the edge of chaos\". Early evidence for this\nhypothesis has been reported e.g. in [1]. The results of numerous computer simulations\ncarried out in these studies suggested that there is a sharp transition between ordered and\nchaotic dynamics. Later on this was confirmed by Derrida and others [2]. They used\nideas from statistical physics to develop an accurate mean-field theory which allowed to\ndetermine the critical parameters analytically. Because of the physical background, this\ntheory focused on the autonomous dynamics of the system, i.e. its relaxation from an\ninitial state (the input) to some terminal state (the output) without any external influences.\nIn contrast to such \"off-line\" computations, we will focus in this article on time-series\ncomputations, i.e. mappings, also called filters, from a time-varying input signal to a time-\nvarying output signal. Such \"online\" or real-time computations describe more adequately\nthe input to output relation of systems like animals or autonomous robots which must react\nin real-time to a continuously changing stream of sensory input.\n\nThe purpose of this paper is to analyze how the computational capabilities of randomly\nconnected recurrent neural networks in the domain of real-time processing and the type\nof dynamics induced by the underlying distribution of synaptic weights are related to each\nother. In particular, we will show that for the types of neural networks considered in this pa-\nper (defined in Sec. 2) there also exists a transition from ordered to chaotic dynamics. This\nphase transition is determined using an extension of the mean-field approach described in\n[3] and [4] (Sec. 3). As the next step we propose a novel complexity measure (Sec. 4) which\n\n\f\n input\n\n\n network activity\n\n\n 10\n\n\n 20\n\n neuron # 30\n\n 0 20 40 0 20 40 0 20 40\n timesteps timesteps timesteps\n\n 0.4\n\n 0.2 0.8\n\n 0\n 0.6\n -0.2 critical\n\n -0.4 ordered chaotic 0.4 mean activity\n -0.6\n\n 0.1 1 10\n 2\n\nFigure 1: Networks of randomly connected threshold gates can exhibit ordered, critical and\nchaotic dynamics. In the upper row examples of the temporal evolution of the network state\nxt are shown (black: xi,t = 1, white: xi,t = 0, input as indicated above) for three different\nnetworks with parameters taken from the ordered, critical and chaotic regime, respectively.\nParameters: K = 5, N = 500, \n u = -0.5, r = 0.3 and and 2 as indicated in the phase\nplot below. The background of the phase plot shows the mean activity a (see Sec. 3) of\nthe networks depending on the parameters and 2.\n\ncan be calculated using the mean-field theory developed in Sec. 3 and serves as a predic-\ntor for the computational capability of a network in the time-series domain. Employing a\nrecently developed framework for analyzing real-time computations [5, 6] we investigate\nin Sec. 5 the relationship between network dynamics and the computational capabilities in\nthe time-series domain. In Sec. 6 of this paper we propose and analyze a synaptic scaling\nrule for self-organized criticality (SOC) for the types of networks considered here. In con-\ntrast to previous work [7], we do not only check that the proposed rule shows adaptation\ntowards critical dynamics, but also show that the computational capabilities of the network\nare actually increased if the rule is applied.\n\nRelation to previous work: In [5], the so-called liquid state machine (LSM) approach was\nproposed and used do analyze the computational capabilities in the time-series domain of\nrandomly connected networks of biologically inspired network models (composed of leaky\nintegrate-and-fire neurons). We will use that approach to demonstrate that only near the\nedge of chaos, complex computations can be performed (see Sec. 5). A similar analysis for\na restricted case (zero mean of synaptic weights) of the network model considered in this\npaper can be found in [4].\n\n2 The Network Model and its Dynamics\nWe consider input driven recurrent networks consisting of N threshold gates with states\nxi {0, 1}. Each node i receives nonzero incoming weights wij from exactly K randomly\nchosen nodes j. Each nonzero connection weight wij is randomly drawn from a Gaussian\ndistribution with mean and variance 2. Furthermore, the network is driven by an exter-\nnal input signal u() which is injected into each node. Hence, in summary, the update of\nthe network state xt = (x1,t, . . . , xN,t) is given by xi,t = ( N w\n j=1 ij xj,t-1 + ut-1)\nwhich is applied to all neurons in parallel and where (h) = 1 if h 0 and (h) = 0\notherwise. In the following we consider a randomly drawn binary input signal u(): at each\n\n\f\ntime step ut assumes the value \n u + 1 with probability r and the value \n u with probability\n1 - r. This network model is similar to the one we have considered in [4]. However it\ndiffers in two important aspects: a) By using states xi {0, 1} we emphasis the asymmet-\nric information encoding by spikes prevalent in biological neural systems and b) it is more\ngeneral in the sense that the Gaussian distribution from which the non-zero weights are\ndrawn is allowed to have an arbitrary mean R. This implies that the network activity\na N\n t = 1 x\n N i=1 i,t can vary considerably for different parameters (compare Fig. 1) and\nenters all the calculations discussed in the rest of the paper.\n\nThe top row of Fig. 1 shows typical examples of ordered, critical and chaotic dynamics (see\nthe next section for a definition of order and chaos). The system parameters corresponding\nto each type of dynamics are indicated in the lower panel (phase plot). We refer to the\n(phase) transition from the ordered to the chaotic regime as the critical line (shown as the\nsolid line in the phase plot). Note that increasing the variance 2 of the weights consistently\nleads to chaotic behavior.\n\n3 The Critical Line: Order and Fading Memory versus Chaos\n\nTo define the chaotic and ordered phase of an input driven network we use an approach\nwhich is similar to that proposed by Derrida and Pomeau [2] for autonomous systems:\nconsider two (initial) network states with a certain (normalized) Hamming distance. These\nstates are mapped to their corresponding successor states (using the same weight matrix)\nwith the same input in each case and the change in the Hamming distance is observed. If\nsmall distances tend to grow this is a sign of chaos whereas if the distance tends to decrease\nthis is a signature of order.\n\nFollowing closely the arguments in [4, 3] we developed a mean-field theory (see [8] for\nall details) which allows to calculate the update dt+1 = f (dt, at, ut) of the normalized\nHamming distance dt = |{i : xi,t = ~\n xi,t}|/N between two states xt and ~\n xt as well as the\nupdate at+1 = A(at, ut) of the network activity in one time step. Note that dt+1 depends\non the input ut (in contrast to [3]) and also on the activity at (in contrast to [4]). Hence the\ntwo-dimensional map Fu(dt, at) := (dt+1, at+1) = (f (dt, at, ut), A(at, ut)) describes\nthe time evolution of dt and at given the input times series u().\n\nLet us consider the steady state of the averaged Hamming distance f as well as the steady\nstate of the averaged network activity a, i.e. (f , a) = limt F tu .1 If f = 0 we\nknow that any state differences will eventually die out and the network is in the ordered\nphase. If on the other hand a small difference is amplified and never dies out we have\nf = 0 and the network is in the chaotic phase. Whether f = 0 or f = 0 can be decided\nby looking at the slope of the function f (, , ) at its fixed point f = 0. Since at does not\ndepend on dt we calculate the averaged steady state activity a and determine the slope \nof the map rf (d, a, \n u + 1) + (1 - r)f (d, a, \n u) at the point (d, a) = (0, a). Accordingly\nwe say that the network is in the ordered, critical or chaotic regime if < 1, = 1 or\n > 1 respectively. In [8] it is shown that the so called critical line = 1 where the\nphase transition from ordered to chaotic behavior occurs is given by\n\n K-1 K - 1 1\n Pbf = an(1 - a)K-1-n(rQ(1, n, \n u + 1) + (1 - r)Q(1, n, \n u)) = (1)\n n K\n n=0\n\nwhere Pbf denotes the probability (averaged over the inputs and the network activity) that\na node will change its output if a single out of its K input bits is flipped.2 Examples of\n\n 1F tu denotes t-fold composition of the map Fu(, ) where in the k-th iteration the input uk is\napplied and denotes the average over all possible initial conditions and all input signals with a\ngiven statistics determined by \n u and r.\n 2The actual single bit-flip probability Q depends on the number n of inputs which are 1 and the\n\n\f\n K = 5 K = 10\n 0.4 0.4\n 0.1\n 0.3 0.3 0.07\n\n 0.2 0.2\n 0.08 0.06\n\n 0.1 0.1\n 0.05\n 0 0\n 0.06\n -0.1 -0.1 0.04\n\n -0.2 0.04 -0.2 0.03\n NM-Separation NM-Separation\n -0.3 -0.3\n 0.02\n -0.4 0.02 -0.4\n 0.01\n -0.5 -0.5\n\n -0.6 0 -0.6 0\n 0.1 1 10 0.1 1 10\n 2 2\n\nFigure 2: N M -separation assumes high values on the critical line. The gray coded image\nshows the N M -separation in dependence on and 2 for K denoted in the panels, r = 0.3,\n\nu = -0.5 and b = 0.1. The solid line marks the critical values for and 2.\n\ncritical lines that were calculated from this formula (marked by the solid lines) can be seen\nin Fig. 2 for K = 5 and K = 10.3\n\nThe ordered phase can also be described by using the notion of fading memory (see [5] and\nthe references therein). Intuitively speaking in a network with fading memory a state xt is\nfully determined by a finite history ut-T , ut-T +1, . . . , ut-1, ut of the input u(). A slight\nreformulation of this property (see [6] and the references therein) shows that it is equivalent\nto the requirement that all state differences vanish, i.e. being in the ordered phase. Fading\nmemory plays an important role in the \"liquid state machine\" framework [5] since together\nwith the separation property (see below) it would in principle allow an appropriate readout\nfunction to deduce the recent input, or any function of it, from the network state. If on\nthe other hand the network does not have fading memory (i.e. is in the chaotic regime)\na given network state xt also contains \"spurious\" information about the initial conditions\nand hence it is hard or even impossible to deduce any features of the recent input.\n\n4 NM-Separation as a Predictor for Computational Power\n\nThe already mentioned separation property [5] is especially important if a network is to\nbe useful for computations on input time-series: only if different input signals separate\nthe network state, i.e. different inputs result in different states, it is possible for a readout\nfunction to respond differently. Hence it is necessary that any two different input time\nseries for which the readout function should produce different outputs drive the recurrent\nnetwork into two sufficiently different states.\n\nThe mean field theory we have developed (see [8]) can be extended to describe the\nupdate dt+1 = s(dt, ...) of the Hamming distance that result from applying differ-\nent inputs u() and ~\n u() with a mean distance of b := Pr {ut = ~\n ut}, i.e. the separa-\ntion. In summary the three-dimensional map Su,~u(dt, at, ~\n at) := (dt+1, at+1, ~at+1) =\n(s(dt, at, ~at, ut, ~\n ut), A(at, ut), A(~at, ~\n ut)) fully describes the time evolution of the Ham-\nming distance and the network activities. Again we consider the steady state of the averaged\nHamming distance s and the network activities a, ~\n a, i.e. (s, a, ~\n a) = limt St .\n u,~\n u\n\nThe overall separation for a given input statistics (determined by \n u, r, and b) is then given\nby s. However, this overall separation measure can not be directly related to the computa-\n\nexternal input u and is given by Q(1, n, u) = -u (, n, n2) 1 - (-u - , , 2) d +\n -\n\n (, n, n2)(-u - , , 2)d where , denote the Gaussian density and cumulative den-\n -u\n\nsity respectively (see [8] for a detailed explanation).\n 3For each value of = -0.6 + k 0.01, k = 0 . . . 100 a search was conducted to find the value\nfor 2 such that = 1. Numerical iterations of the function A were used to determine a.\n\n\f\n A B C\n 3bit parity (K = 5) 3bit parity (K = 10) 5bit random boolean functions\n 0.4\n 5\n 0.8\n 0.2 4\n 4\n\n 0.6\n 0 3\n 3\n \n -0.2 MC (MI) 2 MC (MI) 0.4 mean MI\n 2\n\n\n -0.4 0.2\n 1 1\n\n\n -0.6 0 0 0\n 0.01 0.1 1 10 100 0.01 0.1 1 10 100 0.01 0.1 1 10 100\n 2 2 2\n\nFigure 3: Real-time computation at the edge of chaos. A The gray coded image (an in-\nterpolation between the data points marked with open diamonds) shows the performance\nof trained networks in dependence of the parameters and 2 for the delayed 3-bit par-\nity task. Performance is measured as the memory capacity M C = I(v, y()) where\n \nI(v, y()) is the mutual information between the classifier output v() and the target func-\ntion y() = PARITY(u\n t t- , ut- -1, ut- -2) measured on a test set. B Same as panel A\nbut for K = 10. C Same as panel A but for an average over 50 randomly drawn Boolean\nfunctions f of the last 5 time steps, i.e. yt = f (ut, ut-1, ..., ut-4).\n\ntional power since chaotic networks separate even minor differences in the input to a very\nhigh degree. The part of this separation that is caused by the input distance b and not by the\ndistance of some initial state is therefore given by s - f because f measures the state\ndistance that is caused by differences in the initial states and remains even after long runs\nwith the same inputs (see Sec. 3). Note that f is always zero in the ordered phase and\nnon-zero in the chaotic phase.\n\nSince we want the complexity measure, which we will call N M -separation, to be a\npredictor for computational power we correct s - f by a term which accounts for\nthe separation due to an all-dominant input drive. A suitable measure for this \"imme-\ndiate separation\" i is the average increase in the Hamming distance if the system is\nrun for a long time (t ) with equal inputs u() = ~\n u() and then a single step\nwith an input pair (v, ~\n v) with an average difference of b = Pr {v, = ~\n v} is applied:\ni = lim 1\n t rv(1-r)1-vb|v-~v|(1-b)1-|v-~v| s(, , , v, ~\n v) St -f . Hence\n v,~\n v=0 u,u\na measure of the network mediated separation N Msep due to input differences is given by\n\n N Msep = s - f - i (2)\n\nIn Fig. 2 the N M -separation resulting from an input difference of b = 0.1 is shown in\ndependence of the network parameters and 2.4 Note that the N M -separation peaks\nvery close to the critical line. Because of the computational importance of the separation\nproperty this also suggests that the computational capabilities of the networks will peak at\nthe onset of chaos, which is confirmed in the next section.\n\n5 Real-Time Computations at the Edge of Chaos\n\nTo access the computational power of a network we make use of the so called \"liquid state\nmachine\" framework which was proposed by Maass et.al. [5] and independently by Jaeger\n[6]. They put forward the idea that any complex time-series computation can be imple-\nmented by composing a system which consists of two conceptually different parts: a) a\n\n 4For each value of = -0.6 + k 0.05, k = 0 . . . 20, 10 values for 2 where chosen near\nthe critical line and 10 other values where equally spaced (on a logarithmic scale) over the interval\n[0.02,50]. For each such pair (, 2) extensive numerical iterations of the map S where performed\nto obtain accurate estimates of s, f and i. Hopefully these numerical estimates can be replaced\nby analytic results in the future.\n\n\f\nproperly chosen general-purpose recurrent network with \"rich\" dynamics and b) a read-\nout function that is trained to map the network state to the desired outputs (see [5, 6, 4]\nfor more details). This approach is potentially successful if the general-purpose network\nencodes the relevant features of the input signal in the network state in such a way that\nthe readout function can easily extract it. We will show that near the critical line the net-\nworks considered in this paper encode the input in such a way that a simple linear classifier\nC(xt) = (w xt + w0) suffices to implement a broad range of complex nonlinear fil-\nters. Note that in order to train the network for a given task only the parameters w RN ,\nw0 R of the linear classifier are adjusted such that the actual network output vt = C(xt)\nis as close as possible to the target values yt.\n\nTo access the computational power in a principled way networks with different parameters\nwere tested on a delayed 3-bit parity task for increasing delays and on randomly drawn\nBoolean functions of the last 5 input bits. Note that these tasks are quite complex for\nthe networks considered here since most of them are not linear separable (i.e. the parity\nfunction) and require memory. Hence to achieve good performance it is necessary that a\nstate xt contains information about several input bits ut , t < t in a nonlinear transformed\nform such that a linear classifier C is sufficient to perform the nonlinear computations.\n\nThe results are summarized in Fig. 3 where the performance (measured in terms of mutual\ninformation) on a test set between the network output and the target signal is shown for\nvarious parameter settings (for details see [4]). The highest performance is clearly achieved\nfor parameter values close to the critical line where the phase transition occurs. This has\nbeen noted before [1]. In contrast to these previous results the networks used here are\nnot optimized for any specific task but their computational capabilities are assessed by\nevaluating them for many different tasks. Therefore a network that is specifically designed\nfor a single task will not show a good performance in this setup. These considerations\nsuggest the following hypotheses regarding the computational function of generic recurrent\nneural circuits: to serve as a general-purpose temporal integrator, and simultaneously as a\nkernel (i.e., nonlinear projection into a higher dimensional space) to facilitate subsequent\n(linear) readout of information whenever it is needed.\n\n6 Self-Organized Criticality via Synaptic Scaling\nSince the computational capabilities of a network depend crucially on having almost critical\ndynamics an adaptive system should be able to adjust its dynamics accordingly.\n\nEqu. (1) states that critical dynamics are achieved if the probability Pbf that a single bit-\nflip in the input shows up in the output should on average (over the external and internal\ninput statistics given by \n u, r and a respectively) be equal to 1 . To allow for a rule that\n K\ncan adjust the weights of each node a local estimate of Pbf must be available. This can\nbe accomplished by estimating Pbf from the margin of each node, i.e. the distance of the\ninternal activation from the firing threshold. Intuitively a node with an activation that is\nmuch higher or lower than its firing threshold is rather unlikely to change its output if a\nsingle bit in its input is flipped. Formally P i of node i is given by the average (over the\n bf\ninternal and external input statistics) of the following quantity:\n\n 1 N\n (w\n K ij (1 - 2xj,t-1)(1 - 2xi,t) - mi,t) (3)\n j=1,wij =0\n\nwhere mi,t = N w\n j=1 ij xj,t-1 + ut-1 denotes the margin of node i (see [8] for details).\nEach node now applies synaptic scaling to adjust itself towards the critical line. Accord-\ningly we arrive at the following SOC-rule:\n\n 1 w (t) > 1\n w 1+ ij if P esti\n bf K\n ij (t + 1) = (4)\n (1 + ) wij(t) if P esti(t) < 1\n bf K\n\n\f\nA\n\n\n 50\n\n\n 100\n\n neuron #\n 150\n\n\n 200\n 100 200 300 400 500 600 700\n timesteps\n\n B C\n 1.5 1.5\n\n\n\n\n 1 1\n\n bf bf\n\n K*P K*P\n 0.5 0.5\n\n\n\n\n 0 0\n 0 100 200 300 400 500 600 700 0 100 200 300 400 500 600 700\n timesteps timesteps\n\n\nFigure 4: Self-organized criticality. A Time evolution of the network state xt starting\nin a chaotic regime while the SOC-rule (4) is active (black: xi,t = 1, white: xi,t = 0).\nParameters: N = 500, K = 5, \n u = -0.5, r = 0.3, = 0 and initial 2 = 100. B\nEstimated Pbf . The dotted line shows how the node averaged estimate of Pbf evolves over\ntime for the network shown in A. The running average of this estimate (thick black line) as\nused by the SOC-rule clearly shows that Pbf approaches its critical value (dashed line). C\nSame as B but for K = 10 and initial 2 = 0.01 in the ordered regime.\n\nwhere 0 < 1 is the learning rate and P esti(t) is a running average of the formula\n bf\nin Equ. (3) to estimate P i . Applying this rule in parallel to all nodes of the network is\n bf\nthen able to adjust the network dynamics towards criticality as shown in Fig. 45. The upper\nrow shows the time evolution of the network states xt while the SOC-rule (4) is running.\nIt is clearly visible how the network dynamics changes from chaotic (the initial network\nhad the parameters K = 5, = 0 and 2 = 100) to critical dynamics that respect the\ninput signal. The lower row of Fig. 4 shows how the averaged estimated bit-flip probability\n1 N P esti(t) approaches its critical value for the case of the above network and one\nN i=1 bf\nthat started in the ordered regime (K = 10, = 0, 2 = 0.01).\nSince critical dynamics are better suited for information processing (see Fig. 3) it is ex-\npected that the performance on the 3-bit parity task improves due to SOC. This is con-\nfirmed in Fig. 5 which shows how the memory capacity M C (defined in Fig. 3) grows for\nnetworks that were initialized in the chaotic and ordered regime respectively. Note that the\nperformance reached by these networks using the SOC-rule (4) is as high as for networks\nwhere the critical value for 2 is chosen apriori and stays at this level. This shows that\nrule (4) is stable in the sense that it keeps the dynamics critical and does not destroy the\ncomputational capabilities.\n\n7 Discussion\n\nWe developed a mean-field theory for input-driven networks which allows to determine the\nposition of the transition line between ordered and chaotic dynamics with respect to the\n\n 5Here a learning rate of = 0.01 and an exponentially weighted running average with a time\nconstant of 15 time steps were used.\n\n\f\nA B\n K = 5, start 2 = 100 (chaotic) K = 10, start 2 = 0.01 (ordered)\n\n 5\n\n 4\n 4\n\n 3\n 3\n\n MC [bits] 2 2\n MC [bits]\n\n 1 1\n\n\n 0 0\n 0 500 1000 1500 2000 0 500 1000 1500 2000\n SOC steps SOC steps\n\n\nFigure 5: Time evolution of the performance with activated SOC-rule. A The plot shows\nthe memory capacity M C (see Fig. 3) on the 3-bit parity task averaged over 25 networks (\nstandard deviation as error-bars) evaluated at the indicated time steps. At each evaluation\ntime step the network weights were fixed and the M C was measured as in Fig. 3 by training\nthe corresponding readouts from scratch. The networks were initialized in the chaotic\nregime. B Same as in A but for K = 10 and networks initialized in the ordered regime.\n\nparameters controlling the network connectivity and input statistics. Based on this theory\nwe proposed a complexity measure (called N M -separation) which assumes its highest\nvalues at the critical line and shows a clear correlation with the computational power for\nreal-time time-series processing. These results provide further evidence for the idea of\n\"computation at the edge of chaos\" [1] and support the hypothesis that dynamics near\nthe critical line are expected to be a general property of input driven dynamical systems\nwhich support complex real-time computations. Therefore our analysis and the proposed\ncomplexity measure provide a new approach towards discovering dynamical principles that\nenable biological systems to do sophisticated information processing.\n\nFurthermore we have shown that a local rule for synaptic scaling is able to adjust the\nweights of a network towards critical dynamics. Additionally networks adjusting them-\nselves by this rule have been found to exhibit enhanced computational capabilities. Thereby\nsystems can combine task-specific optimization provided by (supervised) learning rules\nwith self-organization of its dynamics towards criticality. This provides an explanation\nhow specific information can be processed while still being able to react to incoming sig-\nnals in a flexible way.\n\nAcknowledgement This work was supported in part by the PASCAL project #IST-2002-506778\nof the European Community.\n\nReferences\n\n[1] C. G. Langton. Computation at the edge of chaos. Physica D, 42, 1990.\n[2] B. Derrida and Y. Pomeau. Random networks of automata: A simple annealed approximation.\n Europhys. Lett., 1:4552, 1986.\n[3] B. Derrida. Dynamical phase transition in non-symmetric spin glasses. J. Phys. A: Math. Gen.,\n 20:721725, 1987.\n[4] N. Bertschinger and T. Natschlager. Real-time computation at the edge of chaos in recurrent\n neural networks. Neural Computation, 16(7):14131436, 2004.\n[5] W. Maass, T. Natschlager, and H. Markram. Real-time computing without stable states: A new\n framework for neural computation based on perturbations. Neural Computation, 14(11), 2002.\n[6] H. Jaeger and H. Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy\n in wireless communication. Science, 304(5667):7880, 2004.\n[7] S. Bornholdt and T. R ohl. Self-organized critical neural networks. Physical Review E, 67:066118,\n 2003.\n[8] N. Bertschinger and T. Natschlager. Supplementary information to the mean-\n field theory for randomly connected recurrent networks of threshold gates, 2004.\n http://www.igi.tugraz.at/tnatschl/edge-of-chaos/mean-field-supplement.pdf.\n\n\f\n", "award": [], "sourceid": 2671, "authors": [{"given_name": "Nils", "family_name": "Bertschinger", "institution": null}, {"given_name": "Thomas", "family_name": "Natschl\u00e4ger", "institution": null}, {"given_name": "Robert", "family_name": "Legenstein", "institution": null}]}