{"title": "A Synaptical Story of Persistent Activity with Graded Lifetime in a Neural System", "book": "Advances in Neural Information Processing Systems", "page_first": 352, "page_last": 360, "abstract": "Persistent activity refers to the phenomenon that cortical neurons keep firing even after the stimulus triggering the initial neuronal responses is moved. Persistent activity is widely believed to be the substrate for a neural system retaining a memory trace of the stimulus information. In a conventional view, persistent activity is regarded as an attractor of the network dynamics, but it faces a challenge of how to be closed properly. Here, in contrast to the view of attractor, we consider that the stimulus information is encoded in a marginally unstable state of the network which decays very slowly and exhibits persistent firing for a prolonged duration. We propose a simple yet effective mechanism to achieve this goal, which utilizes the property of short-term plasticity (STP) of neuronal synapses. STP has two forms, short-term depression (STD) and short-term facilitation (STF), which have opposite effects on retaining neuronal responses. We find that by properly combining STF and STD, a neural system can hold persistent activity of graded lifetime, and that persistent activity fades away naturally without relying on an external drive. The implications of these results on neural information representation are discussed.", "full_text": "A Synaptical Story of Persistent Activity with\n\nGraded Lifetime in a Neural System\n\nYuanyuan Mi,\n\nLuozheng Li\n\nState Key Laboratory of Cognitive Neuroscience & Learning,\n\nBeijing Normal University, Beijing 100875, China\n\nmiyuanyuan0102@163.com,\n\nliluozheng@mail.bnu.edu.cn\n\nState Key Laboratory of Cognitive Neuroscience & Learning,\n\nSchool of System Science, Beijing Normal University,Beijing 100875, China\n\nDahui Wang\n\nwangdh@bnu.edu.cn\n\nSi Wu\n\nState Key Laboratory of Cognitive Neuroscience & Learning,\n\nIDG/McGovern Institute for Brain Research,\n\nBeijing Normal University ,Beijing 100875, China\n\nwusi@bnu.edu.cn\n\nAbstract\n\nPersistent activity refers to the phenomenon that cortical neurons keep \ufb01ring even\nafter the stimulus triggering the initial neuronal responses is moved. Persistent\nactivity is widely believed to be the substrate for a neural system retaining a mem-\nory trace of the stimulus information. In a conventional view, persistent activity is\nregarded as an attractor of the network dynamics, but it faces a challenge of how\nto be closed properly. Here, in contrast to the view of attractor, we consider that\nthe stimulus information is encoded in a marginally unstable state of the network\nwhich decays very slowly and exhibits persistent \ufb01ring for a prolonged duration.\nWe propose a simple yet effective mechanism to achieve this goal, which utilizes\nthe property of short-term plasticity (STP) of neuronal synapses. STP has two\nforms, short-term depression (STD) and short-term facilitation (STF), which have\nopposite effects on retaining neuronal responses. We \ufb01nd that by properly combin-\ning STF and STD, a neural system can hold persistent activity of graded lifetime,\nand that persistent activity fades away naturally without relying on an external\ndrive. The implications of these results on neural information representation are\ndiscussed.\n\n1 Introduction\n\nStimulus information is encoded in neuronal responses. Persistent activity refers to the phenomenon\nthat cortical neurons keep \ufb01ring even after the stimulus triggering the initial neural responses is\nremoved [1, 2, 3]. It has been widely suggested that persistent activity is the substrate for a neu-\nral system to retain a memory trace of the stimulus information [4]. For instance, in the classical\ndelayed-response task where an animal needs to memorize the stimulus location for a given pe-\nriod of time before taking an action, it was found that neurons in the prefrontal cortex retained\nhigh-frequency \ufb01ring during this waiting period, indicating that persistent activity may serve as the\n\n1\n\n\fneural substrate of working memory [2]. Understanding the mechanism of how persistent activity is\ngenerated in neural systems has been at the core of theoretical neuroscience for decades [5, 6, 7].\nIn a conventional view, persistent activity is regarded as an emergent property of network dynamic-\ns: neurons in a network are reciprocally connected with each other via excitatory synapses, which\nform a positive feedback loop to maintain neural responses in the absence of an external drive; and\nmeanwhile a matched inhibition process suppresses otherwise explosive neural activities. Mathe-\nmatically, this view is expressed as the dynamics of an attractor network, in which persistent activity\ncorresponds to a stationary state (i.e., an attractor) of the network. The notion of attractor dynamics\nis appealing, which qualitatively describes a number of brain functions, but its detailed implementa-\ntion in neural systems remains to be carefully evaluated.\nA long-standing debate on the feasibility of attractor dynamics is on how to properly close the at-\ntractor states in a network: once a neural system is evolved into a self-sustained active state, it will\nstay there forever until an external force pulls it out. Solutions including applying a strong global in-\nhibitory input to shut-down all neurons simultaneously, or applying a strong global excitatory input\nto excite all neurons and force them to fall into the refractory period simultaneously, were suggest-\ned [9], but none of them appears to be natural or feasible in all conditions. From the computational\npoint of view, it is also unnecessary for a neural system to hold a mathematically perfect attractor\nstate lasting forever. In reality, the brain only needs to hold the stimulus information for a \ufb01nite\namount of time necessary for the task. For instance, in the delayed-response task, the animal only\nneeded to memorize the stimulus location for the waiting period [1].\nTo address the above issues, here we propose a novel mechanism to retain persistent activity in neu-\nral systems, which gives up the concept of prefect attractor, but rather consider that a neural system\nis in a marginally unstable state which decays very slowly and exhibits persistent \ufb01ring for a pro-\nlonged period. The proposed mechanism utilizes a general feature of neuronal interaction, i.e., the\nshort-term plasticity (STP) of synapses [10, 11]. STP has two forms: short-term depression (STD)\nand short-term facilitation (STF). The former is due to depletion of neurotransmitters after neural\n\ufb01ring, and the latter is due to elevation of calcium level after neural \ufb01ring which increases the re-\nlease probability of neurotransmitters. STD and STP have opposite effects on retaining prolonged\nneuronal responses: the former weakens neuronal interaction and hence tends to suppress neuronal\nactivities; whereas, the latter strengthens neuronal interaction and tends to enhance neuronal activ-\nities. Interestingly, we \ufb01nd that the interplay between the two processes endows a neural system\nwith the capacity of holding persistent activity with desirable properties, including: 1) the lifetime\nof persistent activity can be arbitrarily long depending on the parameters; and 2) persistent activity\nfades away naturally in a network without relying on an external force. The implications of these\nresults on neural information representation are discussed.\n\n2 The Model\n\nWithout loss of generality, we consider a homogeneous network in which neurons are randomly and\nsparsely connected with each other with a small probability p. The dynamics of a single neuron is\ndescribed by an integrate-and-\ufb01re process, which is given by\n\n= \u2212(vi \u2212 VL) + Rmhi;\n\n(cid:28)\n\ndvi\ndt\n\nfor i = 1 : : : N;\n\n(1)\n\nwhere vi is the membrane potential of the ith neuron and (cid:28) the membrane time constant. VL is the\nresting potential. hi is the synaptic current and Rm the membrane resistance. A neuron \ufb01res when\nits potential exceeds the threshold, i.e., vi > Vth, and after that vi is reset to be VL. N the number\nof neurons.\nThe dynamics of the synaptic current is given by\n\ndhi\ndt\n\n= \u2212hi +\n\n1\nN p\n\n(cid:28)s\n\n(2)\nwhere (cid:28)s is the synaptic time constant, which is about 2 \u223c 5ms. Jij is the absolute synaptic ef\ufb01cacy\nfrom neurons j to i. Jij = J0 if there is a connection from the neurons j to i, and Jij = 0 otherwise.\ntsp\nj denotes the spiking moment of the neuron j. All neurons in the network receive an external input\n\nJiju+\nj x\n\n);\n\nj (cid:14)(t \u2212 tsp\n\u2212\n\nj ) + I ext(cid:14)(t \u2212 text\n\ni\n\n\u2211\n\nj\n\n2\n\n\fin the form of Poisson spike train. I ext represents the external input strength and text\nof the Poisson spike train the neuron i receives.\nThe variables uj and xj measure, respectively, the STF and STD effects on the synapses of the jth\nneuron, whose dynamics are given by [12, 13]\n\nthe moment\n\ni\n\n(cid:28)f\n\n(cid:28)d\n\n(4)\n\nduj\ndt\ndxj\ndt\n\nj )(cid:14)(t \u2212 tsp\n= \u2212uj + (cid:28)f U (1 \u2212 u\n\u2212\nj );\n= 1 \u2212 xj \u2212 (cid:28)du+\nj (cid:14)(t \u2212 tsp\n\u2212\nj x\nj );\n\u2212\nwhere uj is the release probability of neurotransmitters, with u+\nj and u\nj denoting, respectively,\nthe values of uj just after and just before the arrival of a spike. (cid:28)f is the time constant of STF.\n\u2212\nU controls the increment of uj produced by a spike. Upon the arrival of a spike, u+\nj +\nj = u\nU (1 \u2212 u\n\u2212\nj and x\nj denoting,\nrespectively, the values of xj just after and just before the arrival of a spike. (cid:28)d is the recover time\n\u2212\nof neurotransmitters. Upon the arrival of a spike, x+\nj . The time constants (cid:28)f and (cid:28)d\nare typically in the time order of hundreds to thousands of milliseconds, much larger than (cid:28) and (cid:28)s,\nthat is, STP is a slow process compared to neural \ufb01ring.\n\n\u2212\nj ). xj represents the fraction of available neurotransmitters, with x+\n\n(3)\n\n\u2212\nj = x\nj\n\n\u2212 u+\nj x\n\n2.1 Mean-\ufb01eld approximation\n\nAs to be con\ufb01rmed by simulation, neuronal \ufb01rings in the state of persistent activity are irregular and\nlargely independent to each other. Therefore, we can assume that the responses of individual neurons\nare statistically equivalent in the state of persistent activity. Under this mean-\ufb01eld approximation,\nthe dynamics of a single neuron, and so does the mean activity of the network, can be written as [7]\n\n(cid:28)s\n\n(cid:28)f\n\n(cid:28)d\n\ndh\ndt\ndu\ndt\ndx\ndt\n\n= \u2212h + J0uxR + I;\n= \u2212u + (cid:28)f U (1 \u2212 u)R;\n= 1 \u2212 x \u2212 (cid:28)duxR;\n\n(5)\n\n(6)\n\n(7)\n\nwhere the state variables are the same for all neurons. R is the \ufb01ring rate of a neuron, which is also\nthe mean activity of the neuron ensemble. I = I ext(cid:21) denotes the external input with (cid:21) the rate of\nthe Poisson spike train. The exact relationship between the \ufb01ring rate R and the synaptic input h is\ndif\ufb01cult to obtain. Here, we assume it to be of the form,\n\nR = max((cid:12)h; 0);\n\n(8)\n\nwith (cid:12) a positive constant.\n\n3 The Mechanism\n\nBy using the mean-\ufb01eld model, we \ufb01rst elucidate the working mechanism underlying the generation\nof persistent activity of \ufb01nite lifetime. Later we carry out simulation to con\ufb01rm the theoretical\nanalysis.\n\n3.1 How to generate persistent activity of \ufb01nite lifetime\n\nFor the illustration purpose, we only study the dynamics of the \ufb01ring rate R and assume that the\nvariables u and x reach to their steady values instantly. This approximation is in general inaccurate,\nsince u and x are slow variables compared to R. Nevertheless, it gives us insight into understanding\nthe network dynamics.\nBy setting du=dt = 0 and dx=dt = 0 in Eqs.(6,7) and substituting them into Eqs.(5,8), we get that,\nfor I = 0 and R \u2265 0,\n\n= \u2212R +\n\n(cid:28)s\n\ndR\ndt\n\nJ0(cid:12)(cid:28)f U R2\n\n1 + (cid:28)f U R + (cid:28)d(cid:28)f U R2\n\n\u2261 F (R):\n\n(9)\n\n3\n\n\fFigure 1: The steady states of the network, i.e., the solutions of Eq.(9), have three forms depending\non the parameter values. The three lines correspond to the different neuronal connection strenghths,\nwhich are J0 = 4; 4:38; 5, respectively. The other parameters are: (cid:28)s = 5ms; (cid:28)d = 100ms; (cid:28)f =\n700ms; (cid:12) = 1; U = 0:05 and Jc = 4:38.\n\n(\n\n\u221a\n\n)\n\nDe\ufb01ne a critical connection strength Jc \u2261\n=(cid:12), which is the point the network\ndynamics experiences saddle-node bifurcation (see Figure 1). Depending on the parameters, the\nsteady states of the network have three forms\n\n(cid:28)d=((cid:28)f U )\n\n1 + 2\n\n\u2022 When J0 < Jc, F (R) = 0 has only one solution at R = 0, i.e., the network is only stable\nat the silent state;\n\u2022 When J0 > Jc, F (R) = 0 has three solutions, and the network can be stable at the silent\nstate and an active state;\n\u2022 When J0 = Jc, F (R) = 0 has two solutions, one is the stable silent state, and the other is\na neutral stable state, referred to as R\n\n\u2217.\n\n\u2212\nThe interesting behavior occurs at J0 = J\nc , i.e., J0 is slightly smaller than the critical connection\nstrength Jc. In this case, the network is only stable at the silent state. However, since near to the\n\u2217, F (R) is very close to zero (and so does |dR=dt|), the decay of the network activity is very\nstate R\n\u2217, under the\nslow in this region (Figure 2A). Suppose that the network is initially at a state R > R\n\u2217\nnetwork dynamics, the system will take a considerable amount of time to pass through the state R\nbefore reaching to silence. This is manifested by that the decay of the network activity exhibits a\n\u2217 before dropping to silence rapidly (Figure 2B). Thus, persistent activity of\nlong plateau around R\n\ufb01nite lifetime is achieved.\nThe lifetime of persistent activity, which is dominated by the time of the network state passing\nthrough the point R\n\n\u2217, is calculated to be (see Appendix A),\n\n(10)\n) = d2F (R)=d2R|R(cid:3). By varying the STP effects, such as (cid:28)d and (cid:28)f , the value of\n) is changed, and the lifetime of persistent activity can be adjusted.\n\nF (R\u2217)F \u2032\u2032(R\u2217)\n\n;\n\nwhere F\n)F\nF (R\n\n\u2217\n\n\u2032\u2032\n\u2032\u2032\n\n\u2217\n\u2217\n\n(R\n(R\n\n\u221a\n\nT \u223c\n\n2(cid:28)s\n\n3.2 Persistent activity of graded lifetime\n\nWe formally analyze the condition for the network holding persistent activity of \ufb01nite lifetime.\nInspired by the result in the proceeding section, we focus on the parameter regime of J0 = Jc, i.e.,\nthe situation when the network has the stable silent state and a neutral stable active state.\nDenote (R\ndynamics at this point, we obtain\n\n) to be the neutral stable state of the network at J0 = Jc. Linearizing the network\n\n; u\n\n; x\n\n\u2217\n\n\u2217\n\n\u2217\n\n;\n\n(11)\n\n(\n\nd\ndt\n\nR \u2212 R\n\u2217\nu \u2212 u\u2217\nx \u2212 x\u2217\n\n)\n\nR \u2212 R\n\u2217\nu \u2212 u\u2217\nx \u2212 x\n\u2217\n\n)\n\n(\n\n\u2243 A\n\n4\n\n0RF(R) J0<JcJ0>JcJ0=JcR*\f\u2212\nc ,\nFigure 2: Persistent activity of \ufb01nite lifetime. Obtained by solving Eqs.(5-8). (A) When J0 = J\n\u2217. Around this point, the\nthe function F (R), and so does dR=dt, is very close to zero at the state R\n\u2217. (B) An\nnetwork activity decays very slowly. The inset shows the \ufb01ne structure in the vicinity of R\nexternal input (indicated by the red bar) triggers the network response. After removing the external\ninput, the network activity \ufb01rst decays quickly, and then experiences a long plateau before dropping\nto silence rapidly. The parameters are: (cid:28)s = 5ms; (cid:28)d = 10ms; (cid:28)f = 800ms; (cid:12) = 1; U = 0:5,\nI = 10, Jc = 1:316 and J0 = 1:315.\n\n\u2217\n\n\u2217\n\n; x\n\n; u\n\nwhere A is the Jacobian matrix (see Appendix B).\nIt turns out that the matrix A always has one eigenvector with vanishing eigenvalue, a property due to\n\u2217\n) is the neutral stable state of the network dynamics. As demonstrated in Sec.3.1, by\nthat (R\n\u2212\nc , we expect that the network state will decay very slowly along the eigenvector of\nchoosing J0 = J\nvanishing eigenvalue, which we call the decay-direction. To ensure this always happens, it requires\nthat the real parts of the other two eigenvalues of A are negative, so that any perturbation of the\nnetwork state away from the decay-direction will be pulled back; otherwise, the network state may\napproach to silence rapidly via other routes avoiding the state (R\n). This idea is illustrated\nin Fig.3.\nThe condition for the real parts of the other two eigenvalues of A being smaller than zero is calcu-\nlated to be (see Appendix B):\n\n; u\n\n; x\n\n\u2217\n\n\u2217\n\n\u2217\n\n\u221a\n\n\u221a\n\n1\n\n2\n\n(cid:28)f (cid:28)d\n\n+\n\n1\n(cid:28)d\n\n1\n\n+\n\nU\n(cid:28)f (cid:28)d\n\u2212\nc form the condition for the network holding persistent activity\n\n(cid:28)f U\n(cid:28)d\n\n(12)\n\n(cid:28)d(cid:28)s\n\n> 0:\n\n1 +\n\n\u2212 1\n(cid:28)f (cid:28)s\n\nThis inequality together with J0 = J\nof \ufb01nite lifetime.\n\nFigure 3: Illustration of the slow-decaying process of the network activity. The network dynamics\nexperiences a long plateau before dropping to silence quickly. The inset presents a 3-D view of the\nlocal dynamics in the plateau region, where the network state is attracted to the decay-direction to\nensure slow-decaying.\n\nBy solving the network dynamics Eqs.(5-8), we calculate how the lifetime of persistent activity\nchanges with the STP effect. Fig.4A presents the results of \ufb01xing U and J0 and varying (cid:28)d and\n\n5\n\n0RF(R)R*0AB02468R*t(s)R(Hz)tRR*3-D viewDecay-direction\f(cid:28)f , We see that below the critical line J0 = Jc, which is the region for J0 > Jc, the network has\nprefect attractor states never decaying; and above the critical line, the network has only the stable\nsilent state. Close to the critical line, the network activity decays slowly and displays persistent\nactivity of \ufb01nite lifetime. Fig.4B shows a case that when the STF strength ((cid:28)f ) is \ufb01xed, the lifetime\nof persistent activity decreases with the STD strength ((cid:28)d). This is understandable, since STD tends\nto suppress neuronal responses. Fig.4C shows a case that when (cid:28)d is \ufb01xed, the lifetime of persistent\nactivity increases with (cid:28)f , due to that STF enhances neuronal responses. These results demonstrate\nthat by regulating the effects of STF and STD, the lifetime of persistent activity can be adjusted.\n\nFigure 4: (A). The lifetimes of the network states with respect to (cid:28)f and (cid:28)d when U and J0 are \ufb01xed.\nWe use an external input to trigger a strong response of the network and then remove the input. The\nlifetime of a network state is measured from the offset of the external input to the moment when the\nnetwork returns to silence. The white line corresponds to the condition of J0 = Jc, below which\nthe network has attractors lasting forever; and above which, the lifetime of a network state gradually\ndecreases (coded by colour). (B) When (cid:28)f = 1250ms is \ufb01xed, the lifetime of persistent activity\ndecreases with (cid:28)d (the vertical dashed line in A). (C) When (cid:28)d = 260ms is \ufb01xed, the lifetime of\npersistent activity increases with (cid:28)f (the horizontal dashed line in A). The other parameters are:\n(cid:28)s = 5ms, (cid:12) = 1, U = 0:05 and J0 = 5.\n\n4 Simulation Results\n\nWe carry out simulation with the spiking neuron network model given by Eqs.(1-4) to further con\ufb01rm\nthe above theoretical analysis. A homogenous network with N = 1000 neurons is used, and in the\nnetwork neurons are randomly and sparsely connected with each other with a probability p = 0:1.\nAt the state of persistent activity, neurons \ufb01re irregularly (the mean value of Coef\ufb01cient of Variation\nis 1.29)and largely independent to each other(the mean correlation of all spike train pairs is 0.30)\nwith each other (Fig.5A). Fig.5 present the examples of the network holding persistent activity with\nvaried lifetimes, through different combinations of STF and STD satisfying the condition Eq.(12).\n\n5 Conclusions\n\nIn the present study, we have proposed a simple yet effective mechanism to generate persistent\nactivity of graded lifetime in a neural system. The proposed mechanism utilizes the property of STP,\na general feature of neuronal synapses, and that STF and STD have opposite effects on retaining\nneuronal responses. We \ufb01nd that with properly combined STF and STD, a neural system can be in a\nmarginally unstable state which decays very slowly and exhibits persistent \ufb01ring for a \ufb01nite lifetime.\nThis persistent activity fades away naturally without relying on an external force, and hence avoids\nthe dif\ufb01culty of closing an active state faced by the conventional attractor networks.\nSTP has been widely observed in the cortex and displays large diversity in different region-\ns [14, 15, 16]. Compared to static ones, dynamical synapses with STP greatly enriches the response\npatterns and dynamical behaviors of neural networks, which endows neural systems with informa-\ntion processing capacities which are otherwise dif\ufb01cult to implement using purely static synapses.\nThe research on the computational roles of STP is receiving increasing attention in the \ufb01eld [12]. In\n\n6\n\n00.511.50510\u03c4d (s)Decay time (s)00.511.50510\u03c4f (s)Decay time (s)attractorABC\fFigure 5: The simulation results of the spiking neural network. (A) A raster plot of the responses of\n50 example neurons randomly chosen from the network. The external input is applied for the \ufb01rst\n0.5 second. The persistent activity lasts about 1100ms. The parameters are: (cid:28)f = 800ms; (cid:28)d =\n500ms; U = 0:5; J0 = 28:6. (B) The \ufb01ring rate of the network for the case (A). (C) An example\nof persistent activity of negligible lifetime. The parameters are:(cid:28)f = 800ms; (cid:28)d = 1800ms; U =\n0:5; J0 = 28:6. (D) An example of persistent activity of around 400ms lifetime. The parameters\nare:(cid:28)f = 600ms; (cid:28)d = 500ms; U = 0:5; J0 = 28:6. (E) An example of the network holding an\nattractor lasting forever. The parameters are: (cid:28)f = 800ms; (cid:28)d = 490ms; U = 0:5; J0 = 28:6.\n\nterms of information presentation, a number of appealing functions contributed by STP were pro-\nposed. For instances, Mongillo et al. proposed an economical way of using the facilitated synapses\ndue to STF to realize working memory in the prefrontal cortex without recruiting neural \ufb01ring [8];\nP\ufb01ster et al. suggested that STP enables a neuron to estimate the membrane potential information\nof the pre-synaptic neuron based on the spike train it receives [17]. Torres et al. found that STD\ninduces instability of attractor states in a network, which could be useful for memory searching [18];\nFung et al. found that STD enables a continuous attractor network to have a slow-decaying state in\nthe time order of STD, which could serve for passive sensory memory [19]. Here, our study reveals\nthat through combining STF and STD properly, a neural system can hold stimulus information for\nan arbitrary time, serving for different computational purposes. In particular, STF tends to increase\nthe lifetime of persistent activity; whereas, STD tends to decrease the lifetime of persistent activity.\nThis property may justify the diverse distribution of STF and STD in different cortical regions. For\ninstances, in the prefrontal cortex where the stimulus information often needs to be held for a long\ntime in order to realize higher cognitive functions, such as working memory, STF is found to be\ndominating; whereas, in the sensory cortex where the stimulus information will be forwarded to\nhigher cortical regions shortly, STD is found to be dominating. Furthermore, our \ufb01ndings suggest\nthat a neural system may actively regulate the combination of STF and STD, e.g., by applying ap-\npropriate neural modulators [10], so that it can hold the stimulus information for a \ufb02exible amount\nof time depending on the actual computational requirement. Further experimental and theoretical\nstudies are needed to clarify these interesting issues.\n\n6 Acknowledgments\n\nThis work is supported by grants from National Key Basic Research Program of China\n(NO.2014CB846101), and National Foundation of Natural Science of China (No.11305112, Y.Y.M.;\nNo.31261160495, S.W.; No.31271169,D.H.W.), and the Fundamental Research Funds for the cen-\ntral Universities (No.31221003, S.W.), and SRFDP (No.20130003110022, S.W), and Natural Sci-\nence Foundation of Jiangsu Province BK20130282.\n\n7\n\nABCDE\fAppendix A: The lifetime of persistent activity\n\n\u2217\n\n) is slightly smaller than zero (Fig.2A). Starting from a state R > R\n\nConsider the network dynamics Eq.(9). When J0 = Jc, the network has a stable silent state (R = 0)\n\u2212\nand an unstable active state, referred to as R\nc . In this case,\n\u2217, the network will take\nF (R\n\u2217, since dR=dt is very small in this region, and\na considerable amount of time to cross the point R\nthe network exhibits persistent activity for a considerable amount of time. We estimate the time\n\u2217.\nconsuming for the network crossing the point R\nAccording to Eq.(9), we have\n\n\u2217 (Fig.1). We consider that J0 = J\n\nT\u222b\n\ndt =\n\n0\n\n=\n\n=\n\n(cid:3)\n\nR\n\n+\u222b\n\u221a\n\u221a\n\n(cid:3)\n(cid:0)\n\nR\n\ndR \u2248\n\n(cid:28)s\n\nF (R)\n\n2(cid:28)s\n\n2(cid:28)s\n\nF (R\u2217)F \u2032\u2032(R\u2217)\n\nF (R\u2217)F \u2032\u2032(R\u2217)\n\n(cid:3)\n\nR\n\n+\u222b\n[\n\n(cid:3)\n(cid:0)\n\nR\n\n\u2217\n\n);\n\nG(R\n\n(cid:28)sdR\n\nF (R\u2217) + (R \u2212 R\u2217)2F \u2032\u2032(R\u2217)=2\n\u221a\n\n\u2212 R\n\n\u2212 arctg\n\n\u2217\n+\n\nR\n\nF (R\u2217)=F \u2032\u2032(R\u2217)\n\n\u2217\n\n;\n\n\u221a\n\narctg\n\n\u2212 \u2212 R\n\u2217\n\n\u2217\n\nR\n\nF (R\u2217)=F \u2032\u2032(R\u2217)\n\n]\n\n;\n\n(13)\n\n\u2217\n\u2212 denote, respectively, the points slightly larger or smaller than R\n\n(R)=dR|R(cid:3). To get the above result, we used the second-order\n\n\u2217, F\n\n) = dF\n\n) =\n\n(R\n\n\u2217\n\n\u2032\n\n\u2032\n\n\u2217, and the condition F\n\n\u2032\n\n\u2217\n\n(R\n\n) = 0.\n\n\u2217\n+ and R\n\n\u2217\n\n\u2032\u2032\n\nwhere R\ndF (R)=dR|R(cid:3), and F\nTaylor expansion of F (R) at R\n\u2217\nIn the limit of F (R\nis in the order of\n\n(R\n\n) \u2192 0, the value of G(R\n\u221a\n\nT \u223c\n\n\u2217\n\n) is bounded. Thus, the lifetime of persistent activity\n\n2(cid:28)s\n\nF (R\u2217)F \u2032\u2032(R\u2217)\n\n:\n\n(14)\n\nAppendix B: The condition for the network holding persistent activity of \ufb01nite\nlifetime\n\n) to be the neutral stable state of the network when J0 = Jc, which is calculated\n\n\u2217\n\n\u2217\n\nDenote (R\nto be (by solving Eqs.(5-8)),\n\n; x\n\n; u\n\n\u2217\n\n\u221a\n\n\u2217\n\nR\n\n=\n\n1=(cid:28)f (cid:28)dU ; u\n\n\u2217\n\n=\n\n\u2217\n\n(cid:28)f U R\n1 + (cid:28)f U R\u2217 ; x\n\n\u2217\n\n=\n\n\u2217\n\n1 + (cid:28)f U R\n\n1 + (cid:28)f U R\u2217 + (cid:28)f (cid:28)dU R\u22172 :\n)\n\n(\n\nLinearizing the network dynamics at this point, we obtain Eq.(12), in which the Jacobian matrix A\nis given by\n\n\u2217 \u2212 1)=(cid:28)s;\n\u2217\nx\n(J0u\nU (1 \u2212 u\n\u2217\n\u2212u\n\u2217\n\u2217\nx\n\n);\n\n\u2217\n\n\u2217\n=(cid:28)s;\nR\nJ0x\n\u22121=(cid:28)f \u2212 U R\n\u2217\n\u2212x\n\u2217\n\u2217\n\nR\n\n;\n\n\u2217\n\u2217\n=(cid:28)s\nJ0u\n\u22121=(cid:28)d \u2212 u\n\u2217\n\nR\n0\n\nA =\n\n(16)\nThe eigenvalues of the Jacobian matrix satisfy the equality |A \u2212 (cid:21)I| = 0. Utilizing Eqs.(15), this\nequality becomes\n\nR\n\n\u2217\n\n;\n\n;\n\n:\n\n(cid:21)((cid:21)2 + b(cid:21) + c(cid:21)) = 0;\n\n(15)\n\n(17)\n\nwhere the coef\ufb01cients b and c are given by\n\nb =\n\nc =\n\n1\n(cid:28)d\n\n+\n\n1\n(cid:28)f\n\n+ u\n\n\u221a\n\n\u2217\n\n\u2217\n\nR\n\n+ U R\n\n\u2217\n\n;\n\n2\n\n(cid:28)f (cid:28)d\n\n+\n\n1\n(cid:28)d\n\nU\n(cid:28)f (cid:28)d\n\n+\n\n1\n\n(cid:28)d(cid:28)s\n\n1 +\n\n\u221a\n\n1\n\n\u2212 1\n(cid:28)f (cid:28)s\n\n:\n\n(cid:28)f U\n(cid:28)d\n\n(18)\n\n(19)\n\nFrom Eq.(17), we see that the matrix A has three eigenvalues. One eigenvalue, referred to as (cid:21)1, is\nalways zero. The other two eigenvalues satisfy that (cid:21)2 + (cid:21)3 = \u2212b and (cid:21)2(cid:21)3 = c. Since b > 0, the\ncondition for the real parts of (cid:21)2 and (cid:21)3 being negative is c > 0.\n\n8\n\n\fReferences\n[1] J. Fuster and G. Alexander. Neuron activity related to short-term memory. Science 173, 652-\n\n654 (1971).\n\n[2] S. Funahashi, C. J. Bruce and P.S. Goldman-Rakic. Mnemonic coding of visual space in the\n\nmonkeys dorsolateral prefrontal cortex. J. Neurophysiol. 61, 331-349 (1989).\n\n[3] R. Romo, C. D. Brody, A. Hernandez. Lemus L. Neuronal correlates of parametric working\n\nmemory in the prefrontal cortex. Nature 399, 470-473 (1999).\n\n[4] D.J. Amit. Modelling brain function. New York: Cambridge University Press. (1989)\n[5] S. Amari. Dynamics of pattern formation in lateral-inhibition type neural \ufb01elds. Biol. Cybern.\n\n27, 77-87 (1977).\n\n[6] X.J. Wang. Synaptic basis of cortical persistent activity: the importance of NMDA receptors\n\nto working memory. J. Neurosci. 19, 9587-9603 (1999).\n\n[7] O. Barak and M. Tsodyks. Persistent Activity in Neural Networks with Dynamic Synapses.\n\nPLoS Computational Biology.3(2): e35(2007).\n\n[8] G. Mongillo, O. Barak and M. Tsodyks. Synaptic theory of working memory. Science\n\n319,1543-1546(2008).\n\n[9] B. Gutkin, C. Laing, C. Colby, C. Chow, and B. Ermentrout. Turning on and off with excitation:\nthe role of spike-timing asynchrony and synchrony in sustained neural activity. J. Comput.\nNeurosci. 11, 121C134 (2001).\n\n[10] H. Markram and M. Tsodyks. Redistribution of synaptic ef\ufb01cacy between neocortical pyrami-\n\ndal neurons. Nature. 382(6594): 807-810(1996).\n\n[11] L. F. Abbott and W. G. Regehr. Synaptic computation. Nature. 431(7010): 796-803(2004).\n[12] M. Tsodyks and S. Wu. Short-Term Synaptic Plasticity. Scholarpedia, 8(10): 3153(2013).\n[13] M. Tsodyks, K. Pawelzik and H. Markram. Neural Networks with Dynamic Synapses. Neural\n\nComputation. 10(4): 821-835(1998).\n\n[14] H. Markram, Y. Wang and M. Tsodyks. Differential signaling via the same axon of neocor-\ntical pyramidal neurons. Proceedings of the National Academy of Sciences. 95(9): 5323-\n5328(1998).\n\n[15] J. S. Dittman, A. C. Kreitzer and W. G. Regehr. Interplay between facilitation, depression, and\n\nresidual calcium at three presynaptic terminals. J. Neurosci. 20: 1374-1385(2000).\n\n[16] Y. Wang, H. Markram, P. H. Goodman, T. K. Berger, J. Y. Ma and P. S. Goldman-Rakic.\nHeterogeneity in the pyramidal network of the medial prefrontal cortex. Nature Neuroscience.\n9(4): 534-542(2006).\n\n[17] J. P. P\ufb01ster, P. Dayan and M. Lengyel. Synapses with short-term plasticity are optimal estima-\n\ntors of presynaptic membrane potentials. Nature Neuroscience 13,1271-1275(2010).\n\n[18] J.J. Torres, J. M. Cortes, J. Marro and H. J. Kappen. Competition Between Synaptic Depression\nand Facilitation in Attractor Neural Networks. Neural Computation. 19(10): 2739-2755(2007).\n[19] C. C. Fung, K. Y. Michael Wong, H. Wang and S. Wu. Dynamical Synapses Enhance Neural In-\nformation Processing: Gracefulness, Accuracy and Mobility. Neural Computation 24(5):1147-\n1185(2012).\n\n9\n\n\f", "award": [], "sourceid": 256, "authors": [{"given_name": "Yuanyuan", "family_name": "Mi", "institution": "Weizmann Institute of Science"}, {"given_name": "Luozheng", "family_name": "Li", "institution": "Beijing Normal University"}, {"given_name": "Dahui", "family_name": "Wang", "institution": "Beijnng Normal University"}, {"given_name": "Si", "family_name": "Wu", "institution": "Beijing Normal University"}]}