{"title": "A 1, 000-Neuron System with One Million 7-bit Physical Interconnections", "book": "Advances in Neural Information Processing Systems", "page_first": 705, "page_last": 711, "abstract": "", "full_text": "A 1,OOO-Neuron System with One \n\nMillion 7-bit Physical Interconnections \n\nInstitute of Information Sciences and Electronics \n\nYuzo Hirai \n\nUniversity of Tsukuba \n\n1-1-1 Ten-nodai, Tsukuba, Ibaraki 305, Japan \n\ne-mail: hirai@is.tsukuba.ac.jp \n\nAbstract \n\nAn asynchronous PDM (Pulse-Density-Modulating) digital neural \nnetwork system has been developed in our laboratory. It consists \nof one thousand neurons that are physically interconnected via one \nmillion 7-bit synapses. It can solve one thousand simultaneous \nnonlinear first-order differential equations in a fully parallel and \ncontinuous fashion. The performance of this system was measured \nby a winner-take-all network with one thousand neurons. Although \nthe magnitude of the input and network parameters were identi(cid:173)\ncal for each competing neuron, one of them won in 6 milliseconds. \nThis processing speed amounts to 360 billion connections per sec(cid:173)\nond. A broad range of neural networks including spatiotemporal \nfiltering, feedforward, and feedback networks can be run by loading \nappropriate network parameters from a host system. \n\n1 \n\nINTRODUCTION \n\nThe hardware implementation of neural networks is crucial in order to realize the \nreal-time operation of neural functions such as spatiotemporal filtering, learning \nand constraint processings. Since the mid eighties, many VLSI chips and systems \nhave been reported in the literature, e.g. [1] [2]. Most of the chips and the systems \nincluding analog and digital implementations, however, have focused on feedforward \nneural networks. Little attention has been paid to the dynamical aspect of feed(cid:173)\nback neural networks, which is especially important in order to realize constraint \nprocessings, e.g. [3]. Although there were a small number of exceptions that used \nanalog circuits [4] [5], their network sizes were limited as compared to those of their \nfeedforward counterparts because of wiring problems that are inevitable in regard \nto full and physical interconnections. To relax this problem, a pulse-stream system \nhas been used in analog [6] and digital implementations [7]. \n\n\f706 \n\nY. Hirai \n\nThe author developed a fully interconnected 54-neuron system that uses an asyn(cid:173)\nchronous PDM (Pulse-Density-Modulating) digital circuit system [8]. The present \npaper describes a thousand-neuron system in which all of the neurons are physi(cid:173)\ncally interconnected via one million 7-bit synapses in order to create a fully parallel \nfeedback system. The outline of this project was described in [10]. In addition to \nthe enlargement of system size, synapse circuits were improved and time constant \nof each neuron was made variable. The PDM system was used because it can ac(cid:173)\ncomplish faithful analog data transmission between neurons and can relax wiring \nproblems. An asynchronous digital circuit was used because it can solve scaling \nproblems, and we could also use it to connect more than one thousand VLSI chips, \nas described below. \n\n2 NEURON MODEL AND THE CIRCUITS \n\n2.1 SINGLE NEURON MODEL \n\nThe behavior of each neuron in the system can be described by the following non(cid:173)\nlinear first-order differential equation: \n\ndyi(t) \nIti--;];t \n\n= -viet) + L WijYj{t) + li{t), \n\nN \n\nj=l \nYi{ t) =

0 \ntaind by the difference between a fast and a slow neuron. (d) Response of \na winner-take-all network among 1,007 neurons. The responses of a winner neuron \nand 24 of the 1,006 defeated neurons are shown. \n\nporal filters can be implemented in this way. \n\nFigure 3( d) shows the responses of a winner-take-all network among 1,007 neurons. \nThe time courses of the responses of a winner neuron and 24 of the 1,006 defeated \nneurons are shown in the figure. The strength of all of the inhibitory synaptic \nweights between neurons was set to 2 x (- ::), where 2 is an output scale factor. The \nsynaptic weights from a signal neuron to the 1,007 competing ones were identical and \nwere ~;. Although the network parameters and the inputs to all competing neurons \nwere identical, one of them won in 6 msec. Since the system operates asynchronously \nand the spatial summation of the synaptic output pulses is probabilistic, one of the \ncompeting neurons can win in a stochastic manner. \n\nIn order to derive the processing speed in terms of connections per second, the \nsame winner-take-all network was solved by the Euler method on a latest work(cid:173)\nstation. Since it took about 76.2 seconds and 2,736 iterations to converge, the \nprocessing speed of the workstation was about 36 million connections per second \ntimes aster t an t e wor statlOn, \n( \n::::::! \n\n\u2022 mce t 1S system IS \n\nl007xI0<17x2736) S' \n\n. 10000' \n\n76 .2\n\nh' \n\nh \n\nh \n\nk \n\nf \n\n. \n\n! \n\n, \n\n\fA I,OOO-Neuron System with One MillioK7-bit Physical Interconnections \n\n711 \n\nthe processing speed amounts to 360 billion connections per second. \n\nVarious kinds of neural networks including spatiotemporal filtering, feedforward and \nfeedback neural networks can be run in this single system by loading appropriate \nnetwork parameters from the host system. The second version of this system, which \ncan be used via the Internet, will be completed by the end of March, 1998. \n\nAcknowledgements \n\nThe author is grateful to Mr. Y. Kuwabara and Mr. T. Ochiai of Hitachi Micro(cid:173)\ncomputer System Ltd. for their collaboration in developing this system and to Dr. \nM. Yasunaga and Mr. M. Takahashi for their help in testing it. The author is also \ngrateful to Mr. H. Toda for his collaboration in measuring response data. This work \nwas supported by \"Proposal-Based Advanced Industrial Technology R&D Program\" \nfrom NEDO in Japan. \n\nReferences \n\n[1] C. Mead: Analog VLSI and Neural Systems. Addison-Wesley Publishing Com(cid:173)\n\npany, Massachusetts, 1989 \n\n[2] K.W.Przytula and V.K.Prasanna, Eds.: Parallel Digital Implementations of \n\nNeural Networks. Prentice Hall, New Jersey, 1993 \n\n[3] J.J. Hopfield: Neurons with graded response have collective computational \nproperties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A., 81, \npp.3088-3092, 1984 \n\n[4] P. Mueller, J. van der Spiegel, V. Agami, D. Blackman, P. Chance, C. Donham, \nR. Etienne, J. Kim. M. Massa and S. Samarasekera: Design and performance \nof a prototype analog neural computer. Proc. the 2nd International Conf. on \nMicroelectronics for Neural Networks, pp.347-357, 1991 \n\n[5] G. Cauwenberghs: A learning analog neural network chip with continuous-time \n\nrecurrent dynamics. In J. D. Cowan, G. Tesauro and J. Alspector, Eds., Ad(cid:173)\nvances in Neural Information Processing Systems 6, Morgan Kaufmann Pub(cid:173)\nlishers, San Mateo, CA, pp.858-865, 1994 \n\n[6] S. Churcher, D. J. Baxter, A. Hamilton, A. F. Murry, and H. M. Reekie: \nGeneric analog neural computation - The EPSILON chip. In S. J. Hanson, J. \nD. Cowan and C. L. Giles, Eds., Advances in Neural Information Processing \nSystems 6, Morgan Kaufmann Publishers, San Mateo, CA, pp.773-780, 1993 \n\n[7] H. Eguchi, T. Furuta, H. Horiguchi, S. Oteki and T. Kitaguchi: Neural network \nLSI chip with on-chip learning. Proceedings of IJCNN'91 Seattle, Vol.I/453-456, \n1991 \n\n[8] Y. Hirai, et al.: A digital neuro-chip with unlimited connectability for large \nscale neural networks. Proc. International Joint Conf. on Neural Networks'89 \nWashington D.C., Vo1.11/163-169, 1989 \n\n[9] Y.Hirai, VLSI Neural Network Systems (Gordon and Breach Science Publish(cid:173)\n\ners, Birkshire, 1992) \n\n[10] Y. Hirai and M. Yasunaga: A PDM digital neural network system with 1,000 \nneurons fully interconnected via 1,000,000 6-bit synapses. Proc. International \nConference on Neural Information Processings'96, Vo1.ll/1251, 1996 \n\n\f", "award": [], "sourceid": 1395, "authors": [{"given_name": "Yuzo", "family_name": "Hirai", "institution": null}]}