{"title": "An Optimization Network for Matrix Inversion", "book": "Neural Information Processing Systems", "page_first": 397, "page_last": 401, "abstract": null, "full_text": "397 \n\nAN OPTIMIZATION NETWORK FOR MATRIX INVERSION \n\nJu-Seog Jang, S~ Young Lee, and Sang-Yung Shin \n\nKorea Advanced Institute of Science and Technology, \n\nP.O. Box 150, Cheongryang, Seoul, Korea \n\nABSTRACT \n\nInverse matrix calculation can be considered as an optimization. We have \ndemonstrated that this problem can be rapidly solved by highly interconnected \nsimple neuron-like analog processors. A network for matrix inversion based on \nthe concept of Hopfield's neural network was designed, and implemented with \nelectronic hardware. With slight modifications, the network is readily applicable to \nsolving a linear simultaneous equation efficiently. Notable features of this circuit \nare potential speed due to parallel processing, and robustness against variations of \ndevice parameters. \n\nINTRODUCTION \n\nHighly interconnected simple analog processors which mmnc a biological \nneural network are known to excel at certain collective computational tasks. For \nexample, Hopfield and Tank designed a network to solve the traveling salesman \nproblem which is of the np -complete class,l and also designed an AID converter \nof novel architecture2 based on the Hopfield's neural network model?' 4 The net(cid:173)\nwork could provide good or optimum solutions during an elapsed time of only a \nfew characteristic time constants of the circuit. \n\nThe essence of collective computation is the dissipative dynamics in which ini(cid:173)\n\ntial voltage configurations of neuron-like analog processors evolve simultaneously \nand rapidly to steady states that may be interpreted as optimal solutions. Hopfield \nhas constructed the computational energy E (Liapunov function), and has shown \nthat the energy function E of his network decreases in time when coupling coeffi(cid:173)\ncients are symmetric. At the steady state E becomes one of local minima. \n\nIn this paper we consider the matrix inversion as an optimization problem, \n\nand apply the concept of the Hopfield neural network model to this problem. \n\nCONSTRUCTION OF THE ENERGY FUNCTIONS \n\nConsider a matrix equation AV=I, where A is an input n Xn matrix, V is \nthe unknown inverse matrix, and I is the identity matrix. Following Hopfield we \ndefine n energy functions E Ie' k = 1, 2, ... , n, \n\nn \n\nn \n\nn \n\nE 1 = (1I2)[(~ A 1j Vj1 -1)2 + (~A2) Vj1 )2 + ... + (~Anj Vj1)2] \n\n)-1 \nn \n\nj-1 \n\nn \n\nE2 = (1/2)[(~A1)V)2l + (~A2)V)2-1)2 + \n\n)=1 \n\n)=1 \n\n)-1 \n\nn \n\n+ (~An)V}2)2] \n\n}-1 \n\n\u00a9 American Institute of Physics 1988 \n\n\f398 \n\nn \n\nn \n\nn \n\nEn = (1/2)[(~ A1J VJn)2 + (~A2J Vjn )2 + ... + (~An) VJn _1)2] \n\n(1) \n\nj=l \n\n}=1 \n\nJ-1 \n\nwhere AiJ and ViL.are the elements of ith row and jth column of matrix A and \nV, respectively. when A is a nonsingular matrix, the minimum value (=zero) of \neach energy function is unique and is located at a point in the corresponding \nhyperspace whose coordinates are { V u:, V 2k ' \" ' , V nk }, k = 1, 2, \"', n. At \nthis minimum value of each energy function the values of V 11' V 12' ... , Vnn \nbecome the elements of the inverse matrix A -1. When A is a singular matrix the \nminimum value (in general, not zero) of each energy function is not unique and is \nlocated on a contour line of the minimum value. Thus, if we construct a model \nnetwork in which initial voltage configurations of simple analog processors, called \nneurons, converge simultaneously and rapidly to the minimum energy point, we can \nsay the network have found the optimum solution of matrix inversion problem. \nThe optimum solution means that when A is a nonsingular matrix the result is the \ninverse matrix that we want to know, and when A is a singular matrix the result \nis a solution that is optimal in a least-square sense of Eq. (1). \n\nDESIGN OF THE NETWORK AND THE HOPFIELD MODEL \n\nDesigning the network for matrix inversion, we use the Hopfield model \n\nwithout inherent loss terms, that is, \n\n- -= \n\ndt \n\na \naVik \n\n---Ek(V 11' V 2k' . . . , Vnk ) \n\ni,k=1,2, ... ,n \n\n(2) \n\nwhere uik is the input voltage of ith neuron in the kth network, Vik is its output, \nand the function gik \nis the input-output relationship. But the neurons of this \nscheme operate in all the regions of gik differently from Hopfield's nonlinear 2-\nstate neurons of associative memory models.3\u2022 4 \n\nFrom Eq. (1) and Eq. (2), we can define coupling coefficients Tij between \n\nith and jth neurons and rewrite Eq. (2) as \n\n- -= \n\n- ~ TiJ V)k + Aki , \n\ndt \n\nn \n\nj=l \n\nn \n\nTiJ = ~ AliAIJ = Tji ' \n\n1=1 \n\n(3) \n\nIt may be noted that Ti \u00b7 \nis independent of k and only one set of hardware is \nneeded for all k. The implemented network is shown in Fig. 1. The same set of \n\nn \n\nhardware with bias levels, ~ A Ji h), can be used to solve a linear simultaneous \n\n)=1 \n\n\fequation represented by Ax=b for a given vector b. \n\nINPUT \n\n399 \n\nOUTPUT \n\nFig. 1. Implemented network for matrix inversion with externally \ncontrollable coupling coefficients. Nonlinearity between \nthe input and \nthe output of neurons is assumed to be \ndistributed in the adder and the integrator. \n\nThe application of the gradient Hopfield model to this problem gives the result \nthat is similar to the steepest descent method.s But the nonlinearity between the \ninput and the output of neurons is introduced. Its effect to the computational \ncapability will be considered next. \n\nCHARACTERISTICS OF THE NETWORK \n\nFor a simple case of 3 x3 input matrices the network is implemented with \nelectronic hardware and its dynamic behavior is simulated by integration of the \nEq. (3). For nonsingular input matrices, exact realization of Tij connection and \nbias Ali is an important factor for calculation accuracy, but the initial condition \nand other device parameters such as steepness, shape and uniformity of gil are \nnot. Even a complex gik function shown in Fig. 2 can not affect the computa(cid:173)\ntional capability. Convergence time of the output state is determined by the \ncharacteristic time constant of the circuit. An example of experimental results is \nshown in Fig. 3. For singular input matrices, the converged output voltage confi(cid:173)\nguration of the network is dependent upon the initial state and the shape of gil' \n\n\f400 \n\n,...-_____ Vm-t-___ --::==----r A ik > 1 \n= 1 \n< 1 \n\nVm \n\nUi\\< \n\nFig. 2. gile \nAile \n\nfunctions used in computer simulations where \nis the steepness of sigmoid function tanh (Aile uile)' \n\ninput \nmatrix \n\nA = -I \n\n[ 1 2 I] \n\nr 1 \n1 0-1 \n\n(cf) \n\nA-I = \n\n[\n\n0.5 -I \n1 \n\n0 \n\n-o.~] \n\n0.5 -I \n\n-1.5 \n\noutput \nmatrix \n\n0.50 -0.98 -0.49J \nV = 0.02 0.99 1.00 \n0.53 -0.98 - 1.50 \n\n[\n\no \n\n0.5 \n\n\u00b0 \n\nFig. 3. An example of experimental results \n\n\fCOMPLEXITY ANALYSIS \n\nBy counting operations we compare the neural net approach with other well(cid:173)\nknown methods such as Triangular-decomposition and Gauss-Jordan elimination.6 \n\n401 \n\n(1) Triangular-decomposition or Gauss-Jordan elimination method takes 0 (8n 3/3) \nmultiqlications/divisions and additions \ninversion, and \no (2n /3) multiplications/divisions and additions for solving the linear simultaneous \nequation Ax=b. \n\nlarge n Xn matrix \n\nfor \n\n(2) The neural net approach takes the number of operations required to calculate \nTij (nothing but matrix-matrix multiplication), that is, 0 (n 3/2) multiplications and \nadditions for both matrix inversion and solving the linear simultaneous equation. \nAnd the time required for output stablization is about a few times the charac(cid:173)\nteristic time constant of the network. The calculation of coupling coefficients can \nbe directly executed without multiple iterations by a specially designed optical \nmatrix-matrix multiplier,' while the calculation of bias values in solving a linear \nsimultaneous equation can be done by an optical vector-matrix multiplier.8 Thus, \nthis approach has a definite advantage in potential calculation speed due to global \ninterconnection of simple parallel analog processors, though its calculation accu(cid:173)\nracy may be limited by the nature of analog computation. A large number of \ncontrollable Tij interconnections may be easily realized with optoelectronic dev(cid:173)\nices.9 \n\nCONCLUSIONS \n\nWe have designed and implemented a matrix inversion network based on the \nconcept of the Hopfield's neural network model. 1bis network is composed of \nhighly interconnected simple neuron-like analog processors which process the infor(cid:173)\nmation in parallel. The effect of sigmoid or complex nonlinearities on the compu(cid:173)\ntational capability is unimportant in this problem. Steep sigmoid functions reduce \nonly the convergence time of the network. When a nonsingular matrix is given as \nan input, the network converges spontaneously and rapidly to the correct inverse \nmatrix regardless of initial conditions. When a singular matrix is given as an \ninput, the network gives a stable optimum solution that depends upon initial con(cid:173)\nditions of the network. \n\nREFERENCES \n\n1. J. J. Hopfield and D. W. Tank, BioI. Cybern. 52, 141 (1985). \n2. D. W. Tank and J. J. Hopfield, IEEE Trans. Circ. Sys. CAS-33, 533 (1986). \n3. J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982). \n4. J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 81 , 3088 (1984). \n5. G. A. Bekey and W. J. Karplus, Hybrid Computation (Wiley, 1968), P. 244. \n6. M. J. Maron, Numerical Analysis: A Practical Approach (Macmillan, 1982), \n\n7. H . Nakano and K. Hotate, Appl. Opt. 26, 917 (1987). \n8. J. W. Goodman, A. R. Dias, and I. M. Woody, Opt. Lett. ~ 1 (1978). \n9. J. W. Goodman, F. J. Leonberg, S-Y. Kung, and R. A. Athale, IEEE Proc. \n\np. 138. \n\n72, 850 (1984). \n\n\f", "award": [], "sourceid": 45, "authors": [{"given_name": "Ju-Seog", "family_name": "Jang", "institution": null}, {"given_name": "Soo-Young", "family_name": "Lee", "institution": null}, {"given_name": "Sang-Yung", "family_name": "Shin", "institution": null}]}