{"title": "Are Hopfield Networks Faster than Conventional Computers?", "book": "Advances in Neural Information Processing Systems", "page_first": 239, "page_last": 245, "abstract": null, "full_text": "Are Hopfield Networks Faster Than \n\nConventional Computers? \n\nIan Parberry* and Hung-Li Tsengt \n\nDepartment of Computer Sciences \n\nUniversity of North Texas \n\nP.O. Box 13886 \n\nDenton, TX 76203-6886 \n\nAbstract \n\nIt is shown that conventional computers can be exponentiallx faster \nthan planar Hopfield networks: although there are planar Hopfield \nnetworks that take exponential time to converge, a stable state of an \narbitrary planar Hopfield network can be found by a conventional \ncomputer in polynomial time. The theory of 'P.cS-completeness \ngives strong evidence that such a separation is unlikely for nonpla(cid:173)\nnar Hopfield networks, and it is demonstrated that this is also the \ncase for several restricted classes of nonplanar Hopfield networks, \nincluding those who interconnection graphs are the class of bipar(cid:173)\ntite graphs, graphs of degree 3, the dual of the knight's graph, the \n8-neighbor mesh, the hypercube , the butterfly, the cube-connected \ncycles, and the shuffle-exchange graph. \n\n1 \n\nIntroduction \n\nAre Hopfield networks faster than conventional computers? This apparently \nstraightforward question is complicated by the fact that conventional computers \nare universal computational devices, that is, they are capable of simulating any \ndiscrete computational device including Hopfield networks. Thus, a conventional \ncomputer could in a sense cheat by imitating the fastest Hopfield network possible. \n\n* Email: ianGcs. unt .edu. URL: http://hercule .csci. unt. edu/ian. \nt Email: ht sengGponder. csci. unt . edu. \n\n\f240 \n\nI. Parberry and H. Tseng \n\nBut the question remains, is it faster for a computer to imitate a Hopfield network , \nor to use other computational methods? Although the answer is likely to be differ(cid:173)\nent for different benchmark problems, and even for different computer architectures , \nwe can make our results meaningful in the long term by measuring scalability, that \nis, how the running time of Hopfield networks and conventional computers increases \nwith the size of any benchmark problem to be solved. \n\nStated more technically, we are interested in the computational complexity of the \nstable state problem for Hopfield networks, which is defined succinctly as follows : \ngiven a Hopfield network, determine a stable configuration. As previously stated, \nthis stable configuration can be determined by imitation, or by other means. The \nfollowing results are known about the scalability of Hopfield network imitation. Any \nimitative algorithm for the stable state problem must take exponential time on som e \nHopfield networks, since there exist Hopfield networks that require exponential time \nto converge (Haken and Luby [4] , Goles and Martinez [2]) . It is unlikely that even \nnon-imitative algorithms can solve the stable state problem in polynomial time , \nsince the latter is PeS-complete (Papadimitriou , Schaffer, and Yannakakis [9]). \nHowever , the stable state problem is more difficult for some classes of Hopfield \nnetworks than others. Hopfield networks will converge in polynomial time if their \nweights are bounded in magnitude by a polynomial of the number of nodes (for \nan expository proof see Parberry [11 , Corollary 8.3.4]) . In contrast , the stable \nstate problem for Hopfield networks whose interconnection graph is bipartite is \npeS-complete (this can be proved easily by adapting techniques from Bruck and \nGoodman [1]) which is strong evidence that it too requires superpolynomial time \nto solve even with a nonimitative algorithm. \n\nWe show in this paper that although there exist planar Hopfield networks that t ake \nexponential time to converge in the worst case, the stable state problem for planar \nHopfield networks can be solved in polynomial time by a non-imitative algorithm. \nThis demonstrates that imitating planar Hopfield networks is exponentially slower \nthan using non-imitative algorithmic techniques. In contrast , we discover that the \nstable state problem remains peS-complete for many simple classes of nonplanar \nHopfield network, including bipartite networks, networks of degree 3, and some \nnetworks that are popular in neurocomputing and parallel computing. \n\nThe main part of this manuscript is divided into four sections. Section 2 contains \nsome background definitions and references. Section 3 contains our results about \nplanar Hopfield networks. Section 4 describes our peS-completeness results , based \non a pivotal lemma about a nonstandard type of graph embedding. \n\n2 Background \n\nThis section contains some background which are included for completeness but \nmay be skipped on a first reading. It is divided into two subsections, the first on \nHopfield networks, and the second on PeS-completeness. \n\n2.1 Hopfield Networks \n4- Hopfield network [6] is a discrete neural network model with symmetric connec(cid:173)\ntions. Each processor in the network computes a hard binary weighted threshold \n\n\fAre Hopfield Networks Faster than Conventional Computers? \n\n241 \n\nfunction. Only one processor is permitted to change state at any given time. That \nprocessor becomes active if its excitation level exceeds its threshold, and inactive \notherwise. A Hopfield network is said to be in a stable state if the states of all of \nits processors are consistent with their respective excitation levels. It is well-known \nthat all Hopfield networks converge to a stable state. The proof defines a measure \ncalled energy, and demonstrates that energy is positive but decreases with every \ncomputation step. Essentially then, a Hopfield network finds a local minimum in \nsome energy landscape. \n\n2.2 P .cS-completeness \n\nWhile the theory of NP-completeness measures the complexity of global optimiza(cid:173)\ntion, the theory of p.cS-completeness developed by Johnson, Papadimitriou, and \nYannakakis [7] measures the complexity of local optimization. It is similar to the \ntheory of NP-completeness in that it identifies a set of difficult problems known \ncollectively as p.cS-complete problems. These are difficult in the sense that if a \nfast algorithm can be developed for any P .cS-complete problem, then it can be \nused to give fast algorithms for a substantial number of other local optimization \nproblems including many important problems for which no fast algorithms are cur(cid:173)\nrently known. Recently, Papadimitriou, Schaffer, and Yannakakis [9] proved that \nthe problem of finding stable states in Hopfield networks is P .cS-complete. \n\n3 Planar Hopfield Networks \n\nA planar Hopfield network is one whose interconnection graph is planar, that is, can \nbe drawn on the Euclidean plane without crossing edges. Haken and Luby [4] de(cid:173)\nscribe a planar Hopfield network that provably takes exponential time to converge, \nand hence any imitative algorithm for the stable state problem must take exponen(cid:173)\ntial time on some Hopfield network. Yet there exists a nonimitative algorithm for \nthe stable state problem that runs in polynomial time on all Hopfield networks: \n\nTheorem 3.1 The stable state problem for Hopfield networks with planar intercon(cid:173)\nnection pattern can be solved in polynomial time. \n\nPROOF: (Sketch.) The prooffollows from the fact that the maximal cut in a planar \ngraph can be found in polynomial time (see, for example, Hadlock [3]), combined \nwith results of Papadimitriou, Schaffer, and Yannakakis [9]. 0 \n\n4 P .cS-completeness Results \n\nOur P .cS-completeness results are a straightforward consequence of a new result \nthat characterizes the difficulty of the stable state problem of an arbitrary class \nof Hopfield networks based on a graph-theoretic property of their interconnection \npatterns. Let G = (V, E) and H = (V', E') be graphs. An embedding of G into H \nis a function f: V -+ 2 Vi such that the following properties hold. (1) For all v E V, \nthe subgraph of H induced by f(v) is connected. (2) For all (u, v) E E, there exists \na path (which we will denote f(u , v)) in H from a member of f(u) to a member \nof f(v). (3) Each vertex w E H is used at most once, either as a member of f(v) \n\n\f242 \n\nI. Parberry and H. Tseng \n\nfor some v E V, or as an internal vertex in a path feu, v) for some u, v E V. The \ngraph G is called the guest graph, and H is called the host graph. Our definition \nof embedding is different from the standard notion of embedding (see, for example, \nHong, Mehlhorn, and Rosenberg [5]) in that we allow the image of a single guest \nvertex to be a set of host vertices, and we insist in properties (2) and (3) that the \nimages of guest edges be distinct paths. The latter property is crucial to our results, \nand forms the major difficulty in the proofs. \nLet 5, T be sets of graphs. 5 is said to be polynomial-time embeddable into T, \nwritten 5 ::;e T, if there exists polynomials Pl(n),P2(n) and a function f with the \nfollowing properties: (1) f can be computed in time PI(n), and (2) for every G E 5 \nwith n vertices, there exists H E T with at most p2(n) vertices such that G can \nbe embedded into H by f. A set 5 of graphs is said to be pliable if the set of all \ngraphs is polynomial-time embeddable into 5. \n\nLemma 4.1 If 5 is pliable, then the problem of finding a stable state in Hopfield \nnetworks with interconnection graphs in 5 is 'P \u00a3S-complete. \n\n(Sketch.) Let 5 be a set of graphs with the property that the set of all \nPROOF: \ngraphs is polynomial-time embeddable into 5 . By the results of Papadimitriou, \nSchaffer, and Yannakakis [9], it is enough to show that the max-cut problem for \ngraphs in 5 is 'P \u00a3S-complete. \nLet G be an arbitrary labeled graph. Suppose G is embedded into H E 5 under the \npolynomial-time embedding. For each edge e in G of cost c, select one edge from \nthe path connecting the vertices in f( e) and assign it cost c. We call this special \nedge f' ( e ). Assign all other edges in the path cost -00. For all v E V, assign the \nedges linking the vertices in f(v) a cost of -00. Assign all other edges of H a cost \nof zero. \nIt can be shown that every cut in G induces a cut of the same cost in H, as follows. \nSuppose G ~ E is a cut in G, that is, a set of edges that if removed from G, \ndisconnects it into two components containing vertices VI and V2 respectively. Then, \nremoving vertices f'(G) and all zero-cost edges from H will disconnect it into two \ncomponents containing vertices f(VI ) and f(V2 ) respectively. Furthermore, each \ncut of positive cost in H induces a cut of the same cost in G, since a positive cost \ncut in H cannot contain any edges of cost -00, and hence must consist only of f'(e) \nfor some edges e E E. Therefore, every max-cost cut in H induces in polynomial \ntime a max-cost cut in G. 0 \n\nWe can now present our 'P \u00a3S-completeness results. A graph has degree 3 if all \nvertices are connected to at most 3 other vertices each. \n\nTheorem 4.2 The problem of finding stable states in Hopfield networks of degree \n3 is 'P \u00a3S-complete. \n\nPROOF: \n\n(Sketch.) By Lemma 4.1, it suffices to prove that the set of degree-3 \ngraphs is pliable. Suppose G = (V, E) is an arbitrary graph. Replace each degree-k \nvertex x E V by a path consisting of k vertices, and attach each edge incident with \nv by a new edge incident with one of the vertices in the path. Figure 1 shows an \nexample of this embedding. 0 \n\n\fAre Hopfield Networks Faster than Conventional Computers? \n\n243 \n\nFigure 1: A guest graph of degree 5 (left), and the corresponding host of degree 3 \n(right). Shading indicates the high-degree nodes that were embedded into paths. \nAll other nodes were embedded into single nodes. \n\nFigure 2: An 8-neighbor mesh with 25 vertices (left), and the 8 X 8 knight's graph \nsuperimposed on an 8 x 8 board (right). \n\nThe 8-neighbor mesh is the degree-8 graph G = (V, E) defined as follows: V = \n{1,2, ... ,m} x {1,2, ... ,n}, and vertex (u,v) is connected to vertices (u,v\u00b1 1), \n(u \u00b1 1, v), (u \u00b1 1, v \u00b1 1). Figure 2 shows an 8-neighbor mesh with 25 vertices. \n\nTheorem 4.3 The problem of finding stable states in H opfield networks on the \n8-neighbor mesh is peS-complete. \n\n(Sketch.) By Lemma 4.1, it suffices to prove that the 8-neighbor mesh is \nPROOF: \npliable. An arbitrary graph can be embedded on an 8-neighbor mesh by mapping \neach node to a set of consecutive nodes in the bottom row of the grid, and mapping \nedges to disjoint rectilinear paths which use the diagonal edges of the grid for \ncrossovers. 0 \nThe knight's graph for an n X n chessboard is the graph G = (V, E) where V = \n{(i, j) 11 ~ i, j ~ n}, and E = {((i, j), (k, i\u00bb I {Ii - kl, Ij - il} = {I, 2}}. That is, \n\nthere is a vertex for every square of the board and an edge between two vertices \nexactly when there is a knight's move from one to the other. For example, Figure 2 \nshows the knight's graph for the 8 x 8 chessboard. Takefuji and Lee [15] (see also \nParberry [12]) use the dual of the knight's graph for a Hopfield-style network to \nsolve the knight's tour problem. That is, they have a vertex Ve for each edge e of \nthe knight's graph, and an edge between two vertices Vd and Ve when d and e share \na common vertex in the knight's graph. \n\n\f244 \n\nI. Parberry and H. Tseng \n\nTheorem 4.4 The problem of finding stable states in H opfield networks on the dual \nof the knight's graph is pes -complete. \n\nPROOF: (Sketch.) By Lemma4.1, it suffices to prove that the dual of the knight 's \ngraph is pliable. \nIt can be shown that the knight 's graph is pliable using the \ntechnique of Theorem 4.3. It can also he proved that if a set S of graphs is pliable, \nthen the set consisting of the duals of graphs in S is also pliable. 0 \n\nThe hypercube is the graph with 2d nodes for some d, labelled with the binary \nrepresentations of the d-bit natural numbers, in which two nodes are connected by \nan edge iff their labels differ in exactly one bit. The hypercube is an important \ngraph for parallel computation (see, for example, Leighton [8], and Parberry [lOD . \n\nTheorem 4.5 The problem of finding stable states in Hopfield networks on the \nhypercube is peS-complete. \n\nPROOF : (Sketch.) By Lemma 4.1, it suffices to prove that the hypercube is pliable. \nSince the \"~e\" relation is transitive, it further suffices by Theorem 4.2 to show that \nthe set of degree-3 graphs is polynomial-time embeddable into the hypercube. To \nembed a degree-3 graph G into the hypercube, first break it into a degree-1 graph \nG 1 and a degree-2 graph G2 . Since G2 consists of cycles, paths, and disconnected \nvertices, it can easily be embedded into a hypercube (since a hypercube is rich \nin cycles). G 1 can be viewed as a permutation of vertices in G and can hence be \nrealized using a hypercube implementation of Waksman 's permutation network [16] . \no \nWe conclude by stating PeS-completeness results for three more graphs that are \nimportant in the parallel computing literature the butterfly (see, for example, \nLeighton [8]) , the cube-connected cycles (Preparata and Vuillemin [13D , and the \nshuffle-exchange (Stone [14]). The proofs use Lemma 4.1 and Theorem 4.5 , and are \nomitted for conciseness. \n\nTheorem 4.6 The problem of finding stable states in Hopfield networks on the \nbutterfly, the cube-connected cycles, and the shuffle-exchange is peS-complete. \n\nConclusion \n\nAre Hopfield networks faster than conventional computers? The answer seems to be \nthat it depends on the interconnection graph of the Hopfield network. Conventional \nnonimitative algorithms can be exponentially faster than planar Hopfield networks. \nThe theory of peS-completeness shows us that such an exponential separation \nresult is unlikely not only for nonplanar graphs, but even for simple nonplanar \ngraphs such as bipartite graphs, graphs of degree 3, the dual of the knight's graph, \nthe 8-neighbor mesh, the hypercube, the butterfly, the cube-connected cycles, and \nthe shuffle-exchange graph. \n\nAcknowledgements \n\nThe research described in this paper was supported by the National Science Foun(cid:173)\ndation under grant number CCR- 9302917, and by the Air Force Office of Scientific \n\n\fAre Hopfield Networks Faster than Conventional Computers? \n\n245 \n\nResearch, Air Force Systems Command, USAF, under grant number F49620-93-1-\n0100. \n\nReferences \n\n[1] J. Bruck and J. W. Goodman. A generalized convergence theorem for neural \nnetworks. IEEE Transactions on Information Theory, 34(5):1089-1092, 1988. \n[2] E. Goles and S. Martinez. Exponential transient classes of symmetric neural \nnetworks for synchronous and sequential updating. Complex Systems, 3:589-\n597, 1989. \n\n[3] F. Hadlock. Finding a maximum cut of a planar graph in polynomial time. \n\nSIAM Journal on Computing, 4(3) :221-225, 1975. \n\n[4] A. Haken and M. Luby. Steepest descent can take exponential time for sym(cid:173)\n\nmetric conenction networks. Complex Systems, 2:191-196,1988. \n\n[5] J.-W. Hong, K. Mehlhorn, and A.L. Rosenberg. Cost tradeoffs in graph em(cid:173)\n\nbeddings. Journal of the ACM, 30:709-728,1983. \n\n[6] J. J . Hopfield. Neural networks and physical systems with emergent collective \ncomputational abilities. Proc. National Academy of Sciences, 79:2554-2558 , \nApril 1982. \n\n[7] D. S. Johnson , C. H. Papadimitriou, and M. Yannakakis. How easy is local \nIn 26th Annual Symposium on Foundations of Computer Science, \n\nsearch? \npages 39-42. IEEE Computer Society Press, 1985. \n\n[8] F. T. Leighton. Introduction to Parallel Algorithms and Architectures: Arrays \n\n. Trees\u00b7 Hypercubes. Morgan Kaufmann, 1992. \n\n[9] C. H. Papadimitrioll, A. A. Schaffer, and M. Yannakakis. On the complexity \nof local search. In Proceedings of the Twenty Second Annual ACM Symposium \non Theory of Computing, pages 439-445. ACM Press, 1990. \n\n[10] I. Parberry. Parallel Complexity Theory. Research Notes in Theoretical Com(cid:173)\n\nputer Science. Pitman Publishing, London , 1987. \n\n[11] I. Parberry. Circuit Complexity and Neural Networks. MIT Press, 1994. \n[12] I. Patberry. Scalability of a neural network for the knight's tour problem. \n\nNeurocomputing, 12 :19-34, 1996. \n\n[13] F. P. Preparata and J. Vuillemin. The cube-connected cycles: A versatile \nnetwork for parallel computation. Communications of the ACM, 24(5):300-\n309 , 1981. \n\n[14] H. S. Stone. Parallel processing with the perfect shuffle. IEEE Transactions \n\non Computers, C-20(2):153-161, 1971. \n\n[15] Y. Takefuji and K. C. Lee. Neural network computing for knight's tour prob(cid:173)\n\nlems. Neurocomputing, 4(5):249-254 , 1992. \n\n[16] A. Waksman. A permutation network. Journal of the ACM, 15(1):159-163, \n\nJanuary 1968. \n\n\f", "award": [], "sourceid": 1259, "authors": [{"given_name": "Ian", "family_name": "Parberry", "institution": null}, {"given_name": "Hung-Li", "family_name": "Tseng", "institution": null}]}