{"title": "Neural Network Routing for Random Multistage Interconnection Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 722, "page_last": 729, "abstract": null, "full_text": "Neural Network Routing for Random Multistage \n\nInterconnection Networks \n\nMark W. Goudreau \nPrinceton University \n\nand \n\nNEe Research Institute, Inc. \n\n4 Independence Way \nPrinceton, NJ 08540 \n\nc. Lee Giles \n\nNEC Research Institute, Inc. \n\n4 Independence Way \nPrinceton, NJ 08540 \n\nAbstract \n\nA routing scheme that uses a neural network has been developed that can \naid in establishing point-to-point communication routes through multi(cid:173)\nstage interconnection networks (MINs). The neural network is a network \nof the type that was examined by Hopfield (Hopfield, 1984 and 1985). \nIn this work, the problem of establishing routes through random MINs \n(RMINs) in a shared-memory, distributed computing system is addressed. \nThe performance of the neural network routing scheme is compared to two \nmore traditional approaches - exhaustive search routing and greedy rout(cid:173)\ning. The results suggest that a neural network router may be competitive \nfor certain RMIN s. \n\n1 \n\nINTRODUCTION \n\nA neural network has been developed that can aid in establishing point-to(cid:173)\npoint communication routes through multistage interconnection networks (MINs) \n(Goudreau and Giles, 1991). Such interconnection networks have been widely stud(cid:173)\nied (Huang, 1984; Siegel, 1990). The routing problem is of great interest due to \nits broad applicability. Although the neural network routing scheme can accommo(cid:173)\ndate many types of communication systems, this work concentrates on its use in a \nshared-memory, distributed computing system. \nNeural networks have sometimes been used to solve certain interconnection network \n\n722 \n\n\fNeural Network Routing for Random Multistage Interconnection Networks \n\n723 \n\nInput \nPorts \n\nOutput \nPorts \n\nInterconnection \n\nNetwork \n\nControl Bits \n-\n\n:1 \n\n-\n\nI \n: Logic1 \nLogic2: \nL.: _ _ _ _ _ _ _ .... _-_-:--r-_~_ _ _ _'--_ -_-_ -_-J:.J \nI \nI \n\nNeural \nNetwork \n\nInterconnection \n\nNetwork \nController \n\nExterna Control \n\nFigure 1: The communication system with a neural network router. The input \nports (processors) are on the left, while the output ports (memory modules) are on \nthe right. \n\nproblems, such as finding legal routes (Brown, 1989; Hakim and Meadows, 1990) \nand increasing the throughput of an interconnection network (Brown and Liu, 1990; \nMarrakchi and Troudet, 1989). The neural network router that is the subject of \nthis work, however, differs significantly from these other routers and is specially \ndesigned to handle parallel processing systems that have MINs with random inter(cid:173)\nstage connections. Such random MINs are called RMINs. RMINs tend to have \ngreater fault-tolerance than regular MINs. \nThe problem is to allow a set of processors to access a set of memory modules \nthrough the RMIN. A picture of the communication system with the neural network \nrouter is shown in Figure 1. The are m processors and n memory modules. The \nsystem is assumed to be synchronous. At the beginning of a message cycle, some \nset of processors may desire to access some set of memory modules. \nIt is the \njob of the router to establish as many of these desired connections as possible in \na non-conflicting manner. Obtaining the optimal solution is not critical. Stymied \nprocessors may attempt communication again during the subsequent message cycle. \nIt is the combination of speed and the quality of the solution that is important. \nThe object of this work was to discover if the neural network router could be compet(cid:173)\nitive with other types of routers in terms of quality of solution, speed, and resource \n\n\f724 \n\nGoudreau and Giles \n\nRMINI \n\n2 \n\n3 \n\n1 \n\nRMIN2 \n\n4 \n5 \n6 \n\nRMIN3 \n2 \n3 \n\n4 \n\n1 \n\n1 \n2 \n3 \n\n4 \n\n3 \n\n7 \n8 \n\n1 \n2 \n3 \n\n4 \n5 \n6 \n7 \n8 \n9 \n10 \n\n1 \n2 \n\n3 \n4 \n5 \n6 \n\n7 \n8 \n\nFigure 2: Three random multistage interconnection networks. The blocks that are \nshown are crossbar switches, for which each input may be connected to each output. \n\nutilization. To this end, the neural network routing scheme was compared to two \nother schemes for routing in RMINs - namely, exhaustive search routing and greedy \nrouting. So far, the results of this investigation suggest that the neural network \nrouter may indeed be a practicable alternative for routing in RMINs that are not \ntoo large. \n\n2 EXHAUSTIVE SEARCH ROUTING \n\nThe exhaustive search routing method is optimal in terms of the ability of the router \nto find the best solution. There are many ways to implement such a router. One \napproach is described here. \n\nFor a given interconnection network, every route from each input to each output \nwas stored in a database. (The RMIN s that were used as test cases in this paper \nalways had at least one route from each processor to each memory module.) When \na new message cycle began and a new message set was presented to the router, \nthe router would search through the database for a combination of routes for the \nmessage set that had no conflicts. A conflict was said to occur if more than one \nroute in the set of routes used a single bus in the interconnection network. In the \ncase where every combination of routes for the message set had a conflict, the router \nwould find a combination of routes that could establish the largest possible number \nof desired connections. \nIf there are k possible routes for each message, this algorithm needs a memory of \nsize 8( mnk) and, in the worst case, takes exponential time with respect to the size \n\n\fNeural Network Routing for Random Multistage Interconnection Networks \n\n725 \n\nof the message set. Consequently, it is an impractical approach for most RMINs, \nbut it provides a convenient upper bound for the performance of other routers. \n\n3 GREEDY ROUTING \n\nWhen greedy routing is applied, message connections are established one at a time. \nOnce a route is established in a given message cycle, it may not be removed. Greedy \nrouting does not always provide the optimal routing solution. \nThe greedy routing algorithm that was used required the same route database as \nthe exhaustive search router did. However, it selects a combination of routes in \nthe following manner. When a new message set is present, the router chooses one \ndesired message and looks at the first route on that message's list of routes. The \nrouter then establishes that route. Next, the router examines a second message \n(assuming a second desired message was requested) and sees if one of the routes \nin the second message's route list can be established without conflicting with the \nalready established first message. If such a route does exist, the router establishes \nthat route and moves on to the next desired message. \nIn the worst case, the speed of the greedy router is quadratic with respect to the \nsize of the message set. \n\n4 NEURAL NETWORK ROUTING \n\nThe focal point of the neural network router is a neural network of the type that \nwas examined by Hopfield (Hopfield, 1984 and 1985). The problem of establishing \na set of non-conflicting routes can be reduced to a constraint satisfaction problem. \nThe structure of the neural network router is completely determined by the RMIN. \nWhen a new set of routes is desired, only certain bias currents in the network change. \nThe neural network routing scheme also has certain fault-tolerant properties that \nwill not be described here. \nThe neural network calculates the routes by converging to a legal routing array. A \nlegal routing array is 3-dimensional. Therefore, each element of the routing array \nwill have three indices. If element ai,i,k is equal to 1 then message i is routed \nthrough output port k of stage j. We say ai,;,k and a',m,n are in the same row if \ni = I and k = n. They are in the same column if i = I and j = m. Finally, they are \nin the same rod if j = m and k = n. \nA legal routing array will satisfy the following three constraints: \n\n1. one and only one element in each column is equal to 1. \n2. the elements in successive columns that are equal to 1 represent output ports \n\nthat can be connected in the interconnection network. \n\n3. no more than one element in each rod is equal to 1. \n\nThe first restriction ensures that each message will be routed through one and \nonly one output port at each stage of the interconnection network. The second \nrestriction ensures that each message will be routed through a legal path in the \n\n\f726 \n\nGoudreau and Giles \n\ninterconnection network. The third restriction ensures that any resource contention \nin the interconnection network is resolved. In other words, only one message can \nuse a certain output port at a certain stage in the interconnection network. When \nall three of these constraints are met, the routing array will provide a legal route \nfor each message in the message set. \nLike the routing array, the neural network router will naturally have a 3-dimensional \nstructure. Each ai,j,k of a routing array is represented by the output voltage of a \nneuron, V'i,j,k' At the beginning of a message cycle, the neurons have a random \noutput voltage. If the neural network settles in one of the global minima, the \nproblem will have been solved. \nA continuous time mode network was chosen. It was simulated digitally. The neural \nnetwork has N neurons. The input to neuron i is Ui, its input bias current is Ii, and \nits output is Vi. The input Ui is converted to the output Vi by a sigmoid function, \ng(z). Neuron i influences neuron j by a connection represented by 7ji. Similarly, \nneuron j affects neuron i through connection Iij. In order for the Liapunov function \n(Equation 5) to be constructed, Iij must equal7ji. We further assume that Iii = O. \nFor the synchronous updating model, there is also a time constant, denoted by T. \nThe equations which describe the output of a neuron i are: \n\nU\u00b7 LN \n\nduo \n-' = --' + \ndt \n\nT \n\n. 1 \nJ= \n\nT. .. v,. + L\u00b7 \n, \n~ J \n\nT=RC \nV; = g(Uj) \n1 \n\ng(z) = 1 + e-X \n\n(1) \n\n(2) \n(3) \n\n(4) \n\nThe equations above force the neural net into stable states that are the local minima \nof this approximate energy equation \n\niNN \n\nE = - 2 L 2: Iij Vi V; - L V'i Ii \n\nN \n\ni=1j=1 \n\ni=l \n\n(5) \n\nFor the neural network, the weights (Iii's) are set, as are the bias currents (Ii'S). \nIt is the output voltages (V'i's) that vary to to minimize E. \nLet M be the number of messages in a message set, let S be the number of stages \nin the RMIN, and let P be the number of ports per stage (P may be a function \nof the stage number). Below are the energy functions that implement the three \nconstraints discussed above: \n\nA M 8-1 P \n\nE1 = 2' 2: L 2: Vm\",p(-Vm\",p + 2: Vm,3,i) \n\nE2 = 2' 2: 2: 2: Vm,I,p( - Vm,3,p + L V'i,3,p) \n\nm=1 1=1 p=l \nB 8-1 P M \n\n,=1 p=1 m=1 \n\nP \n\ni=1 \n\nM \n\ni=1 \n\n(6) \n\n(7) \n\n\fNeural Network Routing for Random Multistage Interconnection Networks \n\n727 \n\nC M S-l P \n\nEa = \"2 2: 2: 2:( -2Vm\",p + Vm\",p(-Vm\",p + 2: Vm\",i)) \n\nP \n\ni=l \n\nm=l ,=1 p=l \n\nM [S-l P P \n\nD f. ~]; tt d(s, p, i)Vm,,-l,p Vm\",i \n+ &,( d( 1, (JIm, j)Vm,IJ + d( S, j, Pm )Vm,S -IJ )] \n\n(8) \n\n(9) \n\nA, B, C, and D are arbitrary positive constants. l El and Ea handle the first \nconstraint in the routing array. E4 deals with the second constraint. E2 ensures the \nthird. From the equation for E4, the function d(sl,pl,p2) represents the \"distance\" \nbetween output port pI from stage sl - 1 and output port p2 from stage s1. If pI \ncan connect to p2 through stage sl, then this distance may be set to zero. If pI \nand p2 are not connected through stage sl, then the distance may be set to one. \nAlso, am is the source address of message m, while f3m is the destination address \nof message m. \n\nThe entire energy function is: \n\n(10) \n\nSolving for the connection and bias current values as shown in Equation 5 results \nin the following equations: \n\n(11) \n\n-B031 ,,20pl,p2(1 - Oml,m2) \n-D8m1,m2[031+1,,2d(s2,pl,p2) + 8,1,,2+1 d(sl,p2,pl)] \n(12) \n\n1m \",p = C - D[8\"ld(l, am,p) + o\"s-ld(S,p,f3m)] \n\n8i,j is a Kronecker delta (8j,j = 1 when i = j, and 0 otherwise). \nEssentially, this approach is promising because the neural network is acting as a \nparallel computer. The hope is that the neural network will generate solutions much \nfaster than conventional approaches for routing in RMINs. \nThe neural network that is used here has the standard problem - namely, a global \nminimum is not always reached. But this is not a serious difficulty. Typically, \nwhen the globally minimal energy is not reached by the neural network, some of \nthe desired routes will have been calculated while others will not have. Even a \nlocally minimal solution may partially solve the routing problem. Consequently, \nthis would seem to be a particularly encouraging type of application for this type \nof neural network. For this application, the traditional problem of not reaching \nthe global minimum may not hurt the system's performance very much, while the \nexpected speed of the neural network in calculating the solution will be a great \nasset. \n\nIFor the simulations, T = 1.0, A = 0 = D = 3.0, and B = 6.0. These values for A, B, \n\n0, and D were chosen empirically. \n\n\f728 \n\nGoudreau and Giles \n\nTable 1: Routing results for the RMINs shown in Figure 2. The * entries were not \ncalculated due to their computational complexity. \n\nM \n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n\nRMIN1 \n\nRMIN2 \n\nRMIN3 \n\nEgr Enn \nEel \n1.00 1.00 1.00 \n1.86 1.83 1.87 \n2.54 2.48 2.51 \n3.08 2.98 2.98 \n3.53 3.38 3.24 \n3.89 3.67 3.45 \n4.16 3.91 3.66 \n4.33 4.10 3.78 \n\n2.91 \n\nEgr Enn \nEel \n1.00 1.00 1.00 \n1.97 1.97 1.98 \n2.91 \n2.93 \n3.80 3.79 3.80 \n4.65 4.62 4.61 \n5.44 5.39 5.36 \n6.17 6.13 6.13 \n6.86 6.82 6.80 \n\n2.71 \n\nEgr Enn \nEel \n1.00 1.00 1.00 \n1.99 1.88 1.94 \n2.99 \n2.87 \n3.94 3.49 3.72 \n* 4.22 4.54 \n* 4.90 5.23 \n* 5.52 5.80 \n* 6.10 6.06 \n\nThe neural network router uses a large number of neurons. If there are m input \nports, and m output ports for each stage of the RMIN, an upper bound on the \nnumber of neurons needed is m2 S. Often, however, the number of neurons actually \nrequired is much smaller than this upper bound. \nIt has been shown empirically that neural networks of the type used here can con(cid:173)\nverge to a solution in essentially constant time. For example, this claim is made for \nthe neural network described in (Takefuji and Lee, 1991), which is a slight variation \nof the model used here. \n\n5 SIMULATION RESULTS \n\nFigure 2 shows three RMINs that were examined. The routing results for the three \nrouting schemes are shown in Table 1. Eel represents the expected number of \nmessages to be routed using exhaustive search routing. Egr is for greedy routing \nwhile Enn is for neural network routing. These values are functions of the size \nof the message set, M. Only message sets that did not have obvious conflicts \nwere examined. For example, no message set could have two processors trying to \ncommunicate to the same memory module. The table shows that, for at least these \nthree RMINs, the three routing schemes produce solutions that are of similar virtue. \nIn some cases, the neural network router appears to outperform the supposedly \noptimal exhaustive search router. That is because the Eel and Egr values were \ncalculated by testing every message set of size M, while Enn was calculated by \ntesting 1,000 randomly generated message sets of size M. For the neural network \nrouter to appear to perform best, it must have gotten message sets that were easier \nto route than average. \nIn general, the performance of the neural network router degenerates as the size of \nthe RMIN increases. It is felt that the neural network router in its present form will \nnot scale well for large RMINs. This is because other work has shown that large \nneural networks of the type used here have difficulty converging to a valid solution \n(Hopfield, 1985). \n\n\fNeural Network Routing for Random Multistage Interconnection Networks \n\n729 \n\n6 CONCLUSIONS \n\nThe results show that there is not much difference, in terms of quality of solution, for \nthe three routing methodologies working on these relatively small sample RMINs. \nThe exhaustive search approach is clearly not a practical approach since it is too \ntime consuming. But when considering the asymptotic analyses for these three \nmethodologies one should keep in mind the performance degradation of the greedy \nrouter and the neural network router as the size of the RMIN increases. \nGreedy routing and neural network routing would appear to be valid approaches \nfor RMINs of moderate size. But since asymptotic analysis has a very limited \nsignificance here, the best way to compare the speeds of these two routing schemes \nwould be to build actual implementations. \nSince the neural network router essentially calculates the routes in parallel, it can \nreasonably be hoped that a fast, analog implementation for the neural network \nrouter may find solutions faster than the exhaustive search router and even the \ngreedy router. Thus, the neural network router may be a viable alternative for \nRMIN s that are not too large. \n\nReferences \n\nBrown, T. X., (1989), \"Neural networks for switching,\" IEEE Commun. Mag., Vol. \n27, pp. 72-81, Nov. 1989. \nBrown, T. X. and Liu, K. H., (1990), \"Neural network design of a banyan network \ncontroller,\" IEEE J. on Selected Areas of Comm., pp. 1428-1438, Oct. 1990. \nGoudreau, M. W. and Giles, C. L., (1991), \"Neural network routing for multiple \nstage interconnection networks,\" Proc. IJCNN 91, Vol. II, p. A-885, July 1991. \nHakim, N. Z. and Meadows, H. E., (1990), \"A neural network approach to the setup \nof the Benes switch,\" in Infocom 90, pp. 397-402. \nHopfield, J. J., (1984), \"Neurons with graded response have collective computational \nproperties like those of two-state neurons,\" Proc. Natl. Acad. Sci. USA, Vol. 81, \npp. 3088-3092, May 1984. \nHopfield, J. J ., (1985), \"Neural computation on decisions in optimization problems,\" \nBioi. Cybern., Vol. 52, pp. 141-152, 1985. \nHuang, K. and Briggs, F. A., (1984), Computer Architecture and Parallel Processing, \nMcGraw-Hill, New York, 1984. \nMarrakchi, A. M. and Troudet, T., (1989), \"A neural net arbitrator for large cross(cid:173)\nbar packet-switches,\" IEEE Trans. on Cire. and Sys., Vol. 36, pp. 1039-1041, July \n1989. \nSiegel, H. J., (1990), Interconnection Networks for Large Scale Parallel Processing, \nMcGraw-Hill, New York, 1990. \nTakefuji, Y. and Lee, K. C., (1991), \"An artificial hysteresis binary neuron: a model \nsuppressing the oscillatory behaviors of neural dynamics\", Biological Cybernetics, \nVol. 64, pp. 353-356, 1991. \n\n\f", "award": [], "sourceid": 516, "authors": [{"given_name": "Mark", "family_name": "Goudreau", "institution": null}, {"given_name": "C.", "family_name": "Giles", "institution": null}]}