{"title": "Dataflow Architectures: Flexible Platforms for Neural Network Simulation", "book": "Advances in Neural Information Processing Systems", "page_first": 818, "page_last": 825, "abstract": null, "full_text": "818 \n\nSmotroff \n\nDataflow Architectures: \nFlexible Platforms for \n\nNeural Network Simulation \n\nIra G. Smotroff \n\nMITRE-Bedford Neural Network Group \n\nThe MITRE Corporation \n\nBedford, MA 01730 \n\nABSTRACT \n\nDataflow architectures are general computation engines optimized for \nthe execution of fme-grain parallel algorithms. Neural networks can be \nsimulated on  these systems with  certain advantages.  In this paper, we \nreview  dataflow  architectures,  examine  neural  network  simulation \nperformance  on  a  new  generation  dataflow  machine,  compare  that \nperformance to other simulation alternatives, and discuss the benefits \nand drawbacks of the dataflow approach. \n\n1  DATAFLOW ARCHITECTURES \nDataflow research has been conducted at MIT (Arvind &  Culler,  1986) and elsewhere \n(Hiraki,  et.  aI.,  1987)  for  a  number  of  years.  Dataflow  architectures  are  general \ncomputation engines that treat each instruction of a program as a separate task which is \nscheduled in an asynchronous, data-driven fashion.  Dataflow programs are compiled into \ngraphs which explicitly describe the data dependencies of the computation. These graphs \nare directly executed by the machine. Computations which are not linked by a path in the \ngraphs  can  be executed  in  parallel.  Each  machine  has  a  large  number  of processing \nelements  with  hardware  that  is  optimized  to  reduce  task  switching  overhead  to  a \nminimum.  As each  computation  executes  and  produces  a  result,  it  causes  all  of the \nfollowing computations that require the result to be scheduled. In this manner, fine grain \nparallel  computation  is achieved,  with  the limit on  the  amount of possible parallelism \ndetermined by the problem and the number of processing elements in the machine. \n\n\fDataflow Architectures:  Flexible Platforms for Neural Network Simulation \n\n819 \n\n-1  -1 \n\na \n\nFigure 1:  XOR network and its dataflow graph. \n\n1.1 NEURAL NETWORKS &  DATAFLOW \n\nThe most powerful hardware platforms for neural network simulation were enumerated \nin the DARPA Neural Network Study (Lincoln Laboratory, 1988): Supercomputers offer \nprogramming  in  sequential languages at great cost.  Systolic  Arrays such  as  the eMU \nWARP (pomerleau,  1988) and  \"Massively\" Parallel  machines such as the Connection \nMachine  (Hillis,  1987),  offer  power  at  increasingly  reasonable  costs,  but  require \nspecialized low-level programming to map the  algorithm  to  the  hardware.  Specialized \nVLSI and Optical devices (Alspector,  1989) (Farhat,  1987) (Rudnick &  Hammerstrom, \n1989) offer fast implementations of fixed algorithms1. \nAlthough  dataflow architectures were not included on  the  DARPA list, there are good \nreasons for using them  for neural network simulation. First, there is a natural mapping \nbetween neural networks and the dataflow graphs used to encode dataflow programs (see \nFigure  1). By expressing a neural network simulation as a dataflow program, one gains \nthe  data  synchronization  and  the  parallel  execution  efficiencies  that  the  dataflow \narchitecture provides at an appropriate  fine grain of abstraction.  The close mapping may \nallow  simple compilation  of neural  network  specifications  into  executable programs. \nSecond, this ease of programming makes the approach extremely flexible,  so one can get \ngood  performance on a  new  algorithm  the first  time it  is run,  without having  to  spend \nadditional time determining  the best way  to  map it onto the  hardware.  Thus  dataflow \nsimulations  may  be  particularly  appropriate  for  those  who  develop  new  learning \nalgorithms or architectures. Third, high level languages are being developed for dataflow \nmachines, providing environments in which  neural nets can be combined with standard \ncalculations; this can't be done with  much of the  specialized neural network hardware. \nLast, there may be ways to optimize dataflow architectures for neural network simulation. \n\n1 Hammerstrom's device (Rudnick & Hammerstrom,  1989) may be micro-programmable. \n\n\f820 \n\nSmotroff \n\nwait \n\n- match  ~ \n'W \n\nJ \n\nfrom \nnetwo(' 'k \n\n'\" , \nh \n\n\"\"\" , \n\ninstruction ~ \n\nfetch \n\n~ form \ntoken \n\n--\n'\"  to \nnetwork \n\n~ ALU  ~ \n\n~ form  ~ \n\ntag \n\noJ \n\"' \n\nIJ \n\"' \n\nstructure  .oJ \n...... \nmemory \n\n\\ \n\nFigure 2:  Schematic of a tagged-token dataflow processor. \n\n2  TAGGED-TOKEN  DATAFLOW \nThe Tagged-token  dataflow  approach  represents each  computation product as a  token \nwhich  is  passed  to  following  computations.  A  schematic  view  of  a  tagged-token \nprocessor is shown in Figure 2. Execution proceeds in a Wait-Match-Store  cycle which \nachieves data synchronization.  An  instruction  to  be executed waits  in  the  wait-match \nqueue for a token with  its operand. If a match  occurs, the incoming token contains its \noperand and one  of two  things  happens:  for  a  monadic  operation,  the  instruction  is \nexecuted and the result is passed on; for a dyadic operation, a check is made to see if the \noperand is the first or the second one to arrive. If it's  the first,  the location representing \nthe instruction is tagged,  the operand is stored, and the instruction continues to wait. If \nit's  the second  (Le.  the  instruction  is tagged already) the instruction  is  executed and a \ntoken containing the result is sent to  all computations requiring the result.  A schematic \nview  of the  execution  of the  XOR  network  of Figure  1 on  a  tagged-token  dataflow \nmachine is illustrated in Figure 3. \n\n2.1 SPLIT-PHASE TRANSACTIONS \n\nIn  fine-grain  parallel computations distributed over a  number of physical devices,  the \nlarge number of network transactions represent a potential bottleneck. The tagged-token \ndataflow architecture mitigates this problem in a way  that enhances the overall parallel \nexecution time. Each network transaction is split into two phases. A process requests an \nexternal data value and then goes to sleep. When the  token bearing the requested value \nreturns, the process is awakened and the computation proceeds. In standard approaches, a \nprocessor must idle while it waits for a result. This non-blocking approach allows other \ncomputations to proceed while the value is in transit, thus masking memory and network \nlatencies.  Independent threads of computation  may  be interwoven at each cycle, thus \nallowing the maximum amount of parallel execution at each cycle. As long as the amount \nof parallelism in the task (Le.  the length of each processor's task queue) is larger than the \nnetwork latency, the processors never idle. Consequently, massively parallel applications \nsuch as neural simulations benefit most from  the split-phase transaction approach. \n\n\fDataflow Architectures:  Flexible Platforms for Neural Network Simulation \n\n821 \n\n3  NEURAL NETWORK DATAFLOW SIMULATION \nTo  illustrate  neural  network  execution  on  a  dataflow  processor,  the XOR  network  in \nFigure 1 was coded in the dataflow language ID (Nikhil, 1988) and run on the MIT GIT A \n(Q.raph Interpreter for Tagged-token Architecture) simulator (Nikhil,  1988). Figures 4-6 \nare ALU operations profiles with the vertical axis representing the number of processors \nthat could be simultaneously kept busy  (i.e.  the amount of parallelism  in  the  task  at a \nparticular instance) and the horizontal axis representing elapsed computation cycles.  In \naddition, Figures 4 & 5 are ideal simulations with communication latency of zero time \nand  an  infinite  number  of processors  available  at  all  times.  The  ideal  profile  width \nrepresents the absolute minimum  time in which  the dataflow calculation could possibly \nbe performed, and is termed the critical path. Figure 4 shows the execution profile for a \nsingle linear threshold neuron processing its two inputs. The initial peak: activity of eleven \ncorresponds  to  initialization  activities,  with  later  peaks  corresonding  to  actual \ncomputation steps. The complexity of the profile may be attributed to  various dataflow \nsynchronization  mechanisms. In figure  5, the ideal execution profile for  the XOR net, \nnote the initialization peak: similar to the one appearing in  the single neuron profile; the \npeak  parallelism  of fifty-five  corresponds  to  all  five  neuron  initializations  occuring \nsimultaneously.  This illustrates  the  ability of the  dataflow  approach  to  automatically \nexpose the inherent parallelism in the overall computation. Note also that the critical path \nof one hundred fifty one is substantially less than five times the single neuron critical path \nof eighty-five. Wherever possible, the dataflow  approach has performed computation in \nparallel, and the lengthening of the critical path can be attributed to those computations \nwhich had to be delayed until prior computations became available. \nFigure 6 represents the execution of the same XOR net under more realistic conditions in \nwhich each token operation is subject to a finite network delay. The regular spacing of the \nprofile corresponds to the effect of the network delays. The interesting thing to observe is \nthat the overall critical path  length  has only  increased slightly  to one hundred seventy \nbecause the average amount of parallelism  available as tokens come in  from  the net is \nhigher.  Dataflow's ability  to  interleave computations thus compensates for  much of the \nnetwork latency effects. \n\nFigure 4: Ideal parallelism profile for dataflow execution - single threshold neuron unit. \n\nI SS \n\nZQQ \n\n\f822 \n\nSmotroff \n\n1 \n\n1 \n\nFigure  3:  Execution  of  the  XOR  network  of Figure  1 on  a  tagged-token \ndataflow  processor.  The  black  dots  represent  active  tokens,  the  white  dots \nrepresent  waiting  tokens,  and  the shaded boxes  represent  enabled operations \nexecuting. \n\n\fDataflow Architectures:  Flexible Platforms for Neural Network Simulation \n\n823 \n\n55  1 \n\ns91 \n\nI \n\n38 \n\nFigure 5: Ideal parallelism profile for dataflow execution of XOR network. \n\n299 \n\n\"1 \n_a  j \n\nI \n\n39 \n\n29 \n\nla \n\n9~~~~~~~~~~~~~--___ ___ __  _ \n\n9 \n\n19a \n\n289 \n\nFigure 6: Parallelism profile for dataflow execution of XOR with constant \n\ncommunication latency. \n\n3.1 COST OF THE DATAFLOW APPROACH \n\nThe Tagged-Token  Dataflow  machine executing an ID program performs two to  three \ntimes as many instructions as an IBM 370 executing an equivalent FORTRAN program. \nThe overhead  in  dataflow  programs  is  attributable  to  mechanisms  which  manage  the \nasynchronous parallel execution.  Similar overhead would probably exist in specialized \nneural network simulators written for dataflow machines. However, this overhead can be \njustified because the maximum amount of parallelism in the computation is exposed in a \nstraightforward  manner,  which  requires  no  additional  programming  effort.  On \nconventional multiprocessors, parallelism must be selectively tailored for each problem. \nAs the amount of parallelism increases, the associated costs increase as well; often they \nwill eventually surpass the cost of dataflow (Arvind ,Culler &  Ekanadham, 1988). Thus \nthe parallel performance on the dataflow  machine will often surpass that of alternative \nplatforms despite the overhead. \n\n\f824 \n\nSmotroff \n\n4  THE MONSOON ARCHITECTURE \nEarly  dataflow  implementations  using  a  Tagged  Token  approach  had  a  number  of \npractical barriers (papadoupoulos, 1988). While useful results were achieved, the cost and \nexpansion  limits  of  the  associative  memory  used  for  token  matching  made  them \nimpractical. However, the systems did prove the utility of the Tagged Token approach .. \nRecently, the MONSOON architecture (papadoupoulos, 1988) was developed to  remedy \nthe problems encountered with Tagged Token architectures. The token-matching problem \nhas been solved by treating each token descriptor as an address in a global memory space \nwhich  is  partitioned among  the  processors in  the  system;  matching  becomes a  simple \nRAM operation. \nAn  initial  MONSOON prototype  has  been  constructed and  a  8  processor  machine  is \nscheduled to be built in 1990. Processor elements for that machine are CMOS gate-array \nimplementations being fabricated by Motorola. Each processor board will have a  100 ns \ncycle  time and  process  at a  rate of 7-8 MIPS!2-4  MFLOPS.  Total  memory  for  the 8 \nprocessor machine is 256 MBytes.  Interconnect is  provided by  a  100  MByte!s packet \nswitch network. The throughput of the 8 processor machine is estimated at 56-64 MIPS! \n16-32 MFLOPs. This translates to  2-3  million connections per second per processor and \n16-24 million connections per second for  the  machine. Monsoon performance is in  the \nsupercomputer class while the projected Monsoon cost is significantly less due to the use \nof standard process technologies. \nA  256  processor  machine  with  CMOS  VLSI  processors  is  envisioned.  Estimated \nperformance is 40 MIPS  per processor and  10,240  MIPS  for  the machine.  Aggregate \nneural  simulation performance is estimated at  2.5-3.8  billion connections per second, \nassuming an  interconnect  network of suitable performance. \n\n5  CONCLUSIONS \n\ni)  Dataflow  architectures  should  be  cost  effective  and  flexible \nplatforms  for  neural  network  simulation  if they  become  widely \navailable. \n\nii)  As general architectures. their performance will not exceed that of \n\nspecialized neural network architectures. \n\niii)  Maximum  parallelism  is  attained  simply  by  using  the  dataflow \napproach:  no machine or problem-specific tuning is needed. Thus \ndataflow  is  seen  as  an  excellent  tool  for  empirical  simulation. \nExcellent performance may be obtained on cost effective hardware, \nwith no special effort required for performance improvement. \n\niv)  Dataflow  architectures optimized for  neural  network  simulation \n\nperformance may be possible. \n\nReferences \n\nAlspector, J .\u2022  Gupta, B.  and Allen, R.  B.  (1989)  Performance of a Stochastic Learning \nMicrochip.  In D. S. Touretzky (ed.). Advances in Neural Information Processing Systems \n1, 748-760. San Mateo, CA:  Morgan Kaufmann. \n\n\fDataflow Architectures:  Flexible Platforms for Neural Network Simulation \n\n825 \n\nArvind  and  Culler,  D.  E .. (1986)  Dataflow  Architectures,  MIT  Technical  Report \nMIT/LCS/fM-294, Cambridge, MA. \nArvind, Culler, D. E., Ekanadham, K. (1988)  The Price of Asynchronous Parallelism: An \nAnalysis of Dataflow Architectures. MIT Laboratory for Computer Science, Computation \nStructures Group Memo 278. \nDARPA Neural Network Study (1988)  Lincoln Laboratory, MIT, Lexington, MA. \nFarhat,  N.H.,  and  Shai,  Z.  Y.(1987)  Architectures  and  Methodologies  for  Self(cid:173)\nOrganization and Stochastic Learning  in  Opto-Electronic Analogs of Neural  Nets.  In \nProceedings of IEEE First International Conference on Neural Networks, ill:565-576. \nHillis, W. D.(1986) The Connection Machine, Cambridge, MA: The MIT Press. \nHiraki,  K.,  Sekiguchi, S.  and  Shimada, T.  (1987)  System  Architecture of a  Dataflow \nSupercomputer.  Technical  Report,  Computer  Systems  Division,  Electrotechnical \nLaboratory, 1-1-4 Umezono, Sakura-mura, Niihari-gun, lbaraki, 305, Japan. \nNikhil, R.  S.  (1988) Id World Reference Manual, Computational Structures Group, MIT \nLaboratory for Computer Science, Cambridge, MA. \nPomerleau,  D.  A., Gusciora,  G.  L.,  Touretsky  and  D.  S.,  Kung,  H.  T.(1988)  Neural \nSimulation  at  Warp  Speed:  How  we  got  17  Million  Connections  per  Second. \nIn \nProceedings of the IEEE International Conference on Neural Networks, II: 143-150, San \nDiego. \nPapadoupoulos,  G.  M.  (1988)  Implementation  of a  General  Purpose  Dataflow \nMultiprocessor, Phd. Thesis, MIT Department of Electrical Engineering and Computer \nScience, Cambridge, MA. \nRudnick, M.  and Hammerstrom, D.(1989)  An Interconnection Structure for Wafer Scale \nNeurocomputers.  In Proceedings of the 1988 Connectionist Models Summer School.  San \nMateo, CA: Morgan Kaufmann. \n\n\f\fPART X: \n\nHISTORY OF NEURAL NETWORKS \n\n\f", "award": [], "sourceid": 291, "authors": [{"given_name": "Ira", "family_name": "Smotroff", "institution": null}]}