{"title": "Computational Efficiency: A Common Organizing Principle for Parallel Computer Maps and Brain Maps?", "book": "Advances in Neural Information Processing Systems", "page_first": 60, "page_last": 67, "abstract": null, "full_text": "60 \n\nNelson and Bower \n\nComputational Efficiency: \n\nA  Common  Organizing  Principle  for \n\nParallel  Computer  Maps  and  Brain  Maps? \n\nMark E.  Nelson  James  M.  Bower \n\nComputation and Neural Systems Program \n\nDivision of Biology,  216-76 \n\nCalifornia Institute of Technology \n\nPasadena, CA  91125 \n\nABSTRACT \n\nIt is  well-known  that  neural  responses  in  particular  brain  regions \nare  spatially  organized,  but  no  general  principles  have  been  de(cid:173)\nveloped  that  relate  the structure of a  brain  map  to  the nature of \nthe associated computation.  On parallel computers, maps of a sort \nquite similar to brain maps arise when a computation is distributed \nacross  multiple  processors.  In  this  paper  we  will  discuss  the rela(cid:173)\ntionship  between  maps and computations on these  computers and \nsuggest how similar considerations might also apply to maps in the \nbrain. \n\nINTRODUCTION \n\n1 \nA  great  deal  of effort  in  experimental  and  theoretical  neuroscience  is  devoted  to \nrecording  and  interpreting  spatial  patterns  of neural  activity.  A  variety  of map \npatterns have  been observed  in  different  brain regions  and, presumably,  these  pat(cid:173)\nterns reflect  something about  the  nature of the neural  computations being carried \nout  in  these  regions.  To  date,  however,  there  have  been  no  general  principles  for \ninterpreting  the structure of a  brain  map  in  terms of properties  of the  associated \ncomputation.  In the field  of parallel computing, analogous  maps arise when  a  com(cid:173)\nputation is distributed across multiple processors and, in this case, the relationship \n\n\fComputational Eftkiency \n\n61 \n\nbetween  maps  and  computations  is  better  understood.  In  this  paper,  we  will  at(cid:173)\ntempt to relate some of the mapping principles from  the field  of parallel computing \nto the organization of brain  maps. \n\n2  MAPS  ON PARALLEL  COMPUTERS \nThe  basic  idea of parallel  computing  is  to  distribute  the  computational  workload \nfor  a  single  task  across  a  large  number  of  processors  (Dongarra,  1987;  Fox  and \nMessina,  1987).  In principle,  a  parallel computer  has  the potential to deliver  com(cid:173)\nputing power equivalent to the total computing power  of the processors from which \nit  is  constructed;  a  100  processor  machine  can  potentially  deliver  100  times  the \ncomputing power  of a  single  processor.  In practice,  however,  the performance that \ncan be achieved  is  always  less  efficient  than this  ideal.  A  perfectly efficient  imple(cid:173)\nmentation with  N  processors would give a factor  N  speed  up in computation time; \nthe ratio of the actual speedup  (1  to the ideal speedup  N  can serve as a  measure of \nthe efficiency  f  of a  parallel implementation. \n\nf= -\n\n(1 \nN \n\n(1) \n\nFor  a  given computation, one of the factors  that most influences the overall perfor(cid:173)\nmance is the way in which the computation is mapped onto the available processors. \nThe efficiency  of any particular mapping  can be analyzed  in  terms of two  principal \nfactors:  load-balance  and communication  overhead.  Load-balance  is  a  measure  of \nhow uniformly the computational work  load is  distributed among the available pro(cid:173)\ncessors.  Communication overhead, on the other hand, is  related to the cost in  time \nof communicating information between  processors. \n\nOn  parallel  computers,  the  load  imbalance  A  is  defined  in  terms  of the  average \ncalculation time per  processor T atJg  and the maximum calculation time required by \nthe busiest processor  T maz : \n\nA = Tmaz  - T atJg \n\nT atJg \n\n(2) \n\nThe communication overhead 7]  is defined in terms of the maximum calculation time \nT maz  and the maximum communication time Tcomm: \n\nTcomm \n\n7 ]= - - - - - -\nTmaz + Tcomm \n\n(3) \n\nAssuming that the calculation and communication phases of a  computation do not \noverlap in  time, as is the case for  many parallel computers, the relationship between \nefficiency  f,  load-imbalance  A,  and  communicaticn  overhead  7]  is  given  by  (Fox  et \nal.,1988): \n\n\f62 \n\nNelson and Bower \n\n1-7] \n{= -\nl+A \n\n(4) \n\nWhen  both  load-imbalance  A and  communication  overhead  7]  are  small,  the  inef(cid:173)\nficiency  is  approximately  the  sum  of the  contributions  from  load-imbalance  and \ncommunication overhead: \n\n(~l-(7]+A) \n\n(5) \n\nWhen  attempting  to  achieve  maximum  performance  from  a  parallel  computer,  a \nprogrammer tries to find  a  mapping that minimizes  the combined contributions of \nload-imbalance and communication overhead.  In some cases this is accomplished by \napplying simple heuristics (Fox et al.,  1988),  while in others it requires the explicit \nuse  of optimization  techniques  like  simulated  annealing  (Kirkpatrick  et  al.,  1983) \nor  even  artificial  neural  network  approaches  (Fox  and  Furmanski,  1988).  In  any \ncase,  the  optimal  tradeoff  between  load  imbalance  and  communication  overhead \ndepends  on  certain  properties  of the  computation  itself.  Thus  different  types  of \ncomputations give  rise to different  kinds of optimal maps on parallel computers. \n\n2.1  AN  EXAMPLE \n\nIn order to illustrate how different mappings can give rise to different computational \nefficiencies,  we will consider the simulation of a single neuron using a multicompart(cid:173)\nment modeling approach (Segev et al.,  1989).  In such a simulation,  the model neu(cid:173)\nron is divided into a large number of compartments, each of which is assumed to be \nisopotential.  Each compartment is represented by an equivalent electric circuit that \nembodies information about the local membrane properties.  In order to update the \nvoltage of an  individual compartment,  it is  necessary  to know  the local properties \nas well  as the membrane  voltages of the neighboring compartments.  Such  a  model \ngives  rise to a system of differential equations of the following  form: \n\n(6) \n\nwhere em  is the membrane capacitance, Vi  is the membrane voltage of compartment \ni,  9k  and  Ek  are the local conductances and their reversal potentials, and 9i\u00b1l,i  are \ncoupling conductances to neighboring compartments. \n\nWhen carrying out such a simulation on a  parallel computer, where there are more \ncompartments than processors, each processor is assigned responsibility for updating \na  subset of the compartments (Nelson et al.,  1989).  If the compartments represent \nequivalent  computational  loads,  then  the  load-imbalance  will  be  proportional  to \nthe difference  between the maximum and the average number of compartments per \nprocessor.  If the  computer  processors  are  fully  interconnected  by  communication \nchannels,  then  the  communication  overhead  will  be  proportional  to  the  number \nof interprocessor  messages  providing the voltages of neighboring compartments.  If \n\n\fA \n\nComputational Efficiency \n\n63 \n\nc \n\nA= 0.26 \n11 = 0.04 \n\nE = 0.76 \n\n\\'  A= 0.01 \n:,:!'  11 = 0.07 \n:i~  \u00a3  = 0.92 \n\n~ \n\nFigure  1:  Tradeoffs  between  load-imbalance  A and  communication  overhead  7], \ngiving  rise  to  different  efficiencies  \u00a3  for  different  mappings  of  a  multicompart(cid:173)\nment  neuron  model.  (A)  a  minimum-cut  mapping  that  minimizes  communication \noverhead  but  suffers  from  a  significant  load-imbalance,  (B)  a  scattered  mapping \nthat  minimizes  load-imbalance  but  has  a  large  communication  overhead,  and  (C) \na  near-optimal  mapping  that  simultaneously  minimizes  both  load-imbalance  and \ncommunication overhead. \n\nneighboring compartments are mapped to the same processor, then this information \nis available without any interprocessor communication and thus no communication \noverhead is incurred. \n\nFig.  1  shows  three  different  ways  of mapping  a  155  compartment  neuron  model \nonto a  group  of 4  processors.  In  each  case  the load-imbalance and communication \noverhead  are calculated using the assumptions listed above  and the computational \nefficiency is computed using eq. 4.  The map in Fig. 1A minimizes the communication \noverhead  of the' mapping  by  making  a  minimum  number  of cuts  in  the  dendritic \ntree,  but  is  rather  inefficient  because  a  significant  load-imbalance  remains  even \nafter  optimizing the location  of each  cut.  The map  is  Fig.  1B, on  the other  hand, \nminimizes the load-imbalance,  by  using a  scattered  mapping technique  (Fox et al., \n1988),  but  is  inefficient  because  of a  large  communication  overhead.  The  map  in \nFig.  1C strikes a balance between load-imbalance and communication overhead that \nresults in  a  high computational efficiency.  Thus this particular mapping makes the \nbest use of the available computing resources for  this particular computational task. \n\n\f64 \n\nNelson and Bower \n\nA \n\nB \n\nc \n\nFigure  2:  Three  classes  of map  topologies  found  in  the  brain  (of the  rat).  (A) \ncontinuous map of tactile inputs in somatosensory cortex (B) patchy map of tactile \ninputs to cerebellar cortex and (C) scattered mapping of olfactory inputs to olfactory \ncortex as  represented by the unstructured pattern of 2DG uptake in a single section \nof this cortex. \n\n3  MAPS  IN THE BRAIN \nSince some parallel computer maps are clearly more efficient than others for  partic(cid:173)\nular problems, it seems natural to ask whether a similar relationship might hold for \nbrain maps and neural computations.  Namely, for  a given  computational task, does \none  particular  brain map topology  make  more efficient  use  of the  available  neural \ncomputing resources  than  another?  If so,  does  this impose a  significant  constraint \non the evolution and development of brain map  topologies? \n\nIt turns  out  that  there  are  striking  similarities  between  the  kinds  of  maps  that \narise  on  parallel  computers  and  the  types  of  maps  that  have  been  observed  in \nthe  brain.  In  both  cases,  the  map  patterns  can  be  broadly  grouped  into  three \ncategories:  continuous maps,  patchy maps,  and scattered (non-topographic)  maps. \nFig.  2 shows examples of brain maps that fall  into these categories.  Fig.  2A shows \nan  example  of a  smooth  and  continuous  map  representing  the  pattern  of afferent \ntactile  projections  to  the  primary  somatosensory  cortex  of a  rat  (Welker,  1971). \nThe  patchy  map  in  Fig.  2B  represents  the spatial pattern of tactile  projections to \nthe granule cell layer of the rat cerebellar hemispheres (Shambes et aI.,  1978;  Bower \nand Woolston,  1983).  Finally,  Fig.  2C  represents  an extreme case in which  a  brain \nregion shows  no  apparent  topographic organization.  This figure  shows  the pattern \nof metabolic activity in one section of the olfactory (piriform)  cortex,  as assayed by \n2-deoxyglucose  (2DG)  uptake,  in  response  to the  presentation of a  particular odor \n(Sharp et al.,  1977).  As suggested by the uniform label in the cortex, no discernible \n\n\fComputational Eftkiency \n\n6S \n\nodor-specific  patterns are found  in this region of cortex. \n\nOn parallel computers, maps in these different  categories arise  as optimal solutions \nto different classes of computations.  Continuous maps are optimal for computations \nthat are local in the problem space,  patchy maps are optimal for  computations that \ninvolve  a  mixture  of local  and non-local interactions,  and scattered maps  are opti(cid:173)\nmal or near-optimal for  computations characterized by  a  high degree of interaction \nthroughout the problem space, especially if the patterns of interaction are  dynamic \nor  cannot  be  easily  predicted.  Interestingly,  it  turns  out  that  the  intrinsic  neu(cid:173)\nral  circuitry  associated  with  different  kinds  of brain maps also  reflects  these same \npatterns  of interaction.  Brain  regions  with  continuous  maps,  like  somatosensory \ncortex,  tend to have  predominantly local  circuitry;  regions  with  patchy  maps,  like \ncerebellar cortex, tend to have a mixture of local and non-local circuitry; and regions \nwith scattered maps, like olfactory cortex,  tend to be characterized by  wide-spread \nconnectivity. \n\nThe  apparent  correspondence  between  brain  maps  and  computer  maps  raises  the \ngeneral question of whether or not there are correlates of load-imbalance and com(cid:173)\nmunication overhead in the nervous system.  In general, these factors are much more \ndifficult  to identify  and quantify  in  the  brain than on  parallel  computers.  Parallel \ncomputer  systems  are,  after  all,  human-engineered  while  the  nervous  system  has \nevolved  under  a  set  of selection  criteria  and  constraints  that  we  know  very  little \nabout.  Furthermore, fundamental differences in the organization of digital comput(cid:173)\ners  and brains make  it  difficult  to translate ideas from  parallel  computing  directly \ninto  neural equivalents  (c.f.  Nelson  et al.,  1989).  For example,  it  is  far  from  clear \nwhat should be taken as  the neural  equivalent of a single  processor.  Depending on \nthe level of analysis, it might be a localized region of a dendrite, an entire neuron, or \nan assembly of many neurons.  Thus, one must consider multiple levels of processing \nin  the nervous system when  trying to draw  analogies with  parallel  computers. \n\nFirst we  will consider the issue of load-balancing in the brain.  The map in  Fig.  2A, \nwhile smooth and continuous, is obviously quite distorted.  In particular, the regions \nrepresenting  the  lips  and  whiskers  are  disproportionately  large  in  comparison  to \nthe  rest  of the  body.  It  turns  out  that  similar  map  distortions  arise  on  parallel \ncomputers  as  a  result  of load-balancing.  If different  regions  of the  problem space \nrequire more computation than other regions, load-balance is  achieved by distorting \nthe map  until each  processor ends up  with  an equal share of the workload  (Fox et \nal.,  1988).  In  brain  maps,  such  distortions  are  most  often explained  by  variations \nin  the  density  of  peripheral  receptors.  However,  it  has  recently  been  shown  in \nthe  monkey,  that prolonged  increased  use  of a  particular finger  is  accompanied  by \nan  expansion  of the  corresponding  region  of the  map  in  the  somatosensory  cortex \n(Merzenich,  1987).  Presumably this is  not a  consequence of a  change in peripheral \nreceptor  density,  but  instead  reflects  a  use-dependent  remapping  of some  tactile \ncomputation onto available cortical circuitry. \n\nAlthough  such  map  reorganization  phenomena are suggestive of load-balancing ef(cid:173)\nfects,  we  cannot  push  the  analogy  too far  because  we  do  not  know  what  actually \n\n\f66 \n\nNelson and Bower \n\ncorresponds to \"computational load\"  in the brain.  One possibility is  that it is  asso(cid:173)\nciated with  the metabolic load that arises  in response to neural activity  (Yarowsky \nand Ingvar,  1981).  Since metabolic activity necessitates the delivery of an adequate \nsupply of oxygen  and glucose  via a  network of small capillaries,  the efficient  use  of \nthe capillary system might favor  mappings that tend to avoid metabolic  \"hot spots\" \nwhich  would overload the delivery  capabilities of the system. \n\nWhen discussing communication overhead  in  the brain,  we  also  run into the  prob(cid:173)\nlem of not knowing exactly what corresponds to \"communication cost\".  On parallel \ncomputers, communication overhead  is  usually associated with the time-cost of ex(cid:173)\nchanging information between processors.  In the nervous system, the importance of \nsuch time-costs is  probably quite dependent on the behavioral context of the com(cid:173)\nputation.  There is evidence, for example, that some brain regions actually make use \nof transmission  delays  to process information  (Carr  and  Konishi,  1988).  However, \nthere  is  another  aspect  of communication  overhead  that  may  be  more  generally \napplicable having to do with the space-costs of physically connecting processors to(cid:173)\ngether.  In the design of modern parallel computers  and in  the design  of individual \ncomputer processor  chips,  space-costs associated with interconnections pose  a  very \nserious  constraint  for  the  design  engineer.  In  the  nervous  system,  the  extremely \nlarge  numbers  of potential connections combined  with  rather  strict limitations  on \ncranial  capacity are likely  to make space-costs a  very  important factor. \n\n4  CONCLUSIONS \nThe view  that computational efficiency is an important constraint on  the organiza(cid:173)\ntion  of brain  maps  provides  a  potentially  useful  new  perspective  for  interpretting \nthe  structure  of those  maps.  Although  the  available  evidence  is  largely  circum(cid:173)\nstantial, it seems likely  that the topology of a  brain map  affects  the efficiency  with \nwhich  neural  resources  are  utilized.  Furthermore,  it  seems  reasonable  to  assume \nthat  network  efficiency  would  impose  a  constraint  on  the  evolution  and  develop(cid:173)\nment  of map  topologies  that  would  tend  to  favor  maps  that  are  near-optimal  for \nthe  computational tasks  being  performed.  The very  substantial  task  before  us,  in \nthe  case  of the  nervous  system,  is  to  carry  out  further  experiments  to  better  un(cid:173)\nderstand  the  detailed  relationships  between  brain  maps,  neural  architectures  and \nassociated  computations (Bower,  1990). \n\nAcknowledgements \n\nWe  would  like  to acknowledge  Wojtek  Furmanski and  Geoffrey  Fox of the Caltech \nConcurrent  Computation  Program  (CCCP)  for  their  parallel  computing  support. \nWe  would  also like to thank Geoffrey  for  his comments on an earlier version of this \nmanuscript.  This effort  was  supported  by  the  NSF  (ECS-8700064),  the  Lockheed \nCorporation,  and the Department of Energy  (DE-FG03-85ER25009). \n\nReferences \n\nBower, J .M. (1990) Reverse engineering the nervous system:  An anatomical, physio(cid:173)\nlogical,  and computer based approach.  In:  An Introduction  to  Neural and Electronic \n\n\fComputational Efficiency \n\n67 \n\nNetworks.  (S.  Zornetzer, J. Davis, and C.  Lau,  eds), pp.  3-24,  Academic  Press. \nBower, J .M. and D.C. Woolston  (1983)  Congruence of Spatial Organization of Tac(cid:173)\ntile Projections to Granule Cell and Purkinje Cell Layers of Cerebellar Hemispheres \nof the Albino Rat:  Vertical Organization of Cerebellar Cortex.  J.  Neurophysiol.  49, \n745-756. \nCarr, C.E.  and  M.  Konishi  (1988)  Axonal delay  lines for  time measurement in  the \nowl's brain stem.  Proc  Natl Acad Sci  USA  85,  8311-8315. \nDongarra,  J.J.  (1987)  Experimental Parallel  Computing  Architectures,  (Dongarra, \nJ.J., ed.)  North-Holland. \n\nFox,  G.  C.,  M.  Johnson, G.  Lyzenga,  S.  Otto, J. Salmon,  D.  Walker  (1988)  Solving \nProblems  on  Concurrent  Processors,  Prentice Hall. \nFox,  G.C.  and W.  Furmanski  (1988)  Load  Balancing loosely  synchronous problems \nwith  a  neural  network. \nIn:  Proceedings  of the  Third  Conference  on  Hypercube \nConcurrent  Computers  and Applications,  (Fox,  G.C.,  ed.),  pp.241-278,  ACM. \nFox,  G.C.  and  P.  Messina  (1987)  Advanced  Computer  Architectures.  Scientific \nAmerican, October, 66-74. \n\nKirkpatrick,  S.,  C.D.  Gelatt  and  M.P.  Vecchi  (1983)  Optimization  by  Simulated \nAnnealing.  Science,  220, 671-680. \n\nMerzenich,  M.M.  (1987)  Dynamic  neocortical  processes  and  the  origins  of higher \nbrain functions.  In:  The  Neural and  Molecular Bases of Learning,  (Changeux, J .-P. \nand  Konishi,  M.,  eds.),  pp.  337-358,  John Wiley & Sons. \nNelson,  M.E.,  W.  Furmanski  and  J .M.  Bower  (1989)  Modeling  Neurons  and  Net(cid:173)\nworks  on  Parallel  Computers.  In:  Methods  in  Neuronal  Modeling:  From  Synapses \nto  Networks,  (Koch,  C.  and I.  Segev, eds.),  pp.  397-438,  MIT Press. \nSegev, I., J.W. Fleshman and R.E. Burke (1989) Compartmental Models of Complex \nNeurons.  In:  Methods  in  Neuronal Modeling:  From  Synapses  to  Networks,  (Koch, \nC.  and I.  Segev, eds.),  pp.  63-96,  MIT Press. \n\nShambes, G.M., J .M. Gibson and W. Welker (1978) Fractured Somatotopy in Gran(cid:173)\nule  Cell  Tactile  Areas  of Rat  Cerebellar  Hemispheres  Revealed  by  Micromapping. \nBrain  Behav.  Evol.  15, 94-140. \n\nSharp,  F.R.,  J.S . Kauer  and  G.M.  Shepherd  (1977)  Laminar  Analysis of 2-Deoxy(cid:173)\nglucose  Uptake  in  Olfactory  Bulb  and  Olfactory  Cortex  of  Rabbit  and  Rat.  J. \nNeurophysiol.  40, 800-813. \nWelker, C.  (1971)  Microelectrode delineation of fine  grain somatotopic organization \nof SMI  cerebral neocortex in  albino rat.  Brain  Res.  26,  259-275. \n\nYarowsky,  P.J.  and  D.H.  Ingvar  (1981)  Neuronal  activity  and  energy  metabolism. \nFederation  Proc.  40,  2353-2263. \n\n\f", "award": [], "sourceid": 235, "authors": [{"given_name": "Mark", "family_name": "Nelson", "institution": null}, {"given_name": "James", "family_name": "Bower", "institution": null}]}