{"title": "Associative Learning via Inhibitory Search", "book": "Advances in Neural Information Processing Systems", "page_first": 20, "page_last": 28, "abstract": null, "full_text": "20 \n\nASSOCIATIVE LEARNING \nVIA INHIBITORY SEARCH \n\nDavid H.  Ackley \n\nBell  Communications Research \n\nCognitive Science  Research Group \n\nABSTRACT \n\nALVIS is  a  reinforcement-based  connectionist  architecture  that \nlearns  associative  maps  in  continuous  multidimensional  environ(cid:173)\nments.  The  discovered  locations  of  positive  and  negative  rein(cid:173)\nforcements  are  recorded  in  \"do be\"  and  \"don't  be\"  subnetworks, \nrespectively.  The outputs of the subnetworks relevant  to the cur(cid:173)\nrent goal are combined and compared with the current location to \nproduce  an  error  vector.  This  vector  is  backpropagated  through \na  motor-perceptual  mapping  network.  to  produce  an  action  vec(cid:173)\ntor that leads the system towards do-be locations  and  away from \ndon 't-be locations.  AL VIS is  demonstrated with a simulated robot \nposed a  target-seeking task. \n\nINTRODUCTION \n\nThe \"backpropagation algorithm\" or generalized delta rule (Rumelhart, Hinton, & \nWilliams,  1986)  is  sometimes  criticized  on  the  grounds  that  it  is  a  \"supervised\" \nlearning  algorithm,  which  requires  a  \"teacher\"  to  provide  correct  outputs,  and \napparently leaves open  the question  of how  the  teacher learned  the right answers. \nHowever,  work.  by Rumelhart  (personal  communication,  1987)  and  Miyata (1988) \nhas  shown  how  the  environment  that  a  system  is  embedded  in  can  serve  as  the \n\"teacher.\"  If,  as  in  this  paper,  a  backpropagation  network  is  posed  the  task  of \nmapping  from  a  vector  9  of robot  arm joint  angles  to the  resulting  vector  X  of \narm  coordinates  in  space  (the  \"forward  kinematics  problem\"),  then  input-output \ntraining  data  can  be  obtained  by  supplying  sets  of joint angles  to  the  arm  and \nobserving the resulting configurations. \n\nAlthough this \"environment as teacher\" strategy shows how a \"teacher\" can come to \npossess useful information without an infinite regress learning it, it is not a complete \nsolution.  There  are  problems  for  which  the  \"laws  of physics\"  of an  environment \ndo  not  suffice  to  determine  the  solution.  Suppose,  for  example,  that  a  robot  is \nposed  the  problem  of learning to  reach for  different  positions  in  space  depending \non  which  of a  set  of signals  is  currently  presented,  and  that  the  only  feedback \navailable from  the environment is success or  failure information about the  current \narm configuration. \n\n\fAssociative Learning via Inhibitory Search \n\n21 \n\nd \n\nFigure 1.  A  \"trunk\"  robot. \n\nWhat is  needed in such a case is a  mechanism to search through the space of possi(cid:173)\nble arm configurations,  recording the successful configurations associated  with  the \nvarious inputs.  ALVIS - Associative  Learning Via Inhibitory  Search -\nprovides \none such mechanism.  The next section applies backpropagation to the 9  --+  X  map(cid:173)\nping and shows how  the resulting network can sometimes  be used  to solve X  --+  9 \nproblems.  The  third  section,  \"Self-supervision  and  inhibitory  search,\"  integrates \nthat  network  into  the  overall  ALVIS algorithm.  The  final  section  contains  some \ndiscussion  and  conclusions.  An  expanded  version  of this  paper  may  be  found  in \nAckley (1988). \n\nFORWARD AND INVERSE KINEMATICS \n\nThe in verse kinematics problem in controlling an arm is the problem of determining \nwhat joint angles are needed to produce a specific position and orientation of a hand. \nIn the general case it is a difficult problem.  An itch on your back suggests the kinds \nof questions that arise.  Which hand should you use?  Should you go up from around \nyour  waist,  or  down from  over your shoulders?  Can  you  be  sure you  know  what \nwill  work without actually trying it? \n\nFrom a computational standpoint, forward kinematics -\nwill  end up given a  set  of joint angles -\n\nis  an easier problem. \n\ndeciding where your limbs \n\nFigure 1 depicts the planar \"robot\" that was used in this work.  I call it the \"trunk\" \nrobot. \n(The  work  discussed  in  Ackley  (1988)  also  used  a  two-handed  \"pincer\" \nrobot.)  Of course, the trunk is a  far  cry from a  real robot,  and the only significant \nconstraint  is  that  the  possible  joint  angles  are  limited,  but  this  suffices  to  pose \nnon-trivial kinematics  problems.  The trunk has five joints, and  each joint angle  is \nlimited to a  range of 40  to 176 0  with respect to the previous limb. \n\nI  simulated a  backpropagation network with five  real-valued input units  (the joint \nangles),  sixty  hidden  units  in  a  single  layer,  and  twelve  linear  output  units  (the \nCartesian joint positions).  Joint angles  were  expressed in  radians,  so  the  range  of \ninput  unit values  was  from  about  0.07  to about  3.07.  The configuration  vector  X \nwas represented by twelve output units corresponding  to six pairs of (x, y)  coordi-\n\n\f22 \n\nAckley \n\nFigure 2.  A  \"logarithmic strobe\" display of the trunk's asymptotic convergence on \na  specified  position  and orientation.  The  arm position is  displayed after iterations \n1,2,4,8, ... ,256. \n\nnates, one  pair for  each joint a  through f.  With  the trunk robot  (though not with \nthe  \"pincer\")  f:e  and  fy  have constant values, since they  end  up being part of the \n\"anchor.\"  The state of an  output  unit  equals  the sum of its inputs,  and  the  error \npropagated out of an output unit equals the error propagated into it. \n\nErrors  were defined  by the difference  between  the predicted  configuration  and  the \nactual configuration,  and after extensive training on the trunk robot forward  kine(cid:173)\nmatics problem,  the network achieved high  accuracy over most of the joint ranges. \n\nIn typical backpropagation applications, once the desired mapping has been learned, \nthe backward \"error channels\" in the network are no longer used.  However, suppose \nsome  other  error computation,  different  from  that  used  to  train  the  weights,  was \nthen incorporated.  Those errors can be propagated from  the outputs of the trained \nnetwork  all  the  way  back  to  the  inputs.  The  goal is  no  longer  to  change  weights \nin the network -\nbut  to use the \ntrained  network  to  translate  output-space  errors,  however  defined,  into  errors  at \nthe inputs to the network. \n\nsince they already represent a  useful  mapping -\n\nFigure  2  illustrates  one  use  of this  process,  showing  how  a  trained  forward  kine(cid:173)\nmatics  network  can  be  used  to  perform  a  cheap  kind  of inverse  kinematics.  The \nfigure  shows  superimposed  outputs  of the  trained  network  under  the  influence  of \na  task-specific  error  computationj  in  this  case  the  trunk  is  trying  to  reduce  the \ndistances  between  the  front  and  back of a  \"target arrow\"  and  the front  and  back \nof its first  arm section.  The  target  arrow is  defined  by a  head  (h:e,  hy)  and  a  tail \n(t:e, ty).  The errors  for  output units  a:e  and ay  are defined  by e(a:e}  =  h:e  - a:e  and \ne(ay ) = hy-ay, the errors for  output units  b:e  and by  are defined by e(b:e)  = t:e-b:e \nand e(by ) =  ty - by,  and  the errors at all other outputs are set  to zero. \n\nThe algorithm used  to generate this behavior has the following steps: \n\n1.  Compute errors for one or more output units based on the current positions \nof the joints and the desired positioning and orienting information.  If \"close \n\n\fAssociative Learning via Inhibitory Search \n\n23 \n\no \n\nFigure 3.  The  trunk kinking itself. \n\nenough\"  to  the  target,  exit,  otherwise,  store  these  errors  on  the  selected \noutput units,  and set  the other error  terms to zero. \n\n2.  Backpropagate  the  errors  all  the  way  through  the  network  to  the  input \nunits.  This  produces  an error  term e(9i)  for  each joint angle  9i.  Produce \nnew joint angles:  9~ =  9i + ke(9i)  where  k  is  a  scaling  constant.  Clip the \njoint angles against  their minimum and maximum values. \n\n3.  Forward propagate through  the network based on the  new joint angles,  to \n\nproduce new current positions for  the joints.  Go to step  1. \n\nWhereas the training phase has forward propagation of activations (states) followed \nby  back propagation  of errors,  this  usage  reverses  the  order.  Backpropagation of \nerrors is  followed  by  changes  in inputs followed  by  forward  propagation  of activa(cid:173)\ntions.  This is  a  general gradient descent technique usable when a  backpropagation \nnetwork can learn to map from a  control space 9 to an error or evaluation space X. \n\nFigure  3  illustrates  how  gradient  descent's familiar  limitation can  manifest  itself: \nThe  target  is  reachable  but  the  robot  fails  to reach  it.  The  initial  configuration \nwas  such  that  while  approaching  the  target,  the  trunk  kinked  itself too  short  to \nreach.  If the  robot  had  \"thought\"  to  open  93  instead  of closing  it,  it  could  have \nsucceeded.  In  that  sense,  the  problem arises  because  the  error  computation only \nspecified errors for  the tip of the trunk, and not for  the rest of the arm.  If,  instead, \nthere were indications where all of the joints were to be placed, failures due to local \nminima could be greatly reduced. \n\nSELF-SUPERVISION AND INHIBITORY SEARCH \n\nhowever  they are generated -\n\nThe feedback control network of the previous section locally minimizes joint position \nby translating them into joint angle space \nerrors -\nand  moving downhill.  AL VIS uses  the  feedback  control network  for  arm  control; \nthis section  shows how  ALVIS learns  to generate  appropriate joint position  space \nerrors  given  only  a  reinforcement  signal.  There  are  two  key  points.  The first  is \nthis:  Once  an action  producing a  positive reinforcement has somehow been found, \n\n\f24 \n\nAckley \n\nthe problem reduces  to associative mapping between the input and the discovered \ncorrect  output.  In  ALVIS,  \"do-be  units\"  are  used  to record  such  successes.  The \nsecond point is this:  When negative reinforcement occurs,  the current configuration \ncan  be  associated  with  the  input  in  a  behavior-reversed  fashion  -\nas  a  place  to \navoid in the future.  In  ALVIS,  \"don't-be units\"  are used  to record such failures. \n\nThe overall idea,  then, is  to perform inhibitory search  by remembering failures  as \nthey  occur  and  avoiding  them  in  the  future,  and  to  perform  associative learning \nby remembering successful configurations as the search process uncovers them and \nrecreating them in the future.  In effect,  ALVIS constructs input-dependent \"attrac(cid:173)\ntors\" at arm configurations associated with success and \"repellors\" at configurations \nassociated  with failure.  Figure 4  summarizes  the algorithm.  A  few  points  to note \nare these: \n\n\u2022  The do-be and don't-be units use  the  spherical non-linear  function  explored  by \nBurr &  Hanson (1987).  The response of a spherical unit is maximal and equal to \none when  the input vector and the weight vector are identical.  The  response of \nthe  unit  decreases  monotonically  with  the  Euclidean  distance  between  the  two \nvectors,  and  the radius r  governs  the rate of decay. \n\n\u2022  The don't-be units of each subnetwork (i.e.,  relevant  to one  goal)  are in a  com(cid:173)\n\npetitive network  (see,  e.g.,  Feldman  1982).  The don't-be  unit  with  the  largest \nactivation value (which is  a function of both the distance from  the current posi(cid:173)\ntion  and the radius) is  the only don't-be unit  that has effects  on  the rest  of the \nsystem.  In  the simulations reported here,  I  used  m = 4 don't-be units per goal. \n\u2022  In  addition  to  the  parameters  associated  with  spherical  units,  each  do-be  and \ndon't-be  unit  has a  strength  parameter  a  that specifies  how  much influence the \nunit  has  over  the  behavior of the  arm.  Do-be  strength  (at)  grows  logarithmi(cid:173)\ncally  with  positive  reinforcement  and  shrinks  linearly  with  negative;  don't-be \nstrength (a;i)  grows logarithmically with negative reinforcement and shrinks lin(cid:173)\nearly with  positive. \n\nFigure 5 illustrates a situation from early in a  run of the system.  From left to right, \nthe  three displays show  the state of the relevant  do-be  unit,  the  relevant  don't-be \nsubnetwork,  and  the  current  configuration  of the  arm.  Since  this  particular  goal \nhas  never  been  achieved  before,  the  do-be map  provides  no useful  information  -\nits  weight  vector contains small  random values  (as it  happens,  the origin  is  below \nand right  of the display)  and its strength is zero.  The display of the don't-be map \nshows  the positions  of all four  relevant don't-be units,  with  the currently selected \ndon't-be  (unit  number  3)  drawn somewhat darker.  The don't-be  units are  spread \naround  configuration  space,  creating  \"hills\"  that  push  away  the  arm  if it  comes \ntoo close.  As  the arm moves  about without reaching the target,  different don't-be \nunits  win  the  competition  and  take control.  Negative  reinforcement  accrues,  and \nthe winning don't-be consequently moves toward the various current configurations \nand gets stronger,  until the arm is  pushed elsewhere. \n\nFigure 6  illustrates  the  behavior of the system after  more  extensive learning.  The \n\n\fAssociative Learning via Inhibitory Search \n\n25 \n\nFigure 4.  AL VIS \n\nO.  (Initialize)  Given:  a  space  X  of h  dimensions,  a  backpropagation network \ntrained  on  8  -+  X,  a  set  G  of goals  and  a  mapping from  G  to  regions  of \nspace.  Create an  AL VIS network  with  n  =  IGI  goal  units 91, ... ,9n,  n  do(cid:173)\nbe/don't-be  subnetworks  consisting  of one  do-be  unit dt  and  m  don't-be \nunits  dii, ... , dfu.,  and  h  current  position  units  ZI,'\"  Zh.  Create  modi(cid:173)\nfiable  connections  Wzi  from  z's  to  d's,  Wiz  from  d's  to  z's,  and  a  modi(cid:173)\nfiable  strength  s for  each d.  Set  all  do-be  strengths  st and  all  don't-be \nstrengths  sti  to zero.  Set  all  weights  Wzi  and Wiz  to small random values. \nSet 8 to a  random legal  vector and produce a  current configuration X. \n\n1.  (New stimulus) Choose t  at random from  1 ... n. \n2.  (Do's/Don'ts)  Compute  activations  for  do-be's  and  don't-be's  using  the \n\nspherical  function:  d =  1/ (1 + ~JL./:=1 (Zi - w zil2 ).  In  subnetwork t, \nlet dt:  be  the unit with  the largest  activation. \n3.  (Errors)  Let  wt denote  the  weights  from dt  and  w~ denote  the  weights \nfrom  dt:,  and similarly for  strengths  st and st:.  Compute  errors for  each \ncomponent of X:  e(zi) = si(wt - Zi) + Si:.(Zi  - w~). \n\n4.  (Move)  Backpropagate  to  produce  e(8).  Generate  angle  changes:  A8i  = \nmin(q, max( -q, k,.e(8i ))),  with  parameters  q  and  k,..  Generate  new  an(cid:173)\ngles  respecting  the  maximum vt  and  minimum  vi  possible  joint  angles: \n8~  =  min( vt, max(vi ,8i  + A8i )).  Forward  propagate  to  produce  a  new \nconfiguration X'. \n\n5.  (Positive  reinforcement)  Determine  whether  X'  satisfies  goal  t.  If it does \n\nnot, go  to step 6.  Otherwise, \n\nror 1  -\n\n\u2022  -\n\n1 \n,  \u2022\u2022\u2022 , \n\nI \n\nd  +,  -\n\nhIt  +,  -\nI \n,  e  wiz  - zi an  wzi  - zi' \n\n5  1  t;\\ \n. \n5.2  Let  st' = min (5, si + p+ /  (1 + si)), for  positive reinf p+ > O. \n5.3  Let ri = k,/ max(.I, st'),  with  parameter k,. \n5.4  For i  =  1, ... , m, let Sti' = max(O, sti-p+), and rti  = k,f max(.I, sti'l. \n5.5  Go to step  1. \n6.1  For  i = 1, ... , h,  let wi: = w~ + 77(zi  - w~) and w;/  = w;i + e + \n77(Z;  - w~) with  parameter  77,  where e is  a  uniform random  variable \nbetween \u00b1O.O1. \n\n6.  (Negative reinforcement) Perform the following: \n\n6.2  Let S;:'  = min (5, s;:  + p- /  (1 + s;:)),  for  negative reinf p- > O. \n6.3  Let  rt  = k,/ max ( .1, S;:'). \n6.4  Let  st' = max(O, si - p-),  and ri = k,/ max(O.I, Si'l. \n6.5  Go to step 2. \n\n\f26 \n\nAckley \n\n~~~Be~:~0~.~00~O~OO~0 ___________ I_D_on_'_t_~~:_3~1~.~29~4~70~3 ________ I_B~e ____________________ 1 \n\nFigure  5.  A  display  of  the  internal  state  of  the  trunk  robot  in  the  process  of \nlearning  to  associate  a  set  of twelve  arbitrary  stimuli  with  specified  positions  in \nspace.  The current signal  (though  the system  has  not  discovered  this  yet)  means \n\"touch 5\". \n\nDo  Be:  4.998667 \n~~~~~~----------I~~~~~~---------\n\nDon't  Be:  3  0.007976 \n\nBe \n\nFigure  6.  A  display  of  the  internal state  of the  same  trunk  robot  later  in  the \nlearning process.  The previous  goal  was  \"touch *\"  and  the  current goal is  \"touch \n3.\" \n\n\\  do-be  map  now  contains  an  accurate  image  of  a  successful  configuration  for  the \n\"touch  3\"  goal,  and  its  strength  is  high.  The  strength  of the  selected  don't-be \nunit is  low.  The current configuration map in Figure  6  shows each iteration of the \nalgorithm between the  time it achieved  its previous goal and  the time it achieved \nthe current goal. \n\nFinally,  Figure 7 displays the average  time-per-goal as  a function  of the number of \ngoals achieved.  For 75  repetitions,  the  trunk network was initialized and run until \n500 goals  had been achieved, and the resulting time-per-goal data was averaged to \nproduce the graph. \n\nThe  average  time-per-goal  declines  rapidly  as  goals  are  presented,  then  seems  to \nrise  slightly,  and  then  stabilizes  around  an  average  value  of about  300.  To have \nsome kind of standard of comparison, albeit unsophisticated, if the joint angles are \nsimply changed by uniform random values between q and -q (see Figure 4)  on each \niteration,  the average time-per-goal is  observed to be about  490. \n\n\fAssociative Learning via Inhibitory Search \n\n27 \n\n:nOD \n\n24DO \n\n2100 \n\nliDO \n\n15DD \n\n1200 \n\n900 \n\n600 \n\n300 \n\nFigure  7.  A graph of the average time taken per goal as a  function of the number \nof goals  achieved.  The  horizontal line  shows  the  average  performance  of random \njoint changes. \n\nDISCUSSION \n\nALVIS is  a  preliminary,  exploratory  system.  Of course,  the  ALVIS environment \nis  but a  pale shadow of the  real  world,  but  even granting the limited scope  of the \nproblem  formulation,  several aspects  of  AL VIS are incompletely satisfying  and in \nneed of improvement. \n\nTo  my  eye,  the  biggest  drawback  of the  current  implementation is  the local  goal \nrepresentation - which essentially requires  that the set of goals be enumerable at \nnetwork definition time.  Related problems include the inability to share information \nbetween  one  goal  and  another  and  the  inability  to  pursue  more  than  one  goal \nsimultaneously.  To determine behavior, the constraints of goals must be integrated \nwith  the possibilities  of the  current situation.  In  AL VIS this  is  done  as  a  strictly \ntwo-step  process:  the  goal selects  a  subnetwork,  and  the  current situation selects \nunits within the subnetwork.  Work such as Jordan (1986) and Miyata (1988) shows \nhow goal information and context information can be integrated by supplying both \nas inputs to a  single  network. \n\nALVIS is a pure feedback control model, and can suffer from the traditional problem \nof that  approach:  when  the errors  are small,  the  resulting joint angle changes are \nsmall,  and  the  arm  converges  only  slowly.  If the  gain  at  the  joints is  increased \nto speed  con vergence,  overshoot  and  oscillation  become  more  likely.  However,  in \nALVIS oscillations  gradually  die  out,  as  don't-be  units  shift  positions  under  the \nnegative  reinforcement,  and  sometimes  such  temporary  oscillations  actually  help \nwith the search, causing  the tip of the arm to explore  a  variety of different  points. \n\n\f28 \n\nAckley \n\nThe aspect  of AL VIS behavior that I find  most irritating reveals something about \nthe  approach  in  general.  In  some  cases  -\nusually  on  more  \"peripheral\"  targets \n- AL VIS  learns  to  hit  the  very  edge  of  the  target  region.  While  approaching \nsuch  targets,  ALVIS experiences  negative  reinforcement,  and  the  don't-be  units, \nconsequently,  gain  a  little strength.  The  resulting interference occasionally  causes \na  very  long search  for  a  goal  that  had  previously  been  rapidly  achieved.  AL VIS \nhas a  representation only  for  its own  body;  a  better system  would  also be able  to \nrepresent other objects in the world, and useful relations on  the expanded set.  The \n\"mostly  motor\"  emphasis  evident  in  the  present  system  needs  to be  balanced  by \nmore sophistication on  the perceptual side. \n\nThough limited in scope,  ALVIS demonstrates three ideas I  think worth highlight(cid:173)\nmg: \n\n\u2022  The reuse  of the error channel of a  backpropagation network,  after  training,  for \n\ntranslating arbitrary output-space gradients into input-space gradients. \n\n\u2022  The recording of previous  actual outputs to be used as future desired outputs. \n\n\u2022  The use of \"repellors\" (don 't-be units) as well as attractors in defining errors, and \nthe resulting process of search-by-inhibition generated by negative reinforcement. \n\nCharacterizing the behavior of a machine in terms of attractor dynamics is a familiar \nnotion,  but  \"repellor dynamics\"  seems  to be largely unknown territory.  Indeed, in \nALVIS there is an ephemeral quality  to the don't-be units:  When all answers have \nbeen  discovered,  all  strength  accrues  to  the  do-be's,  good  performances  become \nroutine,  and  AL VIS  behavior  is  essentially  attractor-based.  In  watching  such  a \n\"grown-up\"  AL VIS,  it is easy to forget  how it was in the beginning,  when the world \nwas big and answers were scarce,  and  ALVIS was doing well just to discover  a  new \nmistake. \n\nReferences \n\nAckley,  D.H.  (1988).  Associative  learning  via  inhibitory  search.  Teclmical  memorandum  TM \n\nARH-012509. Morristown. NJ:  Bell  Communications Research. \n\nBurr. D.J .\u2022  &  Hanson. S.J. (1987).  Knowledge representation in connectionist networks.  Technical \n\nmemorandum TM-ARH-008733.  Morristown. NJ:  Bell Communications Research. \n\nFeldman. J.A.  (1982).  Dynamic connections in neural networks.  Biological Cybernetics,  36, 193-\n\n202. \n\nJordan.  M.1.  (1986).  Serial  order:  A  parallel.  distributed processing  approach.  Technical report \n\nICS-86M.  La Jolla,  CA:  University of California. Institute for  Cognitive Science. \n\nMiyata, Y.  (1988).  The learning  and planning  of actions.  Unpublished doctoral dissertation  in \n\npsychology, University of California San Diego. \n\nRumelhart,  D.E .\u2022  Hinton.  G.E .\u2022  &  Williams.  R.J.  (1986).  Learning  representations  by  back(cid:173)\n\npropagating errors.  Nature, 3Z3.  533-536. \n\nRumelhart.  D.E. (personal communication. 1987).  Also cited as personal communication in  Miy(cid:173)\n\nata (1988). \n\n\f", "award": [], "sourceid": 140, "authors": [{"given_name": "David", "family_name": "Ackley", "institution": null}]}