{"title": "Range Image Restoration Using Mean Field Annealing", "book": "Advances in Neural Information Processing Systems", "page_first": 594, "page_last": 601, "abstract": null, "full_text": "594 \n\nRange Image Restoration \n\nusing Mean Field Annealing \n\nGriff L.  Bilbro \n\nWesley  E.  Snyder \n\nCenter for  Communications and Signal  Processing \n\nNorth  Carolina State  University \n\nRaleigh,  NC \n\nAbstract \n\nA  new  optimization strategy,  Mean  Field  Annealing, is  presented. \nIts application to MAP restoration of noisy range images is derived \nand experimentally verified. \n\n1 \n\nIntroduction \n\nThe application which motivates this paper is  image analysis; specifically  the anal(cid:173)\nysis  of range  images.  We  [BS86]  [GS87]  and  others  [YA85][BJ88]  have  found  that \nsurface  curvature  has  the  potential for  providing an  excellent,  view-invariant  fea(cid:173)\nture with which to segment range images.  Unfortunately, computation of curvature \nrequires,  in turn,  computation of second  derivatives of noisy  data. \nWe cast this task as a restoration problem:  Given a measurement g(z, y),  we assume \nthat g(z, y)  resulted from  the addition of noise  to some  \"ideal\"  image  fez, y)  which \nwe  must estimate from  three  things: \n\n1.  The measurement  g(z, y). \n2.  The statistics of the  noise,  here  assumed  to be zero  mean with variance  (1'2. \n3.  Some  a  priori knowledge  of the smoothness  of the  underlying surface(s). \n\nWe  will  turn  this  restoration  problem  into  a  minimization,  and  solve  that  mini(cid:173)\nmization using a  strategy called  Mean  Field A nnealing.  A neural net appears to be \nthe  ideal  architecture  for  the  reSUlting  algorithm,  and  some  work  in  this  area  has \nalready  been  reported  [CZVJ88]. \n\n2 \n\nSimulated Annealing and Mean Field Anneal-\n\u2022 Ing \n\nThe strategy  of SSA  may be  summarized  as follows: \nLet  H(f)  be  the  objective  function  whose  minimum  we  se~k,  wher~ /is somt'  pa(cid:173)\nrameter  vector. \nA  parameter  T  controls  the  algorithm.  The  SSA  algorithm  begins  at  a  relatively \nhigh  value  of T  which  is  gradually  reduced.  Under  certain  conditions,  SSA  will \nconverge  to a  global optimum,  [GGB4]  [RS87] \n\nH (f)  =  min{ H (fie)} V fie \n\n(1) \n\n\fRange Image Restoration Using Mean Field Annealing \n\n595 \n\neven  though local minima may occur.  However,  SSA  suffers  from  two drawbacks: \n\n\u2022  It is  slow,  and \n\u2022  there  is  no  way  to directly  estimate  [MMP87]  a  continuously-valued I  or its \n\nderivatives. \n\nThe algorithm presented  in section  2.1  perturbs  (typically)  a  single element  of fat \neach  iteration.  In  Mean  Field  Annealing,  we  perturb  the  entire  vector  f  at each \niteration  by  making  a  deterministic  calculation  which  lowers  a  certain  average  of \nH,  <  H(f)  >,  at  the  current  temperature.  We  thus  perform  a  rather  conventional \nnon-linear  minimization  (e.g.  gradient  descent),  until  a  minimum is  found  at  that \ntemperature.  We  will  refer  to  the  minimization  condition  at  a  given  T  as  the \nequilibrium for  that  T.  Then,  T is reduced,  and the previous equilibrium is  used  as \nthe initial condition for  another minimization. \nMFA  thus converts a  hard optimization problem into a  sequence of easier problems. \nIn  the  next  section,  we justify this approach by  relating it  to SSA. \n\n2.1  Stochastic  Simulated Annealing \nThe  problem  to  be  solved  is  to  find  j  where  j  minimizes  H(f).  SSA  solves  this \nminimization with  the following  strategy: \n\n1.  Define PT  ex  e- H / T . \n2.  Find the equilibrium conditions on PT,  at the current temperature, T. By equi(cid:173)\n\nlibrium, we mean that any statistic ofpT(f) is constant.  These statistics could \nbe derived from the Markov chain which SSA constructs:  jO, p, ... , IN, ... , al(cid:173)\nthough in fact  such statistical analysis is never  done in  normal running of an \nSSA  algorithm. \n\n3.  Reduce T  gradually. \n4.  As T  --+  0,  PT(f)  becomes  sharply  peaked at  j,  the  minimum. \n\n2.2  Mean Field Annealing \n\nIn Mean  Field Annealing,  we  provide an analytic mechanism for  approximating the \nequilibrium at arbitrary T.  In  MFA,  we  define  an error  function, \n\nEMF(Z, T) = Tln--=-H- - + \n\n-H \n\nfe--ordl \nf eTdl \n\n-Hfl \n\nfe  T  (H-Ho)dl \n- --. \n\n- / j ---\n\nf e- TdJ \n\nwhich follows from  Peierl's inequality  [BGZ76]: \n\nF  ~ Fo+  < H  - Ho  > \n\n(2) \n\n(3) \n\nwhere  F  =  -Tlnf e---r-dl  and Fo  =  -Tlnf e  T  dl .  The significance  of EMF  is  as \nfollows:  the  minimum of EMF  determines  the  best  approximation given  the  form \n\n-H \n\n-Hg \n\n\f596 \n\nBilbro and Snyder \n\nof Ho  to  the  equilibrium statistics  of the  SSA-generated  MRF  at T.  We  will  then \nanneal  on  T.  In  the  next  section,  we  choose  a  special  form  for  Ho  to simplify  this \nprocess  even further. \n\n1.  Define some  Ho(f, z)  which  will be  used  to estimate  H(f). \n2.  At  temperature  T,  minimize  EMF(Z)  where  EMF  is  a  functional  of Ho  and \nH  which  characterizes  the  difference  between  Ho  and  H.  The  process  of \nminimizing  EMF  will  result  in  a  value  of the  parameter  z,  which  we  will \ndenote  as  ZT. \n\n3.  Define  HT(f)  =  Ho(f, ZT)  and for(f)  ex  e- iiT /T. \n\n3 \n\nImage  Restoration  Using  MFA \n\nWe  choose  a  Hamiltonian which represents  both  the  noise  in the image,  and our  a \npriori knowledge of the local shape  of the image data. \n\nHN  =  L.J -2 2 (Ii  - gil \n2 \n\n\"\"  1 \n, \n(1' \n\n\u2022 \n\n(4) \n\n(5) \n\nwhere  18(  represents  [Bes86]  the set of values of pixels neighboring pixel  i (e.g.  the \nvalue of I at  i  along with the I values  at  the four  nearest  neighbors of i);  A is  some \nscalar  valued  function  of that set  of pixels  (e.g.  the  5  pixel  approximation  to  the \nLaplacian or the  9  pixel approximation to  the quadratic variation); and \n\n(6) \n\nThe noise term simply says that the image should be similar to the data, given noise \nof variance (1'2.  The prior term drives toward solutions which are locally planar.  Re(cid:173)\ncently, a simpler V(z) =  z2  and a similar A were successfully used to design a  neural \nnet  [CZVJ88]  which  restores  images  consisting  of discrete,  but  256-valued  pixels. \nOur formulation of the  prior term emphasizes  the importance of \"point processes,\" \nas defined  [WP85] by Wolberg and Pavlidis.  While we  accept the eventual necessity \nof incorporating line  processes  [MMP87]  [Mar85]  [GG84]  [Gem87]  into restoration, \nour  emphasis  in  this  paper  is  to  provide  a  rigorous  relationship  between  a  point \nprocess,  the  prior model,  and  the  more  usual  mathematical  properties  of surfaces. \nUsing  range imagery in this problem makes these  relationships direct.  By adopting \nthis  philosophy,  we  can  exploit  the  results  of Grimson  [Gri83]  as  well  as  those  of \nBrady and  Horn  [BH83]  to improve on the  Laplacian. \nThe  Gaussian  functional  form  of V  is  chosen  because  it is  mathematically  conve(cid:173)\nnient for  Boltzmann statistics and beca.use  it reflects  the following shape properties \nrecommended  for  grey  level  images  in  the  literature  and  is  especially  important if \n\n\fRange Image Restoration Using Mean Field Annealing \n\n597 \n\nline  processes  are  to  be  omitted:  Besag  [Bes86]  notes  that  lito  encourage  smooth \nvariation\",  V(A)  \"should  be  strictly  increasing\"  in  the  absolute  value  of its  argu(cid:173)\nment  and  if \"occasional  abrupt  changes\"  are  expected,  it should  \"quickly  reach  a \nmaximum\" . \nRational functions with shapes similar to our V  have been used in recent stochastic \napproaches  to  image processing  [GM85].  In  Eq.  6,  T  is  a  \"soft  threshold\"  which \nrepresents  our  prior  knowledge  of the  probability  of various  values  of  \\7 2 f  (the \nLaplacian  of the  undegraded  image).  For  T  large,  we  imply  that  high  values  of \nthe  Laplacian  are  common - f  is  highly  textured;  for  small values  of T,  we  imply \nthat  f  is  generally  smooth.  We  note  that  for  high  values  of T,  the  prior  term  is \ninsignificant,  and the  best  estimate of the image is simply the  data. \nWe choose  the  Mean  Field  Hamiltonian to be \n\nand find  that the optimal ZT  approximately minimizes \n\n(7) \n\n(8) \n\nboth at very  high and very low  T . We  have found  experimentally that this  approx(cid:173)\nimation  to  ZT  does  anneal  to  a  satisfactory  restoration.  At  each  temperature,  we \nuse  gradient descent  to find  ZT  with the following approximation to the gradient of \n<H>: \n\n(9) \n\nand \n\n-b \n\nV(r\u00b7) -\n\n,  - y'2;(T+T) \n\n.. ? \n\ne- 2( .. h) \n. \n\nDifferentiating Eq.  8  with  this new  notation, we  find \n\nSince 6'+11,;  is  non-zero only when  i + v = i, we  have \n\n8  < H  >  _  :J!j  - gj  L L \n\n2  + \n\n-\n\n(1' \n\n8 \n\n:J! . \n) \n\nII \n\nIT'(.  ) \n\n')+11 \n\n-II \n\nand this  derivative can  be  used  to find  the equilibrium condition. \n\nAlgorithm \n\n(lO) \n\n(11) \n\n(12) \n\n\f598 \n\nBilbro and Snyder \n\n1.  Initially, we  use  the high temperature assumption, which eliminates the prior \n\nterm entirely,  and results in \n\nZ;  =g;;  for \n\nT  = 00. \n\n(13) \n\nThis  will  provide  the  initial  estimate  of z.  Any  other  estimate  quickly  con(cid:173)\nverges  to g. \n2.  Given  an  image  z;,  form  the  image  ri  = (L  \u00ae  z);,  where  the  \u00ae  indicates \n\nconvolution. \n\nP \n\n, \n\n..? \n\n~T+T)T+T \n\n3.  Create  the image V.  = V' (r\u00b7) = - -----l= _-.!'L e - :II(T~ .. )  \u2022 \n4.  Using  12,  perform ordinary  non-linear minimization of < H  > starting from \nthe  current  z.  The  particular  strategy  followed  is  not  critical.  We  have \nsuccessfully  used  steepest  descent  and  more  sophisticated  conjugate  gradi(cid:173)\nent  [PFTV88]  methods.  The  simpler  methods  seem  adequate  fot  Gaussian \nnoise. \n\n5.  Update z  to the minimizing z found  in step 4. \n6.  Reduce  T  and  go  to  2.  When  T  is  sufficiently  close  to  0,  the  algorithm  is \n\ncomplete. \n\nIn  step  6  above,  T  essentially  defines  the  appropriate  low-temperature  stopping \npoint.  In  section  5,  we  will  elaborate  on  the  determination  of T  and  other  such \nconstants. \n\n4  Performance \n\nIn  this  section,  we  describe  the  performance  of the  algorithm  as  it  is  applied  to \nseveral  range images.  We  will use  range images, in  which  the data is  of the form \n\nz  =  z(z, y). \n\n(14) \n\n4.1 \n\nImages With High Levels  of Noise \n\nFigure  1 illustrates a  range image consisting of three objects, a  wedge  (upper left), \na  cylinder  with rounded  end  and hole  (right), and a  trapezoidal block  viewed  from \nthe  top.  The noise  in  this region  is  measured  at  (1'  = 3units out of a  total range  of \nabout 100 units.  Unsophisticated smoothing will not estimate second derivatives of \nsuch  data without  blurring.  Following the  surface  interpolation literature,  [Gri83] \n[BB83]  we  use  the  quadratic variation as  the  argument of the  penalty function  for \nthe prior  term  to \n\n(15) \n\nand  performing  the  derivative  in  a  manner  analogous  to  Eq.  11  and  12.  The \nLaplacian  of the  restoration  is  shown  in  Figure  2.  Figure  3  shows  a  cross-section \ntaken  as indicated by  the red line on Figure 2. \n\n\fFig.  1  Original rallge image \n\nFig.  2  Laplacian of the restored  image \n\nI \n\nn \n\nJ~~ l\\~lll \n\nFig.  3  Cross  section \nThrough  Laplacian along \nRed Line \n\n4.2  Comparison With Existing Techniques \n\nAccurate  computation of surface  derivatives  requires  extremely  good smoothing of \nsurface  noise,  while segmentation requires  preservat.ion  of edges.  One suc.h  adapt.ive \nsmoothing  technique,[Ter87]  iterative Gaussian smoothing (IGS)  has  been  success(cid:173)\nfully  applied  to range  imagery.  [PB87]  Following this  strategy,  step  edges  are  first \ndetected,  and  smoothing is  then  applied  using  a  small center-weighted  kernel.  At \nedges,  an even  smaller  kernel,  called  a  \"molecule\",  is  used  to  smooth  right  up  to \nthe  edge  without  blurring the edge.  The smoothing is  then  iterated. \n\n\f600 \n\nBilbro and Snyder \n\nThe  results,  restoration  and  Laplacian,  of IGS  are  not  nearly  as  sharp  as  those \nshown  in Figure  2. \n\n5  Determining the  Parameters \n\nAlthough the  optimization strategy  described  in section  3  has  no  hard  thresholds, \nseveral  parameters  exist  either  explicitly  in  Eq.  8  or  implicitly  in  the  iteration. \nGood  estimates  of these  parameters  will  result  in  improved  performance,  faster \nconvergence,  or  both.  The parameters are: \n\n(1'  the standard deviation of the  noise \nb  the relative  magnitude of the  prior term \n11  = T  + T  the initial temperature  and \nT  the final  temperature. \n\nThe decrement  in T  which defines  the  annealing schedule  could  also  be  considered \na  parameter.  However,  we  have  observed  that  10%  or less  per  step  is  always good \nenough. \nWe find  that for depth images of polyhedral scenes,  T  = 0 so that only one parameter \nis  problem dependent:  (1'.  For  the  more  realistic  case  of images  which  also contain \ncurved  surfaces,  however,  see  our  technical  report  [BS88],  which  also describes  the \nMFA  derivation in  much  more detail. \nThe  standard  deviation  of the  noise  must  be  determined  independently  for  each \nIt is  straightforward  to  estimate  (1'  to  within  50%,  and  we  have \nproblem  class. \nobserved  experimentally  that  performance  of the  algorithm is  not sensitive  to  this \norder of error. \nWe can analytically show that annealing occurs in the region T::::::  IL12(1'2  and choose \nTJ  = 2ILI2(1'2.  Here,  ILI2  is  the  squared  norm of the  operator  Land  ILI2  = 20  for \nthe usual  Laplacian and  ILI2  = 12.5 for  the quadratic variation. \nFurther  analysis shows  that b = .J2;ILI(1' is  a  good  choice  for  the coefficient  of the \nprior term. \n\nReferences \n\n[Bes86] \n\nJ.  Besag.  On  the  statistical  analysis  of dirty  pictures.  Journal  of the \nRoyal Stati6ticCJl  Society,  B  48(3),  1986. \n\n[BGZ76]  E.  Brezin,  J.  C.  Le  Guillon,  and  J.  Zinn-Justin.  Field  theoretical  ap(cid:173)\n\nproach  to  critical  phenomena.  In  C.  Domb  and  M.S.  Green,  editors, \nPhCJ6e  Tran6ition6  and  Critical Phenomena,  chapter  3,  Academic  Press, \nNew  York,  1976. \nM.  Brady  and  B.K.P  Horn.  Symm~tric operators for  surface  interpola(cid:173)\ntion.  CVGIP,  22,  1983. \nP.J.  Besl  and  R.C.  Jain.  Segmentation  through  variable-order  surface \nfitting.  IEEE PAMI,  10(2),  1988. \n\n[BH83] \n\n[BJ88] \n\n\fRange Image Restoration Using Mean Field Annealing \n\n601 \n\n[BS86] \n\n[BS88] \n\nG.  Bilbro and  W.  Snyder.  A  Linear  Time  Theory  for  Recognizing  Sur(cid:173)\nface6  in 3-D.  Technical Report CCSP-NCSU TR-86/8, Center for  Com(cid:173)\nmunications  and  Signal  Processing,  North  Carolina  State  University, \n1986. \nG.  L.  Bilbro  and  W.  E.  Snyder.  Range  Image  Re6toration  U6ing  Mean \nField  Annealing.  Technical  Report  NETR  88-19,  Available  from  the \nCenter for  Communications and Signal Processing,  North Carolina State \nUniversity,  1988. \n\n[CZVJ88]  R. Chellappa, Y.-T. Zhou,  A. Vaid, and B.  K. Jenkins.  Image restoration \nusing a  neural network.  IEEE  Tran6action6  on  ASSP,  36(7):1141-1151, \nJuly 1988. \n\n[Gem87]  D.  Geman.  Stochastic model for  boundary detection.  Vi6ion  and Image \n\n[GG84] \n\n[GM85] \n\nComputing,  5(2):61-65,  1987. \nD.  Geman  and  S.  Geman.  Stochastic  relaxation,  Gibbs  Distributions, \nand  the  Bayesian  restoration  of images.  IEEE  Tran6action6  on  PAMI, \nPAMI-6(6):721-741,  November  1984. \nS.  Geman  and  D.E.  McClure.  Bayesian image  analysis:  an  application \nto  single  photon  emission  tomography.  Proceeding6  of the  American \nStati6tical A uociation,  Stati6tical  Computing  Section,  12-18,  1985. \n\nrGri83]  W.E.L.  Grimson.  An implementation of computational theory  of visual \n\n[GS87] \n\n[Mar85] \n\nsurface  interpolation.  CVGIP,  22,  1983. \nB.  R.  Groshong and W. E.  Snyder.  Range image segmentation for object \nrecognition.  In 18th Pitt6burgh  Conference  on Modeling  and  Simulation, \nPittsburgh,  PA,  April  1987. \nJ .L.  Marroquin.  Probabili6tic Solution  to  Inver6e  Problem6.  PhD thesis, \nM.LT, Cambridge,  MA,  September  1985. \n\n[MMP87]  J.  Marroquin,  S.  Mitter,  and  T.  Poggio.  Probabilistic  solution  of ill(cid:173)\n\n[PB87] \n\nposed problems in computational vision.  Journal of American Stati6tical \nAuociation, 82(397):76-89, March  1987. \nT. Ponce and M.  Brady.  Toward a  surface  primal sketch.  In T. Kanade, \neditor,  Three  Dimen6ional Machine  Vi6ion,  Kluwer  Press,  1987. \n\n[PFTV88]  W.  H.  Press,  B.  P.  Flannery,  S.  A.  Teukolsky,  and  W.  T.  Vetterling. \n\n[RS87] \n\n[Ter87] \n\n[WP85] \n\n[YA85] \n\nNumerical Recipe6  in  C.  Cambridge  University  Press,  1988. \nF.  Romeo  and  A.  Sangiovanni-Vencentelli.  Probabalistic  hill  climbing \nalgorithms:  properties  and  applications.  In  Chapel  Hill  Conference  on \nVLSI,  Computer  Science  Press,  Chapel Hill,  NC,  1987. \nD.  Terzopoulos.  The  role  of constraints  and  discontiuities  in  visible(cid:173)\nsurface  reconstruction. \npages  1073-1077, 1987. \nG. Wolberg and T. Pavlidis.  Restoration of binary images using stochas(cid:173)\ntic relaxation with annealing.  Pattern Recognition Letter6, 3(6):375-388, \nDecember  1985. \nM.  Brady A.  Yiulle and H.  Asada.  Describing surfaces.  CVGIP, August \n1985. \n\nIn  Proc.  of  7th  International  Conf.  on  AI, \n\n\f", "award": [], "sourceid": 153, "authors": [{"given_name": "Griff", "family_name": "Bilbro", "institution": null}, {"given_name": "Wesley", "family_name": "Snyder", "institution": null}]}