{"title": "Contour Organisation with the EM Algorithm", "book": "Advances in Neural Information Processing Systems", "page_first": 880, "page_last": 886, "abstract": null, "full_text": "Contour Organisation with the EM \n\nAlgorithm \n\nJ.  A.  F.  Leite and  E.  R.  Hancock \n\nDepartment of Computer Science \n\nUniversity of York, York, Y01  5DD, UK. \n\nAbstract \n\nThis paper describes how the early visual process of contour organ(cid:173)\nisation  can  be  realised  using  the  EM  algorithm.  The  underlying \ncomputational representation is  based on fine  spline coverings.  Ac(cid:173)\ncording  to our  EM  approach  the  adjustment  of spline  parameters \ndraws  on  an  iterative  weighted  least-squares  fitting  process.  The \nexpectation  step  of our  EM  procedure  computes  the  likelihood  of \nthe data using a mixture model defined over the set of spline cover(cid:173)\nings.  These splines  are limited in  their  spatial extent  using  Gaus(cid:173)\nsian windowing functions.  The maximisation of the likelihood leads \nto a set of linear equations in the spline parameters which solve the \nweighted least squares problem.  We  evaluate the technique  on  the \nlocalisation of road structures in  aerial infra-red images. \n\n1 \n\nIntroduction \n\nDempster, Laird and Rubin's EM (expectation and maximisation) [1]  algorithm was \noriginally introduced as  a  means  of finding  maximum likelihood solutions to  prob(cid:173)\nlems  posed in  terms  of incomplete  data.  The basic  idea underlying the  algorithm \nis  to iterate between the expectation  and maximisation  modes until convergence is \nreached.  Expectation  involves  computing  a posteriori  model  probabilities  using  a \nmixture  density  specified  in  terms  of  a  series  of model  parameters.  In  the  max(cid:173)\nimisation  phase,  the  model  parameters  are recomputed  to  maximise  the  expected \nvalue of the incomplete data likelihood.  In fact,  when viewed from this perspective, \nthe  updating  of  a posteriori  probabilities  in  the  expectation  phase  would  appear \nto have  much  in  common  with  the  probabilistic relaxation  process  extensively  ex(cid:173)\nploited in  low  and intermediate level  vision  [9,  2] .  Maximisation of the incomplete \n\n\fContour Organisation with the EM Algorithm \n\n881 \n\ndata likelihood is  reminiscent  of robust estimation where  outlier reject is  employed \nin  the iterative re-computation of model  parameters [7]. \n\nIt is  these  observations  that  motivate  the  study  reported  in  this  paper.  We  are \ninterested in  the organisation of the output of local feature enhancement operators \ninto meaningful global contour structures [13,  2].  Despite providing one of the clas(cid:173)\nsical  applications of relaxation labelling  in  low-level  vision  [9],  successful  solutions \nto the iterative curve reinforcement  problem have proved to be surprisingly elusive \n[8,  12,  2].  Recently,  two  contrasting ideas have offered  practical relaxation operat(cid:173)\nors.  Zucker  et  al  [13]  have  sought  biologically  plausible  operators  which  draw  on \nthe idea of computing a global curve organisation potential and locating consistent \nIn  essence  this  biologically \nstructure  using  a  form  of local  snake  dynamics  [11]. \ninspired model  delivers a  fine  arrangement of local splines that minimise the curve \norganisation potential.  Hancock  and  Kittler  [2],  on the other  hand,  appealed to a \nmore  information theoretic motivation  [4].  In an  attempt to overcome some of the \nwell  documented limitations of the original Rosenfeld,  Hummel and Zucker relaxa(cid:173)\ntion operator [9]  they have developed a Bayesian framework for relaxation labelling \n[4].  Of particular significance for the low-level curve enhancement problem is the un(cid:173)\nderlying statistical framework which makes a clear-cut distinction between the roles \nof uncertain image data and prior knowledge of contour structure.  This framework \nhas allowed the output of local image operators to be represented in terms of Gaus(cid:173)\nsian  measurement densities,  while  curve structure is  represented by  a dictionary of \nconsistent contour structures  [2]. \n\nWhile  both  the  fine-spline  coverings  of  Zucker  [13]  and  the  dictionary-based  re(cid:173)\nlaxation  operator of Hancock  and  Kittler  [2]  have  delivered  practical solutions  to \nthe  curve reinforcement  problem,  they  each  suffer  a  number  of shortcomings.  For \ninstance, although the fine  spline operator can  achieve  quasi-global curve organisa(cid:173)\ntion,  it  is  based  on  an  essentially  ad  hoc  local  compatibility  model.  While  being \nmore information theoretic,  the  dictionary-based relaxation operator is  limited by \nvirtue of the fact that in most practical applications the dictionary can only realist(cid:173)\nically be evaluated over at most a  3x3 pixel neighbourhood.  Our aim in  this paper \nis  to  bridge the  methodological divide  between  the  biologically inspired fine-spline \noperator and the statistical framework of dictionary-based relaxation.  We  develop \nan  iterative  spline  fitting  process  using  the  EM  algorithm  of Dempster  et  al  [1] . \nIn  doing  this  we  retain  the  statistical  framework  for  representing  filter  responses \nthat  has  been  used  to  great  effect  in  the  initialisation  of dictionary-based  relaxa(cid:173)\ntion.  However, we  overcome the limited contour representation of the dictionary by \ndrawing on  local cubic splines. \n\n2  Prerequisites \n\nThe  practical  goal  in  this  paper  is  the  detection  of line-features  which  manifest \nthemselves  as  intensity  ridges  of  variable  width  in  raw  image  data.  Each  pixel \nis  characterised  by  a  vector  of measurements,  ?<i  where  i  is  the  pixel-index.  This \nmeasurement  vector  is  computed  by  applying  a  battery of line-detection  filters  of \nvarious  widths  and  orientations  to  the  raw  image.  Suppose  that  the  image  data \nis  indexed  by  the  pixel-set  I.  Associated  with  each  image  pixel  is  a  cubic  spline \nparameterisation which  represents  the  best-fit  contour  that  couples  it  to adjacent \nfeature  pixels.  The  spline  is  represented  by  a  vector  of  parameters  denoted  by \n\n\f882 \n\n1. A.  F.  Leite and E.  R.  Hancock \n\n9.;  =  (q?, q}, q; , qr) T .  Let  (Xi, Yi)  represent  the  position  co-ordinates of the  pixel \nindexed i .  The spline variable,  Si,j  =  X i  - Xj  associated with the contour connecting \nthe pixel indexed j  is  the horizontal displacement between  the pixels indexed i  and \nj.  We  can  write  the  cubic  spline  as  an  inner  product  F(Si ,j, g)  =  g[.Si,j  where \nSi,j  =  (1, Si,j , S~,j' S~,j) T .  Central  to  our  EM  algorithm  will  be  the  comparison  of \nthe predicted vertical spline displacement with  its measured value  Ti, j  =  Yi  - Yj. \n\nIn order to initialise the EM algorithm, we require a set of initial spline probabilities \nwhich  we  denote  by  7l'(q(O\u00bb).  Here  we  use  the  multi-channel  combination  model \nrecently  reported by  Leite  and  Hancock  [5]  to  compute  an  initial  multi-scale  line-\nfeature  probability.  Accordingly,  if  ~ is  the  variance-covariance  matrix  for  the \ncomponents of the filter  bank,  then \n\n- t  \n\n7l'(g~O\u00bb)  =  l-exp [-~~r~-l~i] \n\n(1) \n\nThe remainder of this paper outlines how these initial probabilities are iteratively re(cid:173)\nfined using the EM algorithm.  Because space is limited we only provide an algorithm \nsketch.  Essential algorithm details such as the estimation of spline orientation and \nthe  local  receptive  gating  of the  spline  probabilities  are  omitted  for  clarity.  Full \ndetails can  be found  in  a  forthcoming journal article  [6]. \n\n3  Expectation \n\nOur  basic  model  of the  spline  organisation process  is  as  follows .  Associated  with \neach  image pixel  is  a  spline  parameterisation.  Key  to our  philosophy of exploiting \na  mixture  model  to  describe  the global  contour  structure of the  image  is  the  idea \nthat  the  pixel  indexed  i  can associate to each of the  putative splines  residing  in  a \nlocal  Gaussian  window  N i .  We  commence by developing  a  mixture  model  for  the \nconditional  probability  density  for  the  filter  response  ~i  given  the  current  global \nspline  description.  If ~(n)  =  {q(n), Vi  E  I}  is  the  global  spline  description  at \niteration n  of the EM process,  then we  can expand the mixture distribution  over a \nset  of putative splines that may associate with the image pixel  indexed i \n\n- t  \n\np(~il~(n\u00bb) =  L p(~ilg;n\u00bb)7l'(g;n\u00bb) \n\njENi \n\n(2) \n\n-J \n\n-J \n\nThe  components  of  the  above  mixture  density  are  the  conditional  measurement \ndensities  p(~ilq(n\u00bb)  and  the  spline  mixing  proportions  7l'(q(n\u00bb).  The  conditional \nmeasurement densities represent the likelihood that the datum ~i originates from the \nspline centred on pixel j. The mixing proportions, on the other hand, represent the \nfractional  contribution to the data arising from  the jth parameter vector  i.e.  q(n). \nSince we  are interested in the maximum likelihood estimation of spline parameters, \nwe  turn our attention to the likelihood of the raw data, i.e. \n\n-J \n\nP(~i' Vi  E  II~(n\u00bb) =  IIp(~il~(n)) \n\n(3) \n\niEI \n\nThe expectation step of the EM algorithm is aimed at estimating the log-likelihood \nusing  the  parameters of the  mixture  distribution.  In  other  words,  we  need  to  av(cid:173)\nerage  the  likelihood  over  the  space  of potential  pixel-spline  assignments.  In  fact, \n\n\fContour Organisation with the EM Algorithm \n\n883 \n\nit  was  Dempster,  Laird and Rubin  [1]  who observed that maximising the weighted \nlog-likelihood was equivalent  to maximising the conditional expectation of the like(cid:173)\nlihood  for  a  new  parameter set  given  an old  parameter set.  For  our  spline  fitting \nproblem, maximisation of the expectation of the conditional likelihood is  equivalent \nto maximising the weighted log-likelihood function \n\nQ(~(n+l)I~(n\u00bb) = L L  p(g;n)l~i) lnp(~ilg;n+l\u00bb) \n\n(4) \n\niEI JEN, \n\nThe  a posteriori probabilities p(q(n)l~i) may be computed from  the corresponding \ncomponents of the mixture density p(~ilqJn\u00bb) using the Bayes formula \n\n-1 \n\np(g;n)l~i) = 2: \n\nP(~i Iq(n) )7r( q(n\u00bb) \n\n(11  (n\u00bb))  (  (n\u00bb) \n7r  gk \n\nkEN.  P  ~t gk \n\n(5) \n\nFor notational convenience, and to make the weighting role of the a posteriori prob(cid:173)\nabilities  explicit  we  use  the shorthand w~~) = p(g;n) I~i).  Once updated parameter \nestimates  q(n)  become  available  through  the  maximisation  of this  criterion,  im-\nproved estimates of the mixture components may be obtained by  substitution into \nequation  (6).  The  updated  mixing  proportions,  7r(q(n+l\u00bb),  required  to  determine \nthe  new  weights  w~~) are  computed from  the  newly  available  density  components \nusing  the following  estimator \n\n- t  \n\n- t  \n\n7r(  (n+l\u00bb)  =  \"\" \n\ngi \n\np(z \u00b7lq(n\u00bb)7r(q(n\u00bb) \n\n-)  -i \n\n~ ~  (I (n\u00bb)  (  (n\u00bb) \nJEN.  L.JkEIP  ~j gk \n\n-i \n7r  gk \n\n(6) \n\nIn  order  to  proceed  with  the  development  of  a  spline  fitting  process  we  require \na  model  for  the  mixture  components,  i.e.  p(~ilg;n\u00bb).  Here  we  assume  that  the \nrequired  model  can  be  specified  in  terms  of  Gaussian  distribution  functions. \nIn \nother words,  we  confine our attention to Gaussian mixtures.  The physical variable \nof these distributions is the squared error residual for  the position prediction of the \nith datum delivered by the jth spline.  Accordingly we  write \n\np(~ilg;n\u00bb) = a exp [-,8 (ri,j - F(Si,j, g;n\u00bb)) 2] \n\n(7) \n\nwhere ,8  is  the  inverse  variance  of the fit  residuals.  Rather than  estimating ,8,  we \nuse  it in  the spirit of a  control variable to regulate the effect  of fit  residuals. \n\nEquations (5),  (6)  and (11)  therefore specify a recursive procedure that iterates the \nweighted  residuals  to  compute  a  new  mixing  proportions  based  on  the  quality  of \nthe spline fit. \n\n4  Maximisation \n\nThe maximisation step aims  to optimize the  quantity  Q(~(n+l)I~(n\u00bb) with  respect \nto  the  spline  parameters.  Formally  this  corresponds  to  finding  the  set  of spline \nparameters which satisfy the condition \n\n(8) \n\n\f884 \n\n1. A.  F. Leite and E.  R.  Hancock \n\nWe find a local approximation to this condition by solving the following set of linear \nequations \n\n8Q( <)(n+l) I<)(n\u00bb) \n\n8(qf)(n+l) \n\n=  0 \n\n(9) \n\nfor  each  spline  parameter  (qf)(n+l)  in  turn,  i.e.  for  k=O,1,2,3.  Recovery  of the \nsplines is most conveniently expressed in terms of the following  matrix equation for \nthe components of the parameter-vector q(n) \n\n- t  \n\n(10) \n\nThe elements  of the vector  x(n)  are weighted  cross-moments  between  the parallel \nand perpendicular spline distances in  the Gaussian window,  i.e. \n\nx(n) = \n\n- t  \n\n(11) \n\nThe  elements  of the  matrix A~n), on  the  other hand,  are weighted moments  com(cid:173)\nputed purely in terms of the parallel distance Si ,j.  If k and I are the row and column \nindices,  then the (k, l)th element  of the matrix A~n) is \n[A(n)]k I  =  \"  w(n) sk+I-2 \n\n(12) \n\nt \n\n, \n\nt,) \n\nt,l \n\nL \nJENi \n\n5  Experiments \n\nWe  have  evaluated  our  iterative  spline  fitting  algorithm  on  the  detection  of line(cid:173)\nfeatures in aerial infra-red images.  Figure 1a shows the original picture.  The initial \n71\" { q~O\u00bb))  assigned  according  to  equation  (1)  are  shown \nfeature  probabilities  (i.e. \nin  Figure  lb.  Figure  1c  shows  the  final  contour-map  after  the  EM algorithm  has \nconverged.  Notice  that  the  line  contours  exhibit  good  connectivity  and  that  the \njunctions are well  reconstructed.  We  have  highlighted  a  subregion  of the  original \nimage.  There  are  two  features  in  this  subregion  to  which  we  would  like  to draw \nattention.  The  first  of  these  is  centred  on  the  junction  structure.  The  second \nfeature is  a neighbouring point  on  the descending branch of the road. \n\nFigure  2 shows  the  iterative evolution  of the  cubic  splines  at  these  two  locations. \nThe spline shown in  Figure 2a adjusts to fit  the upper pair of road segments.  Notice \nalso  that  although  initially  displaced,  the  final  spline  passes  directly  through  the \njunction.  In  the case  of the descending road-branch the spline shown  in  Figure 2b \nrecovers from an initially poor orientation estimate to align itself with the underlying \nroad  structure.  Figure  2c  shows  how  the  spline  probabilities  (i.e.  7I\"(q~n\u00bb))  evolve \nwith  iteration  number.  Initially,  the  neighbourhood  is  highly  ambiguous.  Many \nneighbouring splines  compete to account for  the local image structure.  As  a  result \nthe detected junction is several pixels wide.  However, as the fitting process iterates, \nthe splines move from the inconsistent initial estimate to give a good local estimate \nwhich is  less ambiguous.  In other words the two splines illustrated in  Figure 2 have \nsuccessfully arranged themselfs to account for  the junction structure. \n\n\fContour Organisation with the EM Algorithm \n\n885 \n\n(a)  Original  image. \n\n(b)  Probability  map. \n\n(c)  Line  map. \n\nFigure 1:  Infra-red aerial picture with corresponding probability map showing region \ncontaining pixel  under study and correspondent line  map. \n\n6  Conclusions \n\nWe  have demonstrated how the process of parallel iterative contour refinement can \nbe  realised  using  the  classical  EM  algorithm  of  Dempster,  Laird  and  Rubin  [1]. \nThe refinement of curves by relaxation operations has been a  preoccupation in  the \nliterature since  the  seminal  work  of Rosenfeld,  Hummel  and  Zucker  [9].  However, \nit  is  only  recently  that  successful  algorithms  have  been  developed  by  appealing \nto  more  sophisticated  modelling  methodologies  [13,  2].  Our  EM  approach  not \nonly  delivers  comparable  performance,  it  does  so  using  a  very  simple  underlying \nmodel.  Moreover, it allows the contour re-enforcement process to be understood in \na  weighted  least-squares optimisation framework  which  has  many features  in  com(cid:173)\nmon  with  snake  dynamics  [11]  without  being sensitive  on  the initial  positioning of \ncontrol  points.  Viewed  from  the perspective  of classical  relaxation  labelling  [9,  4], \nthe  EM  framework  provides  a  natural  way  of evaluating  support  beyond  the  im(cid:173)\nmediate object neighbourhood.  Moreover,  the framework  for  spline fitting  in  2D  is \nreadily  extendible  to the reconstruction of surface patches in  3D  [10]. \n\nReferences \n[1]  Dempster  A.,  Laird  N.  and  Rubin  D.,  \"Maximum-likelihood  from  incomplete  data \nvia the EM algorithm\", J. Royal Statistical Soc.  Ser.  B  (methodological) ,39,  pp 1-38, \n1977. \n\n[2]  Hancock  E.R  and  Kittler  J.,  \"Edge  Labelling  using  Dictionary-based  Probabilistic \n\nRelaxation\",  IEEE PAMI,  12,  pp.  161-185,  1990. \n\n[3]  Jordan  M.1.  and  Jacobs  RA,  \"Hierarchical  Mixtures  of  Experts  and  the  EM  Al(cid:173)\n\ngorithm\",  Neural  Computation,  6,  pp. 181-214,  1994. \n\n[4]  Kittler  J.  and  Hancock,  E.R,  \"Combining  Evidence  in  Probabilistic  relaxation\", \nInternational  Journal  of  Pattern  Recognition  and  Artificial  Intelligence,  3,  N1,  pp \n29-51,  1989. \n\n[5]  Leite J.A.F. and Hancock, E.R,  \" Statistically Combining and Refining Multichannel \nInformation\" ,  Progress  in Image  Analysis  and  Processing  III:  Edited  by  S  Impedovo, \nWorld  Scientific,  pp . 193-200,  1994. \n\n[6]  Leite  J.A.F.  and  Hancock,  E .R,  \"Iterative  curve  organisation  with  the  EM  al(cid:173)\n\ngorithm\",  to  appear  in  Pattern  Recognition  Letters,  1997. \n\n\f886 \n\n1. A.  F.  Leite and E.  R.  Hancock \n\n(ll' \n\nIili> \n\n\", \n\n\"  ~ + \n:--I-\n~ / '  \\ ~ \n(vi>  + + ~ \n:f\\ ~ ~ \n\"  :+ :+ :+ \n~ I ~''''  ~ \n:+ \n~ (Ie) \nI\u00bb  ~ \n\nx  ~ ,l) \n\n\"\"\"X  (Ix) \n\n\" \n\nMI) \n\n, ) \n\nx \n\nIv) \n\n, \n\n(YIIII \n\n(j) \n\nso \n\nhy) \n\n(i!) \n\n(v) \n\nI,.) \n\n(ril) \n\n<tii> \n\n(i,) \n\n(a) \n\n(b) \n\n(c) \n\nFigure  2:  Evolution  of the  spline  in  the  fitting  process.  The  image  in  (a)  is  the \njunction spline while the image in  (b)  is  the branch spline.  The first  spline is  shown \nin  (i), and the subsequent ones from  (ii)  to (xi).  The evolution of the corresponding \nspline probabilities is  shown in  (c). \n\n[7]  Meer  P.,  Mintz  D.,  Rosenfeld  A .  and  Kim  D.Y.,  \"Robust  Regression  Methods  for \nComputer  Vision  - A  Review\",  International  Journal  of  Computer  vision,  6,  pp. \n59- 70,  1991. \n\n[8]  Peleg  S.  and  Rosenfeld  A\" \n\n\"Determining  Compatibility  Coefficients  for  curve  en(cid:173)\n\nhancement relaxation  processes\",  IEEE SMC,  8, pp.  548-555,  1978. \n\n[9]  Rosenfeld  A.,  Hummel R.A.  and Zucker  S.W.,  \"Scene  labelling  by relaxation  opera(cid:173)\n\ntions\",  IEEE Transactions  SMC,  SMC-6,  pp400-433,  1976. \n\n[10]  Sander P.T.  and Zucker S.W .,  \"Inferring surface  structure and differential structure \n\nfrom  3D images\" , IEEE PAMI,  12,  pp 833-854,  1990. \n\n[11]  Terzopoulos D. ,  \"Regularisation of inverse problems involving discontinuities\" , IEEE \n\nPAMI,  8,  pp  129-139,  1986. \n\n[12]  Zucker, S.W., Hummel R.A., and Rosenfeld A.,  \"An application ofrelaxation labelling \n\nto line and curve enhancement\",  IEEE  TC, C-26,  pp.  394-403,  1977. \n\n[13]  Zucker  S. ,  David  C.,  Dobbins  A.  and  Iverson  L.,  \"The  organisation  of  curve  qe(cid:173)\n\ntection:  coarse  tangent  fields  and  fine  spline  coverings\" ,  Proceedings  of the  Second \nInternational  Conference  on  Computer  Vision,  pp.  577-586,  1988. \n\n\f", "award": [], "sourceid": 1291, "authors": [{"given_name": "Jos\u00e9", "family_name": "Leite", "institution": null}, {"given_name": "Edwin", "family_name": "Hancock", "institution": null}]}