{"title": "Graph Matching with Hierarchical Discrete Relaxation", "book": "Advances in Neural Information Processing Systems", "page_first": 689, "page_last": 695, "abstract": "", "full_text": "Graph Matching with Hierarchical \n\nDiscrete Relaxation \n\nRichard C. Wilson and Edwin R. Hancock \n\nDepartment of Computer Science, University of York \n\nYork, YOl 5DD, UK. \n\nAbstract \n\nOur aim in this paper is to develop a Bayesian framework for match(cid:173)\ning hierarchical relational models. The goal is to make discrete la(cid:173)\nbel assignments so as to optimise a global cost function that draws \ninformation concerning the consistency of match from different lev(cid:173)\nels of the hierarchy. Our Bayesian development naturally distin(cid:173)\nguishes between intra-level and inter-level constraints. This allows \nthe impact of reassigning a match to be assessed not only at its \nown (or peer) level ofrepresentation, but also upon its parents and \nchildren in the hierarchy. \n\nIntrod uction \n\n1 \nHierarchical graphical structures are of critical importance in the interpretation of \nsensory or perceptual data. For instance, following the influential work of Marr [6] \nthere has been sustained efforts at effectively organising and processing hierarchical \ninformation in vision systems. There are a plethora of concrete examples which in(cid:173)\nclude pyramidal hierarchies [3] that are concerned with multi-resolution information \nprocessing and conceptual hierarchies [4] which are concerned with processing at \ndifferent levels of abstraction. Key to the development of techniques for hierarchical \ninformation processing is the desire to exploit not only the intra-level constraints \napplying at the individual levels of representation but also inter-level constraints \noperating between different levels of the hierarchy. If used effectively these inter(cid:173)\nlevel constraints can be brought to bear on the interpretation of uncertain image \nentities in such a way as to improve the fidelity of interpretation achieved by single \nlevel means. Viewed as an additional information source, inter-level constraints can \nbe used to resolve ambiguities that would persist if single-level constraints alone \nwere used. \n\n\f690 \n\nR. C. Wilson and E. R. Hancock \n\nIn the connectionist literature graphical structures have been widely used to repre(cid:173)\nsent probabilistic causation in hierarchical systems [5, 9]. Although this literature \nhas provided a powerful battery of techniques, they have proved to be of limited use \nin practical sensory processing systems. The main reason for this is that the under(cid:173)\npinning independence assumptions and the resulting restrictions on graph topology \nare rarely realised in practice. In particular there are severe technical problems in \ndealing with structures that contain loops or are not tree-like. One way to overcome \nthis difficulty is to edit intractable structures to produce tractable ones [8]. \n\nOur aim in this paper is to extend this discrete relaxation framework to hierarchi(cid:173)\ncal graphical structures. We develop a label-error process to model the violation of \nboth inter-level and intra-level constraints. These two sets of constraints have dis(cid:173)\ntinct probability distributions. Because we are concerned with directly comparing \nthe topology graphical structures rather than propagating causation, the result(cid:173)\ning framework is not restricted by the topology of the hierarchy. In ,particular, \nwe illustrate the effectiveness of the method on amoral graphs used to represent \nscene-structure in an image interpretation problem. This is a heterogeneous struc(cid:173)\nture [2, 4] in which different label types and different classes of constraint operate \nat different levels of abstraction. This is to be contrasted with the more familiar \npyramidal hierarchy which is effectively homogeneous [1, 3]. Since we are deal(cid:173)\ning with discrete entities inter-level information communication is via a symbolic \ninterpretation of the objects under consideration. \n2 Hierarchical Consistency \nThe hierarchy consists of a number of levels, each containing objects which are \nfully described by their children at the level below. Formally each level is described \nby an attributed relational graph GI = (Vi, EI, Xl), Vi E L, with L being the \nindex-set of levels in the hierarchy; the indices t and b are used to denote the \ntop and bottom levels of the hierarchy respectively. According to our notation for \nlevel i of the hierarchy, Vi is the set of nodes, EI is the set of intra-level edges \nand Xl = {~~, Vu E Vi} is a set of unary attributes residing on the nodes. The \nchildren or descendents which form the representation of an element j at a lower \nlevel are denoted by V j . In other words, if U l - I is in Vj then there is a link in the \nhierarchy between element j at level i and element u at level i-I. According to \nour assumptions, the elements of Vj are drawn exclusively from Vi-I. The goal of \nperforming relaxation operations is to find the match between scene graph G1 and \nmodel graph G2 \u2022 At each individual level of the hierarchy this match is represented \nby a mapping function p, Vi E L, where II: Vi -t Vi. \nThe development of a hierarchical consistency measure proceeds along a similar \nline to the Single-level work of Wilson and Hancock [10]. The quantity of interest is \nthe MAP estimate for the mapping function I given the available unary attributes, \ni.e. I = argmaxj P(jt, Vi E LIXI , Vi E L). We factorize the measurement infor(cid:173)\nmation over the set of nodes by application of Bayes rule under the assumption of \nmeasurement independence on the nodes. As a result \n\nP(/, Vi E L/Xl , Vi E L) = \n\n(Xl ~i E L) {II II p(X~ll(u))}P(fI, Vi E L) (1) \n\np , \n\nIELuEVI \n\nThe critical modelling ingredient in developing a discrete relaxation scheme from the \nabove MAP criterion is the joint prior for the mapping function, i.e. p(fl, Vi E L) \n\n\fGraph Matching with Hierarchical Discrete Relaxation \n\n691 \n\nParents \n\nChildren \n\nA \n\nB \n\nc \n\nPossible mappings or children: \n\n1,2,3 -- A,B,C \nC,B,A \n\nFigure 1: Example constrained children mappings \n\nwhich represents the influence of structural information on the matching process. \nThe joint measurement density, p(XI, 'VI E L), on the other hand is a fixed property \nof the hierarchy and can be eliminated from further consideration. \n\nRaw perceptual information resides on the lowest level of the hierarchy. Our task \nis to propagate this information upwards through the hierarchy. To commence our \ndevelopment, we assume that individual levels are conditionally dependent only on \nthe immediately adjacent descendant and ancestor levels. This assumption allows \nthe factorisation of the joint probability in a manner analogous to a Markov chain \n[3]. Since we wish to draw information from the bottom upwards, the factorisa(cid:173)\ntion commences from the highest level of labelling. The expression for the joint \nprobability of the hierarchical labelling is \n\np(fl, 'VI E L) = p(fb) II P(fl+Ill) \n\nIEL,I#t \n\n(2) \n\nWe can now focus our attention on the conditional probabilities P(fI+1lfl). These \nquantities express the probability of a labelling at the level I + 1 given the previously \ndefined labelling at the descendant level l. We develop tractable expressions for \nthese probabilities by decomposing the hierarchical graph into convenient structural \nunits. Here we build on the idea of decomposing Single-level graphs into super(cid:173)\ncliques that was successfully exploited in our previous work [10]. Super-cliques are \nthe sets of nodes connected to a centre-object by intra-level edges. However, in the \nhierarchical case the relational units are more complex since we must also consider \nthe graph-structure conveyed by inter-level edges. \n\nWe follow the philosophy adopted in the single-level case [10] by averaging the super(cid:173)\nclique probabilities to estimate the conditional matching probabilities P(fI+1lfl). \nIf r~ C fl denotes the current match of the super-clique centred on the object \nj E ~l then we write \n\nP(f'lfl-I) = ~l L p(r~lfl-l) \n\nI I jEV' \n\n(3) \n\nIn order to model this probability, we require a dictionary of constraint relations for \nthe corresponding graph sub-units (super-cliques) from the model graph G2 \u2022 The \nallowed mappings between the model graph and the data graph which preserve the \ntopology of the graph structure at a particular level of representation are referred \n\n\f692 \n\nR. C. Wilson and E R. Hancock \n\nto as \"structure preserving mappings\" or SPM's. It is important to note that we \nneed only explore those mappings which are topologically identical to the super(cid:173)\nclique centred on object j and therefore the possible mappings of the child nodes are \nheavily constrained by the mappings of their parents (Figure 1). We denote the set \nof SPM's by P. Since the set P is effectively the state-space of legal matching, we \ncan apply the Bayes theorem to compute the conditional super-clique probability \nin the following manner \n\np(r~I/I-l) = 2: p(r~15,/I-l)P(5Ii-l) \n\n(4) \n\nSEP \n\nAccording to this expression, there are two distinct components to our model. The \nfirst involves the comparison between our mapped realisation of the super-clique \nfrom graph G1 , i.e. q, with the selected unit from graph G 2 and the mapping \nfrom level 1 - 1. Here we take the view that once we have hypothesised a particular \nmapping 5 from P, the mapping P-l provides us with no further information, i.e. \np(r~ 15, /1-1) = p(r~ 15). The matched super-clique r~ is conditionally independent \ngiven a mapping from the set of SPM's and we may write the first term as p(r~15). \nIn other words, this first conditional probability models only intra-level constraints. \nThe second term is the significant one in evaluating the impact inter-level constraints \non the labelling at the previous level. In this term the probability of the hypothesised \nmapping 5 is conditioned according to the match of the child levell. \n\nAll that remains now is to evaluate the conditional probabilities. Under the as(cid:173)\nsumption of memoryless matching errors, the first term may be factorised over \nthe marginal probabilities for the assigned matches lIon the individual nodes of \nthe matched super-clique q given their counterparts Si belonging to the structure \npreserving mapping 5. In other words, \n\np(r;15) = II P('~lsi) \n\n1'! Ef~ \n\n(5) \n\nIn order to proceed we need to specify a probability distribution for the different \nmatching possibilities. There are three cases. Firstly, the match Ii may be to a \ndummy-node d inserted into q to raise it to the same size as 5 so as to facilitate \ncomparison. This process effectively models structural errors in the data-graph. \nThe second and third cases, relate to whether the match is correct or in error. \nAssuming that dummy node insertions may be made with probability Ps and that \nmatching errors occur with probability Pe , then we can write down the following \ndistribution rule \n\nif Ii = d or Si = d \n'f 1 \n1 Ii = Si \notherwise \n\n(6) \n\nThe second term in Equation (5) is more subtle; it represents the conditional prob(cid:173)\nability of the SPM 5 given a previously determined labelling at the level below. \nHowever, the mapping contains labels only from the current levell, not labels from \nlevel I - 1. We can reconcile this difference by noting that selection of a particular \nmapping at level I limits the number of consistent mappings allowed topologically \nat the level below. In other words if one node is mapped to another at level I, \n\n\fGraph Matching with Hierarchical Discrete Relaxation \n\n693 \n\nthe consistent interpretation is that the children of the nodes must match to each \nother. Provided that a set of mappings is available for the child-nodes, then this \nconstraint can be used to model P(SljI-1). The required child-node mappings are \nreferred to as \"Hierarchy Preserving Mappings\" or HPM's . It is these hierarchical \nmappings that lift the requirements for moralization in our matching scheme, since \nthey effectively encode potentially incestuous vertical relations. We will denote the \nset of HPM's for the descendants of the SPM S as Qs and a member of this set as \nQ = {qi, 'Vi E Vj}. Using this model the conditional probability P(SIfI-l) is given \nby \n\np(SIfI-1) = L P(SIQ,/I-1)P(Qll- 1 ) \n\nQEQs \n\n(7) \n\nFollowing our modelling of the intra-level probabilities, in this inter-level case as(cid:173)\nsume that S is conditionally independent of 11- 1 given Q, i.e. P(SIQ, / 1- 1) = \nP(SIQ)\u00b7 \n\nTraditionally, dictionary based hierarchical schemes have operated by using a la(cid:173)\nbelling determined at a preceding level to prune the dictionary set by elimination \nof vertically inconsistent items [4]. This approach can easily be incorporated into \nour scheme by setting P(QI/I-l) equal to unity for consistent items and to zero for \nthose which are inconsistent. However we propose a different approach; by adopt(cid:173)\ning the same kind of label distribution used in Equation 6 we can grade the SPM's \naccording to their consistency with the match at level 1 - 1, i.e. jI-l. The model \nis developed by factorising over the child nodes qi E Q in the following manner \n\nP(Qll- 1 ) = II P(qih,!-1) \n\nqiEQ \n\n(8) \n\nThe conditional probabilities are assigned by a re-application of the distribution \nrule given in Equation (6), i.e. \n\nif dummy node match \n'f \n1 qi = Ii \notherwise \n\n1-1 \n\n(9) \n\nFor the conditional probability of the SPM given the HPM Q, we adopt a simple \nuniform model under the assumption that all legitimate mappings are equivalent, \ni.e. P(SIQ) = P(S) = I~I' \nThe various simplifications can be assembled along the lines outlined in [10] to \ndevelop a discrete update rule for matching the two hierarchical structures. The \nMAP update decision depends only on the label configurations residing on levels \n1 - 1, 1 and 1 + 1 together with the measurements residing on levell. Specifically, \nthe level 1 matching configuration satisfies the condition \n\nII = argm!F{ II p(~~ljt(j)) }P(fI-llil )P(PI11+1 ) \n\nf \n\njEV! \n\n(10) \n\nHere consistency of match between levels land 1 - 1 of the hierarchy is gauged by \n\n\f694 \n\nR. C. Wilson and E. R. Hancock \n\nthe quantity \n\nPUl-llfl) = :1 L L K(rD \n\n1 iEVI SEP Qs \n\n1 \n\nexp \n\n[-(keH(rL S) + ks~(rL S))] \n\nL K(r!-l) exp \n\n[ - (keH (r~-l ,Q) + ks ~(r~-l, Q)) }11) \n\nQEQs \n\nIn the above expression H (r j, S) is the \"Hamming distance\" which counts the num(cid:173)\nber of label conflicts between the assigned match rj and the structure preserving \nmapping S. This quantity measures the consistency of the matched labels. The \nnumber of dummy nodes inserted into r j by the mapping S is denoted by ~ (r j, S). \nThis second quantity measures the structural compatibility of the two hierarchical \ngraphs. The exponential constants ke = In (l-PeMI-P,) and ks = In 1Ft, are re(cid:173)\nlated to the probabilities of structural errors and mis-assignment errors. Finally, \nK(rj) = (1- Pe ){1- Pe)lrjl is a normalisation constant. Finally, it is worth point(cid:173)\ning out that the discrete relaxation scheme of Equation (10) can be applied at any \nlevel in the hierarchy. In other words the process can be operated in top-down or \nbottom-up modes if required. \n3 Matching SAR Data \nIn our experimental evaluation of the discrete relaxation scheme we will focus on \nthe matching of perceptual groupings of line-segments in radar images. Here the \nmodel graph is elicited from a digital map for the same area as the radar image. The \nline tokens extracted from the radar data correspond to hedges in the landscape. \nThese are mapped as quadrilateral field boundaries in the cartographic model. To \nsupport this application, we develop a hierarchical matching scheme based on line(cid:173)\nsegments and corner groupings. The method used to extract these features from \nthe radar images is explained in detail in [10]. Straight line segments extracted \nfrom intensity ridges are organised into corner groupings. The intra-level graph \nis a constrained Delaunay triangulation of the line-segments. Inter-level relations \nrepresent the subsumption of the bottom-level line segments into corners. \n\nThe raw image data used in this study is shown in Figure 2a. The extracted line(cid:173)\nsegments are shown in Figure 2c. The map used for matching is shown in Figure \n2b. The experimental matching study is based on 95 linear segments in the SAR \ndata and 30 segments contained in the map. However only 23 of the SAR segments \nhave feasible matches within the map representation. Figure 2c shows the matches \nobtained by non-hierarchical means. The lines are coded as follows; the black lines \nare correct matches while the grey lines are matching errors. With the same coding \nscheme Figure 2d. shows the result obtained using the hierarchical method outlined \nin this paper. Comparing Figures 2c and 2d it is clear that the hierarchical method \nhas been effective at grouping significant line structure and excluding clutter. To \ngive some idea of relative performance merit, in the case of the non-hierarchical \nmethod, 20 of the 23 matchable segments are correctly identified with 75 incorrect \nmatches. Application of the hierarchical method gives 19 correct matches, only 17 \nresidual clutter segments with 59 nodes correctly labelled as clutter. \n4 Concl usions \nWe have developed graph matching technique which is tailored to hierarchical rela(cid:173)\ntional descriptions. The key element is this development is to quantify the match-\n\n\fGraph Matching with Hierarchical Discrete Relaxation \n\n695 \n\na) \n\nc) \n\n, \n\\ \n\" \n\n~\\ ' \\ \n\" \n\nb) \n\nd) \n\n-\n\n' \n\n\\ \n\"\\ \n\n-\n/\\{-\\'\\(\" \n\nFigure 2: Graph editing: a) Original image, b) Digital map, c) Non hierarchical \nmatch, d) Hierarchical match. \n\ning consistency using the concept of hierarchy preserving mappings between two \ngraphs. Central to the development of this novel technique is the idea of computing \nthe probability of a particular node match by drawing on the topologically allowed \nmappings of the child nodes in the hierarchy. Results on image data with lines and \ncorners as graph nodes reveal that the technique is capable of matching perceptual \ngroupings under moderate levels of corruption. \n\nReferences \n\n[1] F. Cohen and D. Cooper. Simple Parallel Hierarchical and Relaxation Algorithms for \nSegmenting Non-Causal Markovian Random Fields. IEEE PAMI, 9, 1987, pp.195-\n219. \n\n[2] L. Davis and T. Henderson. Hierarchical Constraint Processes for Shape Analysis. \n\nIEEE PAMI, 3, 1981, pp.265-277. \n\n[3] B. Gidas. A Renormalization Group Approach to Image Processing Problems. IEEE \n\nPAMI, 11, 1989, pp.164-180. \n\n[4] T . Henderson. Discrete Relaxation Techniques . Oxford University Press, 1990. \n[5] D.J. Spiegelhalter and S.L. Lauritzen, Sequential updating of conditional probabilities \n\non directed Graphical structures, Networks, 1990, 20, pp.579-605. \n\n[6] D. Marr, Vision. W.H. Freeman and Co., San Francisco. \n[7] J. Pearl, Probabilistic Reasoning in Intelligent Systems, Morgan Kaufmann, 1988. \n[8] M. Meila and M. Jordan, Optimal triangulation with continuous cost functions, Ad(cid:173)\n\nvances in Neural Information Processing Systems 9, to appear 1997. \n\n[9] P.Smyth, D. Heckerman, M.1. Jordan, Probabilistic independence networks for hidden \n\nMarkov probability models, Neural Computation, 9, 1997, pp. 227-269. \n\n[10] R.C . Wilson and E . R. Hancock, Structural Matching by Discrete Relaxation. IEEE \n\nPAMI, 19, 1997, pp.634- 648. \nIEEE PAMI, June 1997. \n\n\f", "award": [], "sourceid": 1454, "authors": [{"given_name": "Richard", "family_name": "Wilson", "institution": null}, {"given_name": "Edwin", "family_name": "Hancock", "institution": null}]}