{"title": "Grouping Contours by Iterated Pairing Network", "book": "Advances in Neural Information Processing Systems", "page_first": 335, "page_last": 341, "abstract": "", "full_text": "Grouping Contours by Iterated Pairing Network \n\nAmnon Shashua \nM.I.T. Artificial Intelligence Lab., NE43-737 \nand Department of Brain and Cognitive Science \nCambridge, MA 02139 \n\nAbstract \n\nShimon Ullman \n\nWe describe in this paper a network that performs grouping of image con(cid:173)\ntours. The input to the net are fragments of image contours, and the \noutput is the partitioning of the fragments into groups, together with a \nsaliency measure for each group. The grouping is based on a measure of \noverall length and curvature. The network decomposes the overall opti(cid:173)\nmization problem into independent optimal pairing problems performed \nat each node. The resulting computation maps into a uniform locally \nconnected network of simple computing elements. \n\n1 The Problenl: Contour Grouping \n\nA problem that often arises in visual information processing is the linking of con(cid:173)\ntour fragments into optimal groups. For example, certain subsets of contours spon(cid:173)\ntaneously form perceptual groups, as illustrated in Fig. 1, and are often detected \nimmediately without scanning the image in a systematic manner. Grouping process \nof this type are likely to play an important role in object recognition by segmenting \nthe image and selecting image structures that are likely to correspond to objects of \ninterest in the scene. \n\n'Ve propose that some form of autonomous grouping is performed at an early stage \nbased on geometrical characteristics, that are independent of the identity of objects \nto be selected. The grouping process is governed by the notion of saliency in a way \nthat priority is given to forming salient groups at the expense of potentially less \nsalient ones. This general notion can again be illustrated by Fig. 1; it appears that \ncertain groups spontaneously emerge, while grouping decisions concerning the less \nsalient parts of the image may remain unresolved. As we shall see, the computation \nbelow exhibits a similar behavior. \n\nWe define a grouping of the image contours as the formation of a set of disjoint \n\n335 \n\n\f336 \n\nShashua and Ullman \n\n\" \n\n'....,-- ... .,.,\"''''4.-.... \n\n~ \\..-\n...... ,' \",-:'--.., ... ' \nI':: \n\n.... :. '\"/...,,) \n\n'''\" '- '\".-1 ;' 'I )_' \n'>,,\" I'........ I'~ '\\ \nI \n\\. ..... ' \n' \n\n' \nA \n\nI \n\nI \n\nl \n\n.,.; \" ' : ; , \" \" ' \" \n\n.... \n~\"\"\\ \n_I \n\n\" ..... ' \n-\",(, - ~ ,~ \" ~/ -\nI \n... \" \n-\n)' , \n\" \nt:' \n~ .''''' \n.... I ' \"\"J I \n... \n,~.... \nr\" / f \nI \n-\nh \nll..' ... :\" \n\n.... ' \n\n\" \n\" \n\\ \\ , -\n.... \n\n\\ \n\n.......... \n\n\" \n,,(I \n~ .... I \n\nJI\" \n\n'-.... _ ~ \nI,. .... ,~,,,.' '''} ... - 1 .... \"( ,,1-\\ \n.....)' \n... / 1 \n- I .... \n\n~ ..... , \n~ \nI \n\n......... </~\" \\ \n\n::;.!\\ \n\n\\ _\". \n\n~\\,.o 1 \" , ,- ,'04.\"'_ \n\n\\ . , \n\nI\"-\n\npi \n\n-\n\n..r \n\\ ,\" \n..... ', ~ \nI \n.... \n,( f'. \n\n\" \n\nI \n\n\",' \nI \n\nt \n\n\\ \n\nI \n\n, \n\nI ' \n\n, \n1 \" . - \" \n\"... \nI \n\n\". \n... \n., \n\n,\\ \n\nI \n\n.... , \\ \n\n\\ \n\n-, (. \n\n\\ \" _ ' , . \n\n., \n\n\" \n\n' \n\n\\ \n\n} . . . . \n\n.. .... ,-,' , \" -\n~ ~ \n,.... \n\n-\n\n;' I \n\n\"' .... ;' \"-t\"'-\n\n,,~, ~ \n\n0('/' ,~ \n\nI \n\nf ~_ ;' \n\n,_ ;' -\n\n-\n\n, ' \\ ; ' I \n\nFigure 1: Contours that spontaneously form perceptual groups with various degrees of \nsaliency. On the left is an edge image of a plane surrounded by a car, a house, trees \nand texture. The image on t.he right contains three circles, having decreasing degrees of \nsaliency, in a background of randomly placed and oriented segments. \n\ngroups, each corresponding to a curve that may have any number of gaps, and \nwhose union covers all the contour fragments in the image. Given a function F(A) \nthat measures some desired property of a group A, we would like to find a disjoint set \nof groups {AI, ... , Am} that maximizes 2:; F(Ad over all possible groupings. Our \ndefinition of the problem is related to, but not identical with, problems studied in \nthe past under headings of \"perceptual organization\" , \"segmentation\" , \"cueing\" and \n\"figure-ground separat.ion\". In our definition of grouping, local grouping decisions \nbased on collinearity of neighboring edge segments may be overridden in favor of \nmore global decisions that are governed by the overall saliency of the groups. \n\nThe paper introduces a novel grouping method having the following properties: (i) \nthe grouping optimizes an overall saliency measure, (ii) the optimization problem \nis mapped onto a uniform locally connected network of simple computing elements, \nand (iii) the network's architecture and its computation are different in several \nrespects from traditional neural network models. \n\n2 Optimal Grouping \n\nFor the purpose of grouping it is convenient to consider the image as a graph of \nedge elements. The vertices of the graph correspond to image pixels, and the arcs to \nelementary edge fragments. The input to the grouping problem is a contour image, \nrepresented by a su bset E r of the elements in the graph. A path in the graph \ncorresponds to a contour in the image having any number of gaps. This implies \nthat the grouping process implicitly bridges across gaps. This filling-in process is \ncritical to any grouping scheme as demonst.rated by the circles in Fig. 1. \n\nThe emphasis in this paper is on 1-D chains of elements such as objects' bounding \ncontours. Grouping is therefore a collections of chains of AI, ... , Am such that Ai n \nAj = 0 i;/; j and Uj Ai 2 Er. To define an optimal grouping we will define a \nfunction F(A) that measures the quality of a group A. An optimal grouping is then \na grouping that maximizes L~I F(Ai) over all possible groupings of the elements. \n\n\fGrouping Contours by Iterated Pairing Network \n\n337 \n\n2.1 The Quality Measure of a Group, F(A) \n\nThe definition of the measure F(A) is motivated by both perceptual and compu(cid:173)\ntational considerations. In agreement with perceptual observations, it is defined \nto favor long smooth contours. Its form is also designed to facilitate distributed \nmultistage optimization, as discussed below. \nTo define F(A) of a chain of elements A = {e1' ... , em}, consider first a single element \nei, and the n preceding elements in the chain. We use first a quantity Sn (i) which \nis the contribution of the n preceding elements to ei, which is: \n\ni \n\nj=max{1.i-n} \n\nUj is defined as 1 when ej corresponds to a contour fragment in the image and 0 for \ngaps. sn(i) is therefore simply a weighted sum of the contributions of the elements \nin the chain. The weighting factor Gij is taken to be a decreasing function of the \ntotal curvature of the pa.th lij between elements ei and ej. This will lead to a \ngrouping that prefers curves with small overall curvature over wiggly ones. Gij is \ngiven by the formula: \n\nGij = e -\n\nJ (dB)2 d \n\n-raj Tr \n\n6 \n\nThe exponent is the squared total curvature of the path between elements ei and ej, \nand the resulting Gij lies between 0 (highly curved contour) and 1 (straight line). \nFor a discrete sampling of the curve, Gij can be approximated by the product: \n\nj+l \n\nGij = II !p.p-1 \n\np=i \n\nwhere !P.q is referred to as the coupling constant between adjacent elements ep and \neq and is given by !P.q = e- atan ~ where (l' is the angle measuring the orientation \nIn a similar manner, one can define Sn(i) , the \ndifference between p and q [3]. \ncontribution of the n elements following ei in the chain. Sn(i) = sn(i) + sn(i) -\nUi measures the contribution to element ei from both direction. This increases \nmonotonically with the length and low total curvature of the curve passing through \nelement ei. Then then the overall quality of the chain A is finally given by \n\nm \n\nFn(A) = L Sn(i) \n\ni=1 \n\nFn(A) increases quadratica.lly with the size of A and is non-linear with respect to \nl\\'Iaximizing 2: F(Aj) over all possible groupings will, \nthe total curvature of A. \ntherefore, prefer groups that are long and smooth. As n increases, the measure Fn \nwill depend on larger portions of the curve surrounding each element, resulting in \na finer discrimination between groups. In practice, we limit the measure to a finite \nn, and the optimal grouping is defined as: \n\nIn = arg max L Fn(Ai) \n\nm \n\nm.Al \u2022...\u2022 Am . \n\n1=1 \n\nwhere the max is taken over all possible groupings. That is, we are looking for \na grouping that will maximize the overall criterion function based on length and \nsmoothness. \n\n\f338 \n\nShashua and Ullman \n\n3 The Optimization Approach \n\nOptimizing In is a nonlinear problem with an energy landscape that can be quite \ncomplex making it difficult to find a global optimum, or even good local optima, \nusing straightforward gradient descent methods. We define below a computation \nthat proceeds in two stages, saliency and pairing stages, of n steps each. In the \nsaliency stage we compute, by iterating a local computation, optimal values of \nSn(i) for all elements in the graph. These values are an upper-bound on the saliency \nvalues achievable by any grouping. In the pairing stage we further update Sn(i) by \nrepeatedly forming local pairings of elements at each node of the graph. The details \nof both stages are given below. \n\n3.1 Saliency Stage \n\nFor any given grouping AI, ... , Am, because they are disjoint, we have that \n\nL Fn (Aj) = L Sn (i) ~ L max Sn (i) \n\n'Y. \n\nm \n\n. \n3=1 \n\nN \n\n. \n1=1 \n\nwhere N is the number of elements in the graph and \"fi is a curve passing through \nelement ei. We denote Sn(i) to be the saliency of element ei with respect to a curve \n\"fi. We therefore have that the maximal saliency value S~ (i) = max'Yi Sn (i) is an \nupper-bound on the saliency value element ei receives on the optimal grouping In. \n\nWe define a local computation on the grid of elements such that each element ei \ncomputes maximal Sn(i) by iterating the following simple computation, at each \nstep taking the maximal contribution of its neighbors. \n\nsoC i) = (fa \n\nSn+l (i) = (fa + m~x Sn (j)fij \n\n3 \n\n(1) \n\nwhere this computation is performed by all elements in parallel. It can be shown \nthat at the n'th iteration Sn (i) is maximal over all possible curves of length n, \nhaving any number of gaps, that come into ei. Since Sn (i) = Sn (i) + sn (i) - (fi, we \nhave found the maximal Sn (i) as well. For further details on the properties of this \ncomputation, see [3]. Note that since the computation is carried by all elements of \nthe net, including gaps ((f equals 0), the gaps are filled-in as a by-product of the \ncomputation. One can show that the filling-in contour between two end-elements \nhas the smallest overall curva.ture, and therefore has the shape of a cubic spline. \n\n3.2 Pairing Stage \nGiven the optimal saliency values S~( i) computed at the saliency stage we would like \nnext to find a near-optimal grouping In. We first note the one-to-one correspondence \nbetween a grouping and a pairing of elements at each node of the 1raph. We define \na pairing to be a partition of the k elements around node Pinto r '21 disjoint pairs. \nA pairing performed over all nodes of the net creates an equivalence relation over \nthe elements of the net and therefore, by transitivity, determines a grouping. We \ntherefore proceed by selecting a pairing at each node of the net that will yield a \nnear optimal grouping In. \nGiven sn(i), the optimal saliency values computed by (1), and a pairing at node P \n\n\fGrouping Contours by Iterated Pairing Network \n\n339 \n\nwe update the saliency values by \n\nsn+l(i) = (fi + Sn(j)iij \n\n(2) \nwhere ei and ej are pairs determined by the pairing. This computation is exactly \nlike (1) with the exception that (2) is applied to a fixed pairing while in (1) each \nelement selects the neighbor with maximal contribution. Further applications of \npairing followed by (2) allows the result of pairing decisions to propagate along \ncurves and influence other pairing decisions. This gives rise to the notion of iterated \npairings, a repetitive pairing procedure applied simultaneously over all nodes of the \ngraph followed by saliency computation (2). We define below a pairing procedure \nthat identifies salient groups in contour images. \n\nFor every node P in the graph with elements el, ... , ek coming into P, we have that \ni = 1, ... , k computed by (1) are measured along optimal, not necessarily \nsn(i) \ndisjoint, curves AI, ... , Ak of length n each. An optimal pairing at node P is defined \nas a disjoint pairing that concatenates AI, ... , Ak into r ~ 1 curves such that the \nsum of their quality measure Fe) is maximal. Because F is defined to prefer \nsmooth curves and because of its non-linearity with respect to total curvature, an \noptimal pairing agrees with the notion of forming salient groups on the expense \nof potentially less salient ones. The following proposition shows that an optimal \npairing can be determined locally without the need to evaluate the quality measure \nof the concatenated curves. \n\nProposition 1 For a given node P, let el, ... ,ek be the elements around P, \nAI, ... , Ak be curves cowing into P that are associated with the non-zero saliency \nvalues Sl(n), ... , Sk(n) with sufficiently large n (at least twice the largest chain Ai), \n7r be a permutation of the indices (1, ... , k) and J = {(I, 2), (3,4), ... , (k-1, k)}, then \n\nargmax ~ Fn (A 11' A1I' 0) =argmax \" \n\n7r \n\nL-,; \n(i,j)E J \n\nl \n\nJ \n\n7r \n\nL..t \n(i,j)E J \n\nW7f 0 7f \n\n\" J \n\nwhere AiAj stands \niij (sn(i)cn(j) + sn(j)cn(i)) where Cn is defined as cn(i) = Lk Ckj where k is taken \nover all elements in the chain A j\n\nthe concatenation of curves Ai, A j , and Wij \n\nfor \n\n. \n\nProof: This is merely a calculation. Fn(AiAj), the measure of group-saliency of the \nchain AiAj, is equal to Fn(Ai) + Fn(Aj) + Wij. Finally, without loss of generality, \nwe can assume that k is even, because we can always add another element with zero \nweights attached to it. 0 \nProposition 1 shows that an optimal pairing of elements can be determined locally on \nthe basis of the saliency values computed in (1). One way to proceed is therefore the \nfollowing. The quantities Cn and therefore Wij can be accumulated and computed \nduring computation (1). Then, the optimal pairing is computed at every node. \nFinding an optimal pairing is equivalent to finding an optimal weighted match in a \ngeneral graph [2], with weights Wij. The weighted matching problem on graphs has \na polynomial algorithm due to Edmonds [1] and therefore its implementation is not \nunwieldly. \n\nBelow we describe an alternative and more biologically plausible scheme that can \nbe implemented in a simple network using iterative local computations . The com(cid:173)\nputation is in fact almost identical to the saliency computation described in (1). \n\n\f340 \n\nShashua and Ullman \n\nSince the saliency values Sn computed by (1) are an upper-bound on the final values \nachievable in any grouping, we would like to find a pairing that will preserve these \nvalues as closely as possible. Suppose that at P, ei receives its maximal contri(cid:173)\nbution from ej, and at the same time ej provides the maximal contribution to ej \n('mutual neighbors'). When performing local pairing at P, it is reasonable to select \nei and ei as a pair. Note that although this is a local decision at P, the values \nsn(i) and snU) already take into account the contribution of extended curves. The \nremaining elements undergo another round of saliency selection and pairing of mu(cid:173)\ntual neighbors, until all elements at P are paired. The following proposition shows \nthat this pairing process is well behaved in the sense that at each selection round \nthere will always be at least one pair that mutually select each other. We therefore \nhave that the number of selection rounds is bounded by r ~ 1, where k is the number \nof elements having non-zero saliency value coming into node P . \nproposition 2 Let Xl, ... , Xk be k positive real numbers, wii = Wii be positive weights \ni, j = 1, ... , k and bi = al'g maxj XjWij, then 3i, j such that bi = j and bi = i (i and \nj are mutual neighbors). \nProof: by induction on k. For k = 3 assume there exists a cycle in the selection \npattern. For any given cycle we can renumber the indecis such that bl = 2, b2 = 3 \nand b3 = 1. Let Wi stand for wi-l ,i where WI = Wk,l. We get (i) X2W2 > X3Wl, (ii) \nX2W3 > XlW2 and (iii) XIWI > X2W3. From (ii) and (iii) we get an inequality that \ncontradicts (i). For the induction hypothesis, assume the claim holds for arbitrary \nk - 1. We must show that the claim holds for k . Given the induction hypothesis \nwe must show that there is no selection pattern that will give rise to a cycle of \nsize k. Assume in contradiction that such a cycle exists. For any given cycle of \nsize k we can renumber t.he indecis such that bi = i + 1 and bk = 1 which implies \nthat XiWi > XjWij for all j =/; i. In particular we have the following k inequalities: \nXiWi > Xi-2Wi-l where i = 1, ... , k. From the k - 1 inequalities corresponding \nto i = 2, ... , k we get, by transitivity, that XIWI < Xk-lWk which contradicts the \nremaining inequality that corresponds to i = 1. 0 \n3.3 Summary of Computation \n\nThe optimization is ma.pped onto a locally connected network with a simple uniform \ncomputation. The computation consists of the following steps. (i) Compute the \nsaliency S~ of each line element using the computation defined in (1). (ii) At each \nnode perform a pairing of the line elements at the node. The pairing is performed \nby repeatedly selecting mutual neighbors. (iii) Update at each node the values Sn \nbased on the newly formed pa.iring (eq. 2). (iv) Go back to step 2. \n\nThese iterated pairings allow pairing decisions to propagate along maximally salient \ncurves and influence other pairing decisions . In the implementation, the number of \niterations n is equal in both stages and as n increases, the finer the pairing would \nbe, resulting in a finer discrimination between groups. During the computation, the \nmore salient groups emerge first, the less salient groups require additional iterations. \nAlthough the process is not guaranteed to converge to an optimal solution, it is a \nvery simple computation that yields in practice good results. Some examples are \nshown in the next section. \n\n\fGrouping Contours by Iterated Pairing Network \n\n341 \n\no \n\nFigure 2: Results after 30 iterations of saliency and pairing on a net of size 128 X 128 with \n16 elements per node. Images from left to right display the saliency map following the \nsaliency and pairing stages and a number of strongest groups. The saliency of elements in \nthe display is represented in terms of brightness and width -\nincreased saliency measure \ncorresponds to increase in brightness and width of element in display. \n\n3.4 Examples \n\nFig. 2 shows the results of the network applied to the images in Fig. 1. The \nsaliency values following the saliency and pairing stages illustrate that perceptually \nsalient curves are also associated with high saliency values (see also [3]). Finally, \nin these examples, the highest saliency value of each group has been propagated \nalong all elements of the group such that each group is now associated with a single \nsaliency value. A number of strongest groups has been pulled out showing the close \ncorrespondence of these groups to objects of interest in the images. \n\nAcknowledgments \n\nThis work was supported by NSF grant IRI-8900267. Part of the work was done \nwhile A.S. was visiting the exploratory vision group at IBM research center, York(cid:173)\ntown Heights. \n\nReferences \n\n[1] J. Edmonds. Path trees and flowers. Can. J. Math., 1:263-271, 1965. \n[2] C.H. Papadimitriou and K. Steiglitz. Combinatorial Optimization: Algorithms \n\nand Complexity. Prentice-Hall, New Jersey, 1982. \n\n[3] A. Shashua and S. Ullman. Structural saliency: The detection of globally salient \n\nstructures using a locally connected network. In Proceedings of the 2nd Inter(cid:173)\nnational Conference on Computer Vision, pages 321-327, 1988. \n\n\f", "award": [], "sourceid": 438, "authors": [{"given_name": "Amnon", "family_name": "Shashua", "institution": null}, {"given_name": "Shimon", "family_name": "Ullman", "institution": null}]}