{"title": "Multiclass Total Variation Clustering", "book": "Advances in Neural Information Processing Systems", "page_first": 1421, "page_last": 1429, "abstract": "Ideas from the image processing literature have recently motivated a new set of clustering algorithms that rely on the concept of total variation. While these algorithms perform well for bi-partitioning tasks, their recursive extensions yield unimpressive results for multiclass clustering tasks. This paper presents a general framework for multiclass total variation clustering that does not rely on recursion. The results greatly outperform previous total variation algorithms and compare well with state-of-the-art NMF approaches.", "full_text": "Multiclass Total Variation Clustering\n\nXavier Bresson\n\nUniversity of Lausanne\nLausanne, Switzerland\n\nxavier.bresson@unil.ch\n\nDavid Uminsky\n\nUniversity of San Francisco\nSan Francisco, CA 94117\nduminsky@usfca.edu\n\nThomas Laurent\n\nLoyola Marymount University\n\nLos Angeles, CA 90045\ntlaurent@lmu.edu\n\nJames H. von Brecht\n\nUniversity of California, Los Angeles\n\nLos Angeles, CA 90095\njub@math.ucla.edu\n\nAbstract\n\nIdeas from the image processing literature have recently motivated a new set of\nclustering algorithms that rely on the concept of total variation. While these al-\ngorithms perform well for bi-partitioning tasks, their recursive extensions yield\nunimpressive results for multiclass clustering tasks. This paper presents a general\nframework for multiclass total variation clustering that does not rely on recursion.\nThe results greatly outperform previous total variation algorithms and compare\nwell with state-of-the-art NMF approaches.\n\n1\n\nIntroduction\n\nMany clustering models rely on the minimization of an energy over possible partitions of the data\nset. These discrete optimizations usually pose NP-hard problems, however. A natural resolution\nof this issue involves relaxing the discrete minimization space into a continuous one to obtain an\neasier minimization procedure. Many current algorithms, such as spectral clustering methods or\nnon-negative matrix factorization (NMF) methods, follow this relaxation approach.\nA fundamental problem arises when using this approach, however; in general the solution of the\nrelaxed continuous problem and that of the discrete NP-hard problem can differ substantially. In\nother words, the relaxation is too loose. A tight relaxation, on the other hand, has a solution that\nclosely matches the solution of the original discrete NP-hard problem. Ideas from the image pro-\ncessing literature have recently motivated a new set of algorithms [17, 18, 11, 12, 4, 15, 3, 2, 13, 10]\nthat can obtain tighter relaxations than those used by NMF and spectral clustering. These new algo-\nrithms all rely on the concept of total variation. Total variation techniques promote the formation of\nsharp indicator functions in the continuous relaxation. These functions equal one on a subset of the\ngraph, zero elsewhere and exhibit a non-smooth jump between these two regions. In contrast to the\nrelaxations employed by spectral clustering and NMF, total variation techniques therefore lead to\nquasi-discrete solutions that closely resemble the discrete solution of the original NP-hard problem.\nThey provide a promising set of clustering tools for precisely this reason.\nPrevious total variation algorithms obtain excellent results for two class partitioning problems\n[18, 11, 12, 3] . Until now, total variation techniques have relied upon a recursive bi-partitioning\nprocedure to handle more than two classes. Unfortunately, these recursive extensions have yet to\nproduce state-of-the-art results. This paper presents a general framework for multiclass total varia-\ntion clustering that does not rely on a recursive procedure. Speci\ufb01cally, we introduce a new discrete\nmulticlass clustering model, its corresponding continuous relaxation and a new algorithm for opti-\nmizing the relaxation. Our approach also easily adapts to handle either unsupervised or transductive\n\n1\n\n\fclustering tasks. The results signi\ufb01cantly outperform previous total variation algorithms and com-\npare well against state-of-the-art approaches [19, 20, 1]. We name our approach Multiclass Total\nVariation clustering (MTV clustering).\n\n2 The Multiclass Balanced-Cut Model\nGiven a weighted graph G = (V, W ) we let V = {x1, . . . , xN} denote the vertex set and W :=\n{wij}1\u2264i,j\u2264N denote the non-negative, symmetric similarity matrix. Each entry wij of W encodes\nthe similarity, or lack thereof, between a pair of vertices. The classical balanced-cut (or, Cheeger\ncut) [7, 8] asks for a partition of V = A \u222a Ac into two disjoint sets that minimizes the set energy\n\nBal(A) :=\n\nCut(A, Ac)\nmin{|A|,|Ac|} =\n\nxi\u2208A,xj\u2208Ac wij\nmin{|A|,|Ac|} .\n\n(1)\n\nA simple rationale motivates this model: clusters should exhibit similarity between data points,\nwhich is re\ufb02ected by small values of Cut(A, Ac), and also form an approximately equal sized parti-\ntion of the vertex set. Note that min{|A|,|Ac|} attains its maximum when |A| = |Ac| = N/2, so that\nfor a given value of Cut(A, Ac) the minimum occurs when A and Ac have approximately equal size.\nWe generalize this model to the multiclass setting by pursuing the same rationale. For a given\nnumber of classes R (that we assume to be known) we formulate our generalized balanced-cut\nproblem as\n\n(cid:80)\n\nR(cid:88)\n\nr=1\n\nMinimize\n\nCut(Ar, Ac\nr)\nmin{\u03bb|Ar|,|Ac\nr|}\n\n\uf8fc\uf8f4\uf8fd\uf8f4\uf8fe\n\n(P)\n\nover all disjoint partitions Ar \u2229 As = \u2205, A1 \u222a \u00b7\u00b7\u00b7 \u222a AR = V of the vertex set.\n\nIn this model the parameter \u03bb controls the sizes of the sets Ar in the partition. Previous work [4]\nhas used \u03bb = 1 to obtain a multiclass energy by a straightforward sum of the two-class balanced-cut\nterms (1). While this follows the usual practice, it erroneously attempts to enforce that each set in\nthe partition occupy half of the total number of vertices in the graph. We instead select the parameter\n\u03bb to ensure that each of the classes approximately occupy the appropriate fraction 1/R of the total\nr| = N \u2212 |Ar|,\nnumber of vertices. As the maximum of min{\u03bb|Ar|,|Ac\nwe see that \u03bb = R \u2212 1 is the proper choice.\nThis general framework also easily incorporates a priori known information, such as a set of labels\nfor transductive learning. If Lr \u2282 V denotes a set of data points that are a priori known to belong\nto class r then we simply enforce Lr \u2282 Ar in the de\ufb01nition of an allowable partition of the vertex\nset. In other words, any allowable disjoint partition Ar \u2229 As = \u2205, A1 \u222a \u00b7\u00b7\u00b7 \u222a AR = V must also\nrespect the given set of labels.\n\nr|} occurs when \u03bb|Ar| = |Ac\n\n3 Total Variation and a Tight Continuous Relaxation\n\nWe derive our continuous optimization by relaxing the set energy (P) to the continuous energy\n\nE(F ) =\n\n(cid:107)fr(cid:107)T V\n\n(cid:107)fr \u2212 med\u03bb(fr)(cid:107) 1,\u03bb\n\n.\n\n(2)\n\nHere F := [f1, . . . , fR] \u2208 MN\u00d7R([0, 1]) denotes the N \u00d7 R matrix that contains in its columns the\nrelaxed optimization variables associated to the R clusters. A few de\ufb01nitions will help clarify the\nmeaning of this formula. The total variation (cid:107)f(cid:107)T V of a vertex function f : V \u2192 R is de\ufb01ned by\n\n(cid:107)f(cid:107)T V =\n\nwij|f (xi) \u2212 f (xj)|.\n\n(3)\n\nR(cid:88)\n\nr=1\n\nn(cid:88)\n\nn(cid:88)\n\ni=1\n\nj=1\n\nAlternatively, if we view a vertex function f as a vector (f (x1), . . . , f (xN ))t \u2208 RN then we can\nwrite\n\n(4)\n\n(cid:107)f(cid:107)T V := (cid:107)Kf(cid:107)1.\n\n2\n\n\fHere K \u2208 MM\u00d7N (R) denotes the gradient matrix of a graph with M edges and N vertices. Each\nrow of K corresponds to an edge and each column corresponds to a vertex. For any edge (i, j) in\nthe graph the corresponding row in the matrix K has an entry wij in the column corresponding to\nthe ith vertex, an entry \u2212wij in the column corresponding to the jth vertex and zeros otherwise.\nTo make sense of the remainder of (2) we must introduce the asymmetric (cid:96)1-norm. This variant of\nthe classical (cid:96)1-norm gives different weights to positive and negative values:\n\n(cid:107)f(cid:107)1,\u03bb =\n\n|f (xi)|\u03bb\n\nwhere\n\n|t|\u03bb =\n\n(5)\n\n(cid:26)\u03bbt\n\n\u2212t\n\nif t \u2265 0\nif t < 0.\n\nn(cid:88)\n\ni=1\n\nFinally we de\ufb01ne the \u03bb-median (or quantile), denoted med\u03bb(f ), as:\n\nmed\u03bb(f ) = the (k + 1)st largest value in the range of f, where k = (cid:98)N/(\u03bb + 1)(cid:99).\n\n(6)\nThese de\ufb01nitions, as well as the relaxation (2) itself, were motivated by the following theorem. Its\nproof, in the supplementary material, relies only the three preceding de\ufb01nitions and some simple\nalgebra.\nTheorem 1. If f = 1A is the indicator function of a subset A \u2282 V then\n\n(cid:107)f(cid:107)T V\n\n(cid:107)f \u2212 med\u03bb(f )(cid:107) 1,\u03bb\n\n=\n\n2 Cut(A, Ac)\nmin{\u03bb|A|,|Ac|} .\n\nThe preceding theorem allows us to restate the original set optimization problem (P) in the equivalent\ndiscrete form\n\nR(cid:88)\n\nr=1\n\nMinimize\n\n(cid:107)fr(cid:107)T V\n\n(cid:107)fr \u2212 med\u03bb(fr)(cid:107) 1,\u03bb\n\nover non-zero functions f1, . . . , fR : V \u2192 {0, 1} such that f1 + . . . + fR = 1V .\nIndeed, since the non-zero functions fr can take only two values, zero or one, they must de\ufb01ne indi-\ncator functions of some nonempty set. The simplex constraint f1 + . . . + fR = 1V then guarantees\nthat the sets Ar := {xi \u2208 V : fr(xi) = 1} form a partition of the vertex set. We obtain the relaxed\nversion (P-rlx) of (P\u2019) in the usual manner by allowing fr \u2208 [0, 1] to have a continuous range. This\nyields\n\n\uf8fc\uf8f4\uf8fd\uf8f4\uf8fe\n\n(P\u2019)\n\n(P-rlx)\n\n\uf8fc\uf8f4\uf8fd\uf8f4\uf8fe\n\n\uf8fc\uf8f4\uf8fd\uf8f4\uf8fe\n\nR(cid:88)\n\nr=1\n\nMinimize\n\n(cid:107)fr(cid:107)T V\n\n(cid:107)fr \u2212 med\u03bb(fr)(cid:107) 1,\u03bb\n\nover functions f1, . . . , fR : V \u2192 [0, 1] such that f1 + . . . + fR = 1V .\n\nThe following two points form the foundation on which total variation clustering relies:\n1 \u2014 As the next subsection details, the total variation terms give rise to quasi-indicator functions.\nThat is, the relaxed solutions [f1, . . . , fR] of (P-rlx) mostly take values near zero or one and exhibit\na sharp, non-smooth transition between these two regions. Since these quasi-indicator functions es-\nsentially take values in the discrete set {0, 1} rather than the continuous interval [0, 1], solving (P-rlx)\nis almost equivalent to solving either (P) or (P\u2019). In other words, (P-rlx) is a tight relaxation of (P).\n2 \u2014 Both functions f (cid:55)\u2192 (cid:107)f(cid:107)T V and f (cid:55)\u2192 (cid:107)f \u2212 med\u03bb(f )(cid:107)1,\u03bb are convex. The simplex constraint\nin (P-rlx) is also convex. Therefore solving (P-rlx) amounts to minimizing a sum of ratios of convex\nfunctions with convex constraints. As the next section details, this fact allows us to use machinery\nfrom convex analysis to develop an ef\ufb01cient, novel algorithm for such problems.\n\n3.1 The Role of Total Variation in the Formation of Quasi-Indicator Functions\n\nTo elucidate the precise role that the total variation plays in the formation of quasi-indicator func-\ntions, it proves useful to consider a version of (P-rlx) that uses a spectral relaxation in place of the\ntotal variation:\n\nR(cid:88)\n\nr=1\n\nMinimize\n\n(cid:107)fr(cid:107)Lap\n\n(cid:107)fr \u2212 med\u03bb(fr)(cid:107) 1,\u03bb\n\nover functions f1, . . . , fR : V \u2192 [0, 1] such that f1 + . . . + fR = 1V\n\n(P-rlx2)\n\n3\n\n\fLap =(cid:80)n\n\nHere (cid:107)f(cid:107)2\ni=1 wij|f (xi) \u2212 f (xj)|2 denotes the spectral relaxation of Cut(A, Ac); it equals\n(cid:104)f, Lf(cid:105) if L denotes the unnormalized graph Laplacian matrix. Thus problem (P-rlx2) relates\nto spectral clustering (and therefore NMF [9]) with a positivity constraint. Note that the only\ndifference between (P-rlx2) and (P-rlx) is that the exponent 2 appears in (cid:107) \u00b7 (cid:107)Lap while the ex-\nponent 1 appears in the total variation. This simple difference of exponent has an important conse-\nquence for the tightness of the relaxations. Figure 1 presents a simple example that illuminates this\ndifference.\nIf we bi-partition the depicted graph, i.e. a line with 20 vertices and edge weights\nwi,i+1 = 1, then the optimal cut lies between vertex 10 and vertex 11 since this gives a per-\nfectly balanced cut. Figure 1(a) shows the vertex function f1 generated by (P-rlx) while \ufb01gure\n1(b) shows the one generated by (P-rlx2). Observe that the solution of the total variation model\ncoincides with the indicator function of the desired cut whereas the the spectral model prefers its\nsmoothed version. Note that both functions in \ufb01gure 1a) and 1b) have exactly the same total vari-\nation (cid:107)f(cid:107)T V = |f (x1) \u2212 f (x2)| + \u00b7\u00b7\u00b7 + |f (x19) \u2212 f (x20)| = f (x1) \u2212 f (x20) = 1 since both\nfunctions are monotonic. The total variation model will therefore prefer the sharp indicator function\nsince it differs more from its \u03bb-median than the smooth indicator function. Indeed, the denominator\n(cid:107)fr \u2212 med\u03bb(fr)(cid:107)1,\u03bb is larger for the sharp indicator function than for the smooth one. A differ-\nent scenario occurs when we replace the exponent one in (cid:107) \u00b7 (cid:107)T V by an exponent two, however. As\n(cid:107)f(cid:107)2\nLap = |f (x1)\u2212f (x2)|2 +\u00b7\u00b7\u00b7+|f (x19)\u2212f (x20)|2 and t2 < t when t < 1 it follows that(cid:107)f(cid:107)Lap\nis much smaller for the smooth function than for the sharp one. Thus the spectral model will prefer\nthe smooth indicator function despite the fact that it differs less from its \u03bb-median. We therefore\nrecognize the total variation as the driving force behind the formation of sharp indicator functions.\n\n(a)\n\n(b)\n\nFigure 1: Top: The graph used for both relaxations. Bottom left: the solution given by the total\nvariation relaxation. Bottom right: the solution given by the spectral relaxation. Position along the\nx-axis = vertex number, height along the y-axis = value of the vertex function.\n\nThis heuristic explanation on a simple, two-class example generalizes to the multiclass case and\nto real data sets (see \ufb01gure 2). In simple terms, quasi-indicator functions arise due to the fact that\nthe total variation of a sharp indicator function equals the total variation of a smoothed version of\nthe same indicator function. The denominator (cid:107)fr \u2212 med\u03bb(fr)(cid:107)1,\u03bb then measures the deviation of\nthese functions from their \u03bb-median. A sharp indicator function deviates more from its median than\ndoes its smoothed version since most of its values concentrate around zero and one. The energy\nis therefore much smaller for a sharp indicator function than for a smooth indicator function, and\nconsequently the total variation clustering energy always prefers sharp indicator functions to smooth\nones. For bi-partitioning problems this fact is well-known. Several previous works have proven that\nthe relaxation is exact in the two-class case; that is, the total variation solution coincides with the\nsolution of the original NP-hard problem [8, 18, 3, 5].\nFigure 2 illustrates the result of the difference between total variation and NMF relaxations on the\ndata set OPTDIGITS, which contains 5620 images of handwritten numerical digits. Figure 2(a)\nshows the quasi-indicator function f4 obtained, before thresholding, by our MTV algorithm while\n2(b) shows the function f4 obtained from the NMF algorithm of [1]. We extract the portion of each\nfunction corresponding to the digits four and nine, then sort and plot the result. The MTV relaxation\nleads a sharp transition between the fours and the nines while the NMF relaxation leads to a smooth\ntransition.\n\n4\n\n\fFigure 2: Left: Solution f4 from our MTV algorithm (before thresholding) plotted over the fours\nand nines. Right: Solution f4 from LSD [1] plotted over the fours and nines.\n\n3.2 Transductive Framework\n\nFrom a modeling point-of-view, the presence of transductive labels poses no additional dif\ufb01culty. In\naddition to the simplex constraint\n\nF \u2208 \u03a3 :=\n\nF \u2208 MN\u00d7R([0, 1]) : fr(xi) \u2265 0,\n\nfr(xi) = 1\n\n(7)\n\n(cid:40)\n\nR(cid:88)\n\nr=1\n\nrequired for unsupervised clustering we also impose the set of labels as a hard constraint.\nIf\nL1, . . . , LR denote the R vertex subsets representing the labeled points, so that xi \u2208 Lr means\nxi belongs to class r, then we may enforce these labels by restricting F to lie in the subset\nF \u2208 \u039b := {F \u2208 MN\u00d7R([0, 1]) : \u2200r, (f1(xi), . . . , fR(xi)) = er \u2200 xi \u2208 Lr } .\n\n(8)\nHere er denotes the row vector containing a one in the rth location and zeros elsewhere. Our model\nfor transductive classi\ufb01cation then aims to solve the problem\n\n(cid:41)\n\n(cid:41)\n\nif f (xi) > med\u03bb(f )\nif f (xi) = med\u03bb(f )\nif f (xi) < med\u03bb(f )\n\nwhere\n\n5\n\nR(cid:88)\n\nr=1\n\nMinimize\n\n(cid:107)fr(cid:107)T V\n\n(cid:107)fr \u2212 med\u03bb(fr)(cid:107) 1,\u03bb\n\nover matrices F \u2208 \u03a3 \u2229 \u039b.\n\n(P-trans)\n\nNote that \u03a3 \u2229 \u039b also de\ufb01nes a convex set, so this minimization remains a sum of ratios of convex\nfunctions subject to a convex constraint. Transductive classi\ufb01cation therefore poses no additional\nalgorithmic dif\ufb01culty, either. In particular, we may use the proximal splitting algorithm detailed in\nthe next section for both unsupervised and transductive classi\ufb01cation tasks.\n\n4 Proximal Splitting Algorithm\n\nThis section details our proximal splitting algorithm for \ufb01nding local minimizers of a sum of ratios\nof convex functions subject to a convex constraint. We start by showing in the \ufb01rst subsection that\nthe functions\n\n(9)\ninvolved in (P-rlx) or (P-trans) are indeed convex. We also give an explicit formula for a subdiffer-\nential of B since our proximal splitting algorithm requires this in explicit form. We then summarize\na few properties of proximal operators before presenting the algorithm.\n\nand B(f ) := (cid:107)f \u2212 med\u03bb(f )1(cid:107)1,\u03bb\n\nT (f ) := (cid:107)f(cid:107)T V\n\n4.1 Convexity, Subgradients and Proximal Operators\nRecall that we may view each function f : V \u2192 R as a vector in RN with f (xi) as the ith component\nof the vector. We may then view T and B as functions from RN to R. The next theorem states that\nboth B and T de\ufb01ne convex functions on RN and furnishes an element v \u2208 \u2202B(f ) by means of an\neasily computable formula. The formula for the subdifferential generalizes a related result for the\nsymmetric case [11] to the asymmetric setting. We provide its proof in the supplementary material.\nTheorem 2. The functions B and T are convex. Moreover, given f \u2208 RN the vector v \u2208 RN\nde\ufb01ned below belongs to \u2202B(f ):\n\n\uf8f1\uf8f2\uf8f3n0 = |{xi \u2208 V : f (xi) = med\u03bb(f )}|\n\nn\u2212 = |{xi \u2208 V : f (xi) < med\u03bb(f )}|\nn+ = |{xi \u2208 V : f (xi) > med\u03bb(f )}|\n\n\uf8f1\uf8f4\uf8f2\uf8f4\uf8f3\u03bb\n\nn\u2212\u2212\u03bbn+\n\u22121\n\nn0\n\nv(xi) =\n\n\fIn the above theorem \u2202B(f ) denotes the subdifferential of B at f and v \u2208 \u2202B(f ) denotes a subgra-\ndient. Given a convex function A : RN \u2192 R, the proximal operator of A is de\ufb01ned by\n\nproxA(g) := argmin\nf\u2208RN\n\nA(f ) +\n\n||f \u2212 g||2\n2.\n\n1\n2\n\n(10)\n\nIf we let \u03b4C denote the barrier function of the convex set C, that is\n\n\u03b4C(f ) = 0 if f \u2208 C and \u03b4C(f ) = +\u221e if f /\u2208 C,\n\nthen we easily see that prox\u03b4C is simply the least-squares projection on C,\nprox\u03b4C\nmapping from RN to RN that generalizes the least-squares projection onto a convex set.\n\n(f ) = projC(f ) := argmin\ng\u2208C\n\n2||f \u2212 g||2\n\n1\n\n(11)\nin other words,\n2. In this manner the proximal operator de\ufb01nes a\n\n4.2 The Algorithm\n\nWe can rewrite the problem (P-rlx) or (P-trans) as\n\nMinimize \u03b4C(F ) +\n\nE(fr) over all matrices F = [f1, . . . , fr] \u2208 MN\u00d7R\n\n(12)\n\nr=1\n\nwhere E(fr) = T (fr)/B(fr) denotes the energy of the quasi-indicator function of the rth cluster.\nThe set C = \u03a3 or C = \u03a3 \u2229 \u039b is the convex subset of MN\u00d7R that encodes the simplex constraint\n(7) or the simplex constraint with labels. The corresponding function \u03b4C(F ), de\ufb01ned in (11), is\nthe barrier function of the desired set. Beginning from an initial iterate F 0 \u2208 C we propose the\nfollowing proximal splitting algorithm:\n\nF k+1 := proxT k+\u03b4C\n\n(F k + \u2202Bk(F k)).\n\n(13)\n\nR(cid:88)\n\nR(cid:88)\n\nHere T k(F ) and Bk(F ) denote the convex functions\n\nT k(F ) :=\n\nck\nr T (fr)\n\nBk(F ) :=\n\nthe constants (ck\n\nr , dk\n\nr ) are computed using the previous iterate\n\nr=1\n\nR(cid:88)\n\nr=1\n\ndk\nr B(fr),\n\nr = \u2206k/B(f k\nck\nr )\n\nand\n\ndk\nr = \u2206kE(f k\n\nr )/B(f k\nr )\n\nr ) yields\nand \u2206k denotes the timestep for the current iteration. This choice of the constants (ck\nBk(F k) = T k(F k), and this fundamental property allows us to derive (see supplementary material)\nthe energy descent estimate:\nTheorem 3 (Estimate of the energy descent). Each of the F k belongs to C, and if Bk\n\nr , dk\n\nr (cid:54)= 0 then\n\nR(cid:88)\n\n(cid:0)Ek\n\nBk+1\nr\nBk\nr\n\nr \u2212 Ek+1\n\nr\n\n(cid:1) \u2265 (cid:107)F k \u2212 F k+1(cid:107)2\n\n\u2206k\n\n(14)\n\nwhere Bk\n\nr , Ek\n\nr=1\nr stand for B(f k\n\nr ), E(f k\n\nr ).\n\nInequality (14) states that the energies of the quasi-indicator functions (as a weighted sum) decrease\nat every step of the algorithm. It also gives a lower bound for how much these energies decrease. As\nthe algorithm progress and the iterates stabilize the ratio Bk+1\nr converges to 1, in which case\nthe sum, rather than a weighted sum, of the individual cluster energies decreases.\nOur proximal splitting algorithm (13) requires two steps. The \ufb01rst step requires computing Gk =\nF k + \u2202Bk(F k), and this is straightforward since theorem 2 provides the subdifferential of B, and\ntherefore of Bk, through an explicit formula. The second step requires computing proxT k+\u03b4C\n(Gk),\nwhich seems daunting at a \ufb01rst glance. Fortunately, minimization problems of this form play an\nimportant role in the image processing literature. Recent years have therefore produced several fast\nand accurate algorithms for computing the proximal operator of the total variation. As T k + \u03b4C con-\nsists of a weighted sum of total variation terms subject to a convex constraint, we can readily adapt\n\n/Bk\n\nr\n\n6\n\n\fthese algorithms to compute the second step of our algorithm ef\ufb01ciently. In this work we use the\nprimal-dual algorithm of [6] with acceleration. This relies on a proper uniformly convex formulation\nof the proximal minimization, which we detail completely in the supplementary material.\n(Gk) produces a sequence of approximate\nThe primal-dual algorithm we use to compute proxT k+\u03b4C\nsolutions by means of an iterative procedure. A stopping criterion is therefore needed to indicate\nwhen the current iterate approximates the actual solution proxT k+\u03b4C\n(Gk) suf\ufb01ciently. Ideally, we\nwould like to terminate F k+1 \u2248 proxT k+\u03b4C\n(Gk) in such a manner so that the energy descent\nproperty (14) still holds and F k+1 always satis\ufb01es the required constraints. In theory we cannot\nguarantee that the energy estimate holds for an inexact solution. We may note, however, that a\nslightly weaker version of the energy estimate (14)\n\nR(cid:88)\n\nr=1\n\n(cid:0)Ek\n\nBk+1\nr\nBk\nr\n\nr \u2212 Ek+1\n\nr\n\n(cid:1) \u2265 (1 \u2212 \u0001)\n\n(cid:107)F k \u2212 F k+1(cid:107)2\n\n\u2206k\n\n(15)\n\nholds after a \ufb01nite number of iterations of the inner minimization. Moreover, this weaker version\nstill guarantees that the energies of the quasi-indicator functions decrease as a weighted sum in\nexactly the same manner as before. In this way we can terminate the inner loop adaptively: we solve\nF k+1 \u2248 proxT k+\u03b4C\n(Gk) less precisely when F k+1 lies far from a minimum and more precisely as\nthe sequence {F k} progresses. This leads to a substantial increase in ef\ufb01ciency of the full algorithm.\nOur implementation of the proximal splitting algorithm also guarantees that F k+1 always satis\ufb01es\nthe required constraints. We accomplish this task by implementing the primal-dual algorithm in\nsuch a way that each inner iteration always satis\ufb01es the constraints. This requires computing the\nprojection projC(F ) exactly at each inner iteration. The overall algorithm remains ef\ufb01cient provided\nwe can compute this projection quickly. When C = \u03a3 the algorithm [14] performs the required\nprojection in at most R steps. When C = \u03a3\u2229 \u039b the computational effort actually decreases, since in\nthis case the projection consists of a simplex projection on the unlabeled points and straightforward\nassignment on the labeled points. Overall, each iteration of the algorithm scales like O(N R2) +\nO(M R) + O(RN log(N )) for the simplex projection, application of the gradient matrix and the\ncomputation of the balance terms, respectively.\nWe may now summarize the full algorithm, including the proximal operator computation. In prac-\nR} and any small \u0001 work well, so we present the\ntice we \ufb01nd the choices \u2206k = max{Bk\nalgorithm with these choices. Recall the matrix K in (4) denotes the gradient matrix of the graph.\n\n1 , . . . , Bk\n\n(cid:105)\n\n\u00afF = F\n\n0(\u03c4 \u22062L2)\u22121\nB(f1) , . . . , \u2206\n\nB(fR)\n\nAlgorithm 1 Proximal Splitting Algorithm\n\nInput: F \u2208 C, P = 0, L = ||K||2, \u03c4 = L\u22121, \u0001 = 10\u22123\nwhile loop not converged do\n\n(cid:104) E(f1)\n\n//Perform outer step Gk = F k + \u2202Bk(F k)\n\u2206 = maxr B(fr) \u22060 = minr B(fr)\nDE = diag\nV = \u2206[\u2202B(f1), . . . , \u2202B(fR)]DE (using theorem 2)\nG = F + V\n//Perform F k+1 \u2248 proxT k+\u03b4C\nwhile (15) fails do\n\nB(f1) , . . . , E(fR)\n\n(cid:104) \u2206\n\nDB = diag\n\n\u03c3 = \u22062\n\n(cid:105)\n\nB(fR)\n\n\u02dcP = P + \u03c3K \u00afF DB\n\u02dcF = F \u2212 \u03c4 K tP DB F = ( \u02dcF + \u03c4 G)/(1 + \u03c4 )\n\u221a\n\u03b8 = 1/\nend while\n\n\u03c3 = \u03c3/\u03b8\n\n\u03c4 = \u03b8\u03c4\n\n1 + 2\u03c4\n\nend while\n\n(Gk) until energy estimate holds\n\nP = \u02dcP / max{| \u02dcP|, 1} (both operations entriwise)\n\nFold = F\n\nF = projC(F )\n\u00afF = (1 + \u03b8)F \u2212 \u03b8Fold\n\n5 Numerical Experiments\n\nWe now demonstrate the MTV algorithm for unsupervised and transductive clustering tasks. We\nselected six standard, large-scale data sets as a basis of comparison. We obtained the \ufb01rst data set\n\n7\n\n\f(4MOONS) and its similarity matrix from [4] and the remaining \ufb01ve data sets and matrices (WE-\nBKB4, OPTDIGITS, PENDIGITS, 20NEWS, MNIST) from [19]. The 4MOONS data set contains\n4K points while the remaining \ufb01ve contain 4.2K, 5.6K, 11K, 20K and 70K points, respectively.\nOur \ufb01rst set of experiments compares our MTV algorithm against other unsupervised approaches.\nWe compare against two previous total variation algorithms [11, 3], which rely on recursive bi-\npartitioning, and two top NMF algorithms [1, 19]. We use the normalized Cheeger cut versions\nof [11] and [3] with default parameters. We used the code available from [19] to test each NMF\nalgorithm. The non-recursive NMF algorithms (LSD [1], NMFR [19]) received two types of initial\ndata: (a) the deterministic data used in [19]; (b) a random procedure leveraging normalized-cut [16].\nProcedure (b) \ufb01rst selects one data point uniformly at random from each computed NCut cluster,\nthen sets fr equal to one at the data point drawn from the rth cluster and zero otherwise. We then\npropagate this initial stage by replacing each fr with (I +L)\u22121fr where L denotes the unnormalized\ngraph Laplacian. Finally, to aid the NMF algorithms, we add a small constant 0.2 to the result (each\nperformed better than without adding this constant). For MTV we use 30 random trials of (b) then\nreport the cluster purity (c.f. [19] for a de\ufb01nition of purity) of the solution with the lowest discrete\nenergy (P). We then use each NMF with exactly the same initial conditions and report simply the\nhighest purity achieved over all 31 runs. This biases the results in favor of the NMF algorithms.\nDue to the non-convex nature of these algorithms, the random initialization gave the best results and\nsigni\ufb01cantly improved upon previously reported results of LSD in particular. For comparison with\n[19], initialization (a) is followed by 10,000 iterations of each NMF algorithm. Each trial of (b) is\nfollowed by 2000 iterations of each non-recursive algorithm. The following table reports the results.\nOur next set of experiments demonstrate our algorithm in a transductive setting. For each data set\n\nAlg/Data\n\n4MOONS WEBKB4 OPTDIGITS\n\nNCC-TV [3]\n1SPEC [11]\n\nLSD [1]\n\nNMFR [19]\n\nMTV\n\n88.75\n73.92\n99.40\n77.80\n99.53\n\n51.76\n39.68\n54.50\n64.32\n59.15\n\n95.91\n88.65\n97.94\n97.92\n98.29\n\nPENDIGITS\n\n73.25\n82.42\n88.44\n91.21\n89.06\n\n20NEWS MNIST\n88.80\n88.17\n95.67\n96.99\n97.60\n\n23.20\n11.49\n41.25\n63.93\n39.40\n\nwe randomly sample either one label per class or a percentage of labels per class from the ground\ntruth. We then run ten trials of initial condition (b) (propagating all labels instead of one) and report\nthe purity of the lowest energy solution as before along with the average computational time (for\nsimple MATLAB code running on a standard desktop) of the ten runs. We terminate the algorithm\nonce the relative change in energy falls below 10\u22124 between outer steps of algorithm 1. The table\nbelow reports the results. Note that for well-constructed graphs (such as MNIST), our algorithm\nperforms remarkably well with only one label per class.\n\nLabels\n\n4MOONS WEBKB4 OPTDIGITS\n99.55/ 3.0s\n1\n1%\n99.55/ 3.1s\n2.5% 99.55/ 1.9s\n99.53/ 1.2s\n5%\n10% 99.55/ 0.8s\n\n56.58/ 1.8s\n58.75/ 2.0s\n57.01/ 1.7s\n58.34/ 1.3s\n62.01/ 1.2s\n\n98.29/ 7s\n98.29/ 4s\n98.35/ 3s\n98.38/ 2s\n98.45/ 2s\n\nPENDIGITS\n89.17/ 14s\n93.73/ 9s\n95.83/ 7s\n97.98/ 5s\n98.22/ 4s\n\n20NEWS\n50.07/ 52s\n61.70/ 54s\n67.61/ 42s\n70.51/ 32s\n73.97/ 25s\n\nMNIST\n97.53/ 98s\n97.59/ 54s\n97.72/ 39s\n97.79/ 31s\n98.05/ 25s\n\nOur non-recursive MTV algorithm vastly outperforms the two previous recursive total variation\napproaches and also compares well with state-of-the-art NMF approaches. Each of MTV, LSD and\nNMFR perform well on manifold data sets such as MNIST, but NMFR tends to perform best on\nnoisy, non-manifold data sets. This results from the fact that NMFR uses a costly graph smoothing\ntechnique while our algorithm and LSD do not. We plan to incorporate such improvements into the\ntotal variation framework in future work. Lastly, we found procedure (b) can help overcome the\nlack of convexity inherent in many clustering approaches. We plan to pursue a more principled and\nef\ufb01cient initialization along these lines in the future as well. Overall, our total variation framework\npresents a promising alternative to NMF methods due to its strong mathematical foundation and\ntight relaxation.\nAcknowledgements: Supported by NSF grant DMS-1109805, AFOSR MURI grant FA9550-10-1-\n0569, ONR grant N000141210040, and Swiss National Science Foundation grant SNSF-141283.\n\n8\n\n\fReferences\n[1] Raman Arora, M Gupta, Amol Kapila, and Maryam Fazel. Clustering by left-stochastic ma-\ntrix factorization. In International Conference on Machine Learning (ICML), pages 761\u2013768,\n2011.\n\n[2] A. Bertozzi and A. Flenner. Diffuse Interface Models on Graphs for Classi\ufb01cation of High\n\nDimensional Data. Multiscale Modeling and Simulation, 10(3):1090\u20131118, 2012.\n\n[3] X. Bresson, T. Laurent, D. Uminsky, and J. von Brecht. Convergence and energy landscape for\ncheeger cut clustering. In Advances in Neural Information Processing Systems (NIPS), pages\n1394\u20131402, 2012.\n\n[4] X. Bresson, X.-C. Tai, T.F. Chan, and A. Szlam. Multi-Class Transductive Learning based on\n\n(cid:96)1 Relaxations of Cheeger Cut and Mumford-Shah-Potts Model. UCLA CAM Report, 2012.\n\n[5] T. B\u00a8uhler and M. Hein. Spectral Clustering Based on the Graph p-Laplacian. In International\n\nConference on Machine Learning (ICML), pages 81\u201388, 2009.\n\n[6] A. Chambolle and T. Pock. A First-Order Primal-Dual Algorithm for Convex Problems with\nApplications to Imaging. Journal of Mathematical Imaging and Vision, 40(1):120\u2013145, 2011.\n[7] J. Cheeger. A Lower Bound for the Smallest Eigenvalue of the Laplacian. Problems in Analy-\n\nsis, pages 195\u2013199, 1970.\n\n[8] F. R. K. Chung. Spectral Graph Theory, volume 92 of CBMS Regional Conference Series in\nMathematics. Published for the Conference Board of the Mathematical Sciences, Washington,\nDC, 1997.\n\n[9] Chris Ding, Xiaofeng He, and Horst D Simon. On the equivalence of nonnegative matrix\nfactorization and spectral clustering. In Proc. SIAM Data Mining Conf, number 4, pages 606\u2013\n610, 2005.\n\n[10] C. Garcia-Cardona, E. Merkurjev, A. L. Bertozzi, A. Flenner, and A. G. Percus. Fast multiclass\n\nsegmentation using diffuse interface methods on graphs. Submitted, 2013.\n\n[11] M. Hein and T. B\u00a8uhler. An Inverse Power Method for Nonlinear Eigenproblems with Applica-\ntions in 1-Spectral Clustering and Sparse PCA. In Advances in Neural Information Processing\nSystems (NIPS), pages 847\u2013855, 2010.\n\n[12] M. Hein and S. Setzer. Beyond Spectral Clustering - Tight Relaxations of Balanced Graph\n\nCuts. In Advances in Neural Information Processing Systems (NIPS), 2011.\n\n[13] E. Merkurjev, T. Kostic, and A. Bertozzi. An mbo scheme on graphs for segmentation and\n\nimage processing. UCLA CAM Report 12-46, 2012.\n\n[14] C. Michelot. A Finite Algorithm for Finding the Projection of a Point onto the Canonical\n\nSimplex of Rn. Journal of Optimization Theory and Applications, 50(1):195\u2013200, 1986.\n\n[15] S. Rangapuram and M. Hein. Constrained 1-Spectral Clustering. In International conference\n\non Arti\ufb01cial Intelligence and Statistics (AISTATS), pages 1143\u20131151, 2012.\n\n[16] J. Shi and J. Malik. Normalized Cuts and Image Segmentation. IEEE Transactions on Pattern\n\nAnalysis and Machine Intelligence (PAMI), 22(8):888\u2013905, 2000.\n\n[17] A. Szlam and X. Bresson. A total variation-based graph clustering algorithm for cheeger ratio\n\ncuts. UCLA CAM Report 09-68, 2009.\n\n[18] A. Szlam and X. Bresson. Total variation and cheeger cuts. In International Conference on\n\nMachine Learning (ICML), pages 1039\u20131046, 2010.\n\n[19] Zhirong Yang, Tele Hao, Onur Dikmen, Xi Chen, and Erkki Oja. Clustering by nonnegative\nmatrix factorization using graph random walk. In Advances in Neural Information Processing\nSystems (NIPS), pages 1088\u20131096, 2012.\n\n[20] Zhirong Yang and Erkki Oja. Clustering by low-rank doubly stochastic matrix decomposition.\n\nIn International Conference on Machine Learning (ICML), 2012.\n\n9\n\n\f", "award": [], "sourceid": 712, "authors": [{"given_name": "Xavier", "family_name": "Bresson", "institution": "City University of Hong Kong"}, {"given_name": "Thomas", "family_name": "Laurent", "institution": "Loyola Marymount University"}, {"given_name": "David", "family_name": "Uminsky", "institution": "University of San Francisco"}, {"given_name": "James", "family_name": "von Brecht", "institution": "UCLA"}]}