{"title": "Fusion of Similarity Data in Clustering", "book": "Advances in Neural Information Processing Systems", "page_first": 723, "page_last": 730, "abstract": "", "full_text": "Fusion of Similarity Data in Clustering\n\nTilman Lange and Joachim M. Buhmann\n(langet,jbuhmann)@inf.ethz.ch\n\nInstitute of Computational Science, Dept. of Computer Sience,\n\nETH Zurich, Switzerland\n\nAbstract\n\nFusing multiple information sources can yield signi\ufb01cant bene\ufb01ts to suc-\ncessfully accomplish learning tasks. Many studies have focussed on fus-\ning information in supervised learning contexts. We present an approach\nto utilize multiple information sources in the form of similarity data for\nunsupervised learning. Based on similarity information, the clustering\ntask is phrased as a non-negative matrix factorization problem of a mix-\nture of similarity measurements. The tradeoff between the informative-\nness of data sources and the sparseness of their mixture is controlled by\nan entropy-based weighting mechanism. For the purpose of model se-\nlection, a stability-based approach is employed to ensure the selection\nof the most self-consistent hypothesis. The experiments demonstrate the\nperformance of the method on toy as well as real world data sets.\n\n1\n\nIntroduction\n\nClustering has found increasing attention in the past few years due to the enormous infor-\nmation \ufb02ood in many areas of information processing and data analysis. The ability of an\nalgorithm to determine an interesting partition of the set of objects under consideration,\nhowever, heavily depends on the available information. It is, therefore, reasonable to equip\nan algorithm with as much information as possible and to endow it with the capability to\ndistinguish between relevant and irrelevant information sources. How to reasonably iden-\ntify a weighting of the different information sources such that an interesting group structure\ncan be successfully uncovered, remains, however, a largely unresolved issue.\nDifferent sources of information about the same objects naturally arise in many application\nscenarios. In computer vision, for example, information sources can consist of plain in-\ntensity measurements, edge maps, the similarity to other images or even human similarity\nassessments. Similarly in bio-informatics: the similarity of proteins,e.g., can be assessed in\ndifferent ways, ranging from the comparison of gene pro\ufb01les to direct comparisons at the\nsequence level using alignment methods.\nIn this work, we use a non-negative matrix factorization approach (nmf) to pairwise clus-\ntering of similarity data that is extended in a second step in order to incorporate a suitable\nweighting of multiple information sources, leading to a mixture of similarities. The latter\nrepresents the main contribution of this work. Algorithms for nmf have recently found a\nlot of attention. Our proposal is inspired by the work in [11] and [5]. Only recently, [18]\nhave also employed a nmf to perform clustering. For the purpose of model selection, we\nemploy a stability-based approach that has already been successfully applied to model se-\n\n\flection problems in clustering (e.g. in [9]). Instead of following the strategy to \ufb01rst embed\nthe similarities into a space with Euclidean geometry and then to perform clustering and,\nwhere required, feature selection/weighting on the stacked feature vector, we advocate an\napproach that is closer to the original similarity data by performing nmf.\nSome work has been devoted to feature selection and weighting in clustering problems. In\n[13] a variant of the k-means algorithm has been studied that employs the Fisher criterion\nto assess the importance of individual features. In [14, 10], Gaussian mixture model-based\napproaches to feature selection are introduced. The more general problem of learning a\nsuitable metric has also been investigated, e.g. in [17]. Similarity measurements represent\na particularly generic form of providing input to a clustering algorithm. Fusing such repre-\nsentations has only recently been studied in the context of kernel-based supervised learning,\ne.g. in [7] using semi-de\ufb01nite programming and in [3] using a boosting procedure. In [1],\nan approach to learning the bandwidth parameter of an rbf-kernel for spectral clustering is\nstudied.\nThe paper is organized as follows: section 2 introduces the nmf-based clustering method\ncombined with a data-source weighting (section 3). Section 4 discusses an out-of-sample\nextension allowing us to predict assignments and to employ the stability principle for model\nselection. Experimental evidence in favor of our approach is given in section 5.\n\n2 Clustering by Non-Negative Matrix Factorization\nSuppose we want to group a \ufb01nite set of objects On := {o1, . . . , on}. Usually, there are\nmultiple ways of measuring the similarity between different objects. Such relations give\nrise to similarities sij := s(oi, oj) 1 where we assume non-negativity sij \u2265 0, symmetry\nsji = sij, and boundedness sij < \u221e. For n objects, we summarize the similarity data in a\nn\u00d7n matrix S = (sij) which is re-normalized to P = S/1t\nnS1n, where 1n := (1, . . . , 1)t.\nThe re-normalized similarities can be interpreted as the probability of the joint occurrence\nof objects i, j.\nWe aim now at \ufb01nding a non-negative matrix factorization of P \u2208 [0, 1]n\u00d7n into a product\nWHt of the n \u00d7 k matrices W and H with non-negative entries for which additionally\nnW1k = 1 and Ht1n = 1k, where k denotes the number of clusters. That is,\nholds 1t\none aims at explaining the overall probability for a co-occurrence by a latent cause, the\nunobserved classes. The constraints ensure, that the entries of both, W and H, can be\nconsidered as probabilities: the entry wi\u03bd of W is the joint probability q(i, \u03bd) of object\ni and class \u03bd whereas hjk in H is the probability q(j|\u03bd). This model implicitly assumes\nindependence of object i and j conditioned on \u03bd. Given a factorization of P in W and H,\nwe can use the maximum a posteriori estimate, arg max\u03bd hi\u03bd\nj wj\u03bd, to arrive at a hard\nassignment of objects to classes.\nIn order to obtain a factorization, we minimize the cross-entropy\n\nP\n\nC(PkWHt) := \u2212X\n\npij logX\n\nwi\u03bdhj\u03bd\n\n(1)\n\ni,j\n\n\u03bd\n\nis inspired by the Expectation-Maximization (EM) algorithm: Let \u03c4\u03bdij \u2265 0 withP\n1. Then, by the convexity of \u2212 log x, we obtain \u2212 logP\n\nwhich becomes minimal iff P = WHt 2 and is not convex in W and H together. Note, that\nthe factorization is not necessarily unique. We resort to a local optimization scheme, which\n\u03bd \u03c4\u03bdij =\n\u03bd \u03c4\u03bdij log wi\u03bd hj\u03bd\n,\n\u03c4\u03bdij\n\n\u03bd wi\u03bdhj\u03bd \u2264 \u2212P\n\n1In the following, we represent objects by their indices.\n2The Kullback-Leibler divergence is D(PkWHt) = \u2212H(P) + C(PkWHt) \u2265 0 with equality\n\niff P = WHt.\n\n\fwhich yields the relaxed objective function:\n\n\u02dcC(PkWHt) := \u2212X\n\npij\u03c4\u03bdij log wi\u03bdhj\u03bd + \u03c4\u03bdij log \u03c4\u03bdij \u2265 C(PkWHt).\n\n(2)\n\ni,j,\u03bd\n\nWith this relaxation, we can employ an alternating minimization scheme for minimizing\nthe bound on C. As in EM, one iterates\n\n1. Given W and H, minimize \u02dcC w.r.t. \u03c4\u03bdij\n2. Given the values \u03c4\u03bdij, \ufb01nd estimates for W and H by minimizing \u02dcC.\n\nuntil convergence, which produces a sequence of estimates\n\nP\n\n\u03c4 (t)\n\u03bdij =\n\n=X\nuse the convention hj\u03bd = 0 wheneverP\n\nw(t)\ni\u03bd h(t)\n\u00b5 w(t)\n\ni\u00b5 h(t)\n\nw(t+1)\n\nj\u00b5\n\nj\u03bd\n\ni\u03bd\n\n,\n\nj\n\nthat converges to a local minimum of \u02dcC. This is an instance of an MM algorithm [8]. We\ni,j pij\u03c4\u03bdij = 0. The per-iteration complexity is\n\nP\nP\n\n\u03bdij\n\ni pij\u03c4 (t)\na,b pab\u03c4 (t)\n\n\u03bdab\n\n(3)\n\npij\u03c4 (t)\n\u03bdij,\n\nh(t+1)\nj\u03bd =\n\nO(n2).\n\n3 Fusing Multiple Data Sources\n\nlarity matrices P1, . . . , PL. We introduce now weights \u03b1l, 1 \u2264 l \u2264 L, withP\ncombination \u00afP = P\n\nMeasuring the similarity of objects in, say, L different ways results in L normalized simi-\nl \u03b1l = 1. For\n\ufb01xed \u03b1\u03b1\u03b1 = (\u03b1l) \u2208 [0, 1]L, the aggregated and normalized similarity becomes the convex\nij , i.e. a\nmixture of different explanations. Again, we seek a good factorization of \u00afP by minimizing\nthe cross-entropy, which then becomes\nmin\n\u03b1\u03b1\u03b1,W,H\n\nl \u03b1lPl. Hence, \u00afpij is a mixture of individual similarities p(l)\n\n(cid:2)C(PlkWHt)(cid:3)\n\nE\u03b1\u03b1\u03b1\n\n(4)\n\nl \u03b1lfl denotes the expectation w.r.t. the discrete distribution \u03b1\u03b1\u03b1. The\nsame relaxation as in the last section can be used, i.e. for all \u03b1\u03b1\u03b1, W and H, we have\nE\u03b1\u03b1\u03b1 [C(PlkWHt)] \u2264 E\u03b1\u03b1\u03b1[ \u02dcC(PlkWHt)]. Hence, we can employ a slightly modi\ufb01ed,\nnested alternating minimization approach: Given \ufb01xed \u03b1\u03b1\u03b1, obtain estimates W and H using\nthe relaxation of the last section. The update equations change to\n\nwhere E\u03b1\u03b1\u03b1[fl] = P\n\nw(t+1)\n\ni\u03bd\n\n\u03b1l\n\nij \u03c4 (t)\np(l)\n\u03bdij,\n\nh(t+1)\nj\u03bd =\n\n.\n\n(5)\n\nc = (cl)l. Minimizing the expression in equation (4) subject to the constraintsP\n\nGiven the current estimates of W and H, we could minimize the objective in equation (4)\nw.r.t. \u03b1\u03b1\u03b1 subject to the constraint k\u03b1\u03b1\u03b1k1 = 1. To this end, set cl := C(PlkWHt) and let\nl \u03b1l = 1\nand \u03b1\u03b1\u03b1 (cid:23) 0, therefore, becomes a linear program (LP) min\u03b1\u03b1\u03b1 ct\u03b1\u03b1\u03b1 such that 1t\nL\u03b1\u03b1\u03b1 = 1, \u03b1\u03b1\u03b1 (cid:23) 0,\nwhere (cid:23) denotes the element-wise \u2265-relation. The LP solution is very sparse since the opti-\nmal solutions for the linear program lie on the corners of the simplex in the positive orthant\nspanned by the constraints. In particular, it lacks a means to control the sparseness of the\ncoef\ufb01cients \u03b1\u03b1\u03b1. We, therefore, use a maximum entropy approach ([6]) for sparseness control:\nthe entropy is upper bounded by log L and measures the sparseness of the vector \u03b1\u03b1\u03b1, since\nthe lower the entropy the more peaked the distribution \u03b1\u03b1\u03b1 can be. Hence, by lower bounding\nthe entropy, we specify the maximal admissible sparseness. This approach is reasonable\nas we actually want to combine multiple (not only identify one) information sources but\nthe best \ufb01t in an unsupervised problem will be usually obtained by choosing only a single\n\n=X\n\nX\n\nl\n\nj\n\nP\nP\n\nP\nP\n\nl \u03b1l\nl \u03b1l\n\n\u03bdij\n\ni p(l)\ni,j p(l)\n\nij \u03c4 (t)\nij \u03c4 (t)\n\n\u03bdij\n\n\fsource. Thus, we modify the objective originally given in eq. (4) to the entropy-regularized\nproblem E\u03b1\u03b1\u03b1[ \u02dcC(PlkWHt)] \u2212 \u03b7H(\u03b1\u03b1\u03b1), so that the mathematical program given above be-\ncomes\n\n(6)\nwhere H denotes the (discrete) entropy and \u03b7 \u2208 R+ is a positive Lagrange parameter. The\noptimization problem in eq. (6) has an analytical solution, namely the Gibbs distribution\n\ns.t. 1t\n\nmin\n\u03b1\u03b1\u03b1\n\nL\u03b1\u03b1\u03b1 = 1, \u03b1\u03b1\u03b1 (cid:23) 0,\n\nct\u03b1\u03b1\u03b1 \u2212 \u03b7H(\u03b1\u03b1\u03b1)\n\n\u03b1l \u221d exp(\u2212cl/\u03b7)\n\n(7)\nFor \u03b7 \u2192 \u221e one obtains \u03b1l = 1/L, while for \u03b7 \u2192 0, the LP solution is recovered and the\nestimates become the sparser the more the individual cl differ. Put differently, the parameter\n\u03b7 enables us to explore the space of different similarity combinations. The issue of selecting\na reasonable value for the parameter \u03b7 will be discussed in the next section.\nIterating this nested procedure will yield a locally optimal solution to the problem of mini-\nmizing the entropy-constrained objective, since (i) we obtain a local minimum of the mod-\ni\ufb01ed objective function and (ii) solving the outer optimization problem can only further\ndecrease the entropy-constrained objective function.\n\n4 Generalization and Model Selection\n\nIn this section, we introduce an out-of-sample extension that allows us to classify objects,\nthat have not been used for learning the parameters \u03b1\u03b1\u03b1, W and H. The extension mech-\nanism can be seen as in spirit of the Nystr\u00a8om extension (c.f. [16]). Introducing such a\ngeneralization mechanism is worthwhile for two reasons: (i) To speed-up the computation\nif the number n of objects under consideration is very large: By selecting a small subset\nof m (cid:28) n objects for the initial \ufb01t followed by the application of a computationally less\nexpensive prediction step, one can realize such a speed-up. (ii) The free parameters of the\napproach, the number of clusters k as well as the sparseness control parameter \u03b7, can be\nestimated using a re-sampling-based stability assessment that relies on the ability of an\nalgorithm to generalize to previously unseen objects.\nOut-of-Sample Extension: Suppose we have to predict class memberships for r (= n \u2212\nm in the hold-out case) additional objects in the r \u00d7 m matrix \u02dcSl. Given the decomposition\ninto W and H, let zik be the \u201cposterior\u201d estimated for the i-th object in the data set used for\nthe original \ufb01t, i.e. zi\u03bd \u221d hi\u03bd\nj wj\u03bd. We can express the weighted, normalized similarity\noj . We approximate now\n\nbetween a new object o and object i as \u02c6pio :=P\n\u02c6zo\u03bd =X\n\nzo\u03bd for a new object o by\n\noi /P\n\nl,j \u03b1l\u02dcs(l)\n\nl \u03b1l\u02dcs(l)\n\nP\n\nzi\u03bd \u02c6pio,\n\n(8)\n\ni\n\nwhich amounts to an interpolation of the zo\u03bd. These values can be obtained using the orig-\ninally computed zi\u03bd which are weighted according to their similarity between object i and\no. In the analogy to the Nystr\u00a8om approximation, the (zi\u03bd) play the role of basis elements\nwhile the \u02c6pio amount to coef\ufb01cients in the basis approximation. The prediction procedure\nrequires O(mr(l + r + k)) steps.\n\nModel Selection: The approach presented so far has two free parameters, the number\nof classes k and the sparseness penalty \u03b7. In [9], a method for determining the number of\nclasses has been introduced, that assesses the variability of clustering solutions. Thus, we\nfocus on selecting \u03b7 using stability. The assessment can be regarded as a generalization\nof cross-validation, as it relies on the dissimilarity of solutions generated from multiple\nsub-samples. In a second step, the solutions obtained from these samples are extended to\nthe complete data set by an appropriate predictor. Multiple classi\ufb01cations of the same data\n\n\f(a)\n\n(b)\n\n(c)\n\nFigure 1: Results on the toy data set (1(a)): The stability assessment (1(b)) suggests the\nrange \u03b7 \u2208 {101, 102, 5\u00b7 102}, which yield solutions matching the ground-truth. In 1(c), the\n\u03b1l are depicted for a sub-sample and \u03b7 in this range.\n\nset are obtained, whose similarity can be measured. For two clustering solutions Y, Y0 \u2208\n{1, . . . , k}n, we de\ufb01ne their disagreement as\n\nnX\n\ni=1\n\nd(Y, Y0) = min\n\u03c0\u2208Sk\n\n1\nn\n\nI{yi6=\u03c0(y0\ni)}\n\n(9)\n\nwhere Sk denotes the set of all permutation on sets of size k and IA is the indicator func-\ntion on the expression A. The measure quanti\ufb01es the 0-1 loss after the labels have been\npermuted, so that the two clustering solutions are in the best possible agreement. Perfect\nagreement up to a permutation of the labels implies d(Y, Y0) = 0. The optimal permuta-\ntion can be determined in O(k3) by phrasing the problem as a weighted bipartite matching\nproblem. Following the approach in [9], we select the \u03b7, given a pre-speci\ufb01ed range of\nadmissible values, such that the average disagreement observed on B sub-samples is min-\nimal. In this sense, the entropy regularization mechanism guides the search for similarity\ncombinations leading to stable grouping solutions. Note that, multiple minima can occur\nand may yield solutions emphasizing different aspects of the data.\n\n5 Experimental Results and Discussion\n\nThe performance of our proposal is explored by analyzing toy and real world data. For\nthe model selection (sec. 4), we have used B = 20 sub-samples with the proposed out-of-\nsample extension for prediction. For the stability assessment, different \u03b7 have been chosen\nby \u03b7 \u2208 {10\u22123, 10\u22122, 10\u22121, .5, 1, 101, 102, 5\u00b7 102, 103, 104}. We compared our results with\nNCut [15] and Lee and Seung\u2019s two NMF algorithms [11] (which measure the approxi-\nmation error of the factorization with (i) the KL divergence and (ii) the squared Frobenius\nnorm) applied to the uniform combination of similarities.\n\nToy Experiment: Figure 1(a) depicts a data set consisting of two nested rings, where\nthe clustering task consists of identifying each ring as a class. We used rbf-kernels\nk(x, y) = exp(\u2212kx \u2212 yk2/2\u03c32) for \u03c3 varying in {10\u22124, 10\u22123, 10\u22122, 100, 101} as well\nas the path kernel introduced in [4]. All methods fail when used with the individual ker-\nnels except for the path-kernel. The non-trivial problem is to detect the correct structure\ndespite the disturbing in\ufb02uence of 5 un-informative kernels. Data sets of size dn/5e have\nbeen generated by sub-sampling. Figure 1(b) depicts the stability assessment, where we see\nvery small disagreements for \u03b7 \u2208 {101, 102, 5 \u00b7 102}. At the minimum, the solution almost\nperfectly matches the ground-truth (1 error). A plot of the resulting \u03b1\u03b1\u03b1-coef\ufb01cients is given\nin \ufb01gure 1(c). NCut as well as the other nmf-methods lead to an error rate of \u2248 0.5 when\napplied to the uniformly combined similarities.\n\n10\u2212310\u2212110010210300.050.10.150.20.250.30.350.40.450.5sparsity parameter havg. disagreement12345600.050.10.150.20.250.30.350.4data source indexprobability al\f(a)\n\n(b)\n\nFigure 2: Images for the segmentation experiments.\n\nImage segmentation example:3 The next task consists of \ufb01nding a reasonable segmen-\ntation of the images depicted in \ufb01gures 2(b) and 2(a). For both images, we measured local-\nized intensity histograms and additionally computed Gabor \ufb01lter responses (e.g. [12]) on 3\nscales for 4 different orientations. For each response image, the same histogramming pro-\ncedure has been used. For all the histograms, we computed the pairwise Jensen-Shannon\ndivergence (e.g. [2]) for all pairs (i, j) of image sites and took the element-wise expo-\nnential of the negative Jensen-Shannon divergences. The resulting similarity matrices have\nbeen used as input for the nmf-based data fusion. For the sub-sampling, m = 500 objects\nhave been employed. Figures 3(a) (for the shell image) and 3(b) (for the bird image) show\nthe stability curves for these examples which exhibit minima for non-trivial \u03b7 resulting in\nnon-uniform \u03b1\u03b1\u03b1. Figure 3(c) depicts the resulting segmentation generated using \u03b1\u03b1\u03b1 indicated\nby the stability assessment, while 3(d) shows a segmentation result, where \u03b1\u03b1\u03b1 is closer to the\nuniform distribution but the stability score for the corresponding \u03b7 is low. Again, we can see\nthat weighting the different similarity measurements has a bene\ufb01cial effect, since it leads\nto improved results. The comparison with the NCut result on the uniformly weighted data\n(\ufb01g. 3(e)) con\ufb01rms that a non-trivial weighting is desirable here. Note that we have used the\nfull data set with NCut. For, the image in \ufb01g. 2(b), we observe similar behavior: the stability\nselected solution (\ufb01g. 3(f)) is more meaningful than the NCut solution (\ufb01g. 3(g)) obtained\non the uniformly weighted data. In this example, the intensity information dominates the\nsolution obtained on the uniformly combined similarities. However, the texture informa-\ntion alone does not yield a sensible segmentation. Only the non-trivial combination, where\nthe in\ufb02uence of intensity information is decreased and that of the texture information is\nincreased, gives rise to the desired result. It is additionally noteworthy, that the prediction\nmechanism employed works rather well: In both examples, it has been able to generalize\nthe segmentation from m = 500 to more than 3500 objects. However, artifacts resulting\nfrom the subsampling-and-prediction procedure cannot always be avoided, as can be seen\nin 3(f). They vanish, however, once the algorithm is re-applied to the full data (\ufb01g. 3(h)).\n\nClustering of Protein Sequences: Our \ufb01nal application is about the functional catego-\nrization of yeast proteins. We partially adopted the data used in [7] 4. Since several of the\n3588 proteins belong to more than one category, we extracted a subset of 1579 proteins ex-\nclusively belonging to one of the three categories cell cycle + DNA processing,transcription\nand protein fate. This step ensures a clear ground-truth for comparison. Of the matrices used\nin [7], we employed a Gauss Kernel derived from gene expression pro\ufb01les, one derived\nfrom Swiss-Waterman alignments, one obtained from comparisons of protein domains as\nwell as two diffusion kernels derived from protein-protein interaction data. Although the\ndata is not very discriminative for the 3-class problem, the solutions generated on the data\ncombined using the \u03b1\u03b1\u03b1 for the most stable \u03b7 lead to more than 10% improvement w.r.t. the\n\n3Only comparisons with NCut reported. The nmf results are slightly worse than those of NCut.\n4The data is available at http://noble.gs.washington.edu/proj/yeast/.\n\n\f(a)\n\n(b)\n\n(c)\n\n(f)\n\n(d)\n\n(g)\n\n(e)\n\n(h)\n\nFigure 3: Stability plots and segmentation results for the images in 2(a) and 2(b) (see text).\n\nground-truth (the disagreement measure of section 4 is used) in comparison with the so-\nlution obtained using the least stable \u03b7-parameter. The latter, however, was hardly better\nthan random guessing by having an overall disagreement of more than 0.60 (more pre-\ncisely, 0.6392 \u00b1 0.0455) on this data. For the most stable \u03b7, we observed a disagreement\naround 0.52 depending on the sub-sample (best 0.5267 \u00b1 0.0403). In this case, the largest\nweight was assigned to the protein-protein interaction data. NCut and the two nmf meth-\nods proposed in [11] lead to rates 0.5953, 0.6080 and 0.6035, respectively, when applied\nto the naive combination. Note, that the clustering results are comparable with some of\nthose obtained in [7], where the protein-protein interaction data has been used to construct\na (supervised) classi\ufb01er.\n\n6 Conclusion\n\nThis work introduced an approach to combining similarity data originating from multiple\nsources for grouping a set of objects. Adopting a pairwise clustering perspective enables\na smooth integration of multiple similarity measurements. To be able to distinguish be-\ntween desired and distractive information, a weighting mechanism is introduced leading\nto a potentially sparse convex combination of the measurements. Here, an entropy con-\nstraint is employed to control the amount of sparseness actually allowed. A stability-based\nmodel selection mechanism is used to select this free parameter. We emphasize, that this\nprocedure represents a completely unsupervised model selection strategy. The experimen-\ntal evaluation on toy and real world data demonstrates that our proposal yields meaningful\npartitions and is able to distinguish between desired and spurious structure in data.\nFuture work will focus on (i) improving the optimization of the proposed model, (ii) the\n\n10\u2212310\u22121100102103\u22120.0500.050.10.150.20.250.3sparsity parameter havg. disagreement10\u2212310\u22121100102103\u22120.0500.050.10.150.20.250.3sparsity parameter havg. disagreement\fintegration of additional constraints and (iii) the introduction of a cluster-speci\ufb01c weighting\nmechanism. The proposed method as well as its relation to other approaches discussed in\nthe literature is currently under further investigation.\n\nReferences\n\n[1] F. R. Bach and M. I. Jordan. Learning spectral clustering. In NIPS, volume 16. MIT\n\nPress, 2004.\n\n[2] J. Burbea and C. R. Rao. On the convexity of some divergence measures based on\n\nentropy functions. IEEE Trans. Inform. Theory, 28(3), 1982.\n\n[3] K. Crammer, J. Keshet, and Y. Singer. Kernel design using boosting. In NIPS, vol-\n\nume 15. MIT Press, 2003.\n\n[4] B. Fischer, V. Roth, and J. M. Buhmann. Clustering with the connectivity kernel. In\n\nNIPS, volume 16. MIT Press, 2004.\n\n[5] Thomas Hofmann. Unsupervised learning by probabilistic latent semantic analysis.\n\nMach. Learn., 42(1-2):177\u2013196, 2001.\n\n[6] E. T. Jaynes. Information theory and statistical mechanics, I and II. Physical Reviews,\n\n106 and 108:620\u2013630 and 171\u2013190, 1957.\n\n[7] G. R. G. Lanckriet, M. Deng, N. Cristianini, M. I. Jordan, and W. S. Noble. Kernel-\nbased data fusion and its application to protein function prediction in yeast. In Paci\ufb01c\nSymposium on Biocomputing, pages 300\u2013311, 2004.\n\n[8] Kenneth Lange. Optimization. Springer Texts in Statistics. Springer, 2004.\n[9] T. Lange, M. Braun, V. Roth, and J.M. Buhmann. Stability-based model selection. In\n\nNIPS, volume 15. MIT Press, 2003.\n\n[10] M. H. C. Law, M. A. T. Figueiredo, and A. K. Jain. Simultaneous feature selec-\ntion and clustering using mixture models. IEEE Trans. Pattern Anal. Mach. Intell.,\n26(9):1154\u20131166, 2004.\n\n[11] Daniel D. Lee and H. Sebastian Seung. Algorithms for non-negative matrix factor-\n\nization. In NIPS, volume 13, pages 556\u2013562, 2000.\n\n[12] B. S. Manjunath and W. Y. Ma. Texture features for browsing and retrieval of image\n\ndata. IEEE Trans. Pattern Anal. Mach. Intell., 18(8):837\u2013842, 1996.\n\n[13] D. S. Modha and W. S. Spangler. Feature weighting in k-means clustering. Mach.\n\nLearn., 52(3):217\u2013237, 2003.\n\n[14] V. Roth and T. Lange. Feature selection in clustering problems. In NIPS, volume 16.\n\nMIT Press, 2004.\n\n[15] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Trans.\n\nPattern Anal. Mach. Intell., 22(8):888\u2013905, 2000.\n\n[16] C. K. I. Williams and M. Seeger. Using the Nystr\u00a8\u0131\u00bf 1\n\nmachines. In NIPS, volume 13. MIT Press, 2001.\n\n2 m method to speed up kernel\n\n[17] E. Xing, A. Ng, M. Jordan, and S. Russell. Distance metric learning with application\n\nto clustering with side-information. In NIPS, volume 15, 2003.\n\n[18] W. Xu, X. Liu, and Y. Gong. Document clustering based on non-negative matrix\n\nfactorization. In SIGIR \u201903, pages 267\u2013273. ACM Press, 2003.\n\n\f", "award": [], "sourceid": 2880, "authors": [{"given_name": "Tilman", "family_name": "Lange", "institution": null}, {"given_name": "Joachim", "family_name": "Buhmann", "institution": null}]}