{"title": "The Local Rademacher Complexity of Lp-Norm Multiple Kernel Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 2438, "page_last": 2446, "abstract": "We derive an upper bound on the local Rademacher complexity of Lp-norm multiple kernel learning, which yields a tighter excess risk bound than global approaches. Previous local approaches analyzed the case p=1 only while our analysis covers all cases $1\\leq p\\leq\\infty$, assuming the different feature mappings corresponding to the different kernels to be uncorrelated. We also show a lower bound that shows that the bound is tight, and derive consequences regarding excess loss, namely fast convergence rates of the order $O(n^{-\\frac{\\alpha}{1+\\alpha}})$, where $\\alpha$ is the minimum eigenvalue decay rate of the individual kernels.", "full_text": "The Local Rademacher Complexity of (cid:96)p-Norm\n\nMultiple Kernel Learning\n\nMarius Kloft\u2217\n\nMachine Learning Laboratory\n\nTU Berlin, Germany\n\nkloft@tu-berlin.de\n\nGilles Blanchard\n\nDepartment of Mathematics\n\nUniversity of Potsdam, Germany\n\ngilles.blanchard@math.uni-potsdam.de\n\nAbstract\n\nWe derive an upper bound on the local Rademacher complexity of (cid:96)p-norm mul-\ntiple kernel learning, which yields a tighter excess risk bound than global ap-\nproaches. Previous local approaches analyzed the case p = 1 only while our\nanalysis covers all cases 1 \u2264 p \u2264 \u221e, assuming the different feature mappings\ncorresponding to the different kernels to be uncorrelated. We also show a lower\nbound that shows that the bound is tight, and derive consequences regarding ex-\ncess loss, namely fast convergence rates of the order O(n\u2212 \u03b1\n1+\u03b1 ), where \u03b1 is the\nminimum eigenvalue decay rate of the individual kernels.\n\n1\n\nIntroduction\n\nKernel methods [24, 21] allow to obtain nonlinear learning machines from simpler, linear ones;\nnowadays they can almost completely be applied out-of-the-box [3]. Nevertheless, after more than\na decade of research it still remains an unsolved problem to \ufb01nd the best abstraction or kernel for\na problem at hand. Most frequently, the kernel is selected from a candidate set according to its\ngeneralization performance on a validation set. Clearly, the performance of such an algorithm is\nlimited by the best kernel in the set. Unfortunately, in the current state of research, there is little\nhope that in the near future a machine will be able to automatically \ufb01nd\u2014or even engineer\u2014the\nbest kernel for a particular problem at hand [25]. However, by restricting to a less general problem,\ncan we hope to achieve the automatic kernel selection?\nIn the seminal work of Lanckriet et al. [18] it was shown that learning a support vector machine\n(SVM) [9] and a convex kernel combination at the same time is computationally feasible. This ap-\nproach was entitled multiple kernel learning (MKL). Research in the subsequent years focused on\nspeeding up the initially demanding optimization algorithms [22, 26]\u2014ignoring the fact that empir-\nical evidence for the superiority of MKL over trivial baseline approaches (not optimizing the kernel)\nwas missing. In 2008, negative results concerning the accuracy of MKL in practical applications ac-\ncumulated: at the NIPS 2008 MKL workshop [6] several researchers presented empirical evidence\nshowing that traditional MKL rarely helps in practice and frequently is outperformed by a reg-\nular SVM using a uniform kernel combination, see http://videolectures.net/lkasok08_\nwhistler/. Subsequent research (e.g., [10]) revealed further negative evidence and peaked in the\nprovocative question \u201cCan learning kernels help performance?\u201d posed by Corinna Cortes in an\ninvited talk at ICML 2009 [5].\nConsequently, despite all the substantial progress in the \ufb01eld of MKL, there remained an unsatis\ufb01ed\nneed for an approach that is really useful for practical applications: a model that has a good chance\nof improving the accuracy (over a plain sum kernel). A \ufb01rst step towards a model of kernel learning\n\u2217Marius Kloft is also with Friedrich Miescher Laboratory, Max Planck Society, T\u00a8ubingen. A part of this\nwork was done while Marius Kloft was with UC Berkeley, USA, and Gilles Blanchard was with Weierstra\u00df In-\nstitute for Applied Analysis and Stochastics, Berlin.\n\n1\n\n\fFigure 1: Result of a typical (cid:96)p-norm MKL experiment in terms of accuracy (LEFT) and kernel weights output\nby MKL (RIGHT).\n\nm=1 \u03b8mkm\n\nthat is useful for practical applications was made in [7, 13, 14]: by imposing an (cid:96)q-norm penalty\n(q > 1) rather than an (cid:96)1-norm one on the kernel combination coef\ufb01cients. This (cid:96)q-norm MKL is\nan empirical minimization algorithm that operates on the multi-kernel class consisting of functions\nf : x (cid:55)\u2192 (cid:104)w, \u03c6k(x)(cid:105) with (cid:107)w(cid:107)k \u2264 D, where \u03c6k is the kernel mapping into the reproducing kernel\nHilbert space (RKHS) Hk with kernel k and norm (cid:107).(cid:107)k, while the kernel k itself ranges over the\n\n(cid:12)(cid:12)(cid:12) (cid:107)\u03b8(cid:107)q \u2264 1, \u03b8 \u2265 0(cid:9). A conceptual milestone going\n\nset of possible kernels(cid:8)k = (cid:80)M\n\nback to the work of [1] and [20] is that this multi-kernel class can equivalently be represented as a\nblock-norm regularized linear class in the product RKHS:\n\nHp,D,M =(cid:8)fw : x (cid:55)\u2192 (cid:104)w, \u03c6(x)(cid:105) (cid:12)(cid:12) w = (w(1), . . . , w(M )),(cid:107)w(cid:107)2,p \u2264 D(cid:9),\n\nwhere there is a one-to-one mapping of q \u2208 [1,\u221e] to p \u2208 [1, 2] given by p = 2q\nq+1.\nIn Figure 1, we show exemplary results of an (cid:96)p-norm MKL experiment, achieved on the protein\nfold prediction dataset used in [4] (see supplementary material A for experimental details). We \ufb01rst\nobserve that, as expected, (cid:96)p-norm MKL enforces strong sparsity in the coef\ufb01cients \u03b8m when p = 1\nand no sparsity at all otherwise (but various degrees of soft sparsity for intermediate p). Crucially,\nthe performance (as measured by the test error) is not monotonic as a function of p; p = 1 (sparse\nMKL) yields the same performance as the regular SVM using a uniform kernel combination, but\noptimal performance is attained for some intermediate value of p\u2014namely, p = 1.14. This is a\nstrong empirical motivation to study theoretically the performance of (cid:96)p-MKL beyond the limiting\ncases p = 1 and p = \u221e.\nClearly, the complexity of (1) will be greater than one that is based on a single kernel only. However,\nit is unclear whether the increase is decent or considerably high and\u2014since there is a free parameter\np\u2014how this relates to the choice of p. To this end, the main aim of this paper is to analyze the sample\ncomplexity of the hypothesis class (1). An analysis of this model, based on global Rademacher\ncomplexities, was developed by [8] for special cases of p. In the present work, we base our main\nlocal Rademacher complexities, which allows to derive improved and\nanalysis on the theory of\nmore precise rates of convergence that cover the whole range of p \u2208 [1,\u221e].\nOutline of the contributions. This paper makes the following contributions:\n\n(1)\n\n(cid:0) 1\np\u2217 \u22121(cid:1)\n\n\u2022 An upper bound on the local Rademacher complexity of (cid:96)p-norm MKL is shown, from\nwhich we derive an excess risk bound that achieves a fast convergence rate of the order\nO(M 1+ 2\n1+\u03b1 ), where \u03b1 is the minimum eigenvalue decay rate of the individ-\nual kernels (previous bounds for (cid:96)p-norm MKL only achieved O(M\n\n\u2022 A lower bound is shown that beside absolute constants matches the upper bounds, showing\n\np\u2217 n\u2212 1\n2 ).\n\n1\n\n1+\u03b1\n\nn\u2212 \u03b1\n\nthat our results are tight.\n\n\u2022 The generalization performance of (cid:96)p-norm MKL as guaranteed by the excess risk bound\nis studied for varying values of p, shedding light on the appropriateness of a small/large p\nin various learning scenarios.\n\nFurthermore, we also present a simpler, more general proof of the global Rademacher bound shown\nin [8] (at the expense of a slightly worse constant). A comparison of the rates obtained with local\nand global Rademacher analysis is carried out in Section 3.\n\n2\n\nCHPZSVL1L4L14L30SW1SW200.10.20.30.40.50.60.70.8Test Set Accuracy SVM (single)1\u2212norm MKL1.07\u2212norm MKL1.14\u2212norm MKL1.33\u2212norm MKLSVM (all)CHPZSVL1L4L14L30SW1SW200.20.40.60.811.21.4Kernel Weights \u03b8i 1\u2212norm MKL1.07\u2212norm MKL1.14\u2212norm MKL1.33\u2212norm MKLSVM\fn k(xi, xj). Also, we denote u = (u(m))M\n\nNotation. We abbreviate Hp = Hp,D = Hp,D,M if clear from the context. We denote the (normal-\nized) kernel matrices corresponding to k and km by K and Km, respectively, i.e., the ijth entry of\nm=1 = (u(1), . . . , u(M )) \u2208 H = H1 \u00d7 . . . \u00d7 HM .\nK is 1\nFurthermore, let P be a probability measure on X i.i.d. generating the data x1, . . . , xn and denote\nby E the corresponding expectation operator. We work with operators in Hilbert spaces and will use\ninstead of the usual vector/matrix notation \u03c6(x)\u03c6(x)(cid:62) the tensor notation \u03c6(x) \u2297 \u03c6(x) \u2208 HS(H),\nwhich is a Hilbert-Schmidt operator H (cid:55)\u2192 H de\ufb01ned as (\u03c6(x) \u2297 \u03c6(x))u = (cid:104)\u03c6(x), u(cid:105) \u03c6(x). The\nspace HS(H) of Hilbert-Schmidt operators on H is itself a Hilbert space, and the expectation\nE\u03c6(x) \u2297 \u03c6(x) is well-de\ufb01ned and belongs to HS(H) as soon as E(cid:107)\u03c6(x)(cid:107)2 is \ufb01nite, which will\nalways be assumed. We denote by J = E\u03c6(x) \u2297 \u03c6(x) and Jm = E\u03c6m(x) \u2297 \u03c6m(x) the uncen-\ntered covariance operators corresponding to variables \u03c6(x) and \u03c6m(x), respectively; it holds that\ntr(J) = E(cid:107)\u03c6(x)(cid:107)2\n\n2 and tr(Jm) = E(cid:107)\u03c6m(x)(cid:107)2\n2.\n\nn\n\n(cid:80)n\nGlobal Rademacher Complexities We \ufb01rst review global Rademacher complexities (GRC) in\nmultiple kernel learning. Let x1, . . . , xn be an i.i.d. sample drawn from P . The global Rademacher\ncomplexity is de\ufb01ned as R(Hp) = E supfw\u2208Hp(cid:104)w, 1\ni=1 \u03c3i\u03c6(xi)(cid:105), where (\u03c3i)1\u2264i\u2264n is an i.i.d.\ndenoted by (cid:98)R(Hp) = E(cid:2)R(Hp)(cid:12)(cid:12)x1, . . . , xn\n(cid:80)n\nfamily (independent of \u03c6(xi)) of Rademacher variables (random signs). Its empirical counterpart is\n(cid:0)cp\u2217(cid:13)(cid:13)(cid:0) tr(Km)(cid:1)M\ni=1 \u03c3i\u03c6(xi)(cid:105).\nIn the recent paper of [8] it was shown (cid:98)R(Hp) \u2264 D\n\n(cid:3) = E\u03c3 supfw\u2208Hp(cid:104)w, 1\n\n(cid:1)1/2 for p \u2208 [1, 2]\n\nand p\u2217 being an integer (where c = 23/44 and p\u2217 := p\np\u22121 is the conjugated exponent). This bound\nis tight and improves a series of loose results that were given for p = 1 in the past (see [8] and\nreferences therein). In fact, the above result can be extended to the whole range of p \u2208 [1,\u221e] (in\nthe supplementary material we present a quite simple proof using c = 1):\nProposition 1 (GLOBAL RADEMACHER COMPLEXITY BOUND). For any p \u2265 1 the empirical\nversion of global Rademacher complexity of the multi-kernel class Hp can be bounded as\n\n(cid:13)(cid:13) p\u2217\n\nm=1\n\nn\n\nn\n\n2\n\n(cid:98)R(Hp) \u2264 min\n\nt\u2208[p,\u221e]\n\nD\n\n(cid:115)\n\n(cid:13)(cid:13)(cid:13)(cid:16) 1\n\nn\n\nt\u2217\nn\n\n(cid:17)M\n\n(cid:13)(cid:13)(cid:13) t\u2217\n\n2\n\n.\n\ntr(Km)\n\nm=1\n\nInterestingly, the above GRC bound is not monotonic in p and thus the minimum is not always\nattained for t := p.\n\n2 The Local Rademacher Complexity of Multiple Kernel Learning\n\n(cid:80)n\ni=1 \u03c3i\u03c6(xi)(cid:105), where P f 2\n\nn\n\nsample drawn from P . We de\ufb01ne the local Rademacher com-\nLet x1, . . . , xn be an i.i.d.\nw\u2264r(cid:104)w, 1\nplexity (LRC) of Hp as Rr(Hp) = E supfw\u2208Hp:P f 2\nw :=\nE(fw(\u03c6(x)))2. Note that it subsumes the global RC as a special case for r = \u221e. We will also\nuse the following assumption in the bounds for the case p \u2208 [1, 2]:\nAssumption (U) (no-correlation). Let x \u223c P . The Hilbert space valued random variables\n\u03c61(x), . . . , \u03c6M (x) are said to be (pairwise) uncorrelated if for any m (cid:54)= m(cid:48) and w \u2208 Hm , w(cid:48) \u2208\nHm(cid:48) , the real variables (cid:104)w, \u03c6m(x)(cid:105) and (cid:104)w(cid:48), \u03c6m(cid:48)(x)(cid:105) are uncorrelated.\nFor example, if X = RM , the above means that the input variable x \u2208 X has independent co-\nordinates, and the kernels k1, . . . , kM each act on a different coordinate. Such a setting was also\nconsidered by [23] (for sparse MKL). To state the bounds, note that covariance operators enjoy\nj=1 \u03bbjuj \u2297 uj and\n)j\u22651 form orthonormal\nbases of H and Hm, respectively. We are now equipped to state our main results:\nTheorem 2 (LOCAL RADEMACHER COMPLEXITY BOUND, p \u2208 [1, 2] ). Assume that the kernels\nare uniformly bounded ((cid:107)k(cid:107)\u221e \u2264 B < \u221e) and that Assumption (U) holds. The local Rademacher\ncomplexity of the multi-kernel class Hp can be bounded for any p \u2208 [1, 2] as\n\ndiscrete eigenvalue-eigenvector decompositions J = E\u03c6(x) \u2297 \u03c6(x) = (cid:80)\u221e\nJm = Ex(m) \u2297 x(m) =(cid:80)\u221e\n(cid:118)(cid:117)(cid:117)(cid:116) 16\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) \u221e(cid:88)\n\nRr(Hp) \u2264 min\nt\u2208[p,2]\n\nBeDM 1\n\nt\u2217 t\u2217\n\nn\n\n.\n\n, where (uj)j\u22651 and (u(m)\n\nt\u2217 , ceD2t\u22172\u03bb(m)\n\nj \u2297 u(m)\n\nj=1 \u03bb(m)\n\nj u(m)\n\n(cid:17)(cid:19)M\n\n(cid:13)(cid:13)(cid:13)(cid:13) t\u2217\n\nrM 1\u2212 2\n\n(cid:16)\n\nmin\n\n\u221a\n\n+\n\nn\n\nj\n\nj\n\nj\n\nj=1\n\nm=1\n\n2\n\n3\n\n\fRr(Hp) \u2264 min\nt\u2208[p,\u221e]\n\nTheorem 3 (LOCAL RADEMACHER COMPLEXITY BOUND, p \u2208 [2,\u221e] ). For any p \u2208 [2,\u221e],\n\nIt is interesting to compare the above bounds for the special case p = 2 with the ones of Bartlett et al.\n[2]. The main term of the bound of Theorem 3 (taking t = p = 2) is then essentially determined by\n\nj=1 min(r, \u03bb(m)\n\nj\n\nn\n\nm=1\n\n(cid:80)M\n\nO(cid:0)(cid:0) 1\n(cid:80)\u221e\nis equivalently of order O(cid:0)(cid:0) 1\n(cid:8)\u03bb(m)\n{\u03bbi, i \u2265 1} =(cid:83)M\n(cid:13)(cid:13)(cid:0)(cid:80)\u221e\nbound Rr(H1) \u2264 (cid:0) 16\n\nm=1\n\ni\n\nn\n\nj=1\n\n\u221e(cid:88)\n\nt\u2217 \u22121\u03bbj).\n\nmin(r, D2M 2\n\n(cid:118)(cid:117)(cid:117)(cid:116) 2\n)(cid:1)1/2(cid:1). If the variables (\u03c6m(x)) are centered and uncorrelated, this\nj=1 min(r, \u03bbj)(cid:1)1/2(cid:1) because spec(J) =(cid:83)M\n(cid:80)\u221e\n(cid:13)(cid:13)\u221e\nj=1 min(cid:0)rM, e3D2(log M )2\u03bb(m)\n(cid:1)1/2\n\nm=1 spec(Jm); that is,\n, i \u2265 1}; this rate is also what we would obtain through Theorem 3, so\n\n(cid:1)(cid:1)M\n\n3\n2 D log(M )\n\nm=1\n\n\u221a\n\n+\n\nBe\n\nn\n\nn\n\nn\n\nboth bounds on the LRC recover the rate shown in [2] for the special case p = 2.\nIt is also interesting to study the case p = 1: by using t = (log(M ))\u2217 in Theorem 2, we obtain the\n, for\nall M \u2265 e2. We now turn to proving Theorem 2. the proof of Theorem 3 is straightforward and\nshown in the supplementary material C.\nProof of Theorem 2. . Note that it suf\ufb01ces to prove the result for t = p as trivially (cid:107)w(cid:107)2,t \u2264 (cid:107)w(cid:107)2,p\nholds for all t \u2265 p so that Hp \u2286 Ht and therefore Rr(Hp) \u2264 Rr(Ht).\nthe no-correlation assumption, we will work in large parts of the proof with the centered class (cid:102)Hp =\n(cid:12)(cid:12) (cid:107)w(cid:107)2,p \u2264 D(cid:9), wherein (cid:101)fw : x (cid:55)\u2192 (cid:104)w,(cid:101)\u03c6(x)(cid:105), and(cid:101)\u03c6(x) := \u03c6(x) \u2212 E\u03c6(x). We start the proof\n(cid:8)(cid:101)fw\nSTEP 1: RELATING THE ORIGINAL CLASS WITH THE CENTERED CLASS.\nby noting that (cid:101)fw(x) = fw(x) \u2212 (cid:104)w, E\u03c6(x)(cid:105) = fw(x) \u2212 E(cid:104)w, \u03c6(x)(cid:105) = fw(\u03c6(x)) \u2212 Efw(\u03c6(x)),\n\nIn order to exploit\n\nj\n\n2\n\np\u2217\n\n2\n\np\u2217\n\n=\n\nm=1\n\nP f 2\n\nJensen\u2264\n\nm=1\np\u2217\n\nFurthermore we note that by Jensen\u2019s inequality\n\nso that, by the bias-variance decomposition, it holds that\n\n(cid:1)2\nw + (cid:0)P fw\nw = Efw(x)2 = E (fw(x) \u2212 Efw(x))2 + (Efw(\u03c6(x)))2 = P(cid:101)f 2\n(cid:19) 1\n(cid:13)(cid:13)E\u03c6(x)(cid:13)(cid:13)2,p\u2217 =\n(cid:10)E\u03c6m(x), E\u03c6m(x)(cid:11) p\u2217\n(cid:115)(cid:13)(cid:13)(cid:13)(cid:16)\n(cid:13)(cid:13)(cid:13) p\u2217\n(cid:17)M\nn(cid:88)\n\u03c3iE\u03c6(x)(cid:11) .\n(cid:10)w,\nConcerning the \ufb01rst term of the above upper bound, using (2) we have P(cid:101)f 2\n\u03c3i(cid:101)\u03c6(xi)(cid:11) = Rr((cid:101)Hp).\n\n(cid:19) 1\n(cid:18) M(cid:88)\n(cid:18) M(cid:88)\n(cid:13)(cid:13)E\u03c6m(x)(cid:13)(cid:13)p\u2217\n(cid:19) 1\n(cid:18) M(cid:88)\nE(cid:10)\u03c6m(x), \u03c6m(x)(cid:11) p\u2217\nn(cid:88)\n(cid:10)w,\n\u03c3i(cid:101)\u03c6(xi)(cid:11) + E sup\nn(cid:88)\nn(cid:88)\n(cid:10)w,\n\u03c3i(cid:101)\u03c6(xi)(cid:11) \u2264 E sup\nP(cid:101)f 2\nfw\u2208Hp,\nw\u2264r\nn(cid:88)\n\u03c3iE\u03c6(x)(cid:11) \u2264 \u221a\n(cid:10)w,\n\nRr(Hp) \u2264 E sup\nfw\u2208Hp,\nw\u2264r\nP f 2\n\nNow to bound the second term, we write\n\nE sup\nfw\u2208Hp,\nw\u2264r\nP f 2\n\n(cid:104)w, E\u03c6(x)(cid:105) .\n\nw \u2264 P f 2\n\n(cid:10)w,\n\nfw\u2208Hp,\nw\u2264r\nP f 2\n\ntr(Jm)\n\n1\nn\n\n1\nn\n\n1\nn\n\n1\nn\n\nm=1\n\nm=1\n\ni=1\n\ni=1\n\ni=1\n\ni=1\n\n=\n\n2\n\n2\n\nE sup\nfw\u2208Hp,\nw\u2264r\nP f 2\nNow observe that we have\n\n1\nn\n\ni=1\n\nw, and thus\n\nn sup\nfw\u2208Hp,\nw\u2264r\nP f 2\n\n(cid:114)(cid:13)(cid:13)(cid:0) tr(Jm)(cid:1)M\n(cid:114)(cid:13)(cid:13)(cid:0) tr(Jm)(cid:1)M\n(cid:13)(cid:13) p\u2217\n\nm=1\n\n2\n\nm=1\n\n(cid:17)\n\n(cid:13)(cid:13) p\u2217\n\n2\n\nas well as (cid:104)w, E\u03c6(x)(cid:105) = Efw(x) \u2264(cid:112)P f 2\n(3)\u2264 (cid:107)w(cid:107)2,p\n(cid:104)w, E\u03c6(x)(cid:105) H\u00a8older\u2264 (cid:107)w(cid:107)2,p (cid:107)E\u03c6(x)(cid:107)2,p\u2217\n(cid:16)\u221a\nRr(Hp) \u2264 Rr((cid:101)Hp) + n\u2212 1\n\n2 min\n\nr, D\n\nw . We \ufb01nally obtain, putting together the steps above,\n\n.\n\n(4)\n\nThis shows that there is no loss in working with the centered class instead of the uncentered one.\n\n4\n\nso that we can express the complexity of the centered class in terms of the uncentered one as follows:\n\n.\n\n(2)\n\n(3)\n\n\fj\n\nSTEP 2: BOUNDING THE COMPLEXITY OF THE CENTERED CLASS.\nIn this step of the proof\nwe generalize the technique of [19] to multi-kernel classes. First we note that, since the (centered)\n\ncovariance operator E(cid:101)\u03c6m(x) \u2297(cid:101)\u03c6m(x) is also a self-adjoint Hilbert-Schmidt operator on Hm, there\nexists an eigendecomposition E(cid:101)\u03c6m(x) \u2297(cid:101)\u03c6m(x) =(cid:80)\u221e\nj=1(cid:101)\u03bb(m)\n(cid:101)u(m)\nj \u2297(cid:101)u(m)\n, wherein ((cid:101)u(m)\nis an orthogonal basis of Hm. Furthermore, the no-correlation assumption (U) entails E(cid:101)\u03c6l(x) \u2297\n(cid:101)\u03c6m(x) = 0 for all l (cid:54)= m. As a consequence, for all j and m,\n(cid:10)wm,(cid:101)\u03c6m(x)(cid:11)(cid:17)2\nw = E((cid:101)fw(x))2 = E(cid:16) M(cid:88)\n(cid:68)\n(cid:69)2\nM(cid:88)\n\u221e(cid:88)\nP(cid:101)f 2\n(cid:101)\u03bb(m)\nwm,(cid:101)u(m)\n(cid:16) 1\n(cid:68)(cid:101)u(m)\n(cid:69)\n(cid:17)(cid:101)u(m)\nE(cid:68) 1\n(cid:101)\u03bb(m)\nn(cid:88)\nn(cid:88)\nE(cid:101)\u03c6m(xi) \u2297(cid:101)\u03c6m(xi)\n\u03c3i(cid:101)\u03c6m(xi),(cid:101)u(m)\nRr((cid:101)Hp)\n\nLet now h1, . . . , hM be arbitrary nonnegative integers. We can express the LRC in terms of the\neigendecompositon as follows\n\nm=1\n1\nn\n\n(cid:69)2\n\n)j\u22651\n\n(6)\n\n(5)\n\nj\nn\n\nm=1\n\nj=1\n\ni=1\n\ni=1\n\n=\n\n=\n\n=\n\nn\n\nn\n\n.\n\n,\n\nj\n\nj\n\nj\n\nj\n\nj\n\nj\n\nj\n\nw,\n\n(cid:68)\n(cid:34)(cid:118)(cid:117)(cid:117)(cid:116) M(cid:88)\nhm(cid:88)\n(cid:28)\n\nsup\n\n= E\n\nfw\u2208(cid:101)Hp:P(cid:101)f 2\nw\u2264r\nC.-S., Jensen\u2264\nP(cid:101)f 2\nsup\nw\u2264r\n+ E sup\nfw\u2208(cid:101)Hp\n\nm=1\n\nj=1\n\nw,\n\n1\nn\n\ni=1\n\n(cid:69)\nn(cid:88)\n\u03c3i(cid:101)\u03c6(xi)\n(cid:101)\u03bb(m)\n(cid:104)w(m),(cid:101)u(m)\n(cid:16) \u221e(cid:88)\nn(cid:88)\n\nj\n\nj\n\n(cid:104) 1\nn\n\nj=hm+1\n\ni=1\n\nso that (5) and (6) yield\n\nRr((cid:101)Hp)\n\n(5), (6),H\u00a8older\u2264\n\n(cid:115)\n\nr(cid:80)M\n\nm=1 hm\nn\n\n+ D E\n\n= E\n\nj\n\nm=1\n\n(cid:105)2\n\nsup\n\n(cid:68)(cid:0)w(m)(cid:1)M\nfw\u2208(cid:101)Hp:P(cid:101)f 2\n(cid:118)(cid:117)(cid:117)(cid:116) M(cid:88)\n(cid:16)(cid:101)\u03bb(m)\n(cid:17)\u22121 E(cid:10) 1\nhm(cid:88)\nw\u2264r\n(cid:29)\n(cid:17)M\n\u03c3i(cid:101)\u03c6m(xi),(cid:101)u(m)\n(cid:105)(cid:101)u(m)\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:16) \u221e(cid:88)\n\nn(cid:88)\n\n(cid:104) 1\nn\n\nm=1\n\nj=1\n\nn\n\nj\n\nj\n\nj=hm+1\n\ni=1\n\n(cid:69)\n\nm=1,(cid:0) 1\nn(cid:88)\n\nn(cid:88)\n\u03c3i(cid:101)\u03c6m(xi)(cid:1)M\n(cid:35)\n(cid:11)2\n\u03c3i(cid:101)\u03c6m(xi),(cid:101)u(m)\n\ni=1\n\nn\n\nj\n\ni=1\n\nm=1\n\n\u03c3i(cid:101)\u03c6m(xi),(cid:101)u(m)\n\nj\n\n(cid:17)M\n\nm=1\n\n(cid:13)(cid:13)(cid:13)(cid:13)2,p\u2217\n\n.\n\n(cid:105)(cid:101)u(m)\n\nj\n\nE(cid:16) \u221e(cid:88)\n\nn\n\n\u2264 (cid:113) p\u2217\n(cid:18) M(cid:88)\n\n(cid:16)(cid:80)M\n\nE(cid:16)(cid:80)\n\nSTEP 3: KHINTCHINE-KAHANE\u2019S AND ROSENTHAL\u2019S INEQUALITIES. We use the Khintchine-\nKahane (K.-K.) inequality (see Lemma B.2 in the supplementary material) to further bound the right\nterm in the above expression as E\n\n(cid:104) 1\n\n(cid:17)M\n\n(cid:105)(cid:101)u(m)\n\nj\n\nm=1\n\n(cid:13)(cid:13)(cid:13)2,p\u2217\n\nm=1\n\n. Note that for p \u2265 2 it holds that\np\u2217/2 \u2264 1, and thus it suf\ufb01ces to employ Jensen\u2019s inequality once again to move the expectation\noperator inside the inner term. In the general case we need a handle on the p\u2217\n2 -th moments and to\nthis end employ Lemma C.1 (Rosenthal + Young; see supplementary material), which yields\n\nj>hm\n\n1\nn\n\nj\n\nj\n\nn\n\np\u2217\n\nj>hm\n\n(cid:13)(cid:13)(cid:13)(cid:16)(cid:80)\n(cid:80)n\ni=1 \u03c3i(cid:101)\u03c6m(xi),(cid:101)u(m)\n(cid:105)2(cid:17) p\u2217\n2 (cid:17) 1\n(cid:80)n\ni=1(cid:104)(cid:101)\u03c6m(xi),(cid:101)u(m)\n2 (cid:19) 1\n(cid:105)2(cid:17) p\u2217\n(cid:104)(cid:101)\u03c6m(xi),(cid:101)u(m)\n(cid:18)(cid:16) B\n(cid:16) \u221e(cid:88)\n(cid:17) p\u2217\n(cid:18) M(cid:88)\n(cid:16) \u221e(cid:88)\n(cid:101)\u03bb(m)\n\n(ep\u2217)\n\n+\n\n+\n\np\u2217\n\nj\n\n2\n\n2\n\n2\n\ni=1\n\n1\nn\n\nn(cid:88)\n(cid:32) M(cid:88)\n(cid:118)(cid:117)(cid:117)(cid:116)ep\u2217\n(cid:32)\n\nm=1\n\np\u2217\n\nm=1\n\nR+Y\u2264\n\nj=hm+1\n\nn(cid:88)\nE(cid:104)(cid:101)\u03c6m(xi),(cid:101)u(m)\np\u2217(cid:33)\n2 (cid:19) 2\n(cid:17) p\u2217\nwhere for (\u2217) we used the subadditivity of p\u2217\u221a\u00b7. Note that \u2200j, m : (cid:101)\u03bb(m)\nMirsky-Wielandt theorem since E\u03c6m(x)\u2297 \u03c6m(x) = E(cid:101)\u03c6m(x)\u2297(cid:101)\u03c6m(x) +E\u03c6m(x)\u2297E\u03c6m(x). Thus\n\nj \u2264 \u03bb(m)\n\nby the Lidskii-\n\nBM\nn\n\nj=hm+1\n\nj=hm+1\n\n(\u2217)\u2264\n\n1\nn\n\nm=1\n\ni=1\n\np\u2217\n\np\u2217\n\nn\n\nj\n\nj\n\nj\n\n2 (cid:19)(cid:33) 1\n(cid:105)2(cid:17) p\u2217\n\n5\n\n\fn\n\n\u2264\n\nj=hm+1\n\nj=hm+1\n\n+\n\nn\n\n+ D\n\nn\n\n\u221a\n\n+\n\n.\n\n(7)\n\n2\n\np\u2217\n\n+\n\nm=1\n\n2\n\nm=1\n\n2\n\n\u03bb(m)\nj\n\n\u03bb(m)\nj\n\n(cid:33)\n\n1\n\np\u2217 p\u2217\n\nBeDM\n\nBM\nn\n\n(cid:19)M\n\nm=1 hm\nn\n\nm=1 hm\nn\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) p\u2217\n\nby the subadditivity of the root function\n\nRr((cid:101)Hp) \u2264\n(cid:115)\n\n(cid:115)\nr(cid:80)M\nr(cid:80)M\n\n(cid:118)(cid:117)(cid:117)(cid:116) ep\u22172\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)\n(cid:32)\n(cid:18) \u221e(cid:88)\n(cid:118)(cid:117)(cid:117)(cid:116) ep\u22172D2\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) p\u2217\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)\n(cid:18) \u221e(cid:88)\n(cid:19)M\n(cid:1)1/2(cid:17) \u2264\nr, D(cid:0)(cid:13)(cid:13)(cid:0) tr(Jm)(cid:1)M\n2 min(cid:0)\u221a\n(cid:13)(cid:13)(cid:0)(cid:80)\u221e\n(cid:13)(cid:13) p\u2217\n(cid:1)M\n(cid:1)1/2\n(cid:0) ep\u2217 2D2\nr, D(cid:0)(cid:13)(cid:13)(cid:0) tr(Jm)(cid:1)M\n(cid:13)(cid:13) p\u2217\nm=1 hm/n(cid:1)1/2 (in case that at least one\n2 min(cid:0)\u221a\n(cid:1)1/2(cid:1) \u2264 (cid:0)r(cid:80)M\nr, D(cid:0)(cid:13)(cid:13)(cid:0) tr(Jm)(cid:1)M\n(cid:1)1/2(cid:1) \u2264\n2 min(cid:0)\u221a\n(cid:13)(cid:13)(cid:0)(cid:80)\u221e\n(cid:13)(cid:13) p\u2217\n(cid:0) r(cid:80)M\n+(cid:0) ep\u2217 2D2\n(cid:1)M\n(cid:1)1/2\n(cid:1)1/2\n(cid:115)\n(cid:118)(cid:117)(cid:117)(cid:116) 4ep\u22172D2\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) p\u2217\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)\n4r(cid:80)M\n(cid:19)M\n(cid:18) \u221e(cid:88)\n\nSTEP 4: BOUNDING THE COMPLEXITY OF THE ORIGINAL CLASS.\nall nonnegative integers hm we either have n\u2212 1\ncase\n\n(8)\nfor all nonnegative integers hm \u2265 0. Later, we will use the above bound (8) for the computation of\nthe excess loss; however, to gain more insight in the bounds\u2019 properties, we express it in terms of\nthe truncated spectra of the kernels at the scale r as follows:\n\nn\u2212 1\nhm is nonzero) so that\n\n. Thus the following preliminary bound\n\nin any case we get n\u2212 1\n\nn\nfollows from (4) by (7):\n\n(cid:13)(cid:13) p\u2217\n(cid:13)(cid:13) p\u2217\n\nNow note that for\n\nj=hm+1 \u03bb(m)\n\nj=hm+1 \u03bb(m)\n\nRr(Hp) \u2264\n\nm=1 hm\nn\n\nm=1\nor\n\nhm are\n\nBeDM\n\n1\n\np\u2217 p\u2217\n\n\u03bb(m)\nj\n\nm=1\n\n2\n\nm=1\n\n2\n\nm=1\n\n2\n\nm=1\n\n2\n\nit\n\nholds\n\nm=1 hm\n\nn\n\nm=1\n\n2\n\nj\n\nn\n\nzero)\n\n\u221a\n\n+\n\n(in\n\nall\n\n,\n\nn\n\nn\n\nj\n\nj=hm+1\n\n+\n\n2\n\nSTEP 5: RELATING THE BOUND TO THE TRUNCATION OF THE SPECTRA OF THE KERNELS.\nNext, we notice that for all nonnegative real numbers A1, A2 and any a1, a2 \u2208 Rm\n+ it holds for all\nq \u2265 1\n\nA1 +\n\n(9)\n(10)\n(the \ufb01rst statement follows from the concavity of the square root function and the second one is\nreadily proved; see Lemma C.3 in the supplementary material) and thus\n\n(cid:112)\nA2 \u2264 (cid:112)2(A1 + A2)\n\n(cid:112)\n(cid:107)a1(cid:107)q + (cid:107)a2(cid:107)q \u2264 21\u2212 1\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)\n\np\u2217 hm + ep\u22172D2\n\nrM 1\u2212 2\n\n(cid:118)(cid:117)(cid:117)(cid:116) 16\n\nn\n\n1\n\np\u2217 p\u2217\n\n,\n\nBeDM\n\nn\n\nRr(Hp)\u2264\n\nq (cid:107)a1 + a2(cid:107)q \u2264 2(cid:107)a1 + a2(cid:107)q\n\u221e(cid:88)\n\n(cid:19)M\nH\u00a8older\u2264 (cid:16)(cid:107)1(cid:107)(p/q)\u2217 (cid:107)aq(cid:107)p/q\n\n(cid:13)(cid:13)(cid:13)(cid:13) p\u2217\n(cid:17)1/q\n\n\u03bb(m)\nj\n\nj=hm+1\n\nm=1\n\n+\n\n2\n\n\u221a\n\nwhere we used that for all non-negative a \u2208 RM and 0 < q < p \u2264 \u221e it holds\n((cid:96)q-to-(cid:96)p conversion)\nSince the above holds for all nonnegative integers hm, the result follows, completing the proof.\n\n(cid:107)a(cid:107)q = (cid:104)1, aq(cid:105) 1\n\np (cid:107)a(cid:107)p .\n\nq \u2212 1\n\n= M\n\n1\n\nq\n\n(11)\n\n2.1 Lower and Excess Risk Bounds\n\nTo investigate the tightness of the presented upper bounds on the LRC of Hp, we consider the case\nwhere \u03c61(x), . . . , \u03c6M (x) are i.i.d; for example, this happens if the original input space X is RM ,\nthe original input variable x \u2208 X has i.i.d. coordinates, and the kernels k1, . . . , kM are identical and\neach act on a different coordinate of x.\nTheorem 4 (LOWER BOUND). Assume that the kernels are centered and i.i.d.. Then, there is an\nabsolute constant c such that if \u03bb(1) \u2265 1\n\nnD2 then for all r \u2265 1\n\nn and p \u2265 1,\n\n(cid:118)(cid:117)(cid:117)(cid:116) c\n\nn\n\n\u221e(cid:88)\n\nj=1\n\nRr(Hp,D,M ) \u2265\n\nmin(rM, D2M 2/p\u2217\n\n\u03bb(1)\nj ).\n\n(12)\n\nComparing the above lower bound with the upper bounds, we observe that\nper bound of Theorem 2 for centered identical\n\nindependent kernels\n\nis of\n\nthe up-\nthe order\n\n6\n\n\fO(cid:0)(cid:113)(cid:80)\u221e\n\nj=1 min(cid:0)rM, D2M\n\n2\n\np\u2217 \u03bb(1)\n\nj\n\n(cid:1)(cid:1), thus matching the rate of the lower bound (the same holds\n\nfor the bound of Theorem 3). This shows that the upper bounds of the previous section are tight.\n\n(cid:80)n\n\nn\u2212 \u03b1\n\n1+\u03b1\n\n2(cid:1) (if \u03b1 \u2248 1\n\n1\n\np\u2217 n\u2212 1\n\n1\nn\n\nP (l \u02c6f \u2212 lf\u2217 ) := E l( \u02c6f (x), y) \u2212 E l(f\u2217(x), y), where f\u2217 := argminf\n\nAs an application of our results to prediction problems such as classi\ufb01cation or regression, we also\nbound the excess loss of empirical minimization, \u02c6f := argminf\ni=1 l(f (xi), yi), w.r.t. to a loss\nE l(f (x), y) .\nfunction l:\nWe use the analysis of Bartlett et al. [2] to show the following excess risk bound under the assump-\ntion of algebraically decreasing eigenvalues of the kernel matrices, i.e. \u2203d > 0, \u03b1 > 1,\u2200m : \u03bb(m)\nj \u2264\nd j\u2212\u03b1 (proof shown in the supplementary material E):\nTheorem 5. Assume that (cid:107)k(cid:107)\u221e \u2264 B and \u2203d > 0, \u03b1 > 1,\u2200m : \u03bb(m)\n\u2264 d j\u2212\u03b1. Let l be a\nLipschitz continuous loss with constant L and assume there is a positive constant F such that \u2200f \u2208\nF : P (f \u2212 f\u2217)2 \u2264 F P (lf \u2212 lf\u2217 ). Then for all z > 0 with probability at least 1 \u2212 e\u2212z the excess\nloss of the multi-kernel class Hp can be bounded for p \u2208 [1, 2] as\n\nj\n\n1+\u03b1\n\n+\n\n47\n\n\u221a\n\n186\n\n1+\u03b1 F\n\nt\u2217 t\u2217\n\n(cid:114) 3 \u2212 \u03b1\n\n\u03b1\u22121\n\u03b1+1 M 1+ 2\n\n1 \u2212 \u03b1\nBDLM 1\n\nP (l \u02c6f \u2212 lf\u2217 ) \u2264 min\nt\u2208[p,2]\n\n(cid:0)dD2L2t\u22172(cid:1) 1\n\n(cid:0) 1\nt\u2217 \u22121(cid:1)\nWe see from the above bound that convergence can be almost as slow as O(cid:0)p\u2217M\nis small ) and almost as fast as O(cid:0)n\u22121(cid:1) (if \u03b1 is large).\n(cid:16)(cid:0)t\u2217D(cid:1) 2\n(cid:16)\n\n\u2200t \u2208 [p, 2] : P (l \u02c6f \u2212 lf\u2217 ) = O\n\n(22BDLM 1\nt\u2217 + 27F )z\nn\n\nInterpretation of Bounds\n\nt\u2217 \u22121(cid:1)\n(cid:0) 1\n(cid:17)\n\n1+\u03b1 M 1+ 2\n\nn\u2212 \u03b1\n\n3\n\n+\n\n1+\u03b1\n\n1+\u03b1\n\nn\n\n(cid:17)\n\n.\n\n1\n\nIn this section, we discuss the rates of Theorem 5 obtained by local analysis bounds, that is\n\n2\n\n.\n\n(cid:16)\n\n(cid:16)\n\nOn the other hand, the global Rademacher complexity directly leads to a bound of the form [8]\n\n\u2200t \u2208 [p, 2] : P (l \u02c6f \u2212 lf\u2217 ) = O\n\n(14)\nTo compare the above rates, we \ufb01rst assume p \u2265 (log M )\u2217 so that the best choice is t = p. Clearly,\nthe rate obtained through local analysis is better in n since \u03b1 > 1. Regarding the rate in the number\nof kernels M and the radius D, a straightforward calculation shows that the local analysis improves\nover the global one whenever M\nn) . Interestingly, this \u201cphase transition\u201d does not\ndepend on \u03b1 (i.e. the \u201ccomplexity\u201d of the kernels), but only on p.\nSecond, if p \u2264 (log M )\u2217, the best choice in (13) and (14) is t = (log M )\u2217 so that\n\n\u221a\n1\np /D = O(\n\nt\u2217DM\n\nt\u2217 n\u2212 1\n\nt\u2217 n\u2212 1\n\nt\u2217DM\n\n(cid:16) 1\n\n(cid:16) M\n\nD log M = O(\n\nP (l \u02c6f \u2212 lf\u2217 ) \u2264 O\nM n\u22121, min\n(15)\nt\u2208[p,2]\n\u221a\nn). Note, that when letting \u03b1 \u2192 \u221e the\nand the phase transition occurs for\nclassical case of aggregation of M basis functions is recovered. This situation is to be com-\npared to the sharp analysis of the optimal convergence rate of convex aggregation of M func-\ntions obtained by [27] in the framework of squared error loss regression, which is shown to be\n. This corresponds to the setting studied here with D = 1, p = 1\nO\nand \u03b1 \u2192 \u221e, and we see that our bound recovers (up to log factors) in this case this sharp bound and\nthe related phase transition phenomenon.\nPlease note that, by introducing an inequality in Eq. (5), Assumption (U)\u2014a similar assumption was\nalso used in [23]\u2014can be relaxed to a more general, RIP-like assumption as used in [16]; this comes\nat the expense of an additional factor in the bounds (details omitted here).\n\n(cid:17)1/2(cid:17)(cid:17)(cid:17)\n\n(cid:16) M\u221a\n\nn log\n\n(cid:17)(cid:17)\n\n(cid:16)\n\nmin\n\nmin\n\nn ,\n\nM\n\nn\n\n1\n\n2\n\n.\n\n(13)\n\nWhen Can Learning Kernels Help Performance? As a practical application of the presented\nbounds, we analyze the impact of the norm-parameter p on the accuracy of (cid:96)p-norm MKL in var-\nious learning scenarios, showing why an intermediate p often turns out to be optimal in practical\napplications. As indicated in the introduction, there is empirical evidence that the performance of\n(cid:96)p-norm MKL crucially depends on the choice of the norm parameter p (for example, cf. Figure 1\n\n7\n\n\f(a) \u03b2 = 2\n\n(b) \u03b2 = 1\n\n(c) \u03b2 = 0.5\n\nFigure 2: Illustration of the three analyzed learning scenarios (TOP) differing in their soft sparsity of the\nBayes hypothesis w\u2217 (parametrized by \u03b2) and corresponding values of the bound factor \u03bdt as a function of p\n(BOTTOM). A soft sparse (LEFT), a intermediate non-sparse (CENTER), and an almost uniform w\u2217 (RIGHT).\nin the introduction). The aim of this section is to relate the theoretical analysis presented here to this\nempirically observed phenomenon.\nTo start with, \ufb01rst note that the choice of p only affects the excess risk bound in the factor (cf.\nTheorem 5 and Equation (13))\n\n(cid:0)Dpt\u2217(cid:1) 2\n\n1+\u03b1 M 1+ 2\n\n1+\u03b1\n\n(cid:0) 1\nt\u2217 \u22121(cid:1)\n\n.\n\n\u03bdt := min\nt\u2208[p,2]\n\nLet us assume that the Bayes hypothesis can be represented by w\u2217 \u2208 H such that the block com-\nponents satisfy (cid:107)w\u2217\nm(cid:107)2 = m\u2212\u03b2, m = 1, . . . , M, where \u03b2 \u2265 0 is a parameter parameterizing the\n\u201csoft sparsity\u201d of the components. For example, the cases \u03b2 \u2208 {0.5, 1, 2} are shown in Figure 2\nfor M = 2 and rank-1 kernels. If n is large, the best bias-complexity trade-off for a \ufb01xed p will\ncorrespond to a vanishing bias, so that the best choice of D will be close to the minimal value such\nthat w\u2217 \u2208 Hp,D, that is, Dp = ||w\u2217||p. Plugging in this value for Dp, the bound factor \u03bdp becomes\n\n\u03bdp := (cid:107)w\u2217(cid:107) 2\n\n1+\u03b1\np\n\nmin\nt\u2208[p,2]\n\nt\u2217 2\n\n1+\u03b1 M 1+ 2\n\n1+\u03b1\n\n(cid:0) 1\nt\u2217 \u22121(cid:1)\n\n.\n\nWe can now plot the value \u03bdp as a function of p \ufb01xing \u03b1, M, and \u03b2. We realized this simulation for\n\u03b1 = 2, M = 1000, and \u03b2 \u2208 {0.5, 1, 2}.The results are shown in Figure 2. Note that the soft sparsity\nof w\u2217 is increased from the left hand to the right hand side. We observe that in the \u201csoft sparsest\u201d\nscenario (LEFT) the minimum is attained for a quite small p = 1.2, while for the intermediate case\n(CENTER) p = 1.4 is optimal, and \ufb01nally in the uniformly non-sparse scenario (RIGHT) the choice\nof p = 2 is optimal, i.e. SVM. This means that if the true Bayes hypothesis has an intermediately\ndense representation (which is frequently encountered in practical applications), our bound gives the\nstrongest generalization guarantees to (cid:96)p-norm MKL using an intermediate choice of p.\n4 Conclusion\nWe derived a sharp upper bound on the local Rademacher complexity of (cid:96)p-norm multiple kernel\nlearning. We also proved a lower bound that matches the upper one and shows that our result is\ntight. Using the local Rademacher complexity bound, we derived an excess risk bound that attains\nthe fast rate of O(n\u2212 \u03b1\n1+\u03b1 ), where \u03b1 is the minimum eigenvalue decay rate of the individual kernels.\nIn a practical case study, we found that the optimal value of that bound depends on the true Bayes-\noptimal kernel weights. If the true weights exhibit soft sparsity but are not strongly sparse, then\nthe generalization bound is minimized for an intermediate p. This is not only intuitive but also\nsupports empirical studies showing that sparse MKL (p = 1) rarely works in practice, while some\nintermediate choice of p can improve performance.\n\nAcknowledgments\nWe thank Peter L. Bartlett and K.-R. M\u00a8uller for valuable comments. This work was supported by the\nGerman Science Foundation (DFG MU 987/6-1, RA 1894/1-1) and by the European Community\u2019s\n7th Framework Programme under the PASCAL2 Network of Excellence (ICT-216886) and under\nthe E.U. grant agreement 247022 (MASH Project).\n\n8\n\nw*w*w*1.01.21.41.61.82.060708090110pbound1.01.21.41.61.82.0404550556065pbound1.01.21.41.61.82.02030405060pbound\fReferences\n[1] F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the SMO\n\nalgorithm. In Proc. 21st ICML. ACM, 2004.\n\n[2] P. L. Bartlett, O. Bousquet, and S. Mendelson. Local Rademacher complexities. Annals of Statistics,\n\n33(4):1497\u20131537, 2005.\n\n[3] R. R. Bouckaert, E. Frank, M. A. Hall, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. WEKA\u2013\nexperiences with a java open-source project. Journal of Machine Learning Research, 11:2533\u20132541,\n2010.\n\n[4] C. Campbell and Y. Ying. Learning with Support Vector Machines. Synthesis Lectures on Arti\ufb01cial\n\nIntelligence and Machine Learning. Morgan & Claypool Publishers, 2011.\n\n[5] C. Cortes.\n\nInvited talk: Can learning kernels help performance?\n\nIn Proceedings of the 26th Annual\nInternational Conference on Machine Learning, ICML \u201909, pages 1:1\u20131:1, New York, NY, USA, 2009.\nACM. Video http://videolectures.net/icml09_cortes_clkh/.\n\n[6] C. Cortes, A. Gretton, G. Lanckriet, M. Mohri, and A. Rostamizadeh. Proceedings of the NIPS Workshop\non Kernel Learning: Automatic Selection of Optimal Kernels, 2008. URL http://videolectures.\nnet/lkasok08_whistler/, Video http://www.cs.nyu.edu/learning_kernels.\n\n[7] C. Cortes, M. Mohri, and A. Rostamizadeh. L2 regularization for learning kernels. In Proceedings of the\n\nInternational Conference on Uncertainty in Arti\ufb01cial Intelligence, 2009.\n\n[8] C. Cortes, M. Mohri, and A. Rostamizadeh. Generalization bounds for learning kernels. In Proceedings,\n\n27th ICML, 2010.\n\n[9] C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20(3):273\u2013297, 1995.\n[10] P. V. Gehler and S. Nowozin. Let the kernel \ufb01gure it out: Principled learning of pre-processing for kernel\nclassi\ufb01ers. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 06 2009.\n[11] R. Ibragimov and S. Sharakhmetov. The best constant in the rosenthal inequality for nonnegative random\n\nvariables. Statistics & Probability Letters, 55(4):367 \u2013 376, 2001.\n\n[12] J.-P. Kahane. Some random series of functions. Cambridge University Press, 2nd edition, 1985.\n[13] M. Kloft, U. Brefeld, S. Sonnenburg, P. Laskov, K.-R. M\u00a8uller, and A. Zien. Ef\ufb01cient and accurate lp-norm\nmultiple kernel learning. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta,\neditors, Advances in Neural Information Processing Systems 22, pages 997\u20131005. MIT Press, 2009.\n\n[14] M. Kloft, U. Brefeld, S. Sonnenburg, and A. Zien. Lp-norm multiple kernel learning. Journal of Machine\n\nLearning Research, 12:953\u2013997, Mar 2011.\n\n[15] V. Koltchinskii. Local Rademacher complexities and oracle inequalities in risk minimization. Annals of\n\nStatistics, 34(6):2593\u20132656, 2006.\n\n[16] V. Koltchinskii and M. Yuan. Sparsity in multiple kernel learning. Annals of Statistics, 38(6):3660\u20133695,\n\n2010.\n\n[17] S. Kwapi\u00b4en and W. A. Woyczy\u00b4nski. Random Series and Stochastic Integrals: Single and Multiple.\n\nBirkh\u00a8auser, Basel and Boston, M.A., 1992.\n\n[18] G. Lanckriet, N. Cristianini, L. E. Ghaoui, P. Bartlett, and M. I. Jordan. Learning the kernel matrix with\n\nsemi-de\ufb01nite programming. JMLR, 5:27\u201372, 2004.\n\n[19] S. Mendelson. On the performance of kernel classes. J. Mach. Learn. Res., 4:759\u2013771, December 2003.\n[20] C. A. Micchelli and M. Pontil. Learning the kernel function via regularization. Journal of Machine\n\nLearning Research, 6:1099\u20131125, 2005.\n\n[21] K.-R. M\u00a8uller, S. Mika, G. R\u00a8atsch, K. Tsuda, and B. Sch\u00a8olkopf. An introduction to kernel-based learning\n\nalgorithms. IEEE Neural Networks, 12(2):181\u2013201, May 2001.\n\n[22] A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet. SimpleMKL. Journal of Machine Learning\n\nResearch, 9:2491\u20132521, 2008.\n\n[23] G. Raskutti, M. J. Wainwright, and B. Yu. Minimax-optimal rates for sparse additive models over kernel\n\nclasses via convex programming. CoRR, abs/1008.3654, 2010.\n\n[24] B. Sch\u00a8olkopf, A. Smola, and K.-R. M\u00a8uller. Nonlinear component analysis as a kernel eigenvalue problem.\n\nNeural Computation, 10:1299\u20131319, 1998.\n\n[25] J. R. Searle. Minds, brains, and programs. Behavioral and Brain Sciences, 3(03):417\u2013424, 1980.\n[26] S. Sonnenburg, G. R\u00a8atsch, C. Sch\u00a8afer, and B. Sch\u00a8olkopf. Large scale multiple kernel learning. Journal of\n\nMachine Learning Research, 7:1531\u20131565, July 2006.\n\n[27] A. Tsybakov. Optimal rates of aggregation. In B. Sch\u00a8olkopf and M. Warmuth, editors, Computational\nLearning Theory and Kernel Machines (COLT-2003), volume 2777 of Lecture Notes in Arti\ufb01cial Intelli-\ngence, pages 303\u2013313. Springer, 2003.\n\n9\n\n\f", "award": [], "sourceid": 1302, "authors": [{"given_name": "Marius", "family_name": "Kloft", "institution": null}, {"given_name": "Gilles", "family_name": "Blanchard", "institution": null}]}