{"title": "Performance analysis for L\\_2 kernel classification", "book": "Advances in Neural Information Processing Systems", "page_first": 833, "page_last": 840, "abstract": "We provide statistical performance guarantees for a recently introduced kernel classifier that optimizes the $L_2$ or integrated squared error (ISE) of a difference of densities. The classifier is similar to a support vector machine (SVM) in that it is the solution of a quadratic program and yields a sparse classifier. Unlike SVMs, however, the $L_2$ kernel classifier does not involve a regularization parameter. We prove a distribution free concentration inequality for a cross-validation based estimate of the ISE, and apply this result to deduce an oracle inequality and consistency of the classifier on the sense of both ISE and probability of error. Our results can also be specialized to give performance guarantees for an existing method of $L_2$ kernel density estimation.", "full_text": "Performance analysis for L2 kernel classi\ufb01cation\n\nJooSeuk Kim\n\nDepartment of EECS\nUniversity of Michigan\nAnn Arbor, MI, USA\n\nstannum@umich.edu\n\nClayton D. Scott\u2217\nDepartment of EECS\nUniversity of Michigan\nAnn Arbor, MI, USA\n\nclayscot@umich.edu\n\nAbstract\n\nWe provide statistical performance guarantees for a recently introduced kernel\nclassi\ufb01er that optimizes the L2 or integrated squared error (ISE) of a difference\nof densities. The classi\ufb01er is similar to a support vector machine (SVM) in that\nit is the solution of a quadratic program and yields a sparse classi\ufb01er. Unlike\nSVMs, however, the L2 kernel classi\ufb01er does not involve a regularization param-\neter. We prove a distribution free concentration inequality for a cross-validation\nbased estimate of the ISE, and apply this result to deduce an oracle inequality and\nconsistency of the classi\ufb01er on the sense of both ISE and probability of error. Our\nresults also specialize to give performance guarantees for an existing method of\nL2 kernel density estimation.\n\n1 Introduction\n\nIn the binary classi\ufb01cation problem we are given realizations (x1, y1), . . . , (xn, yn) of a jointly\ndistributed pair (X, Y ), where X \u2208 Rd is a pattern and Y \u2208 {\u22121, +1} is a class label. The goal\nof classi\ufb01cation is to build a classi\ufb01er, i.e., a function taking X as input and outputting a label, such\nthat some measure of performance is optimized. Kernel classi\ufb01ers [1] are an important family of\nclassi\ufb01ers that have drawn much recent attention for their ability to represent nonlinear decision\nboundaries and to scale well with increasing dimension d. A kernel classi\ufb01er (without offset) has\nthe form\n\n(cid:41)\n\n(cid:40)\n\nn(cid:88)\n\ng(x) = sign\n\n\u03b1iyik(x, xi)\n\n,\n\ni=1\n\nwhere \u03b1i are parameters and k is a kernel function. For example, support vector machines (SVMs)\nwithout offset have this form [2], as does the standard kernel density estimate (KDE) plug-in rule.\nRecently Kim and Scott [3] introduced an L2 or integrated squared error (ISE) criterion to design the\ncoef\ufb01cients \u03b1i of a kernel classi\ufb01er with Gaussian kernel. Their L2 classi\ufb01er performs comparably\nto existing kernel methods while possesing a number of desirable properties. Like the SVM, L2\nkernel classi\ufb01ers are the solutions of convex quadratic programs that can be solved ef\ufb01ciently using\nstandard decomposition algorithms.\nIn addition, the classi\ufb01ers are sparse, meaning most of the\ncoef\ufb01cients \u03b1i = 0, which has advantages for representation and evaluation ef\ufb01ciency. Unlike\nthe SVM, however, there are no free parameters to be set by the user except the kernel bandwidth\nparameter.\nIn this paper we develop statistical performance guarantees for the L2 kernel classi\ufb01er introduced\nin [3]. The linchpin of our analysis is a new concentration inequality bounding the deviation of a\ncross-validation based ISE estimate from the true ISE. This bound is then applied to prove an oracle\ninequality and consistency in both ISE and probability of error. In addition, as a special case of\n\n\u2217Both authors supported in part by NSF Grant CCF-0830490\n\n1\n\n\four analysis, we are able to deduce performance guarantees for the method of L2 kernel density\nestimation described in [4, 5].\nThe ISE criterion has a long history in the literature on bandwidth selection for kernel density esti-\nmation [6] and more recently in parametric estimation [7]. The use of ISE for optimizing the weights\nof a KDE via quadratic programming was \ufb01rst described in [4] and later rediscovered in [5]. In [8],\nan (cid:96)1 penalized ISE criterion was used to aggregate a \ufb01nite number of pre-determined densities.\nLinear and convex aggregation of densities, based on an L2 criterion, are studied in [9], where the\ndensities are based on a \ufb01nite dictionary or an independent sample. In contrast, our proposed method\nallows data-adaptive kernels, and does not require and independent (holdout) sample.\nIn classi\ufb01cation, some connections relating SVMs and ISE are made in [10], although no new algo-\nrithms are proposed. Finally, the \u201cdifference of densities\u201d perspective has been applied to classi\ufb01ca-\ntion in other settings by [11], [12], and [13]. In [11] and [13], a difference of densities are used to\n\ufb01nd smoothing parameters or kernel bandwidths. In [12], conditional densities are chosen among a\nparameterized set of densities to maximize the average (bounded) density differences.\nSection 2 reviews the L2 kernel classi\ufb01er, and presents a slight modi\ufb01cation needed for our analysis.\nOur results are presented in Section 3. Conclusions are offered in the \ufb01nal section, and proofs are\ngathered in an appendix.\n\n2 L2 Kernel Classi\ufb01cation\n\nWe review the previous work of Kim & Scott [3] and introduce an important modi\ufb01cation. For\nconvenience, we relabel Y so that it belongs to {1,\u2212\u03b3} and denote I+ = {i | Yi = +1} and I\u2212 =\n{i | Yi = \u2212\u03b3}. Let f\u2212(x) and f+(x) denote the class-conditional densities of the pattern given the\nlabel. From decision theory, the optimal classi\ufb01er has the form\n\ng\u2217(x) = sign{f+(x) \u2212 \u03b3f\u2212(x)} ,\n\n(1)\nwhere \u03b3 incorporates prior class probabilities and class-conditional error costs (in the Bayesian\nsetting) or a desired tradeoff between false positives and false negatives [14]. Denote the \u201cdifference\nof densities\u201d d\u03b3(x) := f+(x) \u2212 \u03b3f\u2212(x).\nThe class-conditional densities are modelled using the Gaussian kernel as\n\n(cid:88)\n(cid:98)f+ (x; \u03b1) =\n\uf8f1\uf8f2\uf8f3\u03b1 |\n\nA =\n\ni\u2208I+\n\nwith constraints \u03b1 = (\u03b11, . . . , \u03b1n) \u2208 A where\n\nThe Gaussian kernel is de\ufb01ned as\n\nk\u03c3 (x, Xi) =\n\nThe ISE associated with \u03b1 is\n\nISE (\u03b1) = (cid:107)(cid:98)d\u03b3 (x; \u03b1) \u2212 d\u03b3 (x)(cid:107)2\n\n(cid:90) (cid:98)d2\n\n=\n\n\u03b3 (x; \u03b1) dx \u2212 2\n\n\u03b1ik\u03c3 (x, Xi) ,\n\n\u03b1ik\u03c3 (x, Xi)\n\n(cid:98)f\u2212 (x; \u03b1) =\n\n(cid:88)\n\ni\u2208I\u2212\n\n(cid:88)\n(cid:161)\n\ni\u2208I+\n\n\u03b1i =\n\n2\u03c0\u03c32(cid:162)\u2212d/2 exp\n\n.\n\n2\u03c32\n\ni\u2208I\u2212\n\n(cid:88)\n\n\u2212(cid:107)x \u2212 Xi(cid:107)2\n\n\u03b1i = 1, \u03b1i \u2265 0 \u2200i\n\n\uf8fc\uf8fd\uf8fe .\n(cid:190)\n(cid:189)\n(cid:90) (cid:179)(cid:98)d\u03b3 (x; \u03b1) \u2212 d\u03b3 (x)\n(cid:90) (cid:98)d\u03b3 (x; \u03b1) d\u03b3 (x) dx +\n(cid:90)\n(cid:90) (cid:98)d\u03b3 (x; \u03b1) d\u03b3 (x) dx\n(cid:90)\n\nL2 =\n\n\u03b3 (x; \u03b1) dx \u2212 2Hn (\u03b1) +\n\n\u03b3 (x) dx.\nd2\n\n(cid:180)2\n\ndx\n\n\u03b3 (x) dx.\nd2\n\nSince we do not know the true d\u03b3 (x), we need to estimate the second term in the above equation\n\nby Hn (\u03b1) which will be explained in detail in Section 2.1. Then, the empirical ISE is\n\n(2)\n\n(3)\n\nH (\u03b1) (cid:44)\n\n(cid:90) (cid:98)d2\n\n(cid:100)ISE (\u03b1) =\n\n2\n\n\fNow,(cid:98)\u03b1 is de\ufb01ned as\n\nand the \ufb01nal classi\ufb01er will be\n\n2.1 Estimation of H (\u03b1)\n\n(cid:98)\u03b1 = arg min\n(cid:40)\n\n\u03b1\u2208A\n\n(cid:100)ISE (\u03b1)\n(cid:98)d\u03b3 (x;(cid:98)\u03b1) \u2265 0\n(cid:98)d\u03b3 (x;(cid:98)\u03b1) < 0.\n\ng (x) =\n\n+1,\n\u2212\u03b3,\n\n(4)\n\nIn this section, we propose a method of estimating H (\u03b1) in (2). The basic idea is to view H (\u03b1) as\nan expectation and estimate it using a sample average. In [3], the resubstitution estimator for H (\u03b1)\nwas used. However, since this estimator is biased, we use a leave-one-out cross-validation (LOOCV)\nestimator, which is unbiased and facilitates our theoretical analysis. Note that the difference of\ndensities can be expressed as\n\nn(cid:88)\n\n(cid:98)d\u03b3 (x; \u03b1) = (cid:98)f+ (x) \u2212 \u03b3(cid:98)f\u2212 (x) =\n(cid:90) (cid:98)d\u03b3 (x; \u03b1) d\u03b3 (x) dx =\n(cid:90) n(cid:88)\n(cid:90) n(cid:88)\nn(cid:88)\n\n\u03b1iYik\u03c3 (x, Xi) f+ (x) dx \u2212 \u03b3\n\n(cid:90) (cid:98)d\u03b3 (x; \u03b1) f+ (x) dx \u2212 \u03b3\n\ni=1\n\ni=1\n\ni=1\n\n\u03b1iYik\u03c3 (x, Xi) .\n\n(cid:90) (cid:98)d\u03b3 (x; \u03b1) f\u2212 (x) dx\n\n\u03b1iYik\u03c3 (x, Xi) f\u2212 (x) dx\n\nThen,\n\nH (\u03b1) =\n\n=\n\n=\n\nwhere\n\ni=1\n\n\u03b1iYih (Xi)\n\n(cid:90)\n\nh (Xi) (cid:44)\n\nk\u03c3 (x, Xi) f+ (x) dx \u2212 \u03b3\n\nk\u03c3 (x, Xi) f\u2212 (x) dx.\n\n(cid:88)\n\n\uf8f1\uf8f4\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f4\uf8f3\n\n(cid:98)hi (cid:44)\n\n(cid:88)\n\nN+ \u2212 1\n1\nN+\n\nj\u2208I+\n\nj\u2208I+,j(cid:54)=i\nk\u03c3 (Xj, Xi) \u2212\n\nWe estimate each h (Xi) in (5) for i = 1, . . . , n using leave-one-out cross-validation\ni \u2208 I+\n\nk\u03c3 (Xj, Xi) ,\n\n1\n\nk\u03c3 (Xj, Xi) \u2212 \u03b3\nN\u2212\n\nwhere N+ = |I+| , N\u2212 = |I\u2212|. Then, the estimate of H (\u03b1) is Hn (\u03b1) =\n\ni=1 \u03b1iYi\n\n\u03b3\n\nN\u2212 \u2212 1\n\nj\u2208I\u2212,j(cid:54)=i\n\nk\u03c3 (Xj, Xi) ,\n\ni \u2208 I\u2212\n\n(cid:80)n\n\n(5)\n\n(cid:98)hi.\n\n(cid:90)\n(cid:88)\n(cid:88)\n\nj\u2208I\u2212\n\nThe optimization problem (4) can be formulated as a quadratic program. The \ufb01rst term in (3) is\n\n2.2 Optimization\n\n(cid:90) (cid:98)d2\n\n\u03b3 (x; \u03b1) dx =\n\nn(cid:88)\n\nn(cid:88)\n\ni=1\n\nj=1\n\n(cid:90) (cid:195)\n\nn(cid:88)\n(cid:90)\n\ni=1\n\n(cid:33)2\n\n\u03b1iYik\u03c3 (x, Xi)\n\ndx\n\n=\n\n\u03b1i\u03b1jYiYj\n\nk\u03c3 (x, Xi) k\u03c3 (x, Xj) dx =\n\n\u03b1i\u03b1jYiYjk\u221a\n\nn(cid:88)\n\nn(cid:88)\n\ni=1\n\nj=1\n\n(cid:80)n\n2\u03c3 (Xi, Xj) \u2212 n(cid:88)\n\n2\u03c3 (Xi, Xj)\n\n(cid:98)hi. Finally, since\n\nn(cid:88)\n\nn(cid:88)\n\n(cid:98)\u03b1 = arg min\n\n\u03b1\u2208A\n\n1\n2\n\nby the convolution theorem for Gaussian kernels [15]. As we have seen in Section 2.1, the second\nterm Hn (\u03b1) in (3) is linear in \u03b1 and can be expressed as\nthe third term does not depend on \u03b1, the optimization problem (4) becomes the following quadratic\nprogram (QP)\n\ni=1 \u03b1ici where ci = Yi\n\n\u03b1i\u03b1jYiYjk\u221a\n\nci\u03b1i.\n\n(6)\n\ni=1\n\nj=1\n\ni=1\n\nThe QP (6) is similar to the dual QP of the 2-norm SVM with hinge loss [2] and can be solved by a\nvariant of the Sequential Minimal Optimization (SMO) algorithm [3].\n\n3\n\n\f3 Statistical performance analysis\n\nIn this section, we give theoretical performance analysis on our proposed method. We assume that\n{Xi}i\u2208I+ and {Xi}i\u2208I\u2212 are i.i.d samples from f+ (x) and f\u2212 (x), respectively, and treat N+ and\nN\u2212 as deterministic variables n+ and n\u2212 such that n+ \u2192 \u221e and n\u2212 \u2192 \u221e as n \u2192 \u221e.\n\n3.1 Concentration inequality for Hn (\u03b1)\n\nLemma 1. Conditioned on Xi,(cid:98)hi is an unbiased estimator of h (Xi), i.e,\n\nE\n\n= h (Xi) .\n\n(cid:105)\n\n(cid:104)(cid:98)hi | Xi\n(cid:190)\n\nFurthermore, for any \u0001 > 0,\n\n|Hn (\u03b1) \u2212 H (\u03b1)| > \u0001\n\n(cid:179)\n\ne\u2212c(n+\u22121)\u00012 + e\u2212c(n\u2212\u22121)\u00012(cid:180)\n\n\u2264 2n\n\n(cid:189)\n\nP\n\n(cid:161)\u221a\n\nsup\n\u03b1\u2208A\n\n(cid:162)2d\n\n2\u03c0\u03c3\n\n/ (1 + \u03b3)4.\n\nwhere c = 2\nLemma 1 implies that Hn (\u03b1) \u2192 H (\u03b1) almost surely for all \u03b1 \u2208 A simultaneously, provided that\n\u03c3, n+, and n\u2212 evolve as functions of n such that n+\u03c32d/ ln n \u2192 \u221e and n\u2212\u03c32d/ ln n \u2192 \u221e.\n\n3.2 Oracle Inequality\n\nNext, we establish on oracle inequality, which relates the performance of our estimator to that of the\nbest possible kernel classi\ufb01er.\n\n(cid:179)\n\n(cid:180)\n\n(cid:161)\u221a\n\n(cid:162)2d\n\nTheorem 1. Let \u0001 > 0 and set \u03b4 = \u03b4 (\u0001) = 2n\n\n2\n\n2\u03c0\u03c3\n\n/ (1 + \u03b3)4. Then, with probability at least 1 \u2212 \u03b4\n\ne\u2212c(n+\u22121)\u00012 + e\u2212c(n\u2212\u22121)\u00012\n\nwhere c =\n\nISE (\u03b1) + 4\u0001.\n\n\u03b1\u2208A\n\nISE ((cid:98)\u03b1) \u2264 inf\n(cid:175)(cid:175)(cid:175)ISE (\u03b1) \u2212 (cid:100)ISE (\u03b1)\n\n(cid:175)(cid:175)(cid:175) \u2264 2\u0001,\n\nProof. From Lemma 1, with probability at least 1 \u2212 \u03b4\n\n\u2200\u03b1 \u2208 A\n\nfor all \u03b1 \u2208 A, we have\n\nby using the fact ISE (\u03b1)\u2212 (cid:100)ISE (\u03b1) = 2 (Hn (\u03b1) \u2212 H (\u03b1)). Then, with probability at least 1\u2212 \u03b4,\nISE ((cid:98)\u03b1) \u2264 (cid:100)ISE ((cid:98)\u03b1) + 2\u0001 \u2264 (cid:100)ISE (\u03b1) + 2\u0001 \u2264 ISE (\u03b1) + 4\u0001\nwhere the second inequality holds from the de\ufb01nition of(cid:98)\u03b1. This proves the theorem.\nNext, we have a theorem stating that ISE ((cid:98)\u03b1) converges to zero in probability.\n\u03c3 \u2192 0, n+\u03c32d/ ln n \u2192 \u221e, and n\u2212\u03c32d/ ln n \u2192 \u221e, then ISE ((cid:98)\u03b1) \u2192 0 in probability as n \u2192 \u221e\n\nTheorem 2. Suppose that for f = f+ and f\u2212, the Hessian Hf (x) exists and each entry of Hf (x)\nis piecewise continuous and square integrable. If \u03c3, n+, and n\u2212 evolve as functions of n such that\n\nISE consistency\n\n3.3\n\nThis result intuitively follows from the oracle inequality since the standard Parzen window density\nestimate is consistent and uniform weights are among the simplex A. The rigorous proof is omitted\ndue to space limitations.\n\n4\n\n\f3.4 Bayes Error Consistency\n\nIn classi\ufb01cation, we are ultimately interested in minimizing the probability of error. Let us now\nassume {Xi}n\ni=1 is an i.i.d sample from f (x) = pf+ (x) + (1 \u2212 p)f\u2212 (x), where 0 < p < 1 is\nthe prior probability of the positive class. The consistency with respect to the probability of error\ncould be easily shown if we set \u03b3 to \u03b3\u2217 = 1\u2212p\nand apply Theorem 3 in [17]. However, since p is\nunknown, we must estimate \u03b3\u2217. Note that N+ and N\u2212 are binomial random variables, and we may\nestimate \u03b3\u2217 as \u03b3 = N\u2212\n. The next theorem says the L2 kernel classi\ufb01er is consistent with respect to\nN+\nthe probability of error.\nTheorem 3. Suppose that the assumptions in Theorem 2 are satis\ufb01ed. In addition, suppose that\nf\u2212 \u2208 L2 (R), i.e. (cid:107)f\u2212(cid:107)L2 < \u221e. Let \u03b3 = N\u2212/N+ be an estimate of \u03b3\u2217 = 1\u2212p\np . If \u03c3 evolves as\na function of n such that \u03c3 \u2192 0 and n\u03c32d/ ln n \u2192 \u221e as n \u2192 \u221e, then the L2 kernel classi\ufb01er is\nconsistent. In other words, given training data Dn = ((X1, Y1) , . . . , (Xn, Yn)), the classi\ufb01cation\nerror\n\np\n\n(cid:110)\n\n(cid:110)(cid:98)d\u03b3 (X;(cid:98)\u03b1)\n\n(cid:111)\n\nLn = P\n\nsgn\n\n(cid:111)\n\n(cid:54)= Y | Dn\n\nconverges to the Bayes error L\u2217 in probability as n \u2192 \u221e.\n\nThe proof is given in Appendix A.2.\n\n3.5 Application to density estimation\n\nBy setting \u03b3 = 0, our goal becomes estimating f+ and we recover the L2 kernel density estimate\nof [4, 5] using leave-one-out cross-validation. Given an i.i.d sample X1, . . . , Xn from f (x), the L2\nkernel density estimate of f (x) is de\ufb01ned as\n\nn(cid:88)\n\n(cid:98)f (x;(cid:98)\u03b1) =\n(cid:98)\u03b1ik\u03c3 (x, Xi)\n\uf8eb\uf8ed 1\n2\u03c3 (Xi, Xj) \u2212 n(cid:88)\n\n\u03b1i\n\ni=1\n\n\u03b1i\u03b1jk\u221a\n\nn \u2212 1\n\ni=1\n\nj=1\n\ni=1\n\nwith(cid:98)\u03b1i\u2019s optimized such that\nn(cid:88)\nn(cid:88)\n\n(cid:98)\u03b1 = arg min(cid:80)\n\n1\n2\n\n\u03b1i=1\n\u03b1i\u22650\n\n\uf8f6\uf8f8 .\n\nk\u03c3 (Xi, Xj)\n\n(cid:88)\n\nj(cid:54)=i\n\nOur concentration inequality, oracle inequality, and L2 consistency result immediately extend to\nprovide the same performance guarantees for this method. For example, we state the following\ncorollary.\nCorollary 1. Suppose that the Hessian Hf (x) of a density function f (x) exists and each entry of\nHf (x) is piecewise continuous and square integrable. If \u03c3 \u2192 0 and n\u03c32d/ ln n \u2192 \u221e as n \u2192 \u221e,\nthen\n\n(cid:90) (cid:179)(cid:98)f (x;(cid:98)\u03b1) \u2212 f (x)\n\n(cid:180)2\n\ndx \u2192 0\n\nin probability.\n\n4 Conclusion\n\nThrough the development of a novel concentration inequality, we have established statistical per-\nformance guarantees on a recently introduced L2 kernel classi\ufb01er. We view the relatively clean\nanalysis of this classi\ufb01er as an attractive feature relative to other kernel methods. In future work, we\nhope to invoke the full power of the oracle inequality to obtain adaptive rates of convergence, and\nconsistency for \u03c3 not necessarily tending to zero.\n\n5\n\n\fA Appendix\n\nA.1 Proof of Lemma 1\n\nNote that for any given i, (k\u03c3 (Xj, Xi))j(cid:54)=i are independent and bounded by M = 1/\nFor random vectors Z \u223c f+ (x) and W \u223c f\u2212 (x), h (Xi) in (5) can be expressed as\n\n(cid:161)\u221a\n\n2\u03c0\u03c3\n\n(cid:162)d.\n\nh (Xi) = E [k\u03c3 (Z, Xi) | Xi] \u2212 \u03b3E [k\u03c3 (W, Xi) | Xi] .\n\nSince Xi \u223c f+ (x) for i \u2208 I+ and Xi \u223c f\u2212 (x) for i \u2208 I\u2212, it can be easily shown that\n\n(cid:105)\n\n(cid:104)(cid:98)hi | Xi\n\nE\n\n= h (Xi) .\n\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n(cid:190)\n(cid:88)\n\nFor i \u2208 I+,\n\nP\n\n(cid:189)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175) > \u0001\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175)\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175) \u03b3\n(cid:88)\n\nn+ \u2212 1\n\n\u2264 P\n\n+ P\n\n1\n\nn\u2212\n\nj\u2208I\u2212\n\nk\u03c3 (Xj, Xi) \u2212 E [k\u03c3 (Z, Xi) | Xi]\n\nj\u2208I+,j(cid:54)=i\nk\u03c3 (Xj, Xi) \u2212 \u03b3E [k\u03c3 (W, Xi) | Xi]\n\nBy Hoeffding\u2019s inequality [16], the \ufb01rst term in (7) is\n\nk\u03c3 (Xj, Xi) \u2212 (n+ \u2212 1)E [k\u03c3 (Z, Xi) | Xi]\n\n(cid:175)(cid:175)(cid:175)(cid:175) >\n\n\u03b3\u0001\n1 + \u03b3\n\n(cid:175)(cid:175)(cid:175)(cid:175) >\n\n(cid:190)\n\n1 + \u03b3\n\n\u0001\n\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n(cid:190)\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n\n(cid:175)(cid:175)(cid:175)(cid:175) >\n\n(7)\n\n(cid:190)\n(cid:190)\n\n(cid:190)\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n\nk\u03c3 (Xj, Xi) \u2212 E\n\nk\u03c3 (Xj, Xi) \u2212 E\n\nk\u03c3 (Xj, Xi) | Xi\n\nk\u03c3 (Xj, Xi) | Xi\n\n(n+ \u2212 1) \u0001\n\n1 + \u03b3\n\n(n+ \u2212 1) \u0001\n\n1 + \u03b3\n\n(cid:183) (cid:88)\n(cid:183) (cid:88)\n\nj\u2208I+,j(cid:54)=i\n\nj\u2208I+,j(cid:54)=i\n\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175) (cid:88)\n\nP\n\nj\u2208I+,j(cid:54)=i\n\n= P\n\n= P\n\nj\u2208I+,j(cid:54)=i\n\nj\u2208I+,j(cid:54)=i\n\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175) (cid:88)\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175) (cid:88)\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175)(cid:88)\n\n= P\n\nj\u2208I\u2212\n\nj\u2208I\u2212\n\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:175)(cid:88)\n(cid:110)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n\n\u2264 2e\u22122(n+\u22121)\u00012/(1+\u03b3)2M 2\n\n.\n\nThe second term in (7) is\n\nP\n\nk\u03c3 (Xj, Xi) \u2212 n\u2212E [k\u03c3 (W, Xi) | Xi]\n\nk\u03c3 (Xj, Xi) \u2212 E\n\n(cid:175)(cid:175)(cid:175)(cid:175) >\n\nn\u2212\u0001\n1 + \u03b3\nk\u03c3 (Xj, Xi) | Xi\n\n(cid:183)(cid:88)\n\nj\u2208I\u2212\n\n(cid:190)\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n\n(cid:190)\n\n\u2264 2e\u22122n\u2212\u00012/(1+\u03b3)2M 2 \u2264 2e\u22122(n\u2212\u22121)\u00012/(1+\u03b3)2M 2\n\n.\n\n(cid:175)(cid:175)(cid:175) \u2265 \u0001\n(cid:111)\n\n(cid:183)\n\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) \u2265 \u0001\n\nTherefore,\n\nP\n\nP\n\n= E\n\u2264 2e\u22122(n+\u22121)\u00012/(1+\u03b3)2M 2 + 2e\u22122(n\u2212\u22121)\u00012/(1+\u03b3)2M 2\n\n.\n\nIn a similar way, it can be shown that for i \u2208 I\u2212,\n\n(cid:110)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n\n(cid:175)(cid:175)(cid:175) > \u0001\n\n(cid:111)\n\nP\n\n\u2264 2e\u22122(n+\u22121)\u00012/(1+\u03b3)2M 2 + 2e\u22122(n\u2212\u22121)\u00012/(1+\u03b3)2M 2\n\n.\n\n6\n\n1 + \u03b3\n\n(n+ \u2212 1) \u0001\n\n(cid:184)(cid:175)(cid:175)(cid:175)(cid:175) >\n(cid:184)(cid:175)(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = x\n(cid:184)(cid:175)(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175)(cid:175) Xi = X\n\nn\u2212\u0001\n1 + \u03b3\n\n(cid:190)(cid:184)\n\n\f(cid:189)\n\nThen,\n\nP\n\nsup\n\u03b1\u2208A\n\u2264 P\n\n= P\n\n\u2264 P\n\n= P\n\n= P\n\n\u2264\n\ni\u2208I+\n\u2264 n+\n\n= n\n\n\u0001\n\n\u03b1i\u03b3\n\n\u03b1iYi\n\ni=1\n\n\u03b1i\u03b3\n\ni=1\n\ni\u2208I+\n\n\u0001\n\n\u0001\n\ni\u2208I+\n\n+ P\n\n+ P\n\ni\u2208I\u2212\n\ni\u2208I\u2212\n\n\u03b1i\n\n\u03b1i\n\n= P\n\n1 + \u03b3\n\n(cid:41)\n\nmax\ni\u2208I\u2212\n\nmax\ni\u2208I+\n\nsup\n\u03b1\u2208A\n\nsup\n\u03b1\u2208A\n\nsup\n\u03b1\u2208A\n\nsup\n\u03b1\u2208A\n\nsup\n\u03b1\u2208A\n\n\u03b1i |Yi|\n\n1 + \u03b3\n\u0001\n\n|Hn (\u03b1) \u2212 H (\u03b1)| > \u0001\n\n(cid:175)(cid:175)(cid:175)(cid:175)(cid:175) n(cid:88)\n(cid:180)(cid:175)(cid:175)(cid:175)(cid:175)(cid:175) > \u0001\n(cid:40)\n(cid:190)\n(cid:179)(cid:98)hi \u2212 h (Xi)\n(cid:40)\n(cid:41)\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) > \u0001\nn(cid:88)\n(cid:189)\n(cid:190)\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) +\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) > \u0001\nn(cid:88)\nn(cid:88)\n(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:190)\n(cid:189)\n(cid:189)\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) >\nn(cid:88)\nn(cid:88)\n(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:189)\n(cid:190)\n(cid:190)\n(cid:189)\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175) >\n(cid:190)(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:190)(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:190)\n(cid:189)(cid:91)\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:189)(cid:91)\n(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:175)(cid:175)(cid:175)(cid:175) B\n(cid:190)\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:190)\n(cid:189)(cid:175)(cid:175)(cid:175)(cid:98)hi \u2212 h (Xi)\n(cid:175)(cid:175)(cid:175) >\n(cid:175)(cid:175)(cid:175) >\n(cid:88)\n(cid:88)\n2e\u22122(n+\u22121)\u00012/(1+\u03b3)4M 2 + 2e\u22122(n\u2212\u22121)\u00012/(1+\u03b3)4M 2(cid:180)\n(cid:179)\n(cid:179)\n2e\u22122(n+\u22121)\u00012/(1+\u03b3)4M 2 + 2e\u22122(n\u2212\u22121)\u00012/(1+\u03b3)4M 2(cid:180)\n2e\u22122(n+\u22121)\u00012/(1+\u03b3)4M 2 + 2e\u22122(n\u2212\u22121)\u00012/(1+\u03b3)4M 2(cid:180)\n(cid:180)2\n\n(cid:107)(cid:98)d\u03b3 (x;(cid:98)\u03b1) \u2212 d\u03b3\u2217 (x)(cid:107)L2 = (cid:107)(cid:98)d\u03b3 (x;(cid:98)\u03b1) \u2212 d\u03b3 (x) + (\u03b3 \u2212 \u03b3\u2217) f\u2212 (x)(cid:107)L2\n\n(cid:90) (cid:179)(cid:98)d\u03b3 (x;(cid:98)\u03b1) \u2212 d\u03b3\u2217 (x)\n\u2264 (cid:107)(cid:98)d\u03b3 (x;(cid:98)\u03b1) \u2212 d\u03b3 (x)(cid:107)L2 + (cid:107) (\u03b3 \u2212 \u03b3\u2217) f\u2212 (x)(cid:107)L2\n(cid:112)\nISE ((cid:98)\u03b1) + |\u03b3 \u2212 \u03b3\u2217| \u00b7 (cid:107)f\u2212 (x)(cid:107)L2,\n\ndx \u2192 0\n\n+ n\u2212\n\n1 + \u03b3\n\n1 + \u03b3\n\n1 + \u03b3\n\n1 + \u03b3\n\n1 + \u03b3\n\n(cid:179)\n\ni\u2208I\u2212\n\ni\u2208I\u2212\n\n+ P\n\n+\n\nP\n\ni\u2208I+\n\nP\n\n\u0001\n\n\u0001\n\n\u0001\n\n.\n\nA.2 Proof of Theorem 3\n\nFrom Theorem 3 in [17], it suf\ufb01ces to show that\n\nin probability. Since from the triangle inequality\n\n(cid:190)\n\n(cid:175)(cid:175)(cid:175)(cid:175) B\n\n\u03b3\u0001\n1 + \u03b3\n\n(cid:190)\n\n(cid:169)\n\n=\n\nwe need to show that ISE ((cid:98)\u03b1) and \u03b3 converges in probability to 0 and \u03b3\u2217, respectively. The con-\nIn the previous analyses, we have shown the convergence of ISE ((cid:98)\u03b1) by treating N+, N\u2212 and \u03b3\n\nvergence of \u03b3 to \u03b3\u2217 can be easily shown from the strong law of large numbers.\n\nas deterministic variables but now we turn to the case where these variables are random. De\ufb01ne an\nevent D =\n\n. For any \u0001 > 0,\n\n, \u03b3 \u2264 2\u03b3\u2217\n\n(cid:169)\n\n2\n\n(cid:169)\n\n+ P\n\nN+ \u2265 np\n\n(cid:110)\n2 , N\u2212 \u2265 n(1\u2212p)\nP{ISE ((cid:98)\u03b1) > \u0001} \u2264 P\n(cid:175)(cid:175) n+ \u2265 np\n(cid:170)\nISE ((cid:98)\u03b1) > \u0001, D\n2 , n\u2212 \u2265 n(1\u2212p)\n(cid:88)\n(cid:169)\nISE ((cid:98)\u03b1) > \u0001, D\n(cid:88)\n(cid:169)\nISE ((cid:98)\u03b1) > \u0001\n(cid:169)\nISE ((cid:98)\u03b1) > \u0001\n\n(cid:111)\nDc(cid:170)\n(cid:169)\n\u2264 2\u03b3\u2217(cid:170)\n(cid:175)(cid:175) N+ = n+, N\u2212 = n\u2212\n(cid:175)(cid:175) N+ = n+, N\u2212 = n\u2212\n(cid:175)(cid:175) N+ = n+, N\u2212 = n\u2212\n\n(n+,n\u2212)\u2208S\n\n, n\u2212\nn+\n\nmax\n\nP\n\nP\n\nP\n\n2\n\n=\n\n=\n\n\u2264\n\n.\n\nISE ((cid:98)\u03b1) > \u0001, D\n(cid:170) \u00b7 P{N+ = n+, N\u2212 = n\u2212}\n(cid:170) \u00b7 P{N+ = n+, N\u2212 = n\u2212}\n(cid:170)\n\nThe \ufb01rst term converges to 0 from the strong law of large numbers. Let de\ufb01ne a set S =\n(n+, n\u2212)\n\n. Then,\n\n(cid:170)\n\n(8)\n\nP\n\n.\n\n(n+,n\u2212)\u2208S\n\n7\n\n\fProvided that \u03c3 \u2192 0 and n\u03c32d/ ln n \u2192 \u221e, any pair (n+, n\u2212) \u2208 S satis\ufb01es \u03c3 \u2192 0, n+\u03c32d/ ln n \u2192\n\u221e, and n\u2212\u03c32d/ ln n \u2192 \u221e as n \u2192 \u221e and thus the term in (8) converges to 0 from Theorem 2. This\nproves the theorem.\n\nReferences\n[1] B. Sch\u00a8olkopf and A. J. Smola, Learning with Kernels, MIT Press, Cambridge, MA, 2002.\n[2] C. Cortes and V. Vapnik, \u201cSupport-vector networks,\u201d Machine Learning, vol. 20, no. 3, pp. 273\u2013297,\n\n1995.\n\n[3] J. Kim and C. Scott, \u201cKernel classi\ufb01cation via integrated squared error,\u201d IEEE Workshop on Statistical\n\nSignal Processing, August 2007.\n\n[4] D. Kim, Least Squares Mixture Decomposition Estimation, unpublished doctoral dissertation, Dept. of\n\nStatistics, Virginia Polytechnic Inst. and State Univ., 1995.\n\n[5] Mark Girolami and Chao He, \u201cProbability density estimation from optimally condensed data samples,\u201d\nIEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1253\u20131264, OCT\n2003.\n\n[6] B.A. Turlach, \u201cBandwidth selection in kernel density estimation: A review,\u201d Technical Report 9317,\n\nC.O.R.E. and Institut de Statistique, Universit\u00b4e Catholique de Louvain, 1993.\n\n[7] David W.Scott, \u201cParametric statistical modeling by minimum integrated square error,\u201d Technometrics 43,\n\npp. 274\u2013285, 2001.\n\n[8] A.B. Tsybakov F. Bunea and M.H. Wegkamp, \u201cSparse density estimation with l1 penalties,\u201d Proceedings\nof 20th Annual Conference on Learning Theory, COLT 2007, Lecture Notes in Arti\ufb01cial Intelligence,\nv4539, pp. 530\u2013 543, 2007.\n\n[9] Ph. Rigollet and A.B. Tsybakov, \u201cLinear and convex aggregation of density estimators,\u201d https://\n\nhal.ccsd.cnrs.fr/ccsd-00068216, 2004.\n\n[10] Robert Jenssen, Deniz Erdogmus, Jose C.Principe, and Torbj\u00f8rn Eltoft, \u201cTowards a uni\ufb01cation of infor-\nmation theoretic learning and kernel method,\u201d in Proc. IEEE Workshop on Machine Learning for Signal\nProcessing (MLSP2004), Sao Luis, Brazil.\n\n[11] Peter Hall and Matthew P.Wand, \u201cOn nonparametric discrimination using density differeces,\u201d Biometrika,\n\nvol. 75, no. 3, pp. 541\u2013547, Sept 1988.\n\n[12] P. Meinicke, T. Twellmann, and H. Ritter, \u201cDiscriminative densities from maximum contrast estimation,\u201d\n\nin Advances in Neural Information Proceeding Systems 15, Vancouver, Canada, 2002, pp. 985\u2013992.\n\n[13] M. Di Marzio and C.C. Taylor, \u201cKernel density classi\ufb01cation and boosting: an l2 analysis,\u201d Statistics and\n\nComputing, vol. 15, pp. 113\u2013123(11), April 2005.\n\n[14] E. Lehmann, Testing statistical hypotheses, Wiley, New York, 1986.\n[15] M.P. Wand and M.C. Jones, Kernel Smoothing, Chapman & Hall, 1995.\n[16] L. Devroye and G. Lugosi, \u201cCombinatorial methods in density estimation,\u201d 2001.\n[17] Charles T. Wolverton and Terry J. Wagner, \u201cAsymptotically optimal discriminant fucntions for pattern\n\nclassi\ufb01cation,\u201d IEEE Trans. Info. Theory, vol. 15, no. 2, pp. 258\u2013265, Mar 1969.\n\n8\n\n\f", "award": [], "sourceid": 888, "authors": [{"given_name": "Jooseuk", "family_name": "Kim", "institution": null}, {"given_name": "Clayton", "family_name": "Scott", "institution": null}]}