{"title": "$\\ell_1$-regression with Heavy-tailed Distributions", "book": "Advances in Neural Information Processing Systems", "page_first": 1076, "page_last": 1086, "abstract": "In this paper, we consider the problem of linear regression with heavy-tailed distributions. Different from previous studies that use the squared loss to measure the performance, we choose the absolute loss, which is capable of estimating the conditional median. To address the challenge that both the input and output could be heavy-tailed, we propose a truncated minimization problem, and demonstrate that it enjoys an $O(\\sqrt{d/n})$ excess risk, where $d$ is the dimensionality and $n$ is the number of samples. Compared with traditional work on $\\ell_1$-regression, the main advantage of our result is that we achieve a high-probability risk bound without exponential moment conditions on the input and output. Furthermore, if the input is bounded, we show that the classical empirical risk minimization is competent for $\\ell_1$-regression even when the output is heavy-tailed.", "full_text": "(cid:96)1-regression with Heavy-tailed Distributions\n\nLijun Zhang, Zhi-Hua Zhou\n\nNational Key Laboratory for Novel Software Technology\n\nNanjing University, Nanjing 210023, China\n{zhanglj, zhouzh}@lamda.nju.edu.cn\n\nAbstract\n\nIn this paper, we consider the problem of linear regression with heavy-tailed\ndistributions. Different from previous studies that use the squared loss to measure\nthe performance, we choose the absolute loss, which is capable of estimating the\nconditional median. To address the challenge that both the input and output could\nbe heavy-tailed, we propose a truncated minimization problem, and demonstrate\n\nthat it enjoys an (cid:101)O((cid:112)d/n) excess risk, where d is the dimensionality and n is the\n\nnumber of samples. Compared with traditional work on (cid:96)1-regression, the main\nadvantage of our result is that we achieve a high-probability risk bound without\nexponential moment conditions on the input and output. Furthermore, if the input\nis bounded, we show that the classical empirical risk minimization is competent\nfor (cid:96)1-regression even when the output is heavy-tailed.\n\n1\n\nIntroduction\n\nLinear regression used to be a mainstay of statistics, and remains one of our most important tools\nfor data analysis [Hastie et al., 2009]. Let T = {(x1, y1), . . . , (xn, yn)} \u2286 Rd \u00d7 R be a set of\ninput-output pairs that are independently drawn from an unknown distribution P. In linear regression,\nwe assume that the relationship between the input and output can be well modeled by a linear function,\nand aim to discover it from the training set T . For a linear function f (x) = x(cid:62)w, its quality is\nmeasured by the expected prediction error on a random pair (x, y) sampled from P, i.e., the risk:\n\nR(cid:96)(w) = E(x,y)\u223cP(cid:2)(cid:96)(x(cid:62)w, y)(cid:3)\n\nwhere (cid:96)(\u00b7,\u00b7) is a loss that quanti\ufb01es the prediction error. The most popular losses include the squared\nloss (cid:96)2(u, v) = (u \u2212 v)2 and the absolute loss (cid:96)1(u, v) = |u \u2212 v|.\nLet W \u2286 Rd be a domain of linear coef\ufb01cients. The standard approach for linear regression is the\nempirical risk minimization (ERM)\n\n(cid:96)(x(cid:62)\n\ni w, yi)\n\nw\u2208W (cid:98)R(cid:96)(w) =\n\nmin\n\nn(cid:88)\n\ni=1\n\n1\nn\n\nwhich selects the linear function that minimizes the empirical risk on the training set.\nIn the\nliterature, there are plenty of theoretical guarantees on linear regression by ERM, either targeting\nlinear regression directly [Birg\u00e9 and Massart, 1998, Gy\u00f6r\ufb01 et al., 2002], or being derived from the\ngeneral theories of statistical learning [Vapnik, 2000, Koltchinskii, 2011, Zhang et al., 2017]. In order\nto establish high-probability risk bounds, most of the previous analyses rely on the assumption that\nthe input and output are bounded or sub-Gaussian.\nHowever, if the input and output are heavy-tailed, which is commonly encountered in many disciplines\nsuch as \ufb01nance and environment [Finkenst\u00e4dt and Rootz\u00e9n, 2003], existing high-probability bounds\nof ERM become invalid. In fact, for heavy-tailed distributions, it has been explicitly proved that the\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fempirical risk is no longer a good approximation of the risk [Catoni, 2012], which inspires recent\nstudies on learning with heavy-tailed losses [Audibert and Catoni, 2011, Hsu and Sabato, 2014,\nBrownlees et al., 2015]. In particular, for linear regression with squared loss, i.e., (cid:96)2-regression,\nAudibert and Catoni [2011] develop a truncated min-max estimator, and establish an O(d/n) excess\nrisk that holds with high probability even when the input and output are heavy-tailed. This result is a\ngreat breakthrough as it extends the scope of linear regression, but is limited to the squared loss.\nThe motivation of squared loss is to estimate the conditional mean of y given x. Besides the squared\nloss, there also exist other losses for regression. For example, if we are interested in the conditional\nmedian, then absolute loss would be a natural choice [Hastie et al., 2009]. Furthermore, absolute loss\nis more robust in that it is resistant to outliers in the data [Peter J. Rousseeuw, 1987]. In this paper, we\ntarget linear regression with absolute loss, namely (cid:96)1-regression. Inspired by the truncation function\nof Audibert and Catoni [2011], we propose a truncated minimization problem to support heavy-tailed\n\ndistributions. Theoretical analysis shows that our method achieves an (cid:101)O((cid:112)d/n) excess risk, which\n\nholds with high probability. Our theoretical guarantee requires very mild assumptions, and is derived\nfrom standard techniques\u2014the covering number and concentration inequalities. Furthermore, we\nshow that if the input is bounded, the classical ERM is suf\ufb01cient even when the output is heavy-tailed.\nWe highlight the contributions of this paper as follows.\n\n\u2022 We propose a truncated minimization problem for (cid:96)1-regression, which is more simple than\n\n\u2022 This is the \ufb01rst time an (cid:101)O((cid:112)d/n) excess risk is established for (cid:96)1-regression under the\n\nthe truncated min-max formulation of Audibert and Catoni [2011] for (cid:96)2-regression.\n\ncondition that both the input and output could be heavy-tailed. Although Brownlees et al.\n[2015] develop a general theorem for heavy-tailed losses, when applied to (cid:96)1-regression, it\nrequires the input to be bounded.\n\n\u221a\n\u2022 When the input is bounded, we prove that the classical ERM achieves an O(D/\n\nn) excess\nrisk for (cid:96)1-regression, where D is the maximum norm of the input, and does not require any\nassumption on the output.\n\nFinally, we note that although this paper focuses on (cid:96)1-regression, the idea of truncated minimization\nand our analysis can be directly extended to Lipschitz losses [Alquier et al., 2017] that satisfy\n\n|(cid:96)(x(cid:62)w, y) \u2212 (cid:96)(x(cid:62)w(cid:48), y)| \u2264 |x(cid:62)w \u2212 x(cid:62)w(cid:48)|, \u2200w, w(cid:48) \u2208 W.\n\nMany popular losses for classi\ufb01cation, including hinge loss and logistic loss, are Lipschitz losses. We\nwill provide detailed investigations in an extended paper.\n\n2 Related Work\n\nIn this section, we review the recent work on learning with heavy-tailed losses. A distribution F is\nheavy-tailed if and only if [Foss et al., 2013]\n\n(cid:90)\n\ne\u03bbxdF (x) = \u221e, \u2200\u03bb > 0.\n\nThere exist studies that try to equip ERM with theoretical guarantees under heavy-tailed distributions.\nHowever, these theories need additional assumptions, such as the small-ball assumption [Mendelson,\n2014, 2015] and the Bernstein\u2019s condition [Dinh et al., 2016, Alquier et al., 2017], and do not always\nhold with high probability. In the following, we only discuss alternatives of ERM.\n\n2.1 Truncation based Approaches\n\nOur truncated method follows the line of research stemmed from Catoni [2012]. In his seminal work,\nCatoni [2012] considers the estimation of the mean and variance of a real random variable from\nindependent and identically distributed (i.i.d.) samples. Let x1, . . . , xn \u2208 R be i.i.d. samples drawn\n\nfrom some unknown probability distribution P. Catoni [2012] builds the estimator(cid:98)\u03b8 of the mean as\n\nthe solution of the equation\n\n= 0\n\n(1)\n\nn(cid:88)\n\n(cid:104)\n\n\u03c8\n\n(cid:105)\n\n\u03b1(xi \u2212(cid:98)\u03b8)\n\ni=1\n\n2\n\n\f(cid:18)\n\n(cid:19)\n\n(cid:18)\n\nx2\n2\n\n\u2212 log\n\n1 \u2212 x +\n\nwhere \u03b1 > 0 is a positive parameter, and \u03c8(\u00b7) : R (cid:55)\u2192 R is a truncation function that is non-decreasing\nand satis\ufb01es\n\nBy choosing \u03b1 =(cid:112)2/n\u03bd, where \u03bd is the variance of the random variable, Catoni [2012] demonstrates\nthat with high probability, the deviation of(cid:98)\u03b8 from the mean is upper bounded by O((cid:112)\u03bd/n). Thus,\n\n\u2264 \u03c8(x) \u2264 log\n\n(1) can be applied to heavy-tailed distributions, as long as the variance is \ufb01nite.\nAudibert and Catoni [2011] extend the robust estimator of Catoni [2012] to linear regression with\nsquared loss. Speci\ufb01cally, they consider the (cid:96)2-norm regularized (cid:96)2-regression (i.e., ridge regression),\nand propose the following min-max estimator\n\n1 + x +\n\nx2\n2\n\n(2)\n\n.\n\n(cid:19)\n\nwhere (cid:107) \u00b7 (cid:107) denotes the (cid:96)2-norm of vectors and \u03c8(\u00b7) is the truncation function de\ufb01ned as\n\n\u03c8(cid:2)\u03b1(yi \u2212 x(cid:62)\n\nn(cid:88)\n\ni=1\n\ni w)2 \u2212 \u03b1(yi \u2212 x(cid:62)\n\ni u)2(cid:3)\n\nmin\nw\u2208W max\n\nu\u2208W \u03bb(cid:0)(cid:107)w(cid:107)2 \u2212 (cid:107)u(cid:107)2(cid:1) +\n\uf8f1\uf8f4\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f4\uf8f3\n\n\u03c8(x) =\n\n\u2212 log\n\n1\n\u03b1n\n\n(cid:18)\n\n(cid:19)\n\n1 \u2212 x +\n\n, 0 \u2264 x \u2264 1;\n\nx2\n2\nlog(2), x \u2265 1;\n\u2212\u03c8(\u2212x), x \u2264 0.\n\n(3)\n\n(4)\n\nLet ((cid:98)w,(cid:98)u) be the optimal solution to (3). With a suitable choice of \u03b1, Audibert and Catoni [2011]\n\nhave proved that the following risk bound\n\nR(cid:96)2((cid:98)w) + \u03bb(cid:107)(cid:98)w(cid:107)2 \u2212 min\n\nw\u2208W\n\n(cid:8)R(cid:96)2 (w) + \u03bb(cid:107)w(cid:107)2(cid:9) = O\n\n(cid:18) d\n\n(cid:19)\n\nn\n\nholds with high probability even when neither the input nor the output has exponential moments.\nIn a subsequent work, Brownlees et al. [2015] apply the robust estimator of Catoni [2012] to the\ngeneral problem of learning with heavy-tailed losses. Let x be a random variable taking values in\nsome measurable space X and F be a set of nonnegative functions de\ufb01ned on X . Given n independent\nrandom variables x1, . . . , xn, all distributed as x, Brownlees et al. [2015] propose to use the robust\nestimator in (1) to estimate the mean of each function f \u2208 F, and choose the one that minimizes the\nestimator. Formally, the optimization problem is given by\n\ns. t.\n\nmin\n\n1\nn\u03b1\n\nn(cid:88)\n\nf\u2208F (cid:98)\u00b5f\n\uf8f1\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f3 log\n\n\u2212 log\n\ni=1\n\n\u03c8(x) =\n\n\u03c8(cid:2)\u03b1(f (xi) \u2212(cid:98)\u00b5f )(cid:3) = 0\n(cid:18)\n(cid:18)\n\n(cid:19)\n(cid:19)\n\n1 + x +\n\n, x \u2265 0;\n\n1 \u2212 x +\n\n, x \u2264 0.\n\nx2\n2\nx2\n2\n\n(5)\n\nwhere \u03c8(\u00b7) is de\ufb01ned as\n\nBased on the generic chaining method [Talagrand, 2005], Brownlees et al. [2015] develop performance\nbounds of the above problem. However, the theoretical guarantees rely on the condition that the\nfunction space F is bounded in terms of certain distance. When applying their results to (cid:96)1-regression,\nthe linear function needs to be bounded, which means the input vector must be bounded [Brownlees\net al., 2015, Section 4.1.1]. When applying to (cid:96)2-regression, the input also needs to be bounded and\nthe theoretical guarantee is no longer a high-probability bound [Brownlees et al., 2015, Section 4.1.2].\n\n2.2 Median-of-means Approaches\n\nAnother way to deal with heavy-tailed distributions is the median-of-means estimator [Nemirovski\nand Yudin, 1983, Alon et al., 1999, Bubeck et al., 2013, Minsker, 2015]. The basic idea is to divide\n\n3\n\n\fthe data into several groups, calculate the sample mean within each group, and take the median of\nthese means. Recently, Hsu and Sabato [2014, 2016] generalize the median-of-means estimator\nto arbitrary metric spaces, and apply it to the minimization of smooth and strongly convex losses.\n\nSpeci\ufb01cally, for (cid:96)2-regression with heavy-tailed distributions, a high-probability (cid:101)O(d/n) excess\n\nrisk is established, under slightly stronger assumptions than those of Audibert and Catoni [2011].\nFor regression problem, Lugosi and Mendelson [2016] introduce a new procedure, the so-called\nmedian-of-means tournament, which achieves the optimal tradeoff between accuracy and con\ufb01dence\nunder minimal assumptions. The setting of Lugosi and Mendelson [2016] is general in the sense that\nthe function space could be any convex class of functions, not necessary linear, but the performance\nis only measured by the squared loss.\nCompared with truncation based approaches, the advantage of median-of-means approaches is that\nthey do not require prior knowledge of distributional properties. However, the current theoretical\nresults of median-of-means are restricted to the squared loss or strongly convex losses, and thus\ncannot be applied to (cid:96)1-regression considered in this paper.\n\n3 Our Results\n\nIn this section, we \ufb01rst present our truncated minimization problem, then discuss its theoretical\nguarantee, and \ufb01nally study the special setting of bounded inputs.\n\n3.1 Our Formulation\n\nInspired by the truncated min-max estimator of Audibert and Catoni [2011], we propose the following\ntruncated minimization problem for (cid:96)1-regression with heavy-tailed distributions:\n\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\ni w|(cid:1)\n\nn(cid:88)\n\ni=1\n\nmin\nw\u2208W\n\n1\nn\u03b1\n\n(6)\n\nwhere the truncation function \u03c8(\u00b7) is non-decreasing and satis\ufb01es (2), and \u03b1 > 0 is a parameter\nwhose value will be determined later. Note that we can choose (4) or (5) as the truncation function.\nCompared with the min-max problem in (3), our minimization problem in (6) is more simple.\nAlthough (6) is still a non-convex problem, it has a special structure that can be exploited. Because \u03c8(\u00b7)\nis non-decreasing, it is easy to verify that each individual function \u03c8(\u03b1|yi \u2212 x(cid:62)\ni w|) is quasiconvex.\nThus, our problem is to minimize the sum of quasiconvex functions. From the recent developments\nof quasiconvex optimization [Hazan et al., 2015], we may apply (stochastic) normalized gradient\ndescent (NGD) to solve (6). This paper focuses on the statistical property of (6), and we leave the\ndesign of ef\ufb01cient optimization procedures as a future work.\n\n3.2 Theoretical Guarantees\nLet W be a subset of a Hilbert space H, and (cid:107) \u00b7 (cid:107) be the norm associated with the inner product of H.\nWe introduce assumptions that used in our analysis.\nAssumption 1 The domain W is totally bounded such that for any \u03b5 > 0, there exists a \ufb01nite \u03b5-net\nof W.1\n\nAssumption 2 The expectation of the squared norm of x is bounded, that is,\n\nE(x,y)\u223cP(cid:2)(cid:107)x(cid:107)2(cid:3) < \u221e.\n\nAssumption 3 The (cid:96)2-risk of all w \u2208 W is bounded, that is,\n\nE(x,y)\u223cP(cid:2)(y \u2212 x(cid:62)w)2(cid:3) < \u221e.\n\nsup\nw\u2208W\n\nR(cid:96)2(w) = sup\nw\u2208W\n\n1A subset N \u2286 K is called an \u03b5-net of K if for every w \u2208 K one can \ufb01nd a (cid:101)w \u2208 N so that (cid:107)w \u2212 (cid:101)w(cid:107) \u2264 \u03b5.\n\n4\n\n\fRemark 1 We have the following comments regarding our assumptions.\n\nunbounded, which allows us to model heavy-tailed distributions.\n\n\u2022 Although Assumption 1 requires the domain W is bounded, the input and output could be\n\u2022 Because our goal is to bound the (cid:96)1-risk, which is the \ufb01rst-order moment, it is natural\nto require higher-order moment conditions. Thus, in Assumptions 2 and 3, we introduce\nsecond-order moment conditions on inputs and outputs. Our assumptions support heavy-\ntailed distributions in the sense that commonly used heavy-tailed distributions, such as the\nPareto distribution (with parameter \u03b1 > 2) and the log-normal distribution, have \ufb01nite\nsecond-order moment.\n\u2022 By Jensen\u2019s inequality, we have (E[(cid:107)x(cid:107)])2 \u2264 E[(cid:107)x(cid:107)2]. Thus, Assumption 2 implies E[(cid:107)x(cid:107)]\n\u2022 Given Assumptions 1 and 2, Assumption 3 can be relaxed as the (cid:96)2-risk of the optimal\n\nis bounded.\n\nR(cid:96)2 (w) \u22642R(cid:96)2(w\u2217) + 2(w \u2212 w\u2217)(cid:62)E(cid:2)xx(cid:62)(cid:3) (w \u2212 w\u2217)\n\n\u22642R(cid:96)2(w\u2217) + 2(cid:107)w \u2212 w\u2217(cid:107)2(cid:13)(cid:13)E(cid:2)xx(cid:62)(cid:3)(cid:13)(cid:13)2\n(cid:13)(cid:13)E(cid:2)xx(cid:62)(cid:3)(cid:13)(cid:13)2 \u2264 E(cid:2)(cid:13)(cid:13)xx(cid:62)(cid:13)(cid:13)2\n(cid:3) = E(cid:2)(cid:107)x(cid:107)2(cid:3) < \u221e.\n\nwhere (cid:107) \u00b7 (cid:107)2 is the spectral norm of matrices. First, Assumption 1 implies (cid:107)w \u2212 w\u2217(cid:107) is\nbounded. Second, Assumption 2 implies the spectral norm of E[xx(cid:62)] is also bounded, that\nis,\n\nsolution w\u2217 is bounded. To see this, we have\n\nThus, R(cid:96)2 (w) is bounded as long as R(cid:96)2 (w\u2217) is bounded and Assumptions 1 and 2 hold.\ncardinality of the \u03b5-net of W is called the covering number and denoted by N (W, \u03b5). Let (cid:98)w be a\nTo present our theoretical guarantee, we introduce the notation of the covering number. The minimal\nsolution to (6), and w\u2217 \u2208 argminw\u2208W R(cid:96)1(w) be an optimal solution that minimizes the (cid:96)1-risk.\nWe have the following excess risk bound.\nTheorem 1 Let 0 < \u03b4 < 1/2. Under Assumptions 1, 2 and 3, with probability at least 1 \u2212 2\u03b4, we\nhave\n\nR(cid:96)1 ((cid:98)w) \u2212 R(cid:96)1(w\u2217) \u2264 2\u03b5E[(cid:107)x(cid:107)] + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\n3\u03b1\n2\n\nsup\nw\u2208W\n\nR(cid:96)2 (w) +\n\n1\nn\u03b1\n\nlog\n\nN (W, \u03b5)\n\n\u03b42\n\nfor any \u03b5 > 0. Furthermore, by setting\n\n(cid:114)\n\n\u03b1 =\n\n(cid:114)\n\n1\nn\n\nlog\n\nN (W, \u03b5)\n\n,\n\n\u03b42\n\n(cid:18)\n\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\n(cid:19)\n\nR(cid:96)2(w) + 1\n\n.\n\n3\n2\n\nsup\nw\u2208W\n\n1\nn\n\nlog\n\nN (W, \u03b5)\n\n\u03b42\n\nwe have\n\nR(cid:96)1((cid:98)w) \u2212 R(cid:96)1 (w\u2217) \u2264 2\u03b5E[(cid:107)x(cid:107)] +\n\nRemark 2 Note that Theorem 1 is very general in the sense that it can be applied to in\ufb01nite\ndimensional Hilbert spaces, provided the domain W has a \ufb01nite covering number [Cucker and Smale,\n2002]. In contrast, the result of Audibert and Catoni [2011] is limited to \ufb01nite spaces. The difference\nis caused by the different techniques used in the analysis: While Audibert and Catoni [2011] employ\nthe PAC-Bayesian analysis, we make use of the covering number and standard concentrations.\nTo reveal the order of the excess risk, we need to specify the value of the covering number. To this\nend, we consider the special case that W is a bounded subset of Euclidean space, and introduce the\nfollowing condition.\nAssumption 4 The domain W is a subset of Rd and its radius is bounded by B, that is,\n\n(cid:107)w(cid:107) \u2264 B, \u2200w \u2208 W \u2286 Rd.\n\n(7)\nLet Br \u2286 Rd be a ball centered at origin with radius r, and N (Br, \u03b5) be its \u03b5-net with minimal\ncardinality, denoted by N (Br, \u03b5). According to a standard volume comparison argument [Pisier,\n1989], we have\n\nlog N(cid:0)B1, \u03b5(cid:1) \u2264 d log\n\n\u21d2 log N(cid:0)Br, \u03b5(cid:1) \u2264 d log\n\n3r\n\u03b5\n\n.\n\n3\n\u03b5\n\n5\n\n\fSince W \u2286 BB, we have\n\nlog N (W, \u03b5) \u2264 log N\n\n(cid:16)BB,\n\n(cid:17) \u2264 d log\n\n\u03b5\n2\n\n6B\n\u03b5\n\nwhere the \ufb01rst inequality is because the covering numbers are (almost) increasing by inclusion [Plan\nand Vershynin, 2013, (3.2)]. Then, we have the following corollary by setting \u03b5 = 1/n.\n\nCorollary 2 Let 0 < \u03b4 < 1/2, and set\n\n\u03b1 =\n\n(cid:114)\n\n1\nn\n\nlog\n\nN (W, 1/n)\n\n.\n\n\u03b42\n\n1\nn\n\n.\n\n\u2264 2\nn\n\n=O\n\nUnder Assumptions 2, 3 and 4, with probability at least 1 \u2212 2\u03b4, we have\n\nE[(cid:107)x(cid:107)] +\n\nR(cid:96)1((cid:98)w) \u2212 R(cid:96)1 (w\u2217)\n(cid:115)\n(cid:18)\n(cid:33)\n(cid:32)(cid:114)\nIgnoring the logarithmic factor, Corollary 2 shows an (cid:101)O((cid:112)d/n) excess risk that holds\n\n(cid:19)(cid:18) 1\nn2 E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\nd log(6nB) + log\n\nRemark 3\nwith high probability. From the above discussions, we see that the square root dependence on d\ncomes from the upper bound of the covering number. If the domain W has additional structures (e.g.,\nsparse), the covering number may have a smaller dependence on d, and as a result, the dependence\non d could be improved.\n\nR(cid:96)2(w) + 1\n\n3\n2\n\nsup\nw\u2208W\n\n(cid:19)\n\nd log n\n\n1\n\u03b42\n\nn\n\n3.3 Bounded Inputs\n\nIf we only allow the output to be heavy-tailed and the input is bounded, the problem becomes much\neasier. Although both our method and the algorithm of Brownlees et al. [2015] are applicable, at least\nin theory, there is no need to resort to sophisticated methods. In fact, a careful analysis shows that the\nclassical ERM is suf\ufb01cient in this case.\nWe introduce the following assumption.\nAssumption 5 The norm of the random vector x \u2208 Rd is upper bounded by a constant D, that is,\n(8)\n\n(cid:107)x(cid:107) \u2264 D, \u2200(x, y) \u223c P.\n\nThen, we have the following risk bound for ERM.\nTheorem 3 Let\n\nn(cid:88)\n\ni=1\n\n\u00afw \u2208 argmin\nw\u2208W\n\n1\nn\n\n|yi \u2212 x(cid:62)\n\ni w|\n\n(cid:32)\n\n(cid:114) 1\n\n(cid:33)\n\nR(cid:96)1 ( \u00afw) \u2212 R(cid:96)1(w\u2217) \u2264 4BD\u221a\nn\n\n1 +\n\nlog\n\n1\n\u03b4\n\n2\n\n.\n\nbe a solution returned by ERM. Under Assumptions 4 and 5, with probability at least 1 \u2212 \u03b4, we have\n\nRemark 4 When inputs are upper bounded, we do not need any assumption about outputs. That\n\u221a\nis because the absolute loss is 1-Lipschitz continuous, and outputs will be canceled in the analysis.\nTheorem 3 implies ERM achieves an O(D/\nn) excess risk which holds with high probability.\nCompared with the risk bound in Corollary 2, we observe that the new bound is independent from\nthe dimensionality d, but it has a linear dependence on D, which is the upper bound of the norm of\ninputs.\n\n4 Analysis\n\nDue to the limitation of space, we only present the proof of Theorem 1. The proof of Theorem 3 can\nbe found in the full paper [Zhang and Zhou, 2018].\n\n6\n\n\f.\n\n(9)\n\n4.1 Proof of Theorem 1\n\n1\nn\u03b1\n\nTo simplify notations, de\ufb01ne (cid:98)R\u03c8\u25e6(cid:96)1(w) =\nFrom the optimality of (cid:98)w, we have\ni (cid:98)w|(cid:1)\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n(cid:123)(cid:122)\n(cid:125)\n(cid:98)R\u03c8\u25e6(cid:96)1 ((cid:98)w)\n\nn(cid:88)\n\n1\nn\u03b1\n\n(cid:124)\n\ni=1\n\nn(cid:88)\n\ni=1\n\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\ni w|(cid:1).\nn(cid:88)\ni w\u2217|(cid:1)\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n(cid:123)(cid:122)\n(cid:125)\n(cid:98)R\u03c8\u25e6(cid:96)1 (w\u2217)\n\ni=1\n\n\u2264 1\nn\u03b1\n\n(cid:124)\n\nNext, we will discuss how to upper bound (cid:98)R\u03c8\u25e6(cid:96)1(w\u2217) by R(cid:96)1(w\u2217) and lower bound (cid:98)R\u03c8\u25e6(cid:96)1 ((cid:98)w) by\nR(cid:96)1 ((cid:98)w).\nBecause w\u2217 is independent from samples (x1, y1), . . . , (xn, yn), it is easy to relate (cid:98)R\u03c8\u25e6(cid:96)1(w\u2217) with\n\nR(cid:96)1 (w\u2217) by the standard concentration techniques. To this end, we have the following lemma.\nLemma 1 With probability at least 1 \u2212 \u03b4, we have\n\n(cid:98)R\u03c8\u25e6(cid:96)1 (w\u2217) \u2264 R(cid:96)1(w\u2217) +\n\n\u03b1\n2\n\nR(cid:96)2(w\u2217) +\n\n1\nn\u03b1\n\nlog\n\n1\n\u03b4\n\n.\n\nLower bounding (cid:98)R\u03c8\u25e6(cid:96)1((cid:98)w) is more involved because (cid:98)w depends on the sample. To this end, we\nR(cid:96)1((cid:98)w) \u2264 (cid:98)R\u03c8\u25e6(cid:96)1((cid:98)w) + 2\u03b5E[(cid:107)x(cid:107)] + \u03b1 sup\n\ncombine the covering number and concentration inequalities to develop the following lemma.\nLemma 2 With probability at least 1 \u2212 \u03b4, we have\n\nR(cid:96)2 (w) + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\nN (W, \u03b5)\n\nlog\n\n1\nn\u03b1\n\n\u03b4\n\nw\u2208W\n\nfor any \u03b5 > 0.\n\nThen, Theorem 1 is a direct consequence of (9), Lemmas 1 and 2, and the union bound.\n\n4.2 Proof of Lemma 1\n\nFirst, note that our truncation function \u03c8 satis\ufb01es\n\n(cid:18)\n\nThen, we have\n\n(cid:19)\n(cid:32) n(cid:88)\n\nx2\n2\n\n(cid:34)\n\n\u03c8(x) \u2264 log\n\n1 + x +\n\n, \u2200x \u2208 R.\n\n(cid:16)\n(cid:17)(cid:105)\nn\u03b1(cid:98)R\u03c8\u25e6(cid:96)1 (w\u2217)\n(cid:18)\n\n1 + \u03b1|yi \u2212 x(cid:62)\n\ni w\u2217| +\n\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n(cid:19)(cid:35)\n(cid:21)(cid:19)n\n\n= E\n\nexp\n\ni=1\n\n\u03b12(yi \u2212 x(cid:62)\n\ni w\u2217)2\n\n2\n\n\u03b12|y \u2212 x(cid:62)w\u2217|2\n\n1 + \u03b1|y \u2212 x(cid:62)w\u2217| +\n\n1 + \u03b1R(cid:96)1 (w\u2217) +\n\n\u03b12\n2\n\nR(cid:96)2(w\u2217)\n\n(cid:20)\n\nn\n\n\u03b1R(cid:96)1(w\u2217) +\n\nR(cid:96)2(w\u2217)\n\n.\n\n(cid:19)n\n\n2\n\n(cid:21)(cid:19)\n\nE\n\nexp\n\n(10)\u2264 E\n\n(cid:104)\n(cid:34) n(cid:89)\n(cid:18)\n(cid:20)\n(cid:18)\n1+x\u2264ex\u2264 exp\n\n(cid:18)\n\n=\n\n=\n\ni=1\n\nE\n\ni w\u2217|(cid:1)(cid:33)(cid:35)\n\n\u03b12\n2\n\n7\n\n(10)\n\n(11)\n\n\fBy Chernoff\u2019s method [Lugosi, 2009], we have\n\nP\n\n(cid:20)\n\n\u03b1R(cid:96)1 (w\u2217) +\n\n(cid:26)\nn\u03b1(cid:98)R\u03c8\u25e6(cid:96)1 (w\u2217) \u2265 n\n(cid:26)\n(cid:18)\nexp(cid:0)n\u03b1(cid:98)R\u03c8\u25e6(cid:96)1(w\u2217)(cid:1) \u2265 exp\nexp(cid:0)n\u03b1(cid:98)R\u03c8\u25e6(cid:96)1(w\u2217)(cid:1)(cid:105)\n(cid:104)\nexp(cid:0)n(cid:2)\u03b1R(cid:96)1(w\u2217) + \u03b12\n\n(cid:20)\n2 R(cid:96)2(w\u2217)(cid:3) + log 1\n\n\u03b12\n2\n\nE\n\nn\n\n\u03b4\n\n= P\n\n\u2264\n\nR(cid:96)2(w\u2217)\n\n(cid:1) (11)\u2264 \u03b4\n\n(cid:21)\n\n+ log\n\n(cid:27)\n\n1\n\u03b4\n\n\u03b1R(cid:96)1(w\u2217) +\n\nR(cid:96)2 (w\u2217)\n\n\u03b12\n2\n\n(cid:21)\n\n(cid:19)(cid:27)\n\n+ log\n\n1\n\u03b4\n\nwhich completes the proof.\n\n4.3 Proof of Lemma 2\nLet N (W, \u03b5) be an \u03b5-net of W with minimal cardinality N (W, \u03b5). From the de\ufb01nition of \u03b5-net, there\n\nmust exist a (cid:101)w \u2208 N (W, \u03b5) such that (cid:107)(cid:98)w \u2212(cid:101)w(cid:107) \u2264 \u03b5. So, we have\n\nn\u03b1\n\n1\nn\u03b1\n\ni=1\n\n1\nn\u03b1\n\ni=1\n\n|yi \u2212 x(cid:62)\n\nn(cid:88)\n\ni=1\n\n(cid:98)R\u03c8\u25e6(cid:96)1((cid:98)w) =\n\nSince \u03c8(\u00b7) is non-decreasing, we have\n\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1)\n\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n(cid:20)\n\nTo proceed, we develop the following lemma to lower bound the last term in (13).\n\ni (cid:101)w| \u2212 \u03b5(cid:107)xi(cid:107).\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1).\n\ni (cid:98)w| \u2265 |yi \u2212 x(cid:62)\ni ((cid:101)w \u2212(cid:98)w)| \u2265 |yi \u2212 x(cid:62)\ni (cid:101)w| \u2212 |x(cid:62)\nn(cid:88)\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\ni (cid:98)w|(cid:1) (12)\u2265 1\nLemma 3 With probability at least 1 \u2212 \u03b4, for all (cid:101)w \u2208 N (W, \u03b5), we have\nR(cid:96)2 (w) + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\nn(cid:88)\n\u2265R(cid:96)1((cid:101)w) \u2212\n(cid:98)R\u03c8\u25e6(cid:96)1 ((cid:98)w)\n(cid:21)\n\u2265R(cid:96)1((cid:101)w) \u2212\n(cid:21)\n\u2265R(cid:96)1((cid:98)w) \u2212\nR(cid:96)1 ((cid:98)w) = E(cid:2)|y \u2212 x(cid:62)(cid:98)w|(cid:3) \u2264 E(cid:2)|y \u2212 x(cid:62)(cid:101)w| + |x(cid:62)((cid:101)w \u2212(cid:98)w)|(cid:3) \u2264 R(cid:96)1 ((cid:101)w) + \u03b5E[(cid:107)x(cid:107)].\n\nR(cid:96)2(w) + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\nR(cid:96)2(w) + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\nSubstituting (14) into (13), with probability at least 1 \u2212 \u03b4, we have\n\n\u03b5E[(cid:107)x(cid:107)] + \u03b1 sup\nw\u2208W\n2\u03b5E[(cid:107)x(cid:107)] + \u03b1 sup\nw\u2208W\n\nwhere the last step is due to the following inequality\n\n\u03b5E[(cid:107)x(cid:107)] + \u03b1 sup\nw\u2208W\n\n1\nn\u03b1\n1\nn\u03b1\n\n\u03b4\n\nN (W, \u03b5)\n\nN (W, \u03b5)\n\n1\nn\u03b1\n\nlog\n\nN (W, \u03b5)\n\nlog\n\nlog\n\n(cid:21)\n\n.\n\n(cid:20)\n(cid:20)\n\n\u03b4\n\n\u03b4\n\n(12)\n\n(13)\n\n(14)\n\n4.4 Proof of Lemma 3\n\nWe \ufb01rst consider a \ufb01xed (cid:101)w \u2208 N (W, \u03b5) \u2286 W. The proof is similar to that of Lemma 1. Recall that\n\nthe truncation function \u03c8(\u00b7) satis\ufb01es\n\n\u03c8(x) \u2265 \u2212 log\n\n, \u2200x \u2208 R.\n\n(15)\n\n(cid:19)\n\n(cid:18)\n\n1 \u2212 x +\n\nx2\n2\n\n8\n\n\fThen, we have\n\nwhere the last step is due to the basic inequalities 1 + x \u2264 ex and (a + b)2 \u2264 2a2 + 2b2.\nBy Chernoff\u2019s method [Lugosi, 2009], we have\n\n2\n\n2\n\nn\n\nE\n\nE\n\ni=1\n\n=\n\nE\n\ni=1\n\nexp\n\n\u03b12\n2\n\n(15)\u2264 E\n\n=\n\u2264 exp\n\n1 \u2212 \u03b1|yi \u2212 x(cid:62)\n\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n\n(cid:34)\n(cid:32)\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1)(cid:33)(cid:35)\n\u2212 n(cid:88)\n(cid:34) n(cid:89)\n(cid:32)\ni (cid:101)w| \u2212 \u03b5(cid:107)xi(cid:107)(cid:1)2\n\u03b12(cid:0)|yi \u2212 x(cid:62)\ni (cid:101)w| + \u03b1\u03b5(cid:107)xi(cid:107) +\n(cid:34)\n(cid:32)\n(cid:35)(cid:33)n\n\u03b12(cid:0)|y \u2212 x(cid:62)(cid:101)w| \u2212 \u03b5(cid:107)x(cid:107)(cid:1)2\n1 \u2212 \u03b1|y \u2212 x(cid:62)(cid:101)w| + \u03b1\u03b5(cid:107)x(cid:107) +\n(cid:18)\n(cid:104)(cid:0)|y \u2212 x(cid:62)(cid:101)w| \u2212 \u03b5(cid:107)x(cid:107)(cid:1)2(cid:105)(cid:19)n\n1 \u2212 \u03b1R(cid:96)1 ((cid:101)w) + \u03b1\u03b5E[(cid:107)x(cid:107)] +\n(cid:16) \u2212 \u03b1R(cid:96)1((cid:101)w) + \u03b1\u03b5E[(cid:107)x(cid:107)] + \u03b12R(cid:96)2((cid:101)w) + \u03b12\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3)(cid:17)(cid:105)\n(cid:104)\n(cid:40)\n\u2212 n(cid:88)\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1) \u2265\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n(cid:16) \u2212 \u03b1R(cid:96)1 ((cid:101)w) + \u03b1\u03b5E[(cid:107)x(cid:107)] + \u03b12R(cid:96)2((cid:101)w) + \u03b12\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3)(cid:17)\n(cid:32)\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1)(cid:33)\n\u2212 n(cid:88)\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n(cid:20)\n(cid:16) \u2212 \u03b1R(cid:96)1((cid:101)w) + \u03b1\u03b5E[(cid:107)x(cid:107)] + \u03b12R(cid:96)2((cid:101)w) + \u03b12\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3)(cid:17)\nE(cid:2)exp(cid:0)\u2212(cid:80)n\ni=1 \u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1)(cid:1)(cid:3)\n(cid:16) \u2212 \u03b1R(cid:96)1 ((cid:101)w) + \u03b1\u03b5E[(cid:107)x(cid:107)] + \u03b12R(cid:96)2((cid:101)w) + \u03b12\u03b52E [(cid:107)x(cid:107)2]\n\n+ log 1\n\u03b4\n\n(cid:40)\n\n+ log\n\n(cid:17)\n\n(cid:104)\n\nexp\n\nexp\n\n\u2265\n\ni=1\n\ni=1\n\nn\n\nexp\n\nn\n\nn\n\nP\n\n= P\n\n\u2264\n\n(cid:33)(cid:35)\n\n.\n\n(16)\n\n(cid:27)\n\n1\n\u03b4\n\n(cid:21)(cid:27)\n(cid:105) (16)\u2264 \u03b4.\n\n1\n\u03b4\n\n+ log\n\nThus, with probability at least 1 \u2212 \u03b4, we have\n\nn(cid:88)\n\n\u03c8(cid:0)\u03b1|yi \u2212 x(cid:62)\n\ni (cid:101)w| \u2212 \u03b1\u03b5(cid:107)xi(cid:107)(cid:1)\n\n\u2212 1\nn\u03b1\n\n.\n\ni=1\n\n1\n\u03b4\n\nlog\n\nlog\n\nw\u2208W\n\n1\nn\u03b1\n\n1\n\u03b4\n1\nn\u03b1\n\n\u2264 \u2212 R(cid:96)1 ((cid:101)w) + \u03b5E[(cid:107)x(cid:107)] + \u03b1R(cid:96)2 ((cid:101)w) + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\nR(cid:96)2(w) + \u03b1\u03b52E(cid:2)(cid:107)x(cid:107)2(cid:3) +\n\u2264 \u2212 R(cid:96)1 ((cid:101)w) + \u03b5E[(cid:107)x(cid:107)] + \u03b1 sup\nWe complete the proof by taking the union bound over all (cid:101)w \u2208 N (W, \u03b5).\nminimization problem. Under mild assumptions, we prove that our method enjoys an (cid:101)O((cid:112)d/n)\n\nIn this paper, we consider (cid:96)1-regression with heavy-tailed distributions, and propose a truncated\n\n5 Conclusion and Future Work\n\nexcess risk, which holds with high probability. Compared with traditional work on (cid:96)1-regression,\nthe main advantage of our result is that we establish a high-probability bound without exponential\nmoment conditions on the input and output. Furthermore, we demonstrate that when the input is\nbounded, the classical ERM is suf\ufb01cient for (cid:96)1-regression.\nIn the future, we will develop optimization algorithms and theories for the non-convex problem in\n(6). Another future work is to apply the idea of truncated minimization to other losses in machine\nlearning, especially Lipschitz losses.\n\n9\n\n\fAcknowledgments\n\nThis work was partially supported by the NSFC (61751306), YESS (2017QNRC001), and the\nCollaborative Innovation Center of Novel Software Technology and Industrialization. We thank an\nanonymous reviewer of COLT 2018 for helping us simplify the proof of Theorem 1.\n\nReferences\nN. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments.\n\nJournal of Computer and System Sciences, 58(1):137\u2013147, 1999.\n\nP. Alquier, V. Cottet, and G. Lecu\u00e9. Estimation bounds and sharp oracle inequalities of regularized\n\nprocedures with lipschitz loss functions. ArXiv e-prints, arXiv:1702.01402, 2017.\n\nJ.-Y. Audibert and O. Catoni. Robust linear least squares regression. The Annals of Statistics, 39(5):\n\n2766\u20132794, 2011.\n\nL. Birg\u00e9 and P. Massart. Minimum contrast estimators on sieves: exponential bounds and rates of\n\nconvergence. Bernoulli, 4(3):329\u2013375, 1998.\n\nC. Brownlees, E. Joly, and G. Lugosi. Empirical risk minimization for heavy-tailed losses. The\n\nAnnals of Statistics, 43(6):2507\u20132536, 2015.\n\nS. Bubeck, N. Cesa-Bianchi, and G. Lugosi. Bandits with heavy tail.\n\nInformation Theory, 59(11):7711\u20137717, 2013.\n\nIEEE Transactions on\n\nO. Catoni. Challenging the empirical mean and empirical variance: A deviation study. Annales de\n\nl\u2019Institut Henri Poincar\u00e9, Probabilit\u00e9s et Statistiques, 48(4):1148\u20131185, 2012.\n\nF. Cucker and S. Smale. On the mathematical foundations of learning. Bulletin of the American\n\nMathematical Society, 39(1):1\u201349, 2002.\n\nV. C. Dinh, L. S. Ho, B. Nguyen, and D. Nguyen. Fast learning rates with heavy-tailed losses. In\n\nAdvances in Neural Information Processing Systems 29, pages 505\u2013513, 2016.\n\nB. Finkenst\u00e4dt and H. Rootz\u00e9n, editors. Extreme Values in Finance, Telecommunications, and the\n\nEnvironment. Chapman & Hall/CRC, 2003.\n\nS. Foss, D. Korshunov, and S. Zachary. An Introduction to Heavy-Tailed and Subexponential\n\nDistributions. Springer, 2013.\n\nL. Gy\u00f6r\ufb01, M. Kohler, A. Krzy\u02d9zak, and H. Walk. A Distribution-Free Theory of Nonparametric\n\nRegression. Springer, 2002.\n\nT. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer Series in\n\nStatistics. Springer New York, 2009.\n\nE. Hazan, K. Levy, and S. Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex optimization.\n\nIn Advances in Neural Information Processing Systems 28, pages 1594\u20131602, 2015.\n\nD. Hsu and S. Sabato. Heavy-tailed regression with a generalized median-of-means. In Proceedings\n\nof the 31st International Conference on Machine Learning, pages 37\u201345, 2014.\n\nD. Hsu and S. Sabato. Loss minimization and parameter estimation with heavy tails. Journal of\n\nMachine Learning Research, 17(18):1\u201340, 2016.\n\nV. Koltchinskii. Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems.\n\nSpringer, 2011.\n\nG. Lugosi. Concentration-of-measure inequalities. Technical report, Department of Economics,\n\nPompeu Fabra University, 2009.\n\nG. Lugosi and S. Mendelson. Risk minimization by median-of-means tournaments. ArXiv e-prints,\n\narXiv:1608.00757, 2016.\n\n10\n\n\fS. Mendelson. Learning without concentration. In Proceedings of the 27th Annual Conference on\n\nLearning Theory, pages 25\u201339, 2014.\n\nS. Mendelson. Learning without concentration. Journal of the ACM, 62(3):21:1\u201321:25, 2015.\n\nS. Minsker. Geometric median and robust estimation in Banach spaces. Bernoulli, 21(4):2308\u20132335,\n\n2015.\n\nA. Nemirovski and D. B. Yudin. Problem Complexity and Method Ef\ufb01ciency in Optimization. John\n\nWiley & Sons Ltd, 1983.\n\nA. M. L. Peter J. Rousseeuw. Robust Regression and Outlier Detection. John Wiley & Sons Inc,\n\n1987.\n\nG. Pisier. The volume of convex bodies and Banach space geometry. Cambridge Tracts in Mathematics\n\n(No. 94). Cambridge University Press, 1989.\n\nY. Plan and R. Vershynin. One-bit compressed sensing by linear programming. Communications on\n\nPure and Applied Mathematics, 66(8):1275\u20131297, 2013.\n\nM. Talagrand. The Generic Chaining. Springer, 2005.\n\nV. Vapnik. The Nature of Statistical Learning Theory. Springer, second edition, 2000.\n\nL. Zhang and Z.-H. Zhou.\n\narXiv:1805.00616, 2018.\n\n(cid:96)1-regression with heavy-tailed distributions.\n\nArXiv e-prints,\n\nL. Zhang, T. Yang, and R. Jin. Empirical risk minimization for stochastic convex optimization:\nO(1/n)- and O(1/n2)-type of risk bounds. In Proceedings of the 30th Annual Conference on\nLearning Theory, pages 1954\u20131979, 2017.\n\n11\n\n\f", "award": [], "sourceid": 569, "authors": [{"given_name": "Lijun", "family_name": "Zhang", "institution": "Nanjing University (NJU)"}, {"given_name": "Zhi-Hua", "family_name": "Zhou", "institution": "Nanjing University"}]}