{"title": "Proximal Newton-type methods for convex optimization", "book": "Advances in Neural Information Processing Systems", "page_first": 827, "page_last": 835, "abstract": null, "full_text": "Proximal Newton-type methods for convex\n\noptimization\n\nJason D. Lee\u2217and Yuekai Sun\u2217\n\nInstitute for Computational and Mathematical Engineering\n\nStanford University, Stanford, CA\n\n{jdl17,yuekai}@stanford.edu\n\nMichael A. Saunders\n\nDepartment of Management Science and Engineering\n\nStanford University, Stanford, CA\nsaunders@stanford.edu\n\nAbstract\n\nWe seek to solve convex optimization problems in composite form:\n\nminimize\n\nx\u2208Rn\n\nf (x) := g(x) + h(x),\n\nwhere g is convex and continuously differentiable and h : Rn \u2192 R is a convex\nbut not necessarily differentiable function whose proximal mapping can be eval-\nuated ef\ufb01ciently. We derive a generalization of Newton-type methods to handle\nsuch convex but nonsmooth objective functions. We prove such methods are glob-\nally convergent and achieve superlinear rates of convergence in the vicinity of an\noptimal solution. We also demonstrate the performance of these methods using\nproblems of relevance in machine learning and statistics.\n\n1\n\nIntroduction\n\nMany problems of relevance in machine learning, signal processing, and high dimensional statistics\ncan be posed in composite form:\n\nx\u2208Rn\n\nminimize\n\nf (x) := g(x) + h(x),\n\n(1)\nwhere g : Rn \u2192 R is a convex, continuously differentiable loss function, and h : Rn \u2192 R is a\nconvex, continuous, but not necessarily differentiable penalty function. Such problems include: (i)\nthe lasso [23] (ii) multitask learning [14] and (iii) trace-norm matrix completion [6].\nWe describe a family of Newton-type methods tailored to these problems that achieve superlinear\nrates of convergence subject to standard assumptions. These methods can be interpreted as general-\nizations of the classic proximal gradient method that use the curvature of the objective function to\nselect a search direction.\n\n1.1 First-order methods\n\nThe most popular methods for solving convex optimization problems in composite form are \ufb01rst-\norder methods that use proximal mappings to handle the nonsmooth part. SpaRSA is a generalized\nspectral projected gradient method that uses a spectral step length together with a nonmonotone line\n\n\u2217Equal contributors\n\n1\n\n\fsearch to improve convergence [24]. TRIP by Kim et al. also uses a spectral step length but selects\nsearch directions using a trust-region strategy [12]. TRIP performs comparably with SpaRSA and\nthe projected Newton-type methods we describe later.\n\u221a\nA closely related family of methods is the set of optimal \ufb01rst-order methods, also called acceler-\nated \ufb01rst-order methods, which achieve \u0001-suboptimality within O(1/\n\u0001) iterations [22]. The two\nmost popular methods in this family are Auslender and Teboulle\u2019s method [1] and Fast Iterative\nShrinkage-Thresholding Algorithm (FISTA), by Beck and Teboulle [2]. These methods have been\nimplemented in the software package TFOCS and used to solve problems that commonly arise in\nstatistics, machine learning, and signal processing [3].\n\n1.2 Newton-type methods\n\nThere are three classes of methods that generalize Newton-type methods to handle nonsmooth objec-\ntive functions. The \ufb01rst are projected Newton-type methods for constrained optimization [20]. Such\nmethods cannot handle nonsmooth objective functions; they tackle problems in composite form via\nconstraints of the form h(x) \u2264 \u03c4. PQN is an implementation that uses a limited-memory quasi-\nNewton update and has both excellent empirical performance and theoretical properties [19, 18].\nThe second class of these methods by Yu et al. [25] uses a local quadratic approximation to the\nsmooth part of the form\n\nQ(x) := f (x) + sup\n\ng\u2208\u2202f (x)\n\ngT d +\n\n1\n2\n\ndT Hd,\n\nwhere \u2202f (x) denotes the subdifferential of f at x. These methods achieve state-of-the-art perfor-\nmance on many problems of relevance, such as (cid:96)1-regularized logistic regression and (cid:96)2-regularized\nsupport vector machines.\nThis paper focuses on proximal Newton-type methods that were previously studied in [16, 18] and\nare closely related to the methods of Fukushima and Mine [10] and Tseng and Yun [21]. Both use\nsearch directions \u2206x that are solutions to subproblems of the form\n\nminimize\n\nd\n\n\u2207g(x)T d +\n\n1\n2\n\ndT Hd + h(x + d),\n\nwhere H is a positive de\ufb01nite matrix that approximates the Hessian \u22072g(x). Fukushima and Mine\nchoose H to be a multiple of the identity, while Tseng and Yun set some components of the search\ndirection \u2206x to be zero to obtain a (block) coordinate descent direction. Proximal Newton-type\nmethods were \ufb01rst studied empirically by Mark Schmidt in his Ph.D. thesis [18].\nThe methods GLMNET [9] ((cid:96)1-regularized regression), LIBLINEAR [26] ((cid:96)1-regularized classi\ufb01-\ncation), QUIC and recent work by Olsen et al. [11, 15] (sparse inverse covariance estimation) are\nspecial cases of proximal Newton-type methods. These methods are considered state-of-the-art for\ntheir speci\ufb01c applications, often outperforming generic methods by orders of magnitude. QUIC and\nLIBLINEAR also achieve a quadratic rate of convergence, although these results rely crucially on\nthe structure of the (cid:96)1 norm and do not generalize to generic nonsmooth regularizers.\nThe quasi-Newton splitting method developed by Becker and Fadili is equivalent to a proximal\nquasi-Newton method with rank-one Hessian approxiamtion [4]. In this case, they can solve the\nsubproblem via the solution of a single variable root \ufb01nding problem, making their method signi\ufb01-\ncantly more ef\ufb01cient than a generic proximal Newton-type method.\nThe methods described in this paper are a special case of cost approximation (CA), a class of meth-\nods developed by Patriksson [16]. CA requires a CA function \u03d5 and selects search directions via\nsubproblems of the form\n\nminimize\n\nd\n\ng(x) + \u03d5(x + d) \u2212 \u03d5(x) + h(x + d) \u2212 \u2207g(x)T d.\n\nCost approximation attains a linear convergence rate. Our methods are equivalent to using the CA\n2 xT Hx. We refer to [16] for details about cost approximation and its convergence\nfunction \u03d5(x) := 1\nanalysis.\n\n2\n\n\f2 Proximal Newton-type methods\n\nWe seek to solve convex optimization problems in composite form:\nf (x) := g(x) + h(x).\n\n(2)\nWe assume g : Rn \u2192 R is a closed, proper convex, continuously differentiable function, and its\ngradient \u2207g is Lipschitz continuous with constant L1; i.e.\n\nminimize\n\nx\u2208Rn\n\n(cid:107)\u2207g(x) \u2212 \u2207g(y)(cid:107) \u2264 L1(cid:107)x \u2212 y(cid:107)\n\nfor all x and y in Rn. h : Rn \u2192 R is a closed and proper convex but not necessarily everywhere\ndifferentiable function whose proximal mapping can be evaluated ef\ufb01ciently. We also assume the\noptimal value, f (cid:63), is attained at some optimal solution x(cid:63), not necessarily unique.\n\n2.1 The proximal gradient method\n\nThe proximal mapping of a convex function h at x is\n\nproxh(x) = arg min\n\ny\n\nh(y) +\n\n(cid:107)y \u2212 x(cid:107)2.\n\n1\n2\n\nProximal mappings can be interpreted as generalized projections because if h is the indicator func-\ntion of a convex set, then proxh(x) is the projection of x onto the set.\nThe classic proximal gradient method for composite optimization uses proximal mappings to handle\nthe nonsmooth part of the objective function and can be interpreted as minimizing the nonsmooth\nfunction h plus a simple quadratic approximation to the smooth function g during every iteration:\n\nxk+1 = proxtkh (xk \u2212 tk\u2207g(xk))\n\n= arg min\n\ny\n\n\u2207g(xk)T (y \u2212 xk) +\n\n(cid:107)y \u2212 xk(cid:107)2 + h(y),\n\n1\n2tk\n\nwhere tk denotes the k-th step length. We can also interpret the proximal gradient step as a general-\nized gradient step\n\nGf (x) = proxh(x \u2212 \u2207g(x)) \u2212 x.\n\n(3)\nGf (x) = 0 if and only if x minimizes f so (cid:107)Gf (x)(cid:107) generalizes the smooth \ufb01rst-order measure of\noptimality (cid:107)\u2207f (x)(cid:107).\nMany state-of-the-art methods for problems in composite form, such as SpaRSA and the optimal\n\ufb01rst-order methods, are variants of this method. Our method uses a Newton-type approximation in\nlieu of the simple quadratic to achieve faster convergence.\n\n2.2 The proximal Newton iteration\n\nDe\ufb01nition 2.1 (Scaled proximal mappings). Let h be a convex function and H, a positive de\ufb01nite\nmatrix. Then the scaled proximal mapping of h at x is de\ufb01ned to be\n\nproxH\n\nh (x) := arg min\n\ny\n\nh(y) +\n\n(cid:107)y \u2212 x(cid:107)2\nH .\n\n1\n2\n\n(4)\n\nProximal Newton-type methods use the iteration\n\n(5)\n(6)\nwhere tk > 0 is the k-th step length, usually determined using a line search procedure and Hk is an\napproximation to the Hessian of g at xk. We can interpret the search direction \u2206xk as a step to the\nminimizer of the nonsmooth function h plus a local quadratic approximation to g because\n\nxk+1 = xk + tk\u2206xk,\n\n\u2206xk := proxHk\nh\n\n(cid:0)xk \u2212 H\u22121\n\nk \u2207g(xk)(cid:1) \u2212 xk,\n\n(cid:0)xk \u2212 H\u22121\n\nk \u2207g(xk)(cid:1)\n\nproxHk\nh\n\n= arg min\n\nh(y) +\n\ny\n\n1\n2\n\n(cid:107)(y \u2212 xk) + H\u22121\n\n= arg min\n\ny\n\n\u2207g(xk)T (y \u2212 xk) +\n\n1\n2\n\n3\n\nk \u2207g(xk)(cid:107)2\n(y \u2212 xk)T Hk(y \u2212 xk) + h(y).\n\nHk\n\n(7)\n\n\fHence, the search direction solves the subproblem\n\n\u2206xk = arg min\n\nd\n\n= arg min\n\nd\n\n\u2207g(xk)T d +\n\n1\n2\n\ndT Hkd + h(xk + d)\n\nQk(d) + h(xk + d).\n\nTo simplify notation, we shall drop the subscripts and say x+ = x + t\u2206x in lieu of xk+1 =\nxk + tk\u2206xk when discussing a single iteration.\nLemma 2.2 (Search direction properties). If H is a positive de\ufb01nite matrix, then the search direction\n\u2206x = arg mind Q(d) + h(x + d) satis\ufb01es:\n\nf (x+) \u2264 f (x) + t(cid:0)\u2207g(x)T \u2206x + h(x + \u2206x) \u2212 h(x)(cid:1) + O(t2),\n\n\u2207g(x)T \u2206x + h(x + \u2206x) \u2212 h(x) \u2264 \u2212\u2206xT H\u2206x.\n\n(8)\n(9)\n\nLemma 2.2 implies the search direction is a descent direction for f because we can substitute (9)\ninto (8) to obtain\n\nf (x+) \u2264 f (x) \u2212 t\u2206xT H\u2206x + O(t2).\n\nWe use a quasi-Newton approximation to the Hessian and a \ufb01rst-order method to solve the subprob-\nlem for a search direction, although the user is free to use a method of his or her choice. Empirically,\nwe \ufb01nd that inexact solutions to the subproblem yield viable descent directions.\nWe use a backtracking line search to select a step length t that satis\ufb01es a suf\ufb01cient descent condition:\n(10)\n(11)\nwhere \u03b1 \u2208 (0, 0.5). This suf\ufb01cient descent condition is motivated by our convergence analysis but\nit also seems to perform well in practice.\nLemma 2.3 (Step length conditions). Suppose H (cid:23) mI for some m > 0 and \u2207g is the Lipschitz\ncontinuous with constant L1. Then the step lengths\n2m\nL1\n\n\u2206 := \u2207g(x)T \u2206x + h(x + \u2206x) \u2212 h(x),\n\nf (x+) \u2264 f (x) + \u03b1t\u2206\n\nt \u2264 min\n\n(cid:27)\n\n(1 \u2212 \u03b1)\n\n.\n\n(cid:26)\n\n1,\n\n(12)\n\nsatis\ufb01es the suf\ufb01cient descent condition (10).\n\nAlgorithm 1 A generic proximal Newton-type method\nRequire: x0 in dom f\n1: repeat\n2:\n3:\n4:\n5:\n6:\n7: until stopping conditions are satis\ufb01ed\n\nUpdate Hk using a quasi-Newton update rule\nzk \u2190 proxHk\n\u2206xk \u2190 zk \u2212 xk\nConduct backtracking line search to select tk\nxk+1 \u2190 xk + tk\u2206xk\n\nk \u2207g(xk)(cid:1)\n\n(cid:0)xk \u2212 H\u22121\n\nh\n\n3 Convergence analysis\n\n3.1 Global convergence\nWe assume our Hessian approximations are suf\ufb01ciently positive de\ufb01nite; i.e. Hk (cid:23) mI, k =\n1, 2, . . . for some m > 0. This assumption guarantees the existence of step lengths that satisfy the\nsuf\ufb01cient decrease condition.\nLemma 3.1 (First-order optimality conditions). Suppose H is a positive de\ufb01nite matrix. Then x is\na minimizer of f if and only if the search direction is zero at x; i.e.\nQ(d) + h(x + d).\n\n0 = arg min\n\nd\n\n4\n\n\fThe global convergence of proximal Newton-type methods results from the fact that the search\ndirections are descent directions and if our Hessian approximations are suf\ufb01ciently positive de\ufb01nite,\nthen the step lengths are bounded away from zero.\nTheorem 3.2 (Global convergence). Suppose Hk (cid:23) mI, k = 1, 2, . . . for some m > 0. Then the\nsequence {xk} generated by a proximal Newton-type method converges to a minimizer of f.\n\n3.2 Convergence rate\n\nIf g is twice-continuously differentiable and we use the second order Taylor approximation as our\nlocal quadratic approximation to g, then we can prove {xk} converges Q-quadratically to the optimal\nsolution x(cid:63). We assume in a neighborhood of x(cid:63): (i) g is strongly convex with constant m; i.e.\n\n\u22072g(x) (cid:23) mI, x \u2208 N\u0001(x(cid:63))\n\nwhere N\u0001(x(cid:63)) := {x | (cid:107)x \u2212 x(cid:63)(cid:107) \u2264 \u0001}; and (ii) \u22072g is Lipschitz continuous with constant L2.\nThis convergence analysis is similar to that of Fukushima and Min\u00b4e [10] and Patriksson [16]. First,\nwe state two lemmas: (i) that says step lengths of unity satisfy the suf\ufb01cient descent condition after\nsuf\ufb01ciently many iterations and (ii) that the backward step is nonexpansive.\nLemma 3.3. Suppose (i) \u22072g (cid:23) mI and (ii) \u22072g is Lipschitz continuous with constant L2. If we let\nHk = \u22072g(xk), k = 1, 2, . . . , then the step length tk = 1 satis\ufb01es the suf\ufb01cient decrease condition\n(10) for k suf\ufb01ciently large.\n\nWe can characterize the solution of the subproblem using the \ufb01rst-order optimality conditions for\n(4). Let y denote proxH\nh\n\n(cid:0)x \u2212 H\u22121\u2207g(x)(cid:1), then\n\nH(x \u2212 H\u22121\u2207g(x) \u2212 y) \u2208 \u2202h(u).\n\n[H \u2212 \u2207g] (x) \u2208 [H + \u2202h] (y)\n\n(x) and(cid:2) 1\n\nm (H \u2212 \u2207g)(cid:3) (x) respectively, where m is\n\n\u22121 [H \u2212 \u2207g] (x) = R \u25e6 S(x).\n\nor equivalently\n\nLet R(x) and S(x) denote(cid:2) 1\n\nthe smallest eigenvalue of H. Then\n\n.\n\nLemma 3.4. Suppose R(x) =(cid:2) 1\n\nm (H + \u2202h)(cid:3)\u22121\nm H + \u2202h(cid:3)\u22121\n\ny = [H + \u2202h]\n\n(x), where H is positive de\ufb01nite. Then R is \ufb01rmly-\n\nnonexpansive; i.e. for x and y in dom f, R satis\ufb01es\n\n(R(x) \u2212 R(y))T (x \u2212 y) \u2265 (cid:107)R(x) \u2212 R(y)(cid:107)2.\n\nWe note that x(cid:63) is a \ufb01xed point of R \u25e6 S; i.e. R \u25e6 S(x(cid:63)) = x(cid:63), so we can express (cid:107)y \u2212 x(cid:63)(cid:107) as\n\n(cid:107)y \u2212 x(cid:63)(cid:107) = (cid:107)R \u25e6 S(x) \u2212 R \u25e6 S(x(cid:63))(cid:107) \u2264 (cid:107)S(x) \u2212 S(x(cid:63))(cid:107).\n\nTheorem 3.5. Suppose (i) \u22072g (cid:23) mI and (ii) \u22072g is Lipschitz continuous with constant L2. If we\nlet Hk = \u22072g(xk), k = 1, 2, . . . , then {xk} converges to x(cid:63) Q-quadratically; i.e.\n\n(cid:107)xk+1 \u2212 x(cid:63)(cid:107)\n(cid:107)xk \u2212 x(cid:63)(cid:107)2 \u2192 c.\n(cid:13)(cid:13)(cid:0)Hk \u2212 \u22072g(x(cid:63))(cid:1) (xk+1 \u2212 xk)(cid:13)(cid:13)\n\n(cid:107)xk+1 \u2212 xk(cid:107)\n\nWe can also use the fact that the proximal Newton method converges quadratically to prove a prox-\nimal quasi-Newton method converges superlinearly. We assume the quasi-Newton Hessian approx-\nimations satisfy the Dennis-Mor\u00b4e criterion [7]:\n\n\u2192 0.\n\n(13)\n\nWe \ufb01rst prove two lemmas: (i) step lengths of unity satisfy the suf\ufb01cient descent condition after\nsuf\ufb01ciently many iterations and (ii) the proximal quasi-Newton step is close to the proximal Newton\nstep.\n\n5\n\n\fthere exist M \u2265 m > 0 such that mI (cid:22) H (cid:22) M I.\n\nLemma 3.6. Suppose g is twice-continuously differentiable and the eigenvalues of Hk, k = 1, 2, . . .\nIf {Hk} satisfy the\nare bounded; i.e.\nDennis-Mor\u00b4e criterion, then the unit step length satis\ufb01es the suf\ufb01cient descent condition (10) after\nsuf\ufb01ciently many iterations.\nLemma 3.7. Suppose H and \u02c6H are positive de\ufb01nite matrices with bounded eigenvalues; i.e. mI (cid:22)\nH (cid:22) M I and \u02c6mI (cid:22) \u02c6H (cid:22) \u02c6M I. Let \u2206x and \u2206\u02c6x denote the search directions generated using H\nand \u02c6H respectively; i.e.\n\n\u2206x = proxH\nh\n\u2206\u02c6x = prox \u02c6H\nh\n\n(cid:115)\n\nx \u2212 \u02c6H\u22121\u2207g(x)\n\n(cid:0)x \u2212 H\u22121\u2207g(x)(cid:1) \u2212 x,\n(cid:17) \u2212 x.\n(cid:16)\n(cid:13)(cid:13)(cid:13)1/2 (cid:107)\u2206x(cid:107)1/2,\n(cid:13)(cid:13)(cid:13)( \u02c6H \u2212 H)\u2206x\n\nThen these two search directions satisfy\n\n(cid:107)\u2206x \u2212 \u2206\u02c6x(cid:107) \u2264\n\n1 + c(H, \u02c6H)\n\nm\n\nwhere c is a constant that depends on H and \u02c6H.\nTheorem 3.8. Suppose g is twice-continuously differentiable and the eigenvalues of Hk, k =\n1, 2, . . . are bounded. If {Hk} satisfy the Dennis-Mor\u00b4e criterion, then the sequence {xk} converges\nto x(cid:63) Q-superlinearly; i.e.\n\n(cid:107)xk+1 \u2212 x(cid:63)(cid:107)\n(cid:107)xk \u2212 x(cid:63)(cid:107) \u2192 0.\n\n4 Computational experiments\n\n4.1 PNOPT: Proximal Newton OPTimizer\n\nPNOPT1 is a MATLAB package that uses proximal Newton-type methods to minimize convex objec-\ntive functions in composite form. PNOPT can build BFGS and L-BFGS approximation to the Hes-\nsian (the user can also supply a Hessian approximation) and uses our implementation of SpaRSA or\nan optimal \ufb01rst order method to solve the subproblem for a search direction.\nPNOPT uses an early stopping condition for the subproblem solver based on two ideas: (i) the\nsubproblem should be solved to a higher accuracy if Qk is a good approximation to g and (ii) near a\nsolution, the subproblem should be solved almost exactly to achieve fast convergence.\nWe thus require that the solution to the k-th subproblem (7) y(cid:63)\nk satisfy\nk)(cid:107),\nk)(cid:107) \u2264 \u03b7k(cid:107)Gf (y(cid:63)\n(cid:27)\n\n(14)\nwhere Gf (x) denotes the generalized gradient step at x (3) and \u03b7k is a forcing term. We choose\nforcing terms based on the agreement between g and the previous quadratic approximation to g\nQk\u22121. We set \u03b71 := 0.5 and\n\n(cid:107)GQ+h(y(cid:63)\n\n(cid:26)\n\n(cid:107)\u2207g(xk) \u2212 \u2207Qk\u22121(xk)(cid:107)\n\n\u03b7k := min\n\n, k = 2, 3, . . .\n\n(cid:107)\u2207g(xk)(cid:107)\n\n(15)\nThis choice measures the agreement between \u2207g(xk) and \u2207Qk\u22121(xk) and is borrowed from a\nchoice of forcing terms for inexact Newton methods described by Eisenstat and Walker [8]. Empiri-\ncally, we \ufb01nd that this choice avoids \u201coversolving\u201d the subproblem and yields desirable convergence\nbehavior.\nWe compare the performance of PNOPT, our implementation of SpaRSA, and the TFOCS imple-\nmentations of Auslender and Teboulle\u2019s method (AT) and FISTA on (cid:96)1-regularized logistic regres-\nsion and Markov random \ufb01eld structure learning. We used the following settings:\n\n0.5,\n\n1. PNOPT: We use an L-BFGS approximation to the Hessian with L = 50 and set the suf-\n\ufb01cient decrease parameter to \u03b1 = 0.0001. To solve the subproblem, we use the TFOCS\nimplementation of FISTA.\n\n1PNOPT is available at www.stanford.edu/group/SOL/software/pnopt.html.\n\n6\n\n\f(a)\n\n(b)\n\nFigure 1: Figure 1a and 1b compare two variants of proximal Newton-type methods with SpaRSA\nand TFOCS on on the MRF structure learning problem.\n\n2. SpaRSA: We use a nonmonotone line search with a 10 iteration memory and also set the\nsuf\ufb01cient decrease parameter to \u03b1 = 0.0001. Our implementation of SpaRSA is included\nin PNOPT as the default solver for the subproblem.\n\n3. AT/FISTA: We set tfocsOpts.restart = -inf to turn on adaptive restarting and\n\nuse default values for the rest of the settings.\n\nThese experiments were conducted on a machine running the 64-bit version of Ubuntu 12.04 with\nan Intel Core i7 870 CPU and 8 GB RAM.\n\n4.2 Markov random \ufb01eld structure learning\n\nWe seek the maximum likelihood estimates of the parameters of a Markov random \ufb01eld (MRF)\nsubject to a group elastic-net penalty on the estimates. The objective function is given by\n\u03bb1 (cid:107)\u03b8rj(cid:107)2 + \u03bb2 (cid:107)\u03b8rj(cid:107)2\n\n\u2212 (cid:88)\n\n\u03b8rj(xr, xj) + log Z(\u03b8) +\n\n(cid:88)\n\n(cid:16)\n\n.\n\nF\n\n(16)\n\n(cid:17)\n\nminimize\n\n\u03b8\n\n(r,j)\u2208E\n\n(r,j)\u2208E\n\nxr is a k state variable; xj is a l state variable, and each parameter block \u03b8rj is a k \u00d7 l matrix that\nis associated with an edge in the MRF. We randomly generate a graphical model with |V | = 12 and\nn = 300. The edges are sample uniformly with p = 0.3. The parameters of the non-zero edges are\nsampled from a N (0, 1) distribution.\nThe group elastic-net penalty regularizes the solution and promotes solutions with a few non-zero\ngroups \u03b8rj corresponding to edges of the graphical model [27]. The regularization parameters were\n\nset to \u03bb1 =(cid:112)n log |V | and \u03bb2 = .1\u03bb1. These parameter settings are shown to be model selection\n\nconsistent under certain irrepresentable conditions [17].\nThe algorithms for solving (16) require evaluating the value and gradient of the smooth part. For a\ndiscrete graphical model without special structure, the smooth part requires O(k|V |) operations to\nevaluate, where k is the number of states per variable. Thus even for our small example, where k = 3\nand |V | = 12, function and gradient evaluations dominate the computational expense required to\nsolve (16).\nWe see that for maximum likelihood learning in graphical models, it is important to minimize the\nnumber of function evaluations. Proximal Newton-type methods are well-suited to solve such prob-\nlems because the main computational expense is shifted to solving the subproblems that do not\nrequire function evaluations.\n\n7\n\n010020030010\u22125100Iterationlog(f\u2212f*) FistaATPN100PN15SpaRSA02040608010\u22125100Time (sec)log(f\u2212f*) FistaATPN100PN15SpaRSA\f(a)\n\n(b)\n\nFigure 2: Figure 2 compares proximal Newton-type methods with SpaRSA and TFOCS on (cid:96)1-\nregularized logistic regression.\n\n4.3\n\n(cid:96)1-regularized logistic regression\n\nGiven training data (xi, yi), i = 1, 2, . . . , n, (cid:96)1-regularized logistic regression trains a classi\ufb01er via\nthe solution of the convex optimization problem\n\nn(cid:88)\n\ni=1\n\nminimize\n\nw\u2208Rp\n\n1\nn\n\nlog(1 + exp(\u2212yiwT xi)) + \u03bb(cid:107)w(cid:107)1.\n\n(17)\n\nfor a set of parameters w in Rp. The regularization term (cid:107)w(cid:107)1 avoids over\ufb01tting the training data\nand promotes sparse solutions. \u03bb is trades-off between goodness-of-\ufb01t and model complexity.\nWe use the dataset gisette, a handwritten digits dataset from the NIPS 2003 feature se-\nlection challenge. The dataset is available at http://www.csie.ntu.edu.tw/\u223ccjlin/\nlibsvmtools/datasets. We train our classi\ufb01er using the original training set consisting of\n6000 examples starting at w = 0. \u03bb was chosen to match the value reported in [26], where it was\nchosen by \ufb01ve-fold cross validation on the training set.\nThe gisette dataset is quite dense (3 million nonzeros in the 6000 \u00d7 5000 design matrix) and the\nevaluation of the log-likelihood requires many expensive exp/log operations. We see in \ufb01gure 2 that\nPNOPT outperforms the other methods because the computational expense is shifted to solving the\nsubproblems, whose objective functions are cheap to evaluate.\n\n5 Conclusion\n\nProximal Newton-type methods are natural generalizations of \ufb01rst-order methods that account for\ncurvature of the objective function. They share many of the desirable characteristics of traditional\n\ufb01rst-order methods for convex optimization problems in composite form and achieve superlinear\nrates of convergence subject to standard assumptions. These methods are especially suited to prob-\nlems with expensive function evaluations because the main computational expense is shifted to solv-\ning subproblems that do not require function evaluations.\n\n6 Acknowledgements\n\nWe wish to thank Trevor Hastie, Nick Henderson, Ernest Ryu, Ed Schmerling, Carlos Sing-Long,\nand Walter Murray for their insightful comments.\n\n8\n\n01000200030004000500010\u2212610\u2212410\u22122100Function evaluationsRelative suboptimality ATFISTASpaRSAPN010020030040050010\u2212610\u2212410\u22122100Time (sec)Relative suboptimality ATFISTASpaRSAPN\fReferences\n[1] A. Auslender and M. Teboulle, Interior gradient and proximal methods for convex and conic optimization,\n\nSIAM J. Optim., 16 (2006), pp. 697\u2013725.\n\n[2] A. Beck and M. Teboulle , A fast iterative shrinkage-thresholding algorithm for linear inverse problems,\n\nSIAM J. Imaging Sci., 2 (2009), pp. 183\u2013202.\n\n[3] S. R. Becker, M. J. Cand`es, and M. C. Grant, Templates for convex cone problems with applications to\n\nsparse signal recovery, Math. Program. Comput., 3 (2011), pp. 1\u201354.\n\n[4] S. Becker and J. Fadili, A quasi-Newton proximal splitting method, NIPS, Lake Tahoe, California, 2012.\n[5] S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, Cambridge, 2004.\n[6] E. J. Cand`es and B. Recht, Exact matrix completion via convex optimization, Found. Comput. Math, 9\n\n(2009), pp. 717\u2013772.\n\n[7] J. E. Dennis, Jr. and J. J. Mor\u00b4e, A characterization of superlinear convergence and its application to\n\nquasi-Newton methods, Math. Comp., 28, (1974), pp. 549\u2013560.\n\n[8] S. C. Eisenstat and H. F. Walker, Choosing the forcing terms in an inexact Newton method, SIAM J. Sci.\n\nComput., 17 (1996), pp. 16\u201332.\n\n[9] J. Friedman, T. Hastie, H. Hol\ufb01ng, and R. Tibshirani, Pathwise coordinate optimization, Ann. Appl. Stat.\n\n(2007), pp. 302\u2013332\n\n[10] M. Fukushima and H. Mine, A generalized proximal point algorithm for certain non-convex minimization\n\nproblems, Internat. J. Systems Sci., 12 (1981), pp. 989\u20131000.\n\n[11] C. J. Hsieh, M. A. Sustik, P. Ravikumar, and I. S. Dhillon, Sparse inverse covariance matrix estimation\n\nusing quadratic approximation, NIPS, Granada, Spain, 2011.\n\n[12] D. Kim, S. Sra, and I. S. Dhillon, A scalable trust-region algorithm with applications to mixed-norm\n\nregression, ICML, Haifa, Israel, 2010.\n\n[13] Y. Nesterov, Gradient methods for minimizing composite objective function, CORE discussion paper,\n\n2007.\n\n[14] G. Obozinski, B. Taskar, and M. I. Jordan, Joint covariate selection and joint subspace selection for\n\nmultiple classi\ufb01cation problems, Stat. Comput. (2010), pp. 231\u2013252\n\n[15] P. Olsen, F. Oztoprak, J. Nocedal, S. Rennie, Newton-like methods for sparse inverse covariance estima-\n\ntion, NIPS, Lake Tahoe, California, 2012.\n\n[16] M. Patriksson, Nonlinear Programming and Variational Inequality Problems, Kluwer Academic Publish-\n\ners, The Netherlands, 1999.\n\n[17] P. Ravikumar, M. J. Wainwright and J. D. Lafferty, High-dimensional Ising model selection using (cid:96)1-\n\nregularized logistic regression, Ann. Statist. (2010), pp. 1287-1319.\n\n[18] M. Schmidt, Graphical Model Structure Learning with l1-Regularization, Ph.D. Thesis (2010), Univer-\n\nsity of British Columbia\n\n[19] M. Schmidt, E. van den Berg, M. P. Friedlander, and K. Murphy, Optimizing costly functions with simple\nconstraints: a limited-memory projected quasi-Newton algorithm, AISTATS, Clearwater Beach, Florida,\n2009.\n\n[20] M. Schmidt, D. Kim, and S. Sra, Projected Newton-type methods in machine learning, in S. Sra, S.\n\nNowozin, and S. Wright, editors, Optimization for Machine Learning, MIT Press (2011).\n\n[21] P. Tseng and S. Yun, A coordinate gradient descent method for nonsmooth separable minimization, Math.\n\nProg. Ser. B, 117 (2009), pp. 387\u2013423.\n\n[22] P. Tseng, On accelerated proximal gradient methods for convex-concave optimization, submitted to\n\nSIAM J. Optim. (2008).\n\n[23] R. Tibshirani, Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B Stat. Methodol., 58\n\n(1996), pp. 267\u2013288.\n\n[24] S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, Sparse reconstruction by separable approximation,\n\nIEEE Trans. Signal Process., 57 (2009), pp. 2479\u20132493.\n\n[25] J. Yu, S. V. N. Vishwanathan, S. G\u00a8unter, and N. N. Schraudolph, A Quasi-Newton Approach to Nonsmooth\n\nConvex Optimization, ICML, Helsinki, Finland, 2008.\n\n[26] G. X. Yuan, C. H. Ho and C. J. Lin, An improved GLMNET for (cid:96)1-regularized logistic regression and\n\nsupport vector machines, National Taiwan University, Tech. Report 2011.\n\n[27] R. H. Zou and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B\n\nStat. Methodol., 67 (2005), pp. 301\u2013320.\n\n9\n\n\f", "award": [], "sourceid": 4740, "authors": [{"given_name": "Jason", "family_name": "Lee", "institution": null}, {"given_name": "Yuekai", "family_name": "Sun", "institution": null}, {"given_name": "Michael", "family_name": "Saunders", "institution": null}]}