{"title": "Fast and Provable ADMM for Learning with Generative Priors", "book": "Advances in Neural Information Processing Systems", "page_first": 12027, "page_last": 12039, "abstract": "In this work, we propose a (linearized) Alternating Direction\nMethod-of-Multipliers (ADMM) algorithm for minimizing a convex function\nsubject to a nonconvex constraint. We focus on the special case where such\nconstraint arises from the specification that a variable should lie in the\nrange of a neural network. This is motivated by recent successful\napplications of Generative Adversarial Networks (GANs) in tasks like\ncompressive sensing, denoising and robustness against adversarial examples.\nThe derived rates for our algorithm are characterized in terms of\ncertain geometric properties of the generator network, which we show hold for feedforward architectures, under mild assumptions. Unlike gradient\ndescent (GD), it can efficiently handle non-smooth objectives as well as\nexploit efficient partial minimization procedures, thus being faster in\nmany practical scenarios.", "full_text": "Fast and Provable ADMM for\nLearning with Generative Priors\n\nFabian Latorre, Armin Eftekhari and Volkan Cevher\nLaboratory for information and inference systems (LIONS)\n\nEPFL, Lausanne, Switzerland\n\n{firstname.lastname}@epfl.ch\n\nAbstract\n\nIn this work, we propose a (linearized) Alternating Direction Method-of-Multipliers\n(ADMM) algorithm for minimizing a convex function subject to a nonconvex\nconstraint. We focus on the special case where such constraint arises from the\nspeci\ufb01cation that a variable should lie in the range of a neural network. This is\nmotivated by recent successful applications of Generative Adversarial Networks\n(GANs) in tasks like compressive sensing, denoising and robustness against ad-\nversarial examples. The derived rates for our algorithm are characterized in terms\nof certain geometric properties of the generator network, which we show hold for\nfeedforward architectures, under mild assumptions. Unlike gradient descent (GD),\nit can ef\ufb01ciently handle non-smooth objectives as well as exploit ef\ufb01cient partial\nminimization procedures, thus being faster in many practical scenarios.\n\n1\n\nIntroduction\n\nGenerative Adversarial Networks (GANs) [Goodfellow et al., 2014] show great promise for faithfully\nmodeling complex data distributions, such as natural images [Radford et al., 2015, Brock et al.,\n2019] or audio signals [Engel et al., 2019, Donahue et al., 2019]. Understanding and improving the\ntheoretical and practical aspects of their training has thus attracted signi\ufb01cant interest [Lucic et al.,\n2018, Mescheder et al., 2018, Daskalakis et al., 2018, Hsieh et al., 2018, Gidel et al., 2019].\nResearchers have also begun to leverage the modeling power of GANs and other generative models\nlike Variational Auto-encoders [Kingma and Welling, 2013] in applications ranging from compressive\nsensing [Bora et al., 2017], to image denoising [Lipton and Tripathi, 2017, Tripathi et al., 2018], to\nrobustness against adversarial examples [Ilyas et al., 2017, Samangouei et al., 2018].\nThese and other [Dhar et al., 2018, Ulyanov et al., 2018] applications model high-dimensional data\nas the output of the generator network associated with a generative model, and often lead to a highly\nnon-convex optimization problem of the form minz f (G(z)), where the the generator G is nonlinear\nand f is convex. We then \ufb01nd the optimal latent vector z, as illustrated in Section 5 with several\nexamples.\nThis optimization problem involving a generative model poses various dif\ufb01culties for existing \ufb01rst-\norder algorithms. Indeed, to our knowledge, the only existing provable algorithm for solving (1)\nrelies on the existence of a projection oracle, and is limited to the special case of compressive sensing\nwith a generative prior [Shah and Hegde, 2018, Hegde, 2018], see Section 4 for the details. The\nmain computational bottleneck is of course the non-convex projection step, for which no convergence\nanalysis in terms of the geometry of the underlying generator G currently exists.\nOn the other hand, Gradient Descent (GD) and its adaptive variants [Kingma and Ba, 2014] cannot\nef\ufb01ciently handle non-smooth objective functions, as they are entirely oblivious to the composite\nstructure of the problem [Nesterov, 2013b]. A simple example is denoising with the (cid:96)\u221e-norm,\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\ffor which subgradient descent (as the standard non-smooth alternative to GD) fails in practice, as\nobserved in Section 5.\nWith the explosion of generative models in popularity, there is consequently a pressing need for\nprovable and \ufb02exible optimization algorithms to solve the resulting non-convex and (possibly) non-\nsmooth problems. The present work addresses this need by focusing on the general optimization\ntemplate\n\nminimize\nsubject to\n\nw,z\n\nF (w, z) := L(w) + R(w) + H(z)\n\nw = G(z),\n\n(1)\nwhere L : Rd \u2192 R is convex and smooth, R : Rd \u2192 R and H : Rs \u2192 R are convex but not\nnecessarily smooth, and G : Rs \u2192 Rd is differentiable but often non-linear, corresponding to the\ngenerator network associated with a generative model. Even though R and H might not be smooth,\nwe assume throughout that their proximal mappings can be ef\ufb01ciently computed [Parikh et al., 2014].\nFor brevity, we refer to (1) as optimization with a generative prior whenever G is given by the\ngenerator network associated with a generative model [Kingma and Welling, 2013, Goodfellow et al.,\n2014]. In this context, we make three key contributions, summarized below:\n1. Algorithm: We propose an ef\ufb01cient and scalable (linearized) Alternating Direction Method-of-\nMultipliers (ADMM) framework to solve (1), see Algorithm 1. To our knowledge, this is the \ufb01rst\nnon-convex and linearized ADMM algorithm for nonlinear constraints with provable fast rates to\nsolve problem (1), see Section 4 for a detailed literature review.\nWe evaluate this algorithm numerically in the context of denoising with GANs in the presence of\nadversarial or stochastic noise, as well as compressive sensing [Bora et al., 2017]. In particular,\nAlgorithm 1 allows for ef\ufb01cient denoising with the (cid:96)\u221e- and (cid:96)1-norms, with applications in defenses\nagainst adversarial examples [Szegedy et al., 2013] and signal processing, respectively.\n2. Optimization guarantees: We prove fast approximate convergence for Algorithm 1 under the\nassumptions of smoothness and near-isometry of G, as well as strong convexity of L. That is,\nwe distill the key geometric attributes of the generative network G responsible for the success of\nAlgorithm 1. We then show how some common neural network architectures satisfy these geometric\nassumptions.\nWe also establish a close relation between a variant of Algorithm 1 and the gradient descent in\n[Bora et al., 2017] and, in this sense, provide the \ufb01rst rates for it, albeit in a limit case detailed in\nSection 3. Indeed, one key advantage of the primal-dual formulation studied in this paper is exactly\nthis versatility, as well as the ef\ufb01cient handling of non-smooth objectives.\nLastly, we later relax the assumptions on L to restricted strong convexity/smoothness, thus extending\nour results to the broader context of statistical learning with generative priors, which includes\ncompressive sensing [Bora et al., 2017] as a special case.\n3. Statistical guarantees: In the context of statistical learning with generative priors, where L in (1)\nis replaced with an empirical risk, we provide the generalization error associated with Algorithm 1.\nThat is, we use the standard notion of Rademacher complexity [Mohri et al., 2018] to quantify the\nnumber of training data points required for Algorithm 1 to learn the true underlying parameter w(cid:92).\n\n2 Algorithm\n\nIn this section, we adapt the powerful Alternating Descent Method of Multipliers (ADMM) [Glowin-\nski and Marroco, 1975, Gabay and Mercier, 1976, Boyd et al., 2011] to solve the non-convex\nproblem (1). We de\ufb01ne the corresponding augmented Lagrangian with the dual variable \u03bb \u2208 Rp as\n(2)\n\n(cid:107)w \u2212 G(z)(cid:107)2\n2,\nfor a penalty weight \u03c1 > 0. By a standard duality argument, (1) is equivalent to\n\nL\u03c1(w, z, \u03bb) := L(w) + (cid:104)w \u2212 G(z), \u03bb(cid:105) +\n\n\u03c1\n2\n\nmin\nw,z\n\nmax\n\n\u03bb\n\nL\u03c1(w, z, \u03bb) + R(w) + H(z).\n\n(3)\n\nApplied to (3), every iteration of ADMM would minimize the augmented Lagrangian with respect\nto z, then with respect to w, and then update the dual variable \u03bb. Note that L\u03c1(w, z, \u03bb) is often\n\n2\n\n\fnon-convex with respect to z due to the nonlinearity of the generator G : Rs \u2192 Rd and, consequently,\nthe minimization step with respect to z in ADMM is often intractable.\nTo overcome this limitation, we next linearize ADMM. In the following, we let PR and PH denote\nthe proximal maps of R and H, respectively [Parikh et al., 2014].\nThe equivalence of problems (1) and (3) motivates us to consider the following algorithm for the\npenalty weight \u03c1 > 0, the primal step sizes \u03b1, \u03b2 > 0, and the positive dual step sizes {\u03c3t}t\u22650:\n\nzt+1 = P\u03b2H (zt \u2212 \u03b2\u2207zL\u03c1(wt, zt, \u03bbt)) ,\nwt+1 = P\u03b1R (wt \u2212 \u03b1\u2207wL\u03c1(wt, zt+1, \u03bbt)) ,\n\u03bbt+1 = \u03bbt + \u03c3t+1(wt+1 \u2212 G(zt+1)).\n\n(4)\n\nAs opposed to ADMM, to solve (1), the linearized ADMM in (4) takes only one descent step in\nboth z and w, see Algorithm 1 for the summary. The particular choice of the dual step sizes {\u03c3t}t\nin Algorithm 1 ensures that the dual variables {\u03bbt}t remain bounded, see [Bertsekas, 1976] for a\nprecedent in the convex literature.\nAlgorithm 2. Let us introduce an important variant of Algorithm 1. In our setting, L\u03c1(w, z, \u03bb) is\nin fact convex with respect to w and therefore Algorithm 2 replaces the \ufb01rst step in (4) with exact\nminimization over w. This exact minimization step can be executed with an off-the-shelf convex\nsolver, or might sometimes have a closed-form solution. Moreover, Algorithm 2 gradually increases\nthe penalty weight to emulate a multi-scale structure. More speci\ufb01cally, for an integer K, consider\nthe sequences of penalty weights and primal step sizes {\u03c1k, \u03b1k, \u03b2k}K\n\nk=1, speci\ufb01ed as\n\n\u03c1k = 2k\u03c1,\n\n\u03b1k = 2\u2212k\u03b1,\n\n\u03b2k = 2\u2212k\u03b2,\n\nk \u2264 K.\n\nConsider also a sequence of integers {nk}K\n\nk=1, where\n\nnk = 2kn,\n\nk \u2264 K,\n\n(5)\n\n(6)\n\nfor an integer n. At (outer) iteration k, Algorithm 2 executes nk iterations of Algorithm 1 with\nexact minimization over w. Then it passes the current iterates of w, z, and dual step size to the next\n(outer) iteration. Loosely speaking, Algorithm 2 has a multi-scale structure, allowing it to take larger\nsteps initially and then slowing down as it approaches the solution. As discussed in Section 3, the\ntheoretical guarantees for Algorithm 1 also apply to Algorithm 2. The pseudocode for Algorithm 2 is\ngiven in Supplementary I.\nAs the closing remark, akin to the convex case [He et al., 2000, Xu et al., 2017], it is also possible to\ndevise a variant of Algorithm 1 with adaptive primal step sizes, which we leave for a future work.\n\nAlgorithm 1 Linearized ADMM for solving problem (1)\nInput: Differentiable L, proximal-friendly convex regularizers R and H, differentiable prior G,\npenalty weight \u03c1 > 0, primal step sizes \u03b1, \u03b2 > 0, initial dual step size \u03c30 > 0, primal initialization\nw0 and z0, dual initialization \u03bb0, stopping threshold \u03c4c > 0.\n1 for t = 0, 1, . . . , T \u2212 1 do\n2\n3\n\n(primal updates)\n\n(cid:19)\n\n(dual step size)\n\n(dual update)\n\n(stopping criterion)\n\n(cid:107)wt+1 \u2212 G(zt+1)(cid:107)2t log2(t + 1)\n\n\u03c30,\n\n(cid:18)\n\nzt+1 \u2190 P\u03b2H (zt \u2212 \u03b2\u2207zL\u03c1(wt, zt, \u03bbt))\nwt+1 \u2190 P\u03b1R (wt \u2212 \u03b1\u2207wL\u03c1(wt, zt+1, \u03bbt))\n\u03c3t+1 \u2190 min\n\u03bbt+1 \u2190 \u03bbt + \u03c3t+1(wt+1 \u2212 G(zt+1))\ns \u2190 (cid:107)zt+1 \u2212 zt(cid:107)2\n(cid:107)wt+1 \u2212 wt(cid:107)2\n\u03b1\nif s \u2264 \u03c4c then\n\n\u03c30\n\n2\n\n+\n\nreturn (wt+1, zt+1)\n\n2\n\n+ \u03c3t(cid:107)wt \u2212 G(zt)(cid:107)2\n\n2 \u2264 \u03c4c\n\n\u03b2\n\n4\n\n5\n\n6\n\n7\n8\n\n9 return (wT , zT )\n\n3\n\n\f3 Optimization Guarantees\n\nLet us study the theoretical guarantees of Algorithm 1 for solving program (1), whose constraints are\nnonlinear and non-convex (since G is speci\ufb01ed by a neural network). The main contribution of this\nsection is Theorem 1, which is inherently an optimization result stating that Algorithm 1 succeeds\nunder certain assumptions on (1).\nFrom an optimization perspective, to our knowledge, Theorem 1 is the \ufb01rst to provide (fast) rates for\nnon-convex and linearized ADMM, see Section 4 for a detailed literature review. The assumptions\nimposed below on L and the generator G ensure the success of Algorithm 1 and are shortly justi\ufb01ed\nfor our setup, where G is a generator network.\nAssumption 1. strong convexity / smoothness of L: We assume that L in (1) is both strongly\nconvex and smooth, namely, there exist 0 < \u00b5L \u2264 \u03bdL such that\n(cid:107)w \u2212 w(cid:48)(cid:107)2 \u2264 L(w(cid:48)) \u2212 L(w) \u2212 (cid:104)w(cid:48) \u2212 w,\u2207L(w)(cid:105) \u2264 \u03bdL\n2\n\n\u2200w, w(cid:48) \u2208 Rd.\n\n(cid:107)w \u2212 w(cid:48)(cid:107)2,\n\n\u00b5L\n2\n\nAssumption 1 is necessary to establish fast rates for Algorithm 1, and is readily met for L(w) =\n2 with \u00b5L = \u03bdL = 1, which renders Algorithm 1 applicable to (cid:96)2-denoising with generative\n\n(cid:107)w \u2212 (cid:98)w(cid:107)2\nprior in [Tripathi et al., 2018, Samangouei et al., 2018, Ilyas et al., 2017]. Here, (cid:98)w is the noisy image.\n\nIn Supplementary A, we also relax the strong convexity/smoothness in Assumption 1 to restricted\nstrong convexity/smoothness, which enables us to apply Theorem 1 in the context of statistical\nlearning with a generative prior, for example in compressive sensing [Bora et al., 2017].\nUnder Assumption 1, even though L and consequently the objective function of (1) are strongly\nconvex, problem (1) might not have a unique solution, which is in stark contrast with convex\noptimization. Indeed, a simple example is minimizing x2 + y2 with the constraint x2 + y2 = 1. We\nnext state our assumptions on the generator G.\nAssumption 2. Strong smoothness of G: Let DG be the Jacobian of G. We assume that G : Rs \u2192\nRd is strongly smooth, namely, there exists \u03bdG \u2265 0 such that\n\n(cid:107)G(z(cid:48)) \u2212 G(z) \u2212 DG(z) \u00b7 (z(cid:48) \u2212 z)(cid:107)2 \u2264 \u03bdG\n2\n\n(cid:107)z(cid:48) \u2212 z(cid:107)2\n2,\n\n\u2200z, z(cid:48) \u2208 Rs,\n\n(8)\n\nAssumption 3. Near-isometry of G: We assume that the generative prior G is a near-isometric\nmap, namely, there exist 0 < \u03b9G \u2264 \u03baG such that\n\n(7)\n\n\u03b9G(cid:107)z(cid:48) \u2212 z(cid:107)2 \u2264 (cid:107)G(z(cid:48)) \u2212 G(z)(cid:107)2 \u2264 \u03baG(cid:107)z(cid:48) \u2212 z(cid:107)2,\n\n\u2200z, z(cid:48) \u2208 Rs.\n\n(9)\n\nThe invertibility of certain network architectures have been established before in [Ma et al., 2018,\nHand and Voroninski, 2017]. More concretely, Assumptions 2 and 3 hold for a broad class of\ngenerators, as summarized in Proposition 1 and proved in Supplementary B.\nProposition 1. Let G\u039e : D \u2282 Rd \u2192 Rs be a feedforward neural network with weights \u039e \u2208 Rh, k\nlayers, non-decreasing layer sizes s \u2264 s1 \u2264 . . . sk \u2264 d, with \u03c9i as activation function in the i-th\nlayer, and compact domain D. For every layer i, suppose that the activation \u03c9i : R \u2192 R is of class\nC 1 (continuously-differentiable) and strictly increasing. Then, after an arbitrarily small perturbation\nto the weights \u039e, Assumptions 2 and 3 hold almost surely with respect to the Lebesgue measure.\n\nA few comments about the preceding result are in order.\nChoice of the activation function: Strictly-increasing C 1 activation functions in Proposition 1,\nsuch as the Exponential Linear Unit (ELU) [Clevert et al., 2015] or softplus [Dugas et al., 2001],\nachieve similar or better performance compared to the commonly-used (but non-smooth) Recti\ufb01ed\nLinear Activation Unit (ReLU) [Xu et al., 2015, Clevert et al., 2015, Gulrajani et al., 2017, Kumar\net al., 2017, Kim et al., 2018].\nIn our experiments in Section 5, we found that using ELU activations for the generator G does not\nadversely affect the representation power of the trained generator. Lastly, the activation function for\nthe \ufb01nal layer of the generator is typically chosen as the sigmoid or tanh [Radford et al., 2015], for\nwhich the conditions in Proposition 1 are also met.\nCompact domain: The compactness requirement in Proposition 1 is mild. Indeed, even though the\nGaussian distribution is the default choice as the input for the generator in GANs, training has also\n\n4\n\n\fbeen successful using compactly-supported distributions, such as the uniform distribution [Lipton\nand Tripathi, 2017].\nInterestingly, even after training with Gaussian noise, limiting the resulting generator to a truncated\nGaussian distribution can in fact boost the performance of GANs [Brock et al., 2019], as mea-\nsured with common metrics like the Inception Score [Salimans et al., 2016] or Frechet Inception\nDistance [Heusel et al., 2017]. This evidence suggests that obtaining a good generator G with\ncompact domain is straightforward. In the experiments of Section 5, we use truncated Gaussian on an\nEuclidean ball centered at the origin.\nNon-decreasing layer sizes: This is a standard feature of popular generator architectures such as the\nDCGAN [Radford et al., 2015] or infoGAN [Chen et al., 2016]. This property is also exploited in the\nanalysis of the optimization landscape of problem (1) by Hand and Voroninski [2017], Heckel et al.\n[2019] and for showing invertiblity of (de)convolutional generators [Ma et al., 2018].\n\nNecessity of assumptions on G: Assumptions 2 and 3 on the generator G are necessary for the\nprovable success of Algorithm 1. Loosely speaking, Assumption 2 controls the curvature of the\ngenerative prior, without which the dual iterations can oscillate without improving the objective.\nOn the other hand, the lower bound in (9) means that the generative prior G must be stably injective:\nFaraway latent parameters should be mapped to faraway outputs under G. As a pathological example,\nconsider the parametrization of a circle as {(sin z, cos z) : z \u2208 [0, 2\u03c0)}.\nThis stable injectivity property in (9) is necessary for the success of Algorithm 1 and is not an artifact\nof our proof techniques. Indeed, without this condition, the z updates in Algorithm 1 might not reduce\nthe feasibility gap (cid:107)w \u2212 G(z)(cid:107)2. Geometric assumptions on nonlinear constraints have precedent in\nthe optimization literature [Birgin et al., 2016, Flores-Baz\u00e1n et al., 2012, Cartis et al., 2018] and to a\nlesser extent in the literature of neural networks too [Hand and Voroninski, 2017, Ma et al., 2018],\nwhich we further discuss in Section 4.\nHaving stated and justi\ufb01ed our assumptions on L and the generator G in (1), we are now prepared to\npresent the main technical result of this section. Theorem 1 states that Algorithm 1 converges linearly\nto a small neighborhood of a solution, see Supplementary C for the proof.\nTheorem 1. (guarantees for Algorithm 1) Suppose that Assumptions 1-3 hold. Let (w\u2217, z\u2217) be a\nsolution of program (1) and let \u03bb\u2217 be a corresponding optimal dual variable. Let also {wt, zt, \u03bbt}t\u22650\ndenote the output sequence of Algorithm 1. Suppose that the primal step sizes \u03b1, \u03b2 satisfy\n\n\u03b1 \u2264 1\n\u03bd\u03c1\n\n,\n\n\u03b2 \u2264\n\n1\n\n\u03be\u03c1 + 2\u03b1\u03c4 2\n\u03c1\n\n.\n\n\u03c30 \u2264 \u03c30,\u03c1.\n\nThen it holds that\n\n(cid:107)wt \u2212 w\u2217(cid:107)2\n\n2\n\n\u03b1\n\n(cid:107)zt \u2212 z\u2217(cid:107)2\n\n2\n\n\u03b2\n\n+\n\n\u2264 2(1 \u2212 \u03b7\u03c1)t\u22060 +\n\n\u03b7\u03c1\n\u03c1\n\n,\n\n(cid:107)wt \u2212 G(zt)(cid:107)2\n\n2 \u2264 4(1 \u2212 \u03b7\u03c1)t\u22060\n\nThe convergence rate 1 \u2212 \u03b7\u03c1 \u2208 (0, 1) and the quantities \u03bd\u03c1, \u03be\u03c1, \u03c4\u03c1, \u03c30,\u03c1, \u03b7\u03c1,(cid:101)\u03b7\u03c1 above depend on the\n\n(12)\nfor every iteration t. Above, \u22060 = L\u03c1(w0, z0, \u03bb0)\u2212L\u03c1(w\u2217, z\u2217, \u03bb\u2217) is the initialization error, see (2).\nparameters in Assumptions 1-3 and on \u03bb\u2217, as speci\ufb01ed in the proof. As an example, in the regime\nwhere \u00b5L (cid:29) \u03c1 and \u03b92\n\n\u03c12 ,\n\n\u03c1\n\n+ (cid:101)\u03b7\u03c1\n\n(10)\n\n(11)\n\n\u03b1 \u2248 1\n\u03bdL\n\nG (cid:29) \u03bdG, we can take\n\u03b2 \u2248 1\n(cid:19)\n\u03c1\u03ba2\nG\n\n(cid:18) \u00b5L\n\n,\n\n,\n\n\u03b7\u03c1 \u2248 min\n\n\u03b92\nG\n\u03ba2\nG\n\n(cid:19)\n\n,\n\n(cid:18) \u00b52\n(cid:18) \u03bdL\n\nL\n\u03bd2\nL\n\n,\n\n\u03b94\nG\n\u03ba4\nG\n\n(cid:19)\n\n\u03ba2\nG\n\u03b92\nG\n\n(cid:46) \u03c30 (cid:46) \u03c1 min\n\n\u03c1\u03bdG\n\u03ba2\nG\n\n\u03b7\u03c1 \u2248(cid:101)\u03b7\u03c1 \u2248 max\n\n(13)\nAbove, for the sake of clarity, \u2248 and (cid:46) suppress the universal constants, dependence on the initial\ndual \u03bb0 and the corresponding step size \u03c30.\n\n\u00b5L\n\n\u03bdL\n\n,\n\n,\n\n,\n\n.\n\n5\n\n\fA few clarifying comments about Theorem 1 are in order.\nError: According to Theorem 1, if the primal and dual step sizes are suf\ufb01ciently small and Assump-\ntions 1-3 are met, Algorithm 1 converges linearly to a neighborhood of a solution (w\u2217, z\u2217). The\nsize of this neighborhood depends on the penalty weight \u03c1 in (2). For instance, in the example in\nTheorem 1, it is easy to verify that this neighborhood has a radius of O(1/\u03c1), which can be made\nsmaller by increasing \u03c1.\nTheorem 1 is however silent about the behavior of Algorithm 1 within this neighborhood. This is to\nbe expected. Indeed, even in the simpler convex case, where G in program (1) would have been an\naf\ufb01ne map, provably no \ufb01rst-order algorithm could converge linearly to the solution [Ouyang and Xu,\n2018, Agarwal et al., 2010].\nInvestigating the behavior of Algorithm 1 within this neighborhood, while interesting, arguably has\nlittle practical value. For example, in the convex case, ADMM would converge slowly (sublinearly)\nin this neighborhood, which does not appeal to the practitioners.\nAs another example, when Algorithm 1 is applied in the context of statistical learning, there is no\nbene\ufb01t in solving (1) beyond the statistical accuracy of the problem at hand [Agarwal et al., 2010], see\nthe discussion in Supplementary A.1. As such, we defer the study of the local behavior of Algorithm\n1 to a future work.\nFeasibility gap: Likewise, according to (24) in Theorem 1, the feasibility gap of Algorithm 1 rapidly\nreaches a plateau. In the example in Theorem 1, the feasibility gap rapidly reaches O(1/\u03c1), where \u03c1\nis the penalty weight in (2). As before, even in the convex case, no \ufb01rst-order algorithm could achieve\nexact feasibility at linear rate [Ouyang and Xu, 2018, Agarwal et al., 2010].\nIntution: While the exact expressions for the quantities in Theorem 1 are given in Supplementary C,\nthe example provided in Theorem 1 highlights the simple but instructive regime where \u00b5L (cid:29) \u03c1 and\nG (cid:29) \u03bdG, see Assumptions 1-3. Intuitively, \u00b5L (cid:29) \u03c1 means that minimizing the objective of (1)\n\u03b92\nG (cid:29) \u03bdG suggests that the\nis prioritized over reducing the feasibility gap, see (2). In addition, \u03b92\ngenerative prior G is very smooth.\nIn this regime, the primal step size \u03b1 for w updates is determined by how smooth L is, and the primal\nstep size \u03b2 in the latent variable z is determined by how smooth G is, see (13). Similar restrictions\nare standard in \ufb01rst-order algorithms to avoid oscillations [Nesterov, 2013a].\nAs discussed earlier, the algorithm rapidly reaches a neighborhood of size O(1/\u03c1) of a solution and\nthe feasibility gap plateaus at O(1/\u03c1). Note the trade-off here for the choice of \u03c1: the larger the\npenalty weight \u03c1 is, the more accurate Algorithm 1 would be and yet increasing \u03c1 is restricted by the\nassumption \u03c1 (cid:28) \u00b5L. Moreover, in this example, the rate 1 \u2212 \u03b7\u03c1 of Algorithm 1 depends only on the\nregularity of L and G in program (1), see (13). Indeed, the more well-conditioned L is and the more\nnear-isometric G is, the larger \u03b7\u03c1 and the faster the convergence would be.\nGenerally speaking, increasing the penalty weight \u03c1 reduces the bias of Algorithm 1 at the cost of a\nslower rate. Beyond our work, such dependence on the geometry of the constraints has precedent in\nthe literature of optimization [Birgin et al., 2016, Flores-Baz\u00e1n et al., 2012, Cartis et al., 2018] and\nmanifold embedding theory [Eftekhari and Wakin, 2015, 2017].\n\nRelation to simple gradient descent: Consider a variant of Algorithm 1 that replaces the linearized\nupdate for w in (4) with exact minimization with respect to w, which can be achieved with an off-the-\nshelf convex solver or might have a closed-form solution in some cases. The exact minimization over\nw and Lemma 7 together guarantee that Theorem 1 also applies to this variant of Algorithm 1.\nMoreover, as a special case of (1) where R \u2261 0 and H \u2261 0, this variant is closely related to GD [Bora\net al., 2017], presented there without any rates. In Appendix F, we establish that the updates of both\nalgorithms match as the feasibility gap vanishes.\nIn this sense, Theorem 1 provides the \ufb01rst rates for GD, albeit in the limit case of vanishing feasibility\ngap. Indeed, one key advantage of the primal-dual formulation studied in this paper is exactly this\nversatility in providing a family of algorithms, such as Algorithms 1 and 2, that can be tuned for\nvarious scenarios and can also ef\ufb01ciently handle the non-smooth case where R or H are nonzero\nin (1).\n\n6\n\n\f4 Related Work\n\nBora et al. [2017] empirically tune gradient descent for compressive sensing with a generative prior\n\n(cid:107)A \u00b7 G(z) \u2212 b(cid:107)2\n2,\n\nmin\n\nz\n\n(14)\n\nwhich is a particular case of template (1) (without splitting). They also provide a statistical general-\nization error dependent on a certain set restricted isometry property on the matrix A. More generally,\nTheorem 4 in Supplementary A provides statistical guarantees for Algorithm 1 using the standard\nnotion of empirical Rademacher complexity [Mohri et al., 2018].\nHand and Voroninski [2017] analyze the optimization landscape of (14) under the assumption that\nG (i) is composed of linear layers and ReLU activation functions, (ii) is suf\ufb01ciently expansive at\neach layer and (iii) the network\u2019s weights have a Gaussian distribution or an equivalent deterministic\nweight distribution condition. Under such conditions, they show global existence of descent directions\noutside small neighborhoods around two points, but do not provide algorithmic convergence rates.\nTheir analysis requires ReLU activation in all layers of the generator G, including the last one, which\nis often not met in practice.\nOn the other hand, our framework is not restricted to a particular network architecture and instead\nisolates the necessary assumptions on the network G for the success of Algorithm 1. In doing so,\nwe effectively decouple the learning task from the network structure G and study them separately in\nTheorem 1 and Proposition 1, respectively. In particular, our theory in Section 3 (Supplementary A)\napplies broadly to any nonlinear map G that meets Assumptions 1-3 (Assumptions 2-5), respectively.\nIn turn, Proposition 1 establishes that the standard feed forward network with common differentiable\nactivation functions almost surely meets these assumptions. In this sense, let us also point to the\nwork of Oymak et al. [2018], which is limited to linear regression with a nonlinear constraint, with\nits convex analogue studied in [Agarwal et al., 2010, Giryes et al., 2016].\nHeckel et al. [2019] provides a convergence proof for a modi\ufb01ed version of gradient descent, limited\nto (14) and without specifying a rate. We provide the convergence rate for a broad range of learning\nproblems, and study the statistical generalization. Hand et al. [2018] studied the phase retrieval\nproblem, with a non-convex objective function that is not directly covered by (1).\nFor the problem (14), Shah and Hegde [2018], Hegde [2018] proposed to use Projected Gradient\nDescent (PGD) after splitting in a manner similar to our template (1). If the projection (onto the range\nof the prior G) is successful, and under certain additional conditions, the authors establish linear\nconvergence of PGD to a minimizer of (14). However, the projection onto the nonlinear range of G is\nitself a dif\ufb01cult non-convex program without any theoretical guarantees. In contrast, we can solve the\nsame problem without any projections while still providing a convergence rate.\nFrom an optimization perspective, there are no fast rates for linearized ADMM with nonlinear\nconstraints to our knowledge, but convergence to a \ufb01rst-order stationary point and special cases in\na few different settings have been studied [Liu et al., 2017, Shen et al., 2016, Chen and Gu, 2014,\nQiao et al., 2016]. Let us again emphasize that Assumptions 2 and 3 extract the key attributes of G\nnecessary for the success of Algorithm 1, which is therefore not limited to a generator network. It is\nalso worth noting another line of work that applies tools from statistical physics to inference with\ndeep neural networks, see [Manoel et al., 2017, Rezende et al., 2014] and the references therein.\n\n5 Experiments\n\nIn this section we evaluate our algorithms for image recovery tasks with a generative prior. The\ndatasets we consider are the CelebA dataset of face images [Liu et al., 2015] and the MNIST dataset\nof handwritten digits [LeCun and Cortes, 2010]. We train a generator G with ELU activation\nfunctions Clevert et al. [2015], in order to satisfy Assumption 2. The generators are trained using\nthe Wasserstein GAN framework [Arjovsky et al., 2017]. For the CelebA dataset we downsample\nthe images to 64 \u00d7 64 pixels as in Gulrajani et al. [2017] and we use the same residual architecture\n[He et al., 2015] for the generator with four residual blocks followed by a convolutional layer. For\nMNIST, we use the same architecture as one in Gulrajani et al. [2017], which contains one fully\nconnected layer followed by three deconvolutional layers.\n\n7\n\n\fFigure 1: Reconstruction error and measurement error vs time (one tick equals the time of one GD\niteration). MNIST (left) and CelebA (right).\n\nWe recover images on the range of the generator G, by choosing z(cid:63) \u2208 Rs and setting w(cid:63) := G(z(cid:63))\nas the true image to be recovered. This sets the global minimum of our objective functions at zero,\nand allows us to illustrate and compare the convergence rates of various algorithms.\nOur Algorithm 1 mantains iterates {wt, zt}t where wt might not be feasible, namely, wt might not\nbe in the range of G. As the goal in the following tasks is to recover an element in the range of G\n(feasible points of (1)), we plot the objective value at the point G(zt).\nBaseline. We compare to the most widely-used algorithm in the current literature, the gradient\ndescent algorithm (GD) as used in [Bora et al., 2017], where a \ufb01xed number of iterations with constant\nstep size are performed for the function L(G(z)). We tune its learning rate to be as large as possible\nwithout overshooting. (See Supplementary H for details on the hyperparameter tuning).\nOur goal is to illustrate our theoretical results and highlight scenarios where Algorithm 1 can have\nbetter performance than GD in optimization problems with a generative prior. Hence, we do not\ncompare with sparsity-prior based algorithms, such as LASSO [Tibshirani, 1996], or argue about\nGAN vs. sparsity priors as in Bora et al. [2017].\nOur algorithms. We will use (i) (linearized) ADMM (Algorithm 1), and (ii) ADMM with exact\nminimization (Algorithm 2 a.k.a. EADMM), described in Section 2. For both ADMM and EADMM,\nwe choose a starting iterate (random z0 and w0 = G(z0)) and initial dual variable \u03bb0 = 0 (for GD we\nchoose the same z0 as initial iterate). We carefully track the objective function value vs. computation\ntime for a fair comparison.\nCompressive sensing The exact minimization step of EADMM involves the solution of a system\nof linear equations in each iteration. Performing Singular Value Decomposition (SVD) once on the\nmeasurement matrix A, and storing its components in memory, allows us to solve such linear systems\nwith a very low per-iteration complexity (see Supplementary H.3). We plot the objective function\nvalue as well as the reconstruction error with 50% relative measurements in Figure 1(average over 20\nimages (MNIST) and 10 images (CelebA)).\nAdversarial Denoising with (cid:96)\u221e-norm Projection onto the range of a deep-net prior has been\nconsidered by Samangouei et al. [2018], Ilyas et al. [2017] as a defense mechanism against adversarial\nexamples [Szegedy et al., 2013]. In their settings, samples are denoised with a generative prior,\nbefore being fed to a classi\ufb01er. Even though the adversarial noise introduced is typically bounded in\n(cid:96)\u221e-norm, the projection is done in (cid:96)2-norm. Such projection corresponds to F (w, z) = (cid:107)w \u2212 w(cid:92)(cid:107)2\nin (1).\nWe instead propose to project using the (cid:96)\u221e-norm that bounds the adversarial perturbation. To this\n2 +(cid:107)w \u2212 w(cid:92)(cid:107)\u221e in the template (1), for some small value of \u03b3. The\nend we let F (w, z) = \u03b3(cid:107)w \u2212 w(cid:92)(cid:107)2\nproximal of the (cid:96)\u221e norm is ef\ufb01ciently computable [Duchi et al., 2008], allowing us to split F (w, z)\nin its components L(w) = \u03b3(cid:107)w \u2212 w(cid:92)(cid:107)2\n2 and R(w) = (cid:107)w \u2212 w(cid:92)(cid:107)\u221e (Note that the small \u03b3 ensures that\nAssumption 1 holds)\nWe compare the ADAM optimizer [Kingma and Ba, 2014], GD and ADMM (450 iterations and\nfor GD and ADAM, and 300 iterations for EADMM). We use ADAM to solve the (cid:96)2 projection,\nwhile ADMM solves the (cid:96)\u221e projection. We evaluate on a test set of 2000 adversarial examples from\nthe MNIST dataset, obtained with the Projected Gradient Algorithm of Madry et al. [2018] with\n30 iterations, stepsize 0.01 and attack size 0.2. For the classi\ufb01er, we use a standard convolutional\nnetwork trained on clean MNIST samples. We also test ADAM, GD (3000 iterations) and EADMM\n(2000 iterations) on the (cid:96)\u221e denoising task.\n\n8\n\n101102103104time (cpu)101102reconst. errorgdeadmm100101102103104time (cpu)101102103measurement errorgdeadmm\fFigure 2: Test error on denoised adversarial ex-\namples vs computation time (average cpu time(s)\nover the sample).\n\nFigure 3: (cid:96)\u221e reconstruction error per iteration for\nADAM, GD, and EADMM.\n\nThe test error as a function of computation time is in Figure 2. We observe that the (cid:96)\u221e denoising\nperforms better when faced with (cid:96)\u221e bounded attacks, in the sense that it achieves a lower error with\nless computation time. In Figure 3, we plot the (cid:96)\u221e reconstruction error achieved by ADAM, GD and\nEADMM, averaged over 7 images. GD was unable to decrease the initial error, while ADAM takes a\nconsiderable number of iterations to do so. In contrast, our ADMM already achieves the \ufb01nal error of\nADAM within its \ufb01rst 100 iterations.\n\n6 Conclusions and Future Work\n\nIn this work, we have proposed a \ufb02exible linearized ADMM algorithm for the minimization of a\nconvex function subject to a nonlinear constraint given by a neural network. Under mild assumptions\nwe demonstrate a fast convergence rate to a neighborhood of a solution of its Lagrangian formulation\n(3) (Theorem 1). Empirical evaluation shows how it can handle non-smooth terms more ef\ufb01ciently\nwhen compared to gradient descent and its variants.\nSome avenues of research are left open which could yield faster variants of our proposed approach.\nFirst, ADMM-type algorithms admit acceleration and restart schemes with faster convergence rates\nin the convex case [Goldstein et al., 2014] but their adaptation to the nonlinear constraint given\nby a neural network is non-trivial. Secondly, adaptivity in the choice of penalty parameter \u03c1 can\npotentially improve the performance of the method and reduce the need for tuning [He et al., 2000].\nFinally, the denoising with (cid:96)\u221e-norm shows promise as a defense against adversarial examples, and\nits performance on higher dimensional datasets is worth investigating.\n\nAcknowledgements\n\nThis project has received funding from the European Research Council (ERC) under the European\nUnion\u2019s Horizon 2020 research and innovation programme (grant agreement 725594 - time-data), the\nDepartment of the Navy - Of\ufb01ce of Naval Research (ONR) under a grant number N62909-17-1-2111,\nand from the Swiss National Science Foundation (SNSF) under grant number 200021_178865. FL is\nsupported through a PhD fellowship of the Swiss Data Science Center, a joint venture between EPFL\nand ETH Zurich. VC acknowledges the 2019 Google Faculty Research Award.\n\nReferences\nAlekh Agarwal, Sahand Negahban, and Martin J Wainwright. Fast global convergence rates of gradient methods\nfor high-dimensional statistical recovery. In Advances in Neural Information Processing Systems, pages\n37\u201345, 2010.\n\nMartin Arjovsky, Soumith Chintala, and L\u00e9on Bottou. Wasserstein generative adversarial networks. In Doina\nPrecup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning,\nvolume 70 of Proceedings of Machine Learning Research, pages 214\u2013223, International Convention Centre,\nSydney, Australia, 06\u201311 Aug 2017. PMLR. URL http://proceedings.mlr.press/v70/arjovsky17a.\nhtml.\n\nDimitri P Bertsekas. On penalty and multiplier methods for constrained minimization. SIAM Journal on Control\n\nand Optimization, 14(2):216\u2013235, 1976.\n\n9\n\n050100150200250300time020406080100error(%)\u20182adam\u2018\u221eadmmbase100101102103iteration (t)1003\u00d710-14\u00d710-16\u00d710-1\u2018\u221e measurement errorgdadmmadam\fErnesto G Birgin, JL Gardenghi, Jos\u00e9 Mario Mart\u00ednez, SA Santos, and Ph L Toint. Evaluation complexity for\nnonlinear constrained optimization using unscaled kkt conditions and high-order models. SIAM Journal on\nOptimization, 26(2):951\u2013967, 2016.\n\nAshish Bora, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. Compressed Sensing using Generative Models.\narXiv:1703.03208 [cs, math, stat], March 2017. URL http://arxiv.org/abs/1703.03208. arXiv:\n1703.03208.\n\nStephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and\nstatistical learning via the alternating direction method of multipliers. Foundations and Trends R(cid:13) in Machine\nlearning, 3(1):1\u2013122, 2011.\n\nAndrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high \ufb01delity natural image\nsynthesis. In International Conference on Learning Representations, 2019. URL https://openreview.\nnet/forum?id=B1xsqj09Fm.\n\nCoralia Cartis, Nicholas IM Gould, and Ph L Toint. Optimality of orders one to three and beyond: characterization\n\nand evaluation complexity in constrained nonconvex optimization. Journal of Complexity, 2018.\n\nLaming Chen and Yuantao Gu. The convergence guarantees of a non-convex approach for sparse recovery. IEEE\n\nTransactions on Signal Processing, 62(15):3754\u20133767, 2014.\n\nXi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable\nrepresentation learning by information maximizing generative adversarial nets. In D. D. Lee, M. Sugiyama,\nU. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29,\npages 2172\u20132180. Curran Associates, Inc., 2016.\n\nDjork-Arn\u00e9 Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by\n\nexponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.\n\nConstantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with optimism.\nIn International Conference on Learning Representations, 2018. URL https://openreview.net/forum?\nid=SJJySbbAZ.\n\nManik Dhar, Aditya Grover, and Stefano Ermon. Modeling sparse deviations for compressed sensing using\nIn Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International\ngenerative models.\nConference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1214\u2013\n1223, Stockholmsm\u00e4ssan, Stockholm Sweden, 10\u201315 Jul 2018. PMLR. URL http://proceedings.mlr.\npress/v80/dhar18a.html.\n\nChris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Proceedings of the 7th\n\nInternational Conference on Learning Representations, 2019.\n\nJohn Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Ef\ufb01cient projections onto the l1-\nball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine\nLearning, ICML \u201908, pages 272\u2013279, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4. doi:\n10.1145/1390156.1390191. URL http://doi.acm.org/10.1145/1390156.1390191.\n\nCharles Dugas, Yoshua Bengio, Fran\u00e7ois B\u00e9lisle, Claude Nadeau,\n\nand Ren\u00e9 Garcia.\n\ncorporating second-order\nT. G. Dietterich,\ntems\npages\n1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.\npdf.\n\nIn-\nIn T. K. Leen,\nin Neural\nInformation Processing Sys-\nURL http://papers.nips.cc/paper/\n\nfunctional knowledge for better option pricing.\n\n13,\n\nand V. Tresp,\n472\u2013478. MIT Press,\n\neditors, Advances\n\n2001.\n\nArmin Eftekhari and Michael B Wakin. New analysis of manifold embeddings and signal recovery from\n\ncompressive measurements. Applied and Computational Harmonic Analysis, 39(1):67\u2013109, 2015.\n\nArmin Eftekhari and Michael B Wakin. What happens to a manifold under a bi-lipschitz map? Discrete &\n\nComputational Geometry, 57(3):641\u2013673, 2017.\n\nJesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts.\nGANSynth: Adversarial neural audio synthesis. In International Conference on Learning Representations,\n2019. URL https://openreview.net/forum?id=H1xQVn09FX.\n\nFabi\u00e1n Flores-Baz\u00e1n, Fernando Flores-Baz\u00e1n, and Cristi\u00e1n Vera. A complete characterization of strong duality\nin nonconvex optimization with a single constraint. Journal of Global Optimization, 53(2):185\u2013201, 2012.\n\n10\n\n\fDaniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems\nvia \ufb01nite element approximation. Computers & Mathematics with Applications, 2:17\u201340, 12 1976. doi:\n10.1016/0898-1221(76)90003-1.\n\nD. J. H. Garling. A Course in Mathematical Analysis, volume 1. Cambridge University Press, 2014. doi:\n\n10.1017/CBO9781139424516.\n\nGauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, R\u00e9mi Le Priol, Gabriel Huang, Simon Lacoste-\nJulien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In Kamalika Chaudhuri\nand Masashi Sugiyama, editors, Proceedings of Machine Learning Research, volume 89 of Proceedings of\nMachine Learning Research, pages 1802\u20131811. PMLR, 16\u201318 Apr 2019. URL http://proceedings.mlr.\npress/v89/gidel19a.html.\n\nRaja Giryes, Yonina C Eldar, Alex M Bronstein, and Guillermo Sapiro. Tradeoffs between convergence speed\n\nand reconstruction accuracy in inverse problems. arXiv preprint arXiv:1605.09232, 2016.\n\nR. Glowinski and A. Marroco. Sur l\u2019approximation, par \u00e9l\u00e9ments \ufb01nis d\u2019ordre un, et la r\u00e9solution, par\np\u00e9nalisation-dualit\u00e9 d\u2019une classe de probl\u00e8mes de dirichlet non lin\u00e9aires. ESAIM: Mathematical Modelling\nand Numerical Analysis - Mod\u00e9lisation Math\u00e9matique et Analyse Num\u00e9rique, 9(R2):41\u201376, 1975. URL\nhttp://www.numdam.org/item/M2AN_1975__9_2_41_0.\n\nTom Goldstein, Brendan O\u2019Donoghue, Simon Setzer, and Richard Baraniuk. Fast alternating direction optimiza-\n\ntion methods. SIAM Journal on Imaging Sciences, 7(3):1588\u20131623, 2014.\n\nI. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.\nGenerative Adversarial Networks. ArXiv e-prints, June 2014. URL https://arxiv.org/abs/1406.2661.\n\nIshaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training\n\nof wasserstein gans. In Advances in Neural Information Processing Systems, pages 5767\u20135777, 2017.\n\nPaul Hand and Vladislav Voroninski. Global guarantees for enforcing deep generative priors by empirical risk.\n\narXiv preprint arXiv:1705.07576, 2017.\n\nPaul Hand, Oscar Leong, and Vlad Voroninski. Phase retrieval under a generative prior. In S. Bengio, H. Wallach,\nH. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information\nProcessing Systems 31, pages 9154\u20139164. Curran Associates, Inc., 2018. URL http://papers.nips.cc/\npaper/8127-phase-retrieval-under-a-generative-prior.pdf.\n\nBS He, Hai Yang, and SL Wang. Alternating direction method with self-adaptive penalty parameters for\nmonotone variational inequalities. Journal of Optimization Theory and applications, 106(2):337\u2013356, 2000.\n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition.\n\narXiv e-prints, art. arXiv:1512.03385, December 2015.\n\nReinhard Heckel, Wen Huang, Paul Hand, and Vladislav Voroninski. Deep denoising: Rate-optimal recovery of\n\nstructured signals with a deep prior, 2019. URL https://openreview.net/forum?id=SklcFsAcKX.\n\nC. Hegde. Algorithmic Aspects of Inverse Problems Using Generative Models. ArXiv e-prints, October 2018.\n\nMartin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by\na two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio,\nH. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing\nSystems 30, pages 6626\u20136637. Curran Associates, Inc., 2017.\n\nYa-Ping Hsieh, Chen Liu, and Volkan Cevher. Finding Mixed Nash Equilibria of Generative Adversarial\n\nNetworks. arXiv e-prints, art. arXiv:1811.02002, Oct 2018.\n\nAndrew Ilyas, Ajil Jalal, Eirini Asteri, Constantinos Daskalakis, and Alexandros G. Dimakis. The Robust\nManifold Defense: Adversarial Training using Generative Models. arXiv e-prints, art. arXiv:1712.09196,\nDecember 2017.\n\nYoungjin Kim, Minjung Kim, and Gunhee Kim. Memorization precedes generation: Learning unsupervised\nGANs with memory networks. In International Conference on Learning Representations, 2018. URL\nhttps://openreview.net/forum?id=rkO3uTkAZ.\n\nDiederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art.\n\narXiv:1412.6980, December 2014.\n\n11\n\n\fDiederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.6114,\n\nDecember 2013.\n\nAbhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Semi-supervised learning with gans: Manifold invariance\nwith improved inference. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan,\nand R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5534\u20135544. Curran\nAssociates, Inc., 2017.\n\nYann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann.lecun.com/\n\nexdb/mnist/.\n\nZachary C. Lipton and Subarna Tripathi. Precise Recovery of Latent Vectors from Generative Adversarial\nNetworks. arXiv:1702.04782 [cs, stat], February 2017. URL http://arxiv.org/abs/1702.04782. arXiv:\n1702.04782.\n\nQinghua Liu, Xinyue Shen, and Yuantao Gu. Linearized admm for non-convex non-smooth optimization with\n\nconvergence analysis. arXiv preprint arXiv:1705.02502, 2017.\n\nZiwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings\n\nof International Conference on Computer Vision (ICCV), 2015.\n\nMario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet.\n\nAre gans\ncreated equal?\nIn S. Bengio, H. Wallach, H. Larochelle, K. Grau-\nman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Sys-\ntems 31, pages 700\u2013709. Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/\n7350-are-gans-created-equal-a-large-scale-study.pdf.\n\na large-scale study.\n\nFangchang Ma, Ulas Ayaz, and Sertac Karaman. Invertibility of convolutional generative networks from partial\n\nmeasurements. In Advances in Neural Information Processing Systems, pages 9651\u20139660, 2018.\n\nAleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep\nlearning models resistant to adversarial attacks. In International Conference on Learning Representations,\n2018. URL https://openreview.net/forum?id=rJzIBfZAb.\n\nAndre Manoel, Florent Krzakala, Marc M\u00e9zard, and Lenka Zdeborov\u00e1. Multi-layer generalized linear estimation.\n\nIn 2017 IEEE International Symposium on Information Theory (ISIT), pages 2098\u20132102. IEEE, 2017.\n\nLars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually\nconverge? In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference\non Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3481\u20133490, Stock-\nholmsm\u00e4ssan, Stockholm Sweden, 10\u201315 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/\nmescheder18a.html.\n\nM. Mohri, A. Rostamizadeh, A. Talwalkar, and F. Bach. Foundations of Machine Learning. MIT Press, 2018.\n\nISBN 9780262039406. URL https://books.google.ch/books?id=V2B9DwAAQBAJ.\n\nSahand N Negahban, Pradeep Ravikumar, Martin J Wainwright, Bin Yu, et al. A uni\ufb01ed framework for high-\ndimensional analysis of m-estimators with decomposable regularizers. Statistical Science, 27(4):538\u2013557,\n2012.\n\nY. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization. Springer\n\nUS, 2013a. ISBN 9781441988539. URL https://books.google.ch/books?id=2-ElBQAAQBAJ.\n\nYu Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming, 140(1):\n\n125\u2013161, 2013b.\n\nYuyuan Ouyang and Yangyang Xu. Lower complexity bounds of \ufb01rst-order methods for convex-concave bilinear\n\nsaddle-point problems. arXiv preprint arXiv:1808.02901, 2018.\n\nSamet Oymak, Benjamin Recht, and Mahdi Soltanolkotabi. Sharp time\u2013data tradeoffs for linear inverse problems.\n\nIEEE Transactions on Information Theory, 64(6):4129\u20134158, 2018.\n\nNeal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and Trends R(cid:13) in Optimization, 1(3):\n\n127\u2013239, 2014.\n\nLinbo Qiao, Bofeng Zhang, Jinshu Su, and Xicheng Lu. Linearized alternating direction method of multipliers\nfor constrained nonconvex regularized optimization. In Asian Conference on Machine Learning, pages\n97\u2013109, 2016.\n\n12\n\n\fA. Radford, L. Metz, and S. Chintala. Unsupervised Representation Learning with Deep Convolutional\n\nGenerative Adversarial Networks. ArXiv e-prints, November 2015.\n\nDanilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate\n\ninference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.\n\nTim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Im-\nproved techniques for training gans. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, edi-\ntors, Advances in Neural Information Processing Systems 29, pages 2234\u20132242. Curran Associates, Inc., 2016.\nURL http://papers.nips.cc/paper/6125-improved-techniques-for-training-gans.pdf.\n\nPouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-GAN: Protecting classi\ufb01ers against adversarial\nattacks using generative models. In International Conference on Learning Representations, 2018. URL\nhttps://openreview.net/forum?id=BkJ3ibb0-.\n\nViraj Shah and Chinmay Hegde. Solving Linear Inverse Problems Using GAN Priors: An Algorithm with\nProvable Guarantees. arXiv:1802.08406 [cs, stat], February 2018. URL http://arxiv.org/abs/1802.\n08406. arXiv: 1802.08406.\n\nXinyue Shen, Laming Chen, Yuantao Gu, and Hing-Cheung So. Square-root lasso with nonconvex regularization:\n\nAn admm approach. IEEE Signal Processing Letters, 23(7):934\u2013938, 2016.\n\nChristian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob\n\nFergus. Intriguing properties of neural networks. arXiv e-prints, art. arXiv:1312.6199, December 2013.\n\nRobert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society.\nSeries B (Methodological), 58(1):267\u2013288, 1996. ISSN 00359246. URL http://www.jstor.org/stable/\n2346178.\n\nSubarna Tripathi, Zachary C. Lipton, and Truong Q. Nguyen. Correction by Projection: Denoising Images with\n\nGenerative Adversarial Networks. arXiv e-prints, art. arXiv:1803.04477, March 2018.\n\nDmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. 2018 IEEE/CVF Conference on\n\nComputer Vision and Pattern Recognition, pages 9446\u20139454, 2018.\n\nBing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical Evaluation of Recti\ufb01ed Activations in Convolutional\n\nNetwork. arXiv e-prints, art. arXiv:1505.00853, May 2015.\n\nYi Xu, Mingrui Liu, Qihang Lin, and Tianbao Yang. Admm without a \ufb01xed penalty parameter: Faster\nconvergence with new adaptive penalization. In Advances in Neural Information Processing Systems, pages\n1267\u20131277, 2017.\n\n13\n\n\f", "award": [], "sourceid": 6483, "authors": [{"given_name": "Fabian", "family_name": "Latorre", "institution": "EPFL"}, {"given_name": "Armin", "family_name": "eftekhari", "institution": "EPFL"}, {"given_name": "Volkan", "family_name": "Cevher", "institution": "EPFL"}]}