{"title": "A Linearly Convergent Proximal Gradient Algorithm for Decentralized  Optimization", "book": "Advances in Neural Information Processing Systems", "page_first": 2848, "page_last": 2858, "abstract": "Decentralized optimization is a powerful  paradigm that finds applications in engineering and learning design. \tThis work studies decentralized composite optimization problems with  non-smooth regularization terms. Most existing gradient-based proximal decentralized methods are known to converge to the optimal solution with sublinear rates, and it remains unclear whether this family of methods can achieve global linear convergence. To tackle this problem, this work assumes the non-smooth regularization term is common across all networked agents, which is the case for many machine learning problems. Under this condition, we design a proximal gradient decentralized algorithm whose fixed point coincides with the desired minimizer. We then provide a concise proof that establishes its linear convergence. In the absence of the non-smooth term, our analysis technique covers the well known   EXTRA algorithm and provides useful bounds on the convergence rate and step-size.", "full_text": "A Linearly Convergent Proximal Gradient Algorithm\n\nfor Decentralized Optimization\n\nSulaiman A. Alghunaim, Kun Yuan\n\nAli H. Sayed\n\nElectrical and Computer Engineering Department\n\nEcole Polytechnique F\u00e9d\u00e9rale de Lausanne\n\nUniversity of California Los Angeles\n\nLos Angeles, CA, 90095\n\n{salghunaim,kunyuan}@ucla.edu\n\nCH-1015 Lausanne, Switzerland\n\nali.sayed@epfl.ch\n\nAbstract\n\nDecentralized optimization is a powerful paradigm that \ufb01nds applications in engi-\nneering and learning design. This work studies decentralized composite optimiza-\ntion problems with non-smooth regularization terms. Most existing gradient-based\nproximal decentralized methods are known to converge to the optimal solution\nwith sublinear rates, and it remains unclear whether this family of methods can\nachieve global linear convergence. To tackle this problem, this work assumes the\nnon-smooth regularization term is common across all networked agents, which is\nthe case for many machine learning problems. Under this condition, we design a\nproximal gradient decentralized algorithm whose \ufb01xed point coincides with the\ndesired minimizer. We then provide a concise proof that establishes its linear\nconvergence. In the absence of the non-smooth term, our analysis technique covers\nthe well known EXTRA algorithm and provides useful bounds on the convergence\nrate and step-size.\n\n1\n\nIntroduction\n\nMany machine learning problems can be cast as composite optimization problems of the form\n\nN(cid:88)\n\nn=1\n\nmin\nw\u2208RM\n\nJ(w) + R(w), where J(w) =\n\n1\nN\n\nQ(w; xn)\n\n(1)\n\nwhere w is the optimization variable, xn is the n-th data, and N is the size of the dataset. The loss\nfunction Q(w; xn) is assumed to be smooth, and R(w) is a regularization term possibly non-smooth.\nTypical examples of R(w) can be the (cid:96)1-norm, the elastic-net norm, and indicator functions of convex\nsets (e.g., non-negative orthants). Problems of the form (1) arise in different settings including, among\nothers, in model \ufb01tting [1] and economic dispatch problems in power systems [2].\nWhen the data size N is very large, it becomes intractable or inef\ufb01cient to solve problem (1) with\na single machine. To relieve this dif\ufb01culty, one solution is to divide the N data samples across\nmultiple machines and solve problem (1) in a cooperative manner. Many useful distributed algorithms\nexist that solve problem (1) across multiple computing agents such as the distributed alternating\ndirection method of multipliers (ADMM) [1, 3], parallel SGD methods [4, 5], distributed second-order\nmethods [6\u20138], and parallel dual coordinate methods [9, 10]. All these methods are designed for a\ncentralized network topology, e.g., parameter servers [11], where there is a central node connected to\nall computing agents that is responsible for aggregating local variables and updating model parameters.\nThe potential bottleneck of the centralized network is the communication traf\ufb01c jam on the central\nnode [12\u201314]. The performance of these non-decentralized methods can be signi\ufb01cantly degraded\nwhen the bandwidth around the central node is low.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fIn contrast, decentralized optimization methods are designed for any connected network topology\nsuch as line, ring, grid, and random graphs. There exists no central node for this family of methods,\nand each computing agent will exchange information with their immediate neighbors rather than\na remote central server. Decentralized methods to solve problem (1) have been widely studied for\nsome time in the signal processing, control, and optimization communities [14\u201328]. More recently,\nthere have been works in the machine learning community with interest in these problems due to their\nadvantages over centralized methods [12, 13, 29, 30]. Speci\ufb01cally, since the communication can be\nevenly distributed across each link between nodes, the decentralized algorithms converge faster than\ncentralized ones when the network has limited bandwidth or high latency [12, 13].\nFor the smooth case, the convergence rates of decentralized methods are comparable to centralized\nmethods. For example, the decentralized methods in [15, 27, 31] are shown to converge at the\nsublinear rate O(1/i) (where i is the iteration index) for smooth and convex objective functions,\nand at the linear rate O(\u03b3i) (where \u03b3 \u2208 (0, 1)) for smooth and strongly-convex objective functions.\nThese convergence rates match the convergence rates of centralized gradient descent. However,\nsome gap between decentralized and centralized proximal gradient methods continues to exist in the\npresence of a composite non-smooth term. While centralized proximal gradient methods have been\nshown to converge linearly when the objective is strongly convex [32], it remains an open question to\nestablish the linear convergence of decentralized proximal gradient methods. This work closes this\ngap by proposing a proximal gradient decentralized algorithm that is shown to converge linearly to\nthe desired solution. Next we explain the problem set-up and comment on existing related works.\n\n1.1 Problem Set-up\n\nConsider a network of K agents (e.g., machines, processors) connected over some graph. Through\nonly local interactions (i.e., agents only communicate with their immediate neighbors), each node is\ninterested in \ufb01nding a consensual vector, denoted by w(cid:63), that minimizes the following aggregate cost:\n\nJk(w) + R(w)\n\n(2)\n\n(cid:80)L\nn=1 Q(w; xk,n) where {xk,n}L\n\nThe cost function Jk(w) : RM \u2192 R is privately known by agent k and R(w) : RM \u2192 R \u222a {\u221e}\nis a proper1 and lower-semicontinuous convex function (not necessarily differentiable). When\nn=1 is the local data assigned or collected by agent k,\nJk(w) = 1\nL\nand L is the size of the local data, it is easy to see that problem (2) is equivalent to its centralized\ncounterpart (1) for N = KL. We adopt the following assumption throughout this work.\nAssumption 1. (Cost function): There exists a solution w(cid:63) to problem (2). Moreover, each cost\nfunction Jk(w) is convex and \ufb01rst-order differentiable with \u03b4-Lipschitz continuous gradients:\n\nw(cid:63) = arg min\nw\u2208RM\n\n1\nK\n\nK(cid:88)\n\nk=1\n\n(cid:107)\u2207Jk(wo) \u2212 \u2207Jk(w\u2022)(cid:107) \u2264 \u03b4(cid:107)wo \u2212 w\u2022(cid:107),\n\nfor any wo and w\u2022\n\nand the aggregate cost function \u00afJ(w) = 1\nK\n\n(wo \u2212 w\u2022)T(cid:0)\u2207 \u00afJ(wo) \u2212 \u2207 \u00afJ(w\u2022)(cid:1) \u2265 \u03bd(cid:107)wo \u2212 w\u2022(cid:107)2\n\nk=1 Jk(w) is \u03bd-strongly-convex:\n\n(cid:80)K\n\n(3)\n\n(4)\n\nfor any wo and w\u2022. The constants \u03bd and \u03b4 satisfy 0 < \u03bd \u2264 \u03b4.\nNote that from the strong-convexity condition (4), we know the objective function in (2) is also\nstrongly convex and, thus, the global solution w(cid:63) is unique.\n\n2\n\n1.2 Related Works and Contribution\n\nResearch on decentralized/distributed optimization and computation dates back several decades (see,\ne.g., [33\u201336] and the references therein). In recent years, various centralized optimization methods\nsuch as (sub-)gradient descent, proximal gradient descent, (quasi-)Newton method, dual averaging,\nalternating direction method of multipliers (ADMM), and other primal-dual methods have been\nextended to the decentralized setting. The core problem in decentralized optimization is to design\nmethods with convergence rates that are comparable to their centralized counterparts. For the smooth\n\n1The function f (.) is proper if \u2212\u221e < f (x) for all x in its domain and f (x) < \u221e for at least one x.\n\n2\n\n\fcase (R(w) = 0), the decentralized primal methods from [16, 37\u201339] converge linearly to a biased\nsolution and not the exact solution. For convergence to the exact solution, these primal methods\nrequire employing a decaying step-size that slows down the convergence rate making it sublinear at\nO(1/i) in general. The works [18\u201322] established linear convergence to the exact solution albeit for\ndecentralized primal-dual methods based on ADMM or inexact augmented Lagrangian techniques.\nOther works established linear convergence for simpler implementations including EXTRA [15],\ngradient tracking methods [26, 27], exact diffusion [28], NIDS [40], and others. The work [30] study\nthe problem from the dual domain and propose accelerated dual gradient descent to reach an optimal\nlinear convergence rate for smooth strongly-convex problems.\nThere also exist many works on decentralized composite optimization problems with non-smooth\nregularization terms. The work [41] considered a similar set-up to this work and proposed a proximal\ngradient method combined with Nesterov\u2019s acceleration that can achieve O(1/i2) convergence rate;\nhowever, it requires an increasing number of inner loop consensus steps with each iteration leading to\nan expensive solution. Other works focused on the case where each agent k has a local regularizer\nRk(w) possibly different from other agents. For example, a proximal decentralized linearized ADMM\n(DL-ADMM) approach is proposed in [22] to solve such composite problems with convergence\nguarantees, while the work [42] establishes a sublinear convergence rate O(1/i) for DL-ADMM\nwhen each Jk(w) is smooth with Lipschitz continuous gradient. PG-EXTRA [23] extends EXTRA\n[15] to handle non-smooth regularization local terms and it establishes an improved rate o(1/i). The\nNIDS algorithm [40] also has an o(1/i) rate and can use larger step-sizes compared to PG-EXTRA.\nBased on existing results, there is still a clear gap between decentralized algorithms and centralized\nalgorithms for problem (2) when using proximal gradient methods.\nThe work [43] established the asymptotic linear convergence2 of a proximal decentralized algorithm\nfor the special case when all functions {Jk(w), Rk(w)} (possibly different regularizers) are piecewise\nlinear-quadratic (PLQ) functions. While this result is encouraging, it does not cover the global linear\nconvergence rate we seek in this work since their linear rate occurs only after a suf\ufb01ciently large\nnumber of iterations and requires all costs to be PLQ. Another useful work [29] extends the CoCoA\nalgorithm [9] to the COLA algorithm for decentralized settings and shows linear convergence in\nthe presence of a non-differentiable regularizer. Like most other dual coordinate methods, COLA\nconsiders decentralized learning for generalized linear models (e.g., linear regression, logistic\nregression, SVM, etc). This is because COLA requires solving (2) from the dual domain and the\nlinear model facilitates the derivation of the dual functions. Additionally, different from this work,\nCOLA is not a proximal gradient-based method; it requires solving an inner minimization problem\nto a satisfactory accuracy, which is often computationally expensive but necessary for the linear\nconvergence analysis.\nNote \ufb01nally that decentralized optimization problems of the form (2) can be reformulated into a\nconsensus equality constrained optimization problem (see equation (7)). The consensus constraint\ncan then be added to the objective function using an indicator function. Several works have proposed\ngeneral solutions based on this construction using proximal primal-dual methods \u2013 see [44\u201346]\nand references therein. Linear convergence for these methods have been established under certain\nconditions that do not cover decentralized composite optimization problems of the form (2). For\nexample, the works [44, 45] require a smoothness assumption, which does not cover the indicator\nfunction needed for the consensus constraint. The work [46] requires the coef\ufb01cient matrix for the\nnon-smooth terms to be full-row rank, which is not the case for decentralized optimization problems\neven when R(w) = 0.\nContribution. This paper considers the composite optimization problem (2) and has two main\ncontributions. First, for the case of a common non-smooth regularizer R(w) across all computing\nagents, we propose a proximal decentralized algorithm whose \ufb01xed point coincides with the desired\nglobal solution w(cid:63). We then provide a short proof to establish its linear convergence when the\nk=1 Jk(w) is strongly convex. This result closes the existing\ngap between decentralized proximal gradient methods and centralized proximal gradient methods.\nThe second contribution is in our convergence proof technique. Speci\ufb01cally, we provide a concise\nproof that is applicable to general decentralized primal-dual gradient methods such as EXTRA [15]\nwhen R(w) = 0. Our proof provides useful bounds on the convergence rate and step-sizes.\n\naggregate of the smooth functions(cid:80)K\n\n2A sequence {xi}\u221e\n\n(cid:107)xi \u2212 x(cid:63)(cid:107) \u2264 \u03b3iC for some C > 0 and all i \u2265 io.\n\ni=0 has asymptotic linear convergence to x(cid:63) if there exists a suf\ufb01ciently large io such that\n\n3\n\n\fNotation. For a matrix A \u2208 RM\u00d7N , \u03c3max(A) (\u03c3min(A)) denotes the maximum (minimum) singular\nvalue of A, and \u03c3(A) denotes the minimum non-zero singular value. For a vector x \u2208 RM and a\npositive semi-de\ufb01nite matrix C \u2265 0, we let (cid:107)x(cid:107)2\nC = xTCx. The N \u00d7 N identity matrix is denoted by\nIN . We let 1N be a vector of size N with all entries equal to one. The Kronecker product is denoted\nby \u2297. We let col{xn}N\nn=1 denote a column vector (matrix) that stacks the vector (matrices) xn of\nappropriate dimensions on top of each other. The subdifferential \u2202f (x) of a function f (.) : RM \u2192 R\nat some x \u2208 RM is the set of all subgradients \u2202f (x) = {g | gT(y \u2212 x) \u2264 f (y) \u2212 f (x),\u2200 y \u2208 RM}.\nThe proximal operator with parameter \u00b5 > 0 of a function f (x) : RM \u2192 R is\n\nprox\u00b5f (x) = arg min\n\nz\n\nf (z) +\n\n(cid:107)z \u2212 x(cid:107)2\n\n1\n2\u00b5\n\n(5)\n\n2 Proximal Decentralized Algorithm\n\nIn this section, we derive the algorithm and list its decentralized implementation.\n\n2.1 Algorithm Derivation\n\nWe start by introducing the network weights that are used to implement the algorithm in a decentral-\nized manner. Thus, we let ask denote the weight used by agent k to scale information arriving from\nagent s with ask = 0 if s is not a direct neighbor of agent k, i.e., there is no edge connecting them.\nLet A = [ask] \u2208 RK\u00d7K denote the weight matrix associated with the network. Then, we assume A\nto be symmetric and doubly stochastic, i.e., A1K = 1K and 1T\nK. We also assume that A is\nprimitive, i.e., there exists an integer p such that all entries of Ap are positive. Note that as long as\nthe network is connected, there exist many ways to generate such weight matrices in a decentralized\nfashion \u2013 [14, 47, 48]. Under these conditions, it holds from the Perron-Frobenius theorem [49] that\nA has a single eigenvalue at one with all other eigenvalues being strictly less than one. Therefore,\n(IK \u2212 A)x = 0 if, and only if, x = c1K for any c \u2208 R. If we let wk \u2208 RM denote a local copy of\nthe global variable w available at agent k and introduce the network quantities:\n\nKA = 1T\n\nW \u2206= col{w1,\u00b7\u00b7\u00b7 , wK} \u2208 RKM , B \u2206=\n\n(6)\nthen, it holds that BW = 0 if, and only if, wk = ws for all k, s. Note that since A is symmetric\nwith eigenvalues between (\u22121, 1], the matrix B is positive semi-de\ufb01nite with eigenvalues in [0, 1).\nProblem (2) is equivalent to the following constrained problem:\ns.t. B 1\n\nJ (W) + R(W),\n\n(IKM \u2212 A \u2297 IM )\n\n2 W = 0\n\n1\n2\n\n(7)\n\nwhere J (W) \u2206= (cid:80)K\n\nminimize\nW\u2208RKM\n\nk=1 Jk(wk), R(W) \u2206= (cid:80)K\n\n2 is the square root of the positive\nsemi-de\ufb01nite matrix B. To solve problem (7), we introduce \ufb01rst the following equivalent saddle-point\nproblem:\n\nk=1 R(wk) and B 1\n\nmin\nW\n\nmax\n\nY\n\nL\u00b5(W, Y) \u2206= J (W) + R(W) + YTB 1\n\n2 W +\n\n(cid:107)B 1\n\n2 W(cid:107)2\n\n1\n2\u00b5\n\n(8)\n\nwhere Y \u2208 RM K is the dual variable and \u00b5 > 0 is the coef\ufb01cient for the augmented Lagrangian. By\nintroducing J\u00b5(W) = J (W) + 1/2\u00b5(cid:107)B 1\n\n2 W(cid:107)2, it holds that\n\nTo solve the saddle point problem in (8), we propose the following recursion. For i \u2265 0:\n\nL\u00b5(W, Y) = J\u00b5(W) + R(W) + YTB 1\n\n\uf8f1\uf8f4\uf8f2\uf8f4\uf8f3 Zi = Wi\u22121 \u2212 \u00b5\u2207J\u00b5(Wi\u22121) \u2212 B 1\n\nYi = Yi\u22121 + \u03b1B 1\nWi = prox\u00b5R(Zi)\n\n2 Zi\n\n2 W.\n\n2 Yi\u22121\n\n(9)\n\n(10a)\n(10b)\n(10c)\n\nwhere \u03b1 > 0 is the dual step-size (a tunable parameter). We will next show that with the initialization\nY0 = 0, we can implement this algorithm in a decentralized manner.\n\n4\n\n\fRemark 1 (CONVENTIONAL UPDATE). When R(w) = 0 and \u03b1 = 1, recursions (10a)\u2013(10c) recover\nthe primal-dual form of the EXTRA algorithm from [15]. However, when R(w) (cid:54)= 0, recursions\n(10a)\u2013(10c) differ from PG-EXTRA [23] in the dual update (10b). Different from conventional dual\nupdates that use Wi (e.g., see [50] for the primal-dual form of PG-EXTRA), we use Zi instead of\nWi. This subtle difference changes the complexity of the algorithm and allows us to close the linear\nconvergence gap between centralized and decentralized algorithms for problems of the form (2). 2\n\n2.2 The Decentralized Implementation\nFrom the de\ufb01nition of J\u00b5(W), we have \u2207J\u00b5(W) = \u2207J (W) + 1/\u00b5 BW. Substituting \u2207J\u00b5(W) into\n(10a), we have\n\n(11)\n\n(12)\n\nZi = (IKM \u2212 B)Wi\u22121 \u2212 \u00b5\u2207J (Wi\u22121) \u2212 B 1\n\nWith the above relation, we have for i \u2265 1\n\nZi\u2212Zi\u22121 = (I\u2212B)(Wi\u22121\u2212Wi\u22122)\u2212\u00b5(cid:0)\u2207J (Wi\u22121)\u2212\u2207J (Wi\u22122)(cid:1)\u2212B 1\n\n2 Yi\u22121.\n\nFrom (10b) we have Yi\u22121 \u2212 Yi\u22122 = \u03b1B 1\n\n2 (Yi\u22121 \u2212 Yi\u22122)\n2 Zi\u22121. Substituting this relation into (12), we reach\n\nZi = (I \u2212 \u03b1B)Zi\u22121 + (I \u2212 B)(Wi\u22121\u2212Wi\u22122)\u2212\u00b5(cid:0)\u2207J (Wi\u22121)\u2212\u2207J (Wi\u22122)(cid:1)\nZi = (I \u2212 \u03b1B)Zi\u22121 + (I \u2212 B)(Wi\u22121\u2212Wi\u22122)\u2212\u00b5(cid:0)\u2207J (Wi\u22121)\u2212\u2207J (Wi\u22122)(cid:1)\n\n(13)\nfor i \u2265 1. For initialization, we can repeat a similar argument to show that the proximal primal-dual\nmethod (10a)\u2013(10c) with Y0 = 0 is equivalent to the following algorithm. Let Z0 = W\u22121 = 0, set\n\u2207J (W\u22121) \u2190 0, and W0 to any arbitrary value. Repeat for i = 1,\u00b7\u00b7\u00b7\n\n(14a)\n(14b)\nSince B has network structure, recursion (14) can be implemented in a decentralized way. This\nalgorithm only requires each agent to share one vector at each iteration; a per agent implementation\nof resulting proximal primal-dual diffusion (P2D2) algorithm is listed in (15).\n\nWi = prox\u00b5R(Zi)\n\n(cid:88)\n\nAlgorithm (Proximal Primal-Dual Diffusion \u2013 P2D2)\nSetting: Let B = 0.5(I \u2212 A) = [bsk] and choose step-sizes \u00b5 and \u03b1. Set all initial variables to zero\nand repeat for i = 1, 2,\u00b7\u00b7\u00b7\n\nbsk(\u03b1zs,i\u22121 + ws,i\u22121 \u2212 ws,i\u22122)\n\ns\u2208Nk\n\n\u03c6k,i =\n\u03c8k,i = wk,i\u22121 \u2212 \u00b5\u2207Jk(wk,i\u22121)\nzk,i = zk,i\u22121 + \u03c8k,i \u2212 \u03c8k,i\u22121 \u2212 \u03c6k,i\nwk,i = prox\u00b5R(zk,i)\n\n(Communication Step)\n\n(15a)\n\n(15b)\n(15c)\n(15d)\n\n3 Main Results\n\nIn this section, we establish the linear convergence of the proximal primal-dual diffusion (P2D2)\nalgorithm (10a)\u2013(10c), which is equivalent to (15). To this end, we establish auxiliary lemmas leading\nto the main convergence result.\n\n3.1 Optimality condition\n\nWe start by showing the existence and properties of a \ufb01xed point for recursions (10a)\u2013(10c).\nLemma 1 (FIXED POINT OPTIMILATY). Under Assumption 1, a \ufb01xed point (W(cid:63), Y(cid:63), Z(cid:63)) exists for\nrecursions (10a)\u2013(10c), i.e., it holds that\n\n\uf8f1\uf8f4\uf8f2\uf8f4\uf8f3 Z(cid:63) = W(cid:63) \u2212 \u00b5\u2207J\u00b5(W(cid:63)) \u2212 B 1\n\n0 = B 1\nW(cid:63) = prox\u00b5R(Z(cid:63))\n\n2 Z(cid:63)\n\n2 Y(cid:63)\n\n5\n\n(16a)\n(16b)\n(16c)\n\n\fMoreover, W(cid:63) and Z(cid:63) are unique and each block element of W(cid:63) = col{w(cid:63)\nthe unique solution w(cid:63) to problem (2), i.e., w(cid:63)\n\nk = w(cid:63) for all k.\n\n1,\u00b7\u00b7\u00b7 , w(cid:63)\n\nK} coincides with\n\nProof. The existence of a \ufb01xed point is shown in Section A in the supplementary material. We now\nestablish the optimality of W(cid:63). Since Z(cid:63) satis\ufb01es (16b), it holds that the block elements of Z(cid:63) are\nK, and we denote each block element by z(cid:63). Therefore, from\nequal to each other, i.e. z(cid:63)\n(16c) and the de\ufb01nition of the proximal operator it holds that\n\n1 = \u00b7\u00b7\u00b7 = z(cid:63)\n\nwhere we used z(cid:63)\nany k. It is easy to verify that (17) implies\n\nk = z(cid:63) for each k. Thus, we must have w(cid:63)\n\nK. We denote w(cid:63)\n\nk = w(cid:63) for\n\nw(cid:63)\n\nk = arg min\n\nwk\n\n{R(wk) + (cid:107)wk \u2212 z(cid:63)(cid:107)2/2\u00b5}\n1 = \u00b7\u00b7\u00b7 = w(cid:63)\n\n0 \u2208 \u2202R(w(cid:63)) + (w(cid:63) \u2212 z(cid:63))/\u00b5.\n\n(17)\n\n(18)\n\nK(cid:88)\n\nMultiplying (1K \u2297 IM )T from the left to both sides of equation (16a), we get\n\nKz(cid:63) = Kw(cid:63) \u2212 \u00b5\n\n\u2207Jk(w(cid:63)).\n\n(19)\n\nK\n\nk=1\n\n(cid:80)K\nCombining (18) and (19), we get 0 \u2208 1\nk=1 \u2207Jk(w(cid:63)) + \u2202R(w(cid:63)), which implies that w(cid:63) is the\nunique solution to problem (2). Due to the uniqueness of w(cid:63), we see from (19) that z(cid:63) is unique.\nConsequently, W(cid:63) = 1K \u2297 w(cid:63) and Z(cid:63) = 1K \u2297 z(cid:63) must be unique.\nRemark 2 (PARTICULAR FIXED POINT). From Lemma 1, we see that although W(cid:63) and Z(cid:63) are\nunique, there can be multiple \ufb01xed points. This is because from (16a), Y(cid:63) is not unique due the\nrank de\ufb01ciency of B 1\n2 . However, by following similar arguments to the ones from [18], it can be\nveri\ufb01ed that there exists a particular \ufb01xed point (W(cid:63), Y(cid:63)\nb is a\nunique vector that belongs to the range space of B 1\n2 . In the following we will show that the iterates\n(Wi, Yi, Zi) converge linearly to this particular \ufb01xed point (W(cid:63), Y(cid:63)\n2\n\nb , Z(cid:63)) satisfying (16a)\u2013(16c) where Y(cid:63)\n\nb , Z(cid:63)).\n\n3.2 Linear Convergence\n\n\uf8f1\uf8f4\uf8f2\uf8f4\uf8f3\n\n\u2206= Wi \u2212 W(cid:63),\n\nTo establish the linear convergence of the proximal primal-dual diffusion (P2D2) (10a)\u2013(10c) we\nintroduce the error quantities:\n\n(cid:101)Wi\n(cid:101)Zi = Zi \u2212 Z(cid:63)\n(cid:101)Yi\n(cid:101)Zi = (cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1) \u2212 B 1\n2(cid:101)Yi\u22121\n(cid:101)Yi =(cid:101)Yi\u22121 + \u03b1B 1\n2(cid:101)Zi\n(cid:101)Wi = prox\u00b5R(Zi) \u2212 prox\u00b5R(Z(cid:63))\n\n\u2206= Yi \u2212 Y(cid:63)\n(20)\nb ,\nBy subtracting (16a)\u2013(16c) from (10a)\u2013(10c) with Y(cid:63) = Y(cid:63)\nb, we reach the following error recursions\n(21a)\n(21b)\n(21c)\nWe let \u03c3max and \u03c3 denote the maximum singular value and minimum non-zero singular value of\nthe matrix B. Notice that from (6), B is symmetric and, thus, its singular values are equal to its\neigenvalues and are in [0, 1) (i.e., \u03c3min = 0 < \u03c3 \u2264 \u03c3max < 1). The following result follows from\n[15, Proposition 3.6].\nLemma 2 (AUGMENTED COST). Under Assumption 1, the penalized augmented cost J (W) +\n2(cid:107)W(cid:107)2B with any \u03c1 > 0 is restricted strongly-convex with respect to W(cid:63):\n\n(W \u2212 W(cid:63))T(cid:0)\u2207J (W) \u2212 \u2207J (W(cid:63))(cid:1) + \u03c1(cid:107)W \u2212 W(cid:63)(cid:107)2B \u2265 \u03bd\u03c1(cid:107)W \u2212 W(cid:63)(cid:107)2\n(cid:17)\n\nwhere\n\n(cid:26)\n\n(cid:27)\n\n\u03c1\n\nfor any c \u2208(cid:16)\n\n\u03bd\u03c1 = min\n\n0,\nfor any W with W(cid:63) = 1 \u2297 w(cid:63) and where w(cid:63) denotes the minimizer of (2).\nUsing the previous result, the following lemma establishes a useful inequality for later use.\n\n> 0,\n\n(22)\n\n(23)\n\n\u03bd \u2212 2\u03b4c,\n\n\u03c1\u03c3(B)c2\n4(c2 + 1)\n\n\u03bd\n2\u03b4\n\n6\n\n\fLemma 3 (DESCENT INEQUALITY). Under Assumption 1 and step-size conditions \u00b5 < (1\u2212\u03c3max)\n\nand \u03b1 \u2264 1, it holds that(cid:13)(cid:13)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1)(cid:13)(cid:13)2 \u2264 \u03b31(cid:107)(cid:101)Wi\u22121(cid:107)2Q\n\n(24)\nwhere Q = I \u2212 \u03b1B and \u03b31 = 1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4) < 1 for some \u03c1 > 0 with \u03bd\u03c1 given in (23).\n\n\u03b4\n\nProof. Since \u2207J\u00b5(W) = \u2207J (W) + 1\n\n(cid:13)(cid:13)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1)(cid:13)(cid:13)2\n= (cid:107)(cid:101)Wi\u22121(cid:107)2\u22122\u00b5(cid:101)WT\n\ni\u22121\n\n(cid:0)\u2207J (Wi\u22121)\u2212\u2207J (W(cid:63))(cid:1)\u22122(cid:107)(cid:101)Wi\u22121(cid:107)2B +\u00b52(cid:107)\u2207J\u00b5(Wi\u22121)\u2212\u2207J\u00b5(W(cid:63))(cid:107)2\n\n\u00b5BW, it holds that\n\nNote that \u2207J (W) + 1\n\n\u00b5BW is \u03b4\u00b5 = \u03b4 + 1\n\n\u00b5 \u03c3max-Lipschitz, thus it holds that [51, Theorem 2.1.5]:\n\n1\n\u00b5\n\nB(cid:101)Wi\u22121(cid:107)2\n(cid:0)\u2207J (Wi\u22121) \u2212 \u2207J (W(cid:63)) +\n\n(cid:107)\u2207J\u00b5(Wi\u22121)\u2212\u2207J\u00b5(W(cid:63))(cid:107)2 = (cid:107)\u2207J (Wi\u22121) \u2212 \u2207J (W(cid:63)) +\n\n\u2264 \u03b4\u00b5(cid:101)WT\nB(cid:101)Wi\u22121\n(cid:13)(cid:13)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1)(cid:13)(cid:13)2\n(cid:0)\u2207J (Wi\u22121)\u2212\u2207J (W(cid:63))(cid:1)\u2212(2 \u2212 \u00b5\u03b4\u00b5)(cid:107)(cid:101)Wi\u22121(cid:107)2B\n\u2264 (cid:107)(cid:101)Wi\u22121(cid:107)2\u2212\u00b5(2 \u2212 \u00b5\u03b4\u00b5)(cid:101)WT\n\u2264(cid:0)1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u00b5\u03b4\u00b5)(cid:1)(cid:107)(cid:101)Wi\u22121(cid:107)2\u2212(2 \u2212 \u00b5\u03b4\u00b5)(1 \u2212 \u03c1\u00b5)(cid:107)(cid:101)Wi\u22121(cid:107)2B\n\n(cid:1)\n\ni\u22121\n\ni\u22121\n\n1\n\u00b5\n\nSubstituting the previous inequality into (25) we get\n\n(27)\nwhere the last inequality follows from (22) and 2 \u2212 \u00b5\u03b4\u00b5 = 2 \u2212 \u03c3max \u2212 \u00b5\u03b4 > 0 for \u00b5 < 2\u2212\u03c3max\n\nLetting \u03b31 = 1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u00b5\u03b4\u00b5) and adding and subtracting \u03b1\u03b31(cid:107)(cid:101)Wi\u22121(cid:107)B to the right hand side of\n(cid:13)(cid:13)(cid:101)Wi\u22121\u2212\u00b5(cid:0)\u2207J\u00b5(Wi\u22121)\u2212\u2207J\u00b5(W(cid:63))(cid:1)(cid:13)(cid:13)2 \u2264 \u03b31(cid:107)(cid:101)Wi\u22121(cid:107)2Q\u2212(cid:0)(2 \u2212 \u00b5\u03b4\u00b5)(1 \u2212 \u03c1\u00b5)\u2212\u03b1\u03b31\n(cid:1)(cid:107)(cid:101)Wi\u22121(cid:107)2B (28)\n\nthe previous inequality gives:\n\n\u03b4\n\n.\n\nwhere Q = I \u2212 \u03b1B. If we can ensure that\n\n\u2212(cid:0)(2 \u2212 \u00b5\u03b4\u00b5)(1 \u2212 \u00b5\u03c1) \u2212 \u03b1\u03b31\n\n(cid:1)(cid:107)(cid:101)Wi\u22121(cid:107)2B \u2264 0\n\nthen inequality (28) can be upper bounded by (24). To ensure inquality (29), it is suf\ufb01cient to \ufb01nd \u00b5\nand \u03c1 such that\n\n(2 \u2212 \u00b5\u03b4\u00b5)(1 \u2212 \u00b5\u03c1) \u2212 \u03b31\u03b1 = (2 \u2212 \u03c3max \u2212 \u00b5\u03b4)(1 \u2212 \u00b5\u03c1) \u2212 \u03b31\u03b1 \u2265 0\n\n(25)\n\n(26)\n\n(29)\n\n(30)\nand using \u03b1 \u2264 1, the above\n\nBy noting that \u03b31 = 1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4) < 1 for \u00b5 < 1\u2212\u03c3max\ninequality is guaranteed to hold if\n\n\u03b4\n\n0 < \u03c1 \u2264 1 \u2212 \u03c3max \u2212 \u00b5\u03b4\n\u00b5(2 \u2212 \u03c3max \u2212 \u00b5\u03b4)\n\n(31)\n\nThe previous lemma will be used to establish the following primal-dual error bound.\nLemma 4 (ERROR BOUND). Under Assumption 1, if Y0 = 0 and the step-sizes satisfy \u00b5 < (1\u2212\u03c3max)\nand \u03b1 \u2264 1, it holds that\n\n\u03b4\n\n(cid:107)(cid:101)Zi(cid:107)2Q + (cid:107)(cid:101)Yi(cid:107)2\n\n\u03b1 I \u2264 \u03b31(cid:107)(cid:101)Wi\u22121(cid:107)2Q + \u03b32(cid:107)(cid:101)Yi\u22121(cid:107)2\n\n(32)\nwhere Q = I \u2212 \u03b1B > 0, \u03b31 = 1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4) < 1, and \u03b32 = 1 \u2212 \u03b1\u03c3 < 1 for some \u03c1 > 0\nwith \u03bd\u03c1 given in (23).\n\n1\n\u03b1 I\n\n1\n\n7\n\n\fand\n\nProof. Squaring both sides of (21a) and (21b) we get\n\n(cid:107)(cid:101)Yi(cid:107)2 = (cid:107)(cid:101)Yi\u22121 + \u03b1B 1\n\n\u2212 2(cid:101)YT\n2(cid:101)Zi(cid:107)2 = (cid:107)(cid:101)Yi\u22121(cid:107)2 + \u03b12(cid:107)B 1\n\n(cid:107)(cid:101)Zi(cid:107)2 = (cid:107)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1)(cid:107)2 + (cid:107)B 1\n2(cid:101)Yi\u22121(cid:107)2\n2(cid:0)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1)(cid:1)\ni\u22121B 1\n2(cid:101)Zi(cid:107)2 + 2\u03b1(cid:101)YT\n2(cid:101)Zi\ni\u22121B 1\n= (cid:107)(cid:101)Yi\u22121(cid:107)2 + \u03b12(cid:107)(cid:101)Zi(cid:107)2B \u2212 2\u03b1(cid:107)B 1\n2(cid:101)Yi\u22121(cid:107)2\n2(cid:0)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207WJ\u00b5(W(cid:63))(cid:1)(cid:1)\n+ 2\u03b1(cid:101)YT\n\u03b1 I =(cid:107)(cid:101)Wi\u22121\u2212\u00b5(cid:0)\u2207J\u00b5(Wi\u22121)\u2212\u2207J\u00b5(W(cid:63))(cid:1)(cid:107)2 +(cid:107)(cid:101)Yi\u22121(cid:107)2\n2(cid:101)Yi\u22121(cid:107)2\n\u03b1 I \u2264 (cid:107)(cid:101)Wi\u22121 \u2212 \u00b5(cid:0)\u2207J\u00b5(Wi\u22121) \u2212 \u2207J\u00b5(W(cid:63))(cid:1)(cid:107)2 +(1 \u2212 \u03b1\u03c3)(cid:107)(cid:101)Yi\u22121(cid:107)2\n\n\u03b1 I\u2212\u03b1(cid:107)B 1\n2 , it holds that (cid:107)B 1\n\nb lie in the range space3 of B 1\n\nMultiplying equation (34) by 1\n\n(cid:107)(cid:101)Zi(cid:107)2Q +(cid:107)(cid:101)Yi(cid:107)2\n\u03c3(cid:107)(cid:101)Yi\u22121(cid:107)2 \u2013 see [18]. Thus, we can bound (35) by\n(cid:107)(cid:101)Zi(cid:107)2Q + (cid:107)(cid:101)Yi(cid:107)2\n\nwhere Q = I \u2212 \u03b1B. Since both Yi and Y(cid:63)\n\n\u03b1 and adding to (33), we get\n\ni\u22121B 1\n\n(21a)\n\n1\n\n1\n\n1\n\n(36)\nUnder the conditions given in Lemma 3, we can substitute inequality (24) in the above relation\nand get (32). Note that that \u03b31 = 1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4) < 1 if \u00b5 < (1 \u2212 \u03c3max)/\u03b4. Moreover,\nQ = I \u2212 \u03b1B > 0 and \u03b32 = 1 \u2212 \u03b1\u03c3 < 1 for \u03b1 \u2264 1 since \u03c3max < 1.\n\n1\n\u03b1 I\n\n1\n\u03b1 I\n\n(35)\n\n2(cid:101)Yi\u22121(cid:107)2 \u2265\n\n(33)\n\n(34)\n\nThe next Theorem establishes the linear convergence of our proposed algorithm.\nTheorem 1 (LINEAR CONVERGENCE). Under Assumption 1, Y0 = 0, and if step-sizes satisfy\n\n(1 \u2212 \u03c3max)\n\nIt holds that (cid:107)(cid:101)Wi(cid:107)2 \u2264 C\u03b3i where C > 0 and\n\n\u00b5 <\n\n\u03b4\n\n, \u03b1 \u2264 min{1, \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4)} .\n\n\u03b3 \u2206= max{1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4)/(1 \u2212 \u03b1\u03c3max), 1 \u2212 \u03b1\u03c3} < 1\n\nfor some \u03c1 > 0 with \u03bd\u03c1 given in (23).\nProof. Assume \u03b1 \u2264 1 and note that Q = I \u2212 \u03b1B. Thus, it holds that \u03c3min(Q) = 1 \u2212 \u03b1\u03c3max and\n\u03c3max(Q) = 1. This implies that (1 \u2212 \u03b1\u03c3max)(cid:107)x(cid:107)2 \u2264 (cid:107)x(cid:107)2Q \u2264 (cid:107)x(cid:107)2 for any x \u2208 RKM . Therefore,\nit holds from Lemma 4 that\n\nwhen \u00b5 < (1\u2212\u03c3max)\n\n\u03b4\n\n. Dividing by \u03b2 \u2206= 1 \u2212 \u03b1\u03c3max both sides of the above inequality, we have\n\n(1 \u2212 \u03b1\u03c3max)(cid:107)(cid:101)Zi(cid:107)2 + (cid:107)(cid:101)Yi(cid:107)2\n\n(cid:107)(cid:101)Zi(cid:107)2 + (cid:107)(cid:101)Yi(cid:107)2\n\n\u03b1\u03b2 I \u2264 \u03b31\n\n1\n\n\u03b2\n\n1\n\n\u03b1 I \u2264 \u03b31(cid:107)(cid:101)Wi\u22121(cid:107)2 + \u03b32(cid:107)(cid:101)Yi\u22121(cid:107)2\n(cid:107)(cid:101)Wi\u22121(cid:107)2 + \u03b32(cid:107)(cid:101)Yi\u22121(cid:107)2\n\n\u03b1\u03b2 I .\n\n1\n\n1\n\u03b1 I\n\nClearly, \u03b2 \u2208 (0, 1) when \u03b1 \u2264 1. Now we evaluate \u03b31/\u03b2. It is easy to verify that\n\nwhen \u03b1 \u2264 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4) < \u00b5\u03bd\u03c1(2\u2212\u03c3max\u2212\u00b5\u03b4)\nthe proximal operator we have:\n\n\u03c3max\n\n. Next, from the non-expansiveness property of\n\nBy substituting (42) into (40) and letting \u03b3 \u2206= max{\u03b31/\u03b2, \u03b32}, we reach\n\n= (1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4)) /(1 \u2212 \u03b1\u03c3max) < 1\n\n\u03b31\n\u03b2\n\n(cid:107)(cid:101)Wi(cid:107)2 = (cid:107)prox\u00b5R(Zi) \u2212 prox\u00b5R(Z(cid:63))(cid:107)2 \u2264 (cid:107)(cid:101)Zi(cid:107)2.\n(cid:17)\n(cid:16)(cid:107)(cid:101)Wi\u22121(cid:107)2 + (cid:107)(cid:101)Yi\u22121(cid:107)2\n(cid:107)(cid:101)Wi(cid:107)2 + (cid:107)(cid:101)Yi(cid:107)2\n\u03b1\u03b2 I \u2264 \u03b3\n\u03b1\u03b2 I \u2264 \u03b3i((cid:107)(cid:101)W0(cid:107)2 + (cid:107)(cid:101)Y0(cid:107)2\n(cid:107)(cid:101)Wi(cid:107)2 \u2264 (cid:107)(cid:101)Wi(cid:107)2 + (cid:107)(cid:101)Yi(cid:107)2\n\n1\n\u03b1\u03b2 I\n\n1\n\n1\n\n\u03b1\u03b2 I ),\n\n1\n\n(37)\n\n(38)\n\n(39)\n\n(40)\n\n(41)\n\n(42)\n\n(43)\n\n(44)\n\nwhen step-sizes \u00b5 and \u03b1 satisfy condition (37). We iterate the above inequality and get\n\nwhich concludes the proof.\n\n3Since Y0 = 0 and Yi = Yi\u22121 + \u03b1B 1\n\n2 Zi, we know Yi \u2208 range(B 1\n\n2 ) for any i.\n\n8\n\n\fNext we show that when R(w) = 0, we can have a better upper bound for the dual step-size, which\ncovers the EXTRA algorithm [15].\nTheorem 2 (LINEAR CONVERGENCE WHEN R(w) = 0). Under Assumption 1, if R(w) = 0,\nY0 = 0, and the step-sizes satisfy \u00b5 < (1\u2212\u03c3max)\nC > 0, Q = I \u2212 \u03b1B > 0, and\n\nand \u03b1 \u2264 1, it holds that (cid:107)(cid:101)Wi(cid:107)2Q \u2264 C\u03b3i where\n\n\u03b3 = max(cid:8)1 \u2212 \u00b5\u03bd\u03c1(2 \u2212 \u03c3max \u2212 \u00b5\u03b4), 1 \u2212 \u03b1\u03c3(cid:9) < 1\n\n\u03b4\n\nfor some \u03c1 > 0 with \u03bd\u03c1 given in (23).\nProof. From lemma 4, we know when \u00b5 < (1\u2212\u03c3max)\n\nR(w) = 0, we know Wi = Zi from recursion (10c) and hence(cid:101)Wi =(cid:101)Zi. By letting \u03b3 = max{\u03b31, \u03b32},\nand \u03b1 \u2264 1 that inequality (32) holds. Since\ninequality (32) becomes (cid:107)(cid:101)Wi(cid:107)2Q + (cid:107)(cid:101)Yi(cid:107)2\n\u03b1 I \u2264 \u03b3((cid:107)(cid:101)Wi\u22121(cid:107)2Q + (cid:107)(cid:101)Yi\u22121(cid:107)2\npositive de\ufb01nite when \u03b1 \u2264 1, we reach the linear convergence of (cid:101)Wi.\n\u03b1 I ). Since Q = I \u2212 \u03b1B is\n\n\u03b4\n\n1\n\n1\n\nIn the above Theorem, we see that the convergence rate bound is upper bounded by two terms, one\nterm is from the cost function and the other is from the network. This bound shows how the network\naffects the convergence rate of the algorithm. For example, in Theorem 2, assume that \u03b1 = 1 and the\nnetwork term dominates the convergence rate so that \u03b3 = 1 \u2212 \u03b1\u03c3 = 1 \u2212 \u03c3. Recall that \u03c3 = \u03c3(B)\nis the smallest non-zero singular value (or eigenvalue) of the matrix 0.5(I \u2212 A). Thus, the effect\nof the network on the convergence rate is evident through the term 1 \u2212 \u03c3, which becomes close to\none as the network becomes more sparse. Note when \u03b1 = 1, the algorithm recovers EXTRA as\nhighlighted in Remark 1. In this case, our step-size condition is on the order of O((1 \u2212 \u03c3max)/\u03b4).\nNote that the in the original EXTRA proof in [15, Theorem 3.7], the step-size bound is on the order\nof O(\u03bd\u03c1(1 \u2212 \u03c3max)/\u03b42)), which scales badly for ill-conditioned problems, i.e., if \u03b4 is much larger\nthan \u03bd\u03c1. We remark that simulations of the proposed algorithm are provided in Section B in the\nsupplementary material.\n\nAcknowledgments\n\nThis work was supported in part by NSF grant CCF-1524250. We would like to thank the anonymous\nreviewers for their insightful comments.\n\nReferences\n[1] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning\n\nvia alternating direction method of multipliers. Found. Trends Mach. Lear., 3(1):1\u2013122, Jan. 2011.\n\n[2] A. D. Dominguez-Garcia, S. T. Cady, and C. N. Hadjicostis. Decentralized optimal dispatch of distributed\nenergy resources. In 51st IEEE Conference on Decision and Control (CDC), pages 3688\u20133693, Maui, HI,\nUSA, Dec. 2012.\n\n[3] W. Deng, M.-J. Lai, Z. Peng, and W. Yin. Parallel multi-block ADMM with o (1/k) convergence. Journal\n\nof Scienti\ufb01c Computing, 71(2):712\u2013736, 2017.\n\n[4] M. Zinkevich, M. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. In Advances in\n\nNeural Information Processing Systems (NIPS), pages 2595\u20132603, Vancouver, Canada, 2010.\n\n[5] A. Agarwal and J. C. Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information\n\nProcessing Systems (NIPS), pages 873\u2013881, Granada Spain, 2011.\n\n[6] O. Shamir, N. Srebro, and T. Zhang. Communication-ef\ufb01cient distributed optimization using an approxi-\nmate Newton-type method. In International Conference on Machine Learning (ICML), pages 1000\u20131008,\nBeijing, China, 2014.\n\n[7] Y. Zhang and X. Lin. Disco: Distributed optimization for self-concordant empirical loss. In International\n\nConference on Machine Learning (ICML), pages 362\u2013370, Lille, France, 2015.\n\n[8] C.-P. Lee, C. H. Lim, and S. J. Wright. A distributed quasi-newton algorithm for empirical risk minimization\nwith nonsmooth regularization. In Proc. ACM SIGKDD, pages 1646\u20131655, London, United Kingdom,\n2018.\n\n9\n\n\f[9] V. Smith, S. Forte, M. Chenxin, M. Takac, M. I. Jordan, and M. Jaggi. CoCoA: A general framework for\ncommunication-ef\ufb01cient distributed optimization. Journal of Machine Learning Research, 18(230):1\u201349,\n2018.\n\n[10] Z. Peng, Y. Xu, M. Yan, and W. Yin. ARock: an algorithmic framework for asynchronous parallel\n\ncoordinate updates. SIAM Journal on Scienti\ufb01c Computing, 38(5):A2851\u2013A2879, 2016.\n\n[11] M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y.\nSu. Scaling distributed machine learning with the parameter server. In 11th Symposium on Operating\nSystems Design and Implementation (OSDI), pages 583\u2013598, Broom\ufb01eld, Denver, Colorado, 2014.\n\n[12] X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu. Can decentralized algorithms outperform\ncentralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in\nNeural Information Processing Systems (NIPS), pages 5330\u20135340, Long Beach, CA, USA, 2017.\n\n[13] X. Lian, W. Zhang, C. Zhang, and J. Liu. Asynchronous decentralized parallel stochastic gradient descent.\n\nIn International Conference on Machine Learning (ICML), pages 1\u201310, Stockholm, Sweden, 2018.\n\n[14] A. H. Sayed. Adaptation, learning, and optimization over neworks. Foundations and Trends in Machine\n\nLearning, 7(4-5):311\u2013801, 2014.\n\n[15] W. Shi, Q. Ling, G. Wu, and W. Yin. EXTRA: An exact \ufb01rst-order algorithm for decentralized consensus\n\noptimization. SIAM Journal on Optimization, 25(2):944\u2013966, 2015.\n\n[16] A. Nedic and A. Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transac-\n\ntions on Automatic Control, 54(1):48\u201361, 2009.\n\n[17] A. H. Sayed. Adaptive networks. Proceedings of the IEEE, 102(4):460\u2013497, Apr. 2014.\n\n[18] W. Shi, Q. Ling, K. Yuan, G. Wu, and W. Yin. On the linear convergence of the ADMM in decentralized\n\nconsensus optimization. IEEE Trans. Signal Process., 62(7):1750\u20131761, 2014.\n\n[19] D. Jakoveti\u00b4c, J. M. Moura, and J. Xavier. Linear convergence rate of a class of distributed augmented\n\nLagrangian algorithms. IEEE Transactions on Automatic Control, 60(4):922\u2013936, 2015.\n\n[20] F. Iutzeler, P. Bianchi, P. Ciblat, and W. Hachem. Explicit convergence rate of a distributed alternating\n\ndirection method of multipliers. IEEE Transactions on Automatic Control, 61(4):892\u2013904, 2016.\n\n[21] Q. Ling, W. Shi, G. Wu, and A. Ribeiro. DLM: Decentralized linearized alternating direction method of\n\nmultipliers. IEEE Transactions on Signal Processing, 63:4051\u20134064, 2015.\n\n[22] T.-H. Chang, M. Hong, and X. Wang. Multi-agent distributed optimization via inexact consensus ADMM.\n\nIEEE Transactions on Signal Processing, 63(2):482\u2013497, Jan. 2015.\n\n[23] W. Shi, Q. Ling, G. Wu, and W. Yin. A proximal gradient algorithm for decentralized composite\n\noptimization. IEEE Transactions on Signal Processing, 63(22):6013\u20136023, 2015.\n\n[24] P. Di Lorenzo and G. Scutari. Next: In-network nonconvex optimization. IEEE Transactions on Signal\n\nand Information Processing over Networks, 2(2):120\u2013136, 2016.\n\n[25] J. Xu, S. Zhu, Y. C. Soh, and L. Xie. Augmented distributed gradient methods for multi-agent optimization\nunder uncoordinated constant stepsizes. In Proc. 54th IEEE Conference on Decision and Control (CDC),\npages 2055\u20132060, Osaka, Japan, 2015.\n\n[26] A. Nedic, A. Olshevsky, and W. Shi. Achieving geometric convergence for distributed optimization over\n\ntime-varying graphs. SIAM Journal on Optimization, 27(4):2597\u20132633, 2017.\n\n[27] G. Qu and N. Li. Harnessing smoothness to accelerate distributed optimization. IEEE Transactions on\n\nControl of Network Systems, 5(3):1245\u20131260, Sept. 2018.\n\n[28] K. Yuan, B. Ying, X. Zhao, and A. H. Sayed. Exact diffusion for distributed optimization and learning-Part\n\nI: Algorithm development. IEEE Transactions on Signal Processing, 67(3):708\u2013723, Feb. 2019.\n\n[29] L. He, A. Bian, and M. Jaggi. COLA: Decentralized linear learning. In Advances in Neural Information\n\nProcessing Systems (NeurIPS), pages 4536\u20134546, Montreal, Canada, 2018.\n\n[30] K. Scaman, F. Bach, S. Bubeck, Y. T. Lee, and L. Massoulie. Optimal algorithms for smooth and strongly\nconvex distributed optimization in networks. In International Conference on Machine Learning (ICML),\npages 3027\u20133036, Stockholm, Sweden, 2017.\n\n10\n\n\f[31] K. Yuan, B. Ying, X. Zhao, and A. H. Sayed. Exact diffusion for distributed optimization and learning-Part\n\nII: Convergence analysis. IEEE Transactions on Signal Processing, 67(3):724\u2013739, Feb. 2019.\n\n[32] L. Xiao and T. Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM\n\nJournal on Optimization, 24(4):2057\u20132075, 2014.\n\n[33] D. Chazan and W. Miranker. Chaotic relaxation. Linear Algebra and its Applications, 2(2):199\u2013222, 1969.\n\n[34] G. M. Baudet. Asynchronous iterative methods for multiprocessors. Journal of the ACM (JACM), 25(2):\n\n226\u2013244, 1978.\n\n[35] D. P. Bertsekas. Distributed asynchronous computation of \ufb01xed points. Mathematical Programming, 27\n\n(1):107\u2013120, 1983.\n\n[36] J. Tsitsiklis, D. Bertsekas, and M. Athans. Distributed asynchronous deterministic and stochastic gradient\n\noptimization algorithms. IEEE Transactions on Automatic Control, 31(9):803\u2013812, 1986.\n\n[37] J. C. Duchi, A. Agarwal, and M. J. Wainwright. Dual averaging for distributed optimization: Convergence\n\nanalysis and network scaling. IEEE Transactions on Automatic Control, 57(3):592\u2013606, 2012.\n\n[38] K. Yuan, Q. Ling, and W. Yin. On the convergence of decentralized gradient descent. SIAM Journal on\n\nOptimization, 26(3):1835\u20131854, 2016.\n\n[39] J. Chen and A. H. Sayed. Distributed Pareto optimization via diffusion strategies. IEEE J. Sel. Topics\n\nSignal Process., 7(2):205\u2013220, April 2013.\n\n[40] Z. Li, W. Shi, and M. Yan. A decentralized proximal-gradient method with network independent step-sizes\nand separated convergence rates. IEEE Transactions on Signal Processing, 67(17):4494\u20134506, Sept. 2019.\n\n[41] A. I. Chen and A. Ozdaglar. A fast distributed proximal-gradient method. In Annual Allerton Conference\n\non Communication, Control, and Computing, pages 601\u2013608, Monticello, IL, USA, Oct. 2012.\n\n[42] N. S. Aybat, Z. Wang, T. Lin, and S. Ma. Distributed linearized alternating direction method of multipliers\nfor composite convex consensus optimization. IEEE Transactions on Automatic Control, 63(1):5\u201320, 2018.\n\n[43] P. Latafat, N. M. Freris, and P. Patrinos. A new randomized block-coordinate primal-dual proximal\nalgorithm for distributed optimization. IEEE Transactions on Automatic Control, 64(10):4050\u20134065, Oct.\n2019.\n\n[44] R. I. Bot, E. R. Csetnek, A. Heinrich, and C. Hendrich. On the convergence rate improvement of a\nprimal-dual splitting algorithm for solving monotone inclusion problems. Mathematical Programming,\n150(2):251\u2013279, 2015.\n\n[45] A. Chambolle and T. Pock. On the ergodic convergence rates of a \ufb01rst-order primal\u2013dual algorithm.\n\nMathematical Programming, 159(1-2):253\u2013287, Sept. 2016.\n\n[46] P. Chen, J. Huang, and X. Zhang. A primal-dual \ufb01xed point algorithm for convex separable minimization\n\nwith applications to image restoration. Inverse Problems, 29(2):025011, Jan. 2013.\n\n[47] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. Equation of state\n\ncalculations by fast computing machines. The Journal of Chemical Physics, 21(6):1087\u20131092, 1953.\n\n[48] L. Xiao and S. Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1):\n\n65\u201378, 2004.\n\n[49] S. U. Pillai, T. Suel, and S. Cha. The Perron-Frobenius theorem: Some of its applications. IEEE Signal\n\nProcessing Magazine, 22(2):62\u201375, 2005.\n\n[50] Z. Li and M. Yan. A primal-dual algorithm with optimal stepsizes and its application in decentralized\n\nconsensus optimization. available on arXiv:1711.06785, Nov. 2017.\n\n[51] Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Volume 87, Springer, 2013.\n\n11\n\n\f", "award": [], "sourceid": 1614, "authors": [{"given_name": "Sulaiman", "family_name": "Alghunaim", "institution": "UCLA"}, {"given_name": "Kun", "family_name": "Yuan", "institution": "Alibaba Inc."}, {"given_name": "Ali", "family_name": "Sayed", "institution": "Ecole Polytechnique F\u00e9d\u00e9rale de Lausanne"}]}