{"title": "Understanding Weight Normalized Deep Neural Networks with Rectified Linear Units", "book": "Advances in Neural Information Processing Systems", "page_first": 130, "page_last": 139, "abstract": "This paper presents a general framework for norm-based capacity control for $L_{p,q}$ weight normalized deep neural networks. We establish the upper bound on the Rademacher complexities of this family. With an $L_{p,q}$ normalization where $q\\le p^*$ and $1/p+1/p^{*}=1$, we discuss properties of a width-independent capacity control, which only depends on the depth by a square root term. We further analyze the approximation properties of $L_{p,q}$ weight normalized deep neural networks. In particular, for an $L_{1,\\infty}$ weight normalized network, the approximation error can be controlled by the $L_1$ norm of the output layer, and the corresponding generalization error only depends on the architecture by the square root of the depth.", "full_text": "Understanding Weight Normalized Deep Neural\n\nNetworks with Recti\ufb01ed Linear Units\n\nYixi Xu\n\nDepartment of Statistics\n\nPurdue University\n\nWest Lafayette, IN 47907\n\nxu573@purdue.edu\n\nXiao Wang\n\nDepartment of Statistics\n\nPurdue University\n\nWest Lafayette, IN 47907\nwangxiao@purdue.edu\n\nAbstract\n\nThis paper presents a general framework for norm-based capacity control for Lp,q\nweight normalized deep neural networks. We establish the upper bound on the\nRademacher complexities of this family. With an Lp,q normalization where q \u2264 p\u2217\nand 1/p+1/p\u2217 = 1, we discuss properties of a width-independent capacity control,\nwhich only depends on the depth by a square root term. We further analyze the\napproximation properties of Lp,q weight normalized deep neural networks. In\nparticular, for an L1,\u221e weight normalized network, the approximation error can be\ncontrolled by the L1 norm of the output layer, and the corresponding generalization\nerror only depends on the architecture by the square root of the depth.\n\n1\n\nIntroduction\n\nDuring the past decade, deep neural networks (DNNs) have demonstrated an amazing performance in\nsolving many complex arti\ufb01cial intelligence tasks such as object recognition and identi\ufb01cation, text\nunderstanding and translation, question answering, and more [11]. The capacity of unregularized fully\nconnected DNNs, as a function of the network size and depth, is fairly well understood [1, 4, 23]. By\nbounding the L2 norm of the incoming weights of each unit, [22] is able to accelerate the convergence\nof stochastic gradient descent optimization across applications in supervised image recognition,\ngenerative modeling, and deep reinforcement learning. However, theoretical investigations on such\nnetworks are less explored in the literature, and a few exceptions are [4, 5, 10, 18, 19, 25]. There is a\ncentral question waiting for an answer: Can we bound the capacity of fully connected DNNs with\nbias neurons by weight normalization alone, which has the least dependence on the architecture?\nIn this paper, we focus on networks with recti\ufb01ed linear units (ReLU) and study a more general\nweight normalized deep neural network (WN-DNN), which includes all layer-wise Lp,q weight\nnormalizations. In addition, these networks have a bias neuron per hidden layer, while prior stud-\nies [4, 5, 10, 18, 19, 25] either exclude the bias neuron, or only include the bias neuron in the\ninput layer, which differs from the practical application. We establish the upper bound on the\nRademacher complexities of this family and study the theoretical properties of WN-DNNs in terms\nof the approximation error.\nWe \ufb01rst examine how the Lp,q WN-DNN architecture in\ufb02uences their generalization properties.\nSpeci\ufb01cally, for Lp,q normalization where q \u2264 p\u2217 and 1/p + 1/p\u2217 = 1, we obtain a complexity\nbound that is independent of width and only has a square root dependence on the depth. To the\nbest of our knowledge, this is the \ufb01rst theoretical result for the fully connected DNNs including a\nbias neuron for each hidden layer in terms of generalization. We will demonstrate later that it is\nnontrivial to extend the existing results to the DNNs with bias neurons. Even excluding the bias\nneurons, existing generalization bounds for DNNs depend on either width or depth logarithmically\n[5], polynomially[10, 18], or even exponentially [19, 25]. Even for [5], the logarithmic dependency\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fis not always guaranteed, as the margin bound is\n\n\uf8eb\uf8edlog(max d)/\n\n\u221a\n\nn\n\nO\n\n(cid:13)(cid:13)(cid:13)WT\n\ni \u2212 MT\n\ni\n\n(cid:13)(cid:13)(cid:13)2/3\n\n2,1\n\n/(cid:107)Wi(cid:107)2/3\n\u03c3\n\n(cid:33)3/2\uf8f6\uf8f8 ,\n\n(cid:107)Wi(cid:107)\u03c3\n\ni=1\n\nk(cid:89)\n(cid:13)(cid:13)(cid:13)WT\n\n(cid:32) k(cid:88)\n(cid:13)(cid:13)(cid:13)2,1\n\ni=1\n\nwhere (cid:107)\u00b7(cid:107)\u03c3 is the spectral norm, and Mi is a collection of predetermined reference matrix. The\nbound will worsen, when the Wi moves farther from Mi. For example, if\n\nfor some constant A0, then the above bound will rely on the network size by O(cid:0)log(max d)k3/2(cid:1).\n\n/(cid:107)Wi(cid:107)\u03c3 \u2265 A0\n\ni \u2212 MT\n\ni\n\nWe also examine the approximation error of WN-DNNs. It is shown that the L1,\u221e WN-DNN is\nable to approximate any Lipschitz continuous function arbitrarily well by increasing the norm of\nits output layer and growing its size. Early work on neural network approximation theory includes\nthe universal approximation theorem [8, 13, 20], indicating that a fully connected network with a\nsingle hidden layer can approximate any continuous functions. More recent work expands the result\nof shallow networks to deep networks with an increased interest in the expressive power of deep\nnetworks especially for some families of \"hard\" functions [2, 9, 16, 21, 26, 27]. For instance, [26]\nshows that for any positive integer l, there exist neural networks with \u0398(l3) layers and \u0398(1) nodes per\nlayer, which can not be approximated by networks with \u0398(l) layers unless they possess \u2126(2l) nodes.\nThese results on the other hand request for an arti\ufb01cial neural network of which the generalization\nbounds grow slowly with depth and even avoid explicit dependence on depth.\nThe contributions of this paper are summarized as follows.\n\n1. We extend the L2,\u221e weight normalization [22] to the more general Lp,q WN-DNNs and\n2. We include a bias node not only in the input layer but also in every hidden layer. As\n\nrelate these classes to those represented by unregularized DNNs.\n\ndiscussed in Claim 1, it is nontrivial to extend prior research to study this case.\n3. We study the Rademacher complexities of WN-DNNs. Especially, with any Lp,q normaliza-\ntion satisfying that q \u2264 p\u2217, we have a capacity control that is independent of the width and\ndepends on the depth by O(\n\nk).\n\n\u221a\n\n4. We analyze the approximation property of Lp,q WN-DNNs and further show the theoretical\n\nadvantage of L1,\u221e WN-DNNs.\n\nThe paper is organized as follows. In Section 2, we de\ufb01ne the Lp,q WN-DNNs and analyze the\ncorresponding function class. Section 3 gives the Rademacher complexities. In Section 4, we provide\nthe error bounds for the approximation error of Lipschitz continuous functions.\n\n2 Preliminaries\n\nIn this section, we de\ufb01ne the WN-DNNs, of which the weights and biases for all layers are scaled\nby some norm up to a normalization constant c. Furthermore, we demonstrate how it surpasses\nunregularized DNNs theoretically.\nA neural network on Rd0 \u2192 Rdk+1 with k hidden layers is de\ufb01ned by a set of k + 1 af\ufb01ne trans-\nformations T1 : Rd0 \u2192 Rd1, T2 : Rd1 \u2192 Rd2 ,\u00b7\u00b7\u00b7 , Tk+1 : Rdk \u2192 Rdk+1 and the ReLU activation\n\u03c3(u) = (u)+ = uI{u > 0}. The af\ufb01ne transformations are parameterized by Ti(u) = W T\ni u + Bi,\nwhere W i \u2208 Rdi\u22121\u00d7di, Bi \u2208 Rdi for i = 1,\u00b7\u00b7\u00b7 , k + 1. The function represented by this neural\nnetwork is\n\nf (x) = Tk+1 \u25e6 \u03c3 \u25e6 Tk \u25e6 \u00b7\u00b7\u00b7 \u25e6 \u03c3 \u25e6 T1 \u25e6 x\n\nBefore introducing Lp,q WN-DNNs, we build an augmented layer for each hidden layer by appending\nthe bias neuron 1 to the original layer, then combine the weight matrix and the bias vector as a new\nmatrix.\nDe\ufb01ne f\u22170 (x) = (1, xT )T . Then the \ufb01rst hidden layer\n\nf1(x) = T1 \u25e6 x (cid:44) \u02dcV\n\nT\n1 f\u22170 (x),\n\n2\n\n\fwhere \u02dcV1 = (B1, W T\n\n1 )T \u2208 R(d0+1)\u00d7d1. De\ufb01ne the augmented \ufb01rst hidden layer as\n\nf\u22171 (x) = (1, (f1(x))T )T \u2208 Rd1+1.\n\nThen f\u22171 (x) (cid:44) VT\nRd0+1. Sequentially for i = 2,\u00b7\u00b7\u00b7 , k, de\ufb01ne the ith hidden layer as\n\n1 f\u22170 (x), where V1 = (e10, \u02dcV1) \u2208 R(d0+1)\u00d7(d1+1) and e10 = (1, 0,\u00b7\u00b7\u00b7 , 0)T \u2208\n\nfi(x) = Ti \u25e6 \u03c3 \u25e6 fi\u22121(x) (cid:44) (cid:104) \u02dcVi, \u03c3 \u25e6 f\u2217i\u22121(x)(cid:105),\n\nwhere \u02dcVi = (Bi, W T\nThe augmented ith hidden layer is\n\ni )T \u2208 R(di\u22121+1)\u00d7di. Note that \u03c3(1) = 1, thus (1, \u03c3 \u25e6 fi\u22121(x)) = \u03c3 \u25e6 f\u2217i\u22121(x).\n\n(1)\n\n(2)\n\n(3)\n\n(4)\n\nf\u2217i (x) = (1, (fi(x))T )T \u2208 Rdi+1,\n\nand f\u2217i (x) (cid:44) (cid:104)Vi, \u03c3 \u25e6 f\u2217i\u22121(x)(cid:105), where\nand e1i = (1, 0,\u00b7\u00b7\u00b7 , 0)T \u2208 Rdi\u22121+1. The output layer is\n\nVi = (e1i, \u02dcVi) \u2208 R(di\u22121+1)\u00d7(di+1),\n\nf (x) = Tk+1 \u25e6 \u03c3 \u25e6 f\u2217k (x) (cid:44) (cid:104) \u02dcVk+1, \u03c3 \u25e6 f\u2217k (x)(cid:105),\nk+1)T \u2208 R(dk+1)\u00d7dk+1.\n\nwhere \u02dcVk+1 = (Bk+1, W T\nThe Lp, q Norm. The Lp, q norm of a s1 \u00d7 s2 matrix A is de\ufb01ned as\n\n\uf8eb\uf8ed s2(cid:88)\n\n(cid:32) s1(cid:88)\n\nj=1\n\ni=1\n\n(cid:33)q/p\uf8f6\uf8f81/q\n\n,\n\n|aij|p\n\n(cid:107)A(cid:107)p,q =\n\n(cid:18) s1(cid:80)\n\n(cid:19)1/p\n\n(cid:48)\n\n(cid:13)(cid:13)(cid:13)(cid:13) T1\n\n(cid:13)(cid:13)(cid:13)(cid:13)\n\nwhere 1 \u2264 p < \u221e and 1 \u2264 q \u2264 \u221e. When q = \u221e, (cid:107)A(cid:107)p,\u221e\np = q = 2, the Lp,q is the Frobenius norm.\nWe motivate our introduction of WN-DNNs with a negative result when directly applying existing\nstudies on fully connected DNNs with bias neurons.\nA Motivating Example. As shown in Figure 1a, de\ufb01ne f = T2 \u25e6 \u03c3 \u25e6 T1 : R \u2192 R, where\nT1(x) = (\u2212x + 1,\u2212x \u2212 1) (cid:44) \u02dcV\n2 (1, u1, u2)T . Consider\nf\n\n1 (1, x)T and T2(u) = 1 \u2212 u1 \u2212 u2 (cid:44) \u02dcV\n\n100 T1, as shown in Figure 1b . Then\n\n= 100T2 \u25e6 \u03c3 \u25e6 1\n\n|aij|p\n\n. When\n\n= supj\n\ni=1\n\nT\n\nT\n\n(cid:48)\n\n(cid:48)\n\n(x) = 100 \u2212 \u03c3(\u2212x + 1) \u2212 \u03c3(\u2212x \u2212 1) = 99 + f (x)\n\nf\n\nNote that the product of the norms of all layers for f\n\nremains the same as that for f:\n\n,\n\n\u2217\n\n\u2217\n\n(cid:13)(cid:13)(cid:13)\n\n\u2217 (cid:107)T1(cid:107)\n\u2217\n\n(cid:13)(cid:13)(cid:13) \u02dcVi\n\n(cid:107)100T2(cid:107)\n\u2217\n\n= (cid:107)T2(cid:107)\n\u2217\n100\n\u2217\nwhere the norm of the af\ufb01ne transformation (cid:107)Ti(cid:107)\nis de\ufb01ned as the norm of its corresponding linear\n\u2217\ntransformation matrix\nfor i = 1, 2. Using a similar trick, we could replace the 100 in this\nexample with any positive number. This on the other hand suggests an unbounded output even when\nthe product of the norms of all layers is small.\nFurthermore, a negative result will be presented in terms of Rademacher complexity in the following\nclaim.\nClaim 1. De\ufb01ne N k,d\n\u03b3\u2217\u2264\u03b3 as a function class that contains all functions representable by some neural\nnetwork of depth k+1 and widths d: f = Tk+1\u25e6\u03c3\u25e6Tk\u25e6\u00b7\u00b7\u00b7\u25e6\u03c3\u25e6T1\u25e6x, where d = (m1, d1,\u00b7\u00b7\u00b7 , dk, 1),\ni (1, uT )T , for i = 1,\u00b7\u00b7\u00b7 , k + 1, such that\n(cid:107)\u00b7(cid:107)\n\u2217\n\u2264 \u03b3.\nThen for a \ufb01xed n and any sample S = {x1,\u00b7\u00b7\u00b7 , xn} \u2286 Rm1,\n\nis an arbitrary norm, and Ti(u) : Rdi\u22121 \u2192 Rdi = \u02dcV\n\n\u03b3\u2217 =\n\nT\n\n(cid:13)(cid:13)(cid:13) \u02dcVi\n(cid:13)(cid:13)(cid:13)\nk+1(cid:89)\n(cid:98)RS(N k,d\n\u03b3\u2217\u2264\u03b3) = \u221e.\n\ni=1\n\n\u2217\n\n3\n\n\f(a) Visualization of f.\n\n(b) Visualization of f\n\n(cid:48)\n\n.\n\nFigure 1: The motivating example.\n\ni=1\n\nk+1(cid:81)\n\n(cid:107)Wi(cid:107)\n\u2217\n\nClaim 1 shows the failure of current norm-based constraints on fully connected neural networks with\nthe bias neuron in each hidden layer. Prior studies [4, 5, 10, 18, 19, 25] included the bias neuron only\nin the input layer and considered layered networks parameterized by a sequence of weight matrices\nonly, that is Bi = 0 for all i = 1,\u00b7\u00b7\u00b7 , k + 1. While \ufb01xing the architecture of neural networks, these\nworks imply that\nis suf\ufb01cient to control the Rademacher complexity of the function class\nrepresented by these DNNs, where (cid:107)\u00b7(cid:107)\nis the spectral norm in [5, 18], the L1,\u221e norm in [4, 25], the\nL1,\u221e/L2,2 norm in [10], and the Lp,q norm in [19] for any p \u2208 [1,\u221e), q \u2208 [1,\u221e]. However, this\n\u2217\nkind of control fails once the bias neuron is added to each hidden layer, demonstrating the necessity\nto use WN-DNNs instead.\nThe Lp,q WN-DNNs. An Lp,q WN-DNN by a normalization constant c \u2265 1 with k hidden layers\nis de\ufb01ned by a set of k + 1 af\ufb01ne transformations T1 : Rd0 \u2192 Rd1, T2 : Rd1 \u2192 Rd2,\u00b7\u00b7\u00b7 , Tk+1 :\nRdk \u2192 Rdk+1 and the ReLU activation, where Ti(u) = \u02dcV\ni (1, uT )T , \u02dcVi \u2208 R(di\u22121+1)\u00d7di and\n(cid:107)Ti(cid:107)p,q\nDe\ufb01ne N k,d\nwith the normalization constant c satisfying:\n\n, for i = 1,\u00b7\u00b7\u00b7 , k + 1. In addition, (cid:107)Ti(cid:107)p,q \u2261 c for i = 1,\u00b7\u00b7\u00b7 , k.\n\np,q,c,co as the collection of all functions that could be represented by an Lp,q WN-DNN\n\n(cid:44)(cid:13)(cid:13)(cid:13) \u02dcVi\n\n(cid:13)(cid:13)(cid:13)p,q\n\nT\n\n(a) The number of neurons in the ith hidden layer is di for i = 1, 2,\u00b7\u00b7\u00b7 , k. The dimension of\n\ninput is d0, and output dk+1;\n\n(b) It has k hidden layers;\n(c) (cid:107)Ti(cid:107)p,q \u2261 c for i = 1,\u00b7\u00b7\u00b7 , k;\n(d) (cid:107)Tk+1(cid:107)p,q \u2264 co.\n\nThe following theorem provides some useful observations regarding N k,d\nTheorem 1. Let c, co, c1, c2, c1\n(d0, d1 \u00b7\u00b7\u00b7 , dk+1) \u2208 Nk+2\nNk2+2\n\no > 0, p \u2208 [1,\u221e), q \u2208 [1,\u221e], k, k1, k2 \u2208 N, d =\nk2+1) \u2208\n0, d1\n\n.\n(a) A function f : Rd0 \u2192 Rdk+1 = Tk+1 \u25e6 \u03c3 \u25e6 Tk \u25e6 \u00b7\u00b7\u00b7 \u25e6 \u03c3 \u25e6 T1 \u25e6 x, where Ti(u) =\n, as long as (cid:107)Ti(cid:107)p,q \u2264 c for i = 1,\u00b7\u00b7\u00b7 , k\n\ni u + Bi : Rdi\u22121 \u2192 Rdi. Then f \u2208 N k,d\n\nk1+1) \u2208 Nk1+2\n\n+ , d1 = (d1\n\n, and d2 = (d2\n\n1 \u00b7\u00b7\u00b7 , d2\n\n1 \u00b7\u00b7\u00b7 , d1\n\np,q,c,co.\n\n0, d2\n\no, c2\n\np,q,c,co\n\n+\n\n+\n\nW T\nand (cid:107)Tk+1(cid:107)p,q \u2264 co.\n\u2286 N k,d\n\np,q,c1,co\n\n(b) N k,d\n\ncog \u2208 N k,d\n\np,q,c,co\n\np,q,c2,co\n.\n\nif c1 \u2264 c2. N k,d\n\np,q,c,c1\no\n\n\u2286 N k,d\n\np,q,c,c2\no\n\nif c1\n\no \u2264 c2\n\no. If g \u2208 N k,d\n\np,q,c,1, then\n\n(c) N k,d\n\np1,q,c,co\n\n\u2286 N k,d\n\np2,q,c,co\n\nif 1 \u2264 p1 \u2264 p2 < \u221e. N k,d\n\np,q1,c,co\n\n\u2286 N k,d\n\np,q2,c,co\n\nif 1 \u2264 q1 \u2264 q2 \u2264 \u221e.\n\n4\n\n1x11\u221211\u22121y1\u22121\u221211x10.01\u22120.010.01\u22120.01y100\u2212100\u2212100\fN k,d\np,\u221e,c,co\nwhen dk+1 = 1, \u02dcco = co.\n\n\u2286 N k,d\n\np,q,\u02dcc,\u02dcco\n\n, where \u02dcc = c max\n\n1\n\nq {d1, d2 \u00b7\u00b7\u00b7 , dk} and \u02dcco = d\n\n1\nq\n\nk+1co. Especially,\n\n(d) N k1,d1\n\n\u2286 N k2,d2\nfor i > k1, and d2\n\np,q,c,co\n\np,q,c,co\n\nk2+1 = d1\n\nk1+1 = 1.\n\nif c \u2265 1, k1 \u2264 k2, d2\n\n0 = d1\n\n0, d2\n\ni \u2265 d1\n\ni for i = 1,\u00b7\u00b7\u00b7 , k1, d2\n\ni \u2265 d1\n\nk1+1\n\nIn particular, Part (a) connects normalized neural networks to unregularized DNNs. Part (b) shows\nthe increased expressive power of neural networks by increasing the normalization constant or the\noutput layer norm constraint. Part (c) discusses the in\ufb02uence of the choice of Lp,q normalization on\nits representation capacity. Part (d) describes the gain in representation power by either widening or\ndeepening the neural networks.\n\n3 Estimating the Rademacher Complexities of N k,d\n\np,q,c,co\n\nIn this section, we bound the Rademacher complexities of N k,d\np,q,c,co, where d0 = m1 and dk+1 = 1.\nWithout loss of generality, assume the input space X = [\u22121, 1]m1 in the following sections. Further\nde\ufb01ne p\u2217 by 1/p + 1/p\u2217 = 1.\nProposition 1. Fix q \u2265 1, k \u2265 0, c, co > 0, di \u2208 N+ for i = 1,\u00b7\u00b7\u00b7 , k, then for any set S =\n{x1,\u00b7\u00b7\u00b7 , xn} \u2286 X , we have\n\n(cid:98)RS(N k,d\n\n1,q,c,co) \u2264 co\u221a\n(cid:112)\n\nn\n\n(cid:16)\nk(cid:88)\n\ni=0\n\n(cid:112)\n\n(cid:112)\n\nmin\n\n2 max(1, ck)\n\nk + 2 + log(m1 + 1),\n\n(cid:33)\n\n(cid:112)\n\nk log 16\n\nci + ck(\n\n2 log(2m1) +\n\nk log 16)\n\n.\n\nProof sketch. As \u03c3(1) = 1, we could treat the bias neuron in the ith hidden layer as a hidden neuron\ncomputed from the (i \u2212 1)th hidden layer by\n\u03c3(eT\n\n1if\u2217i\u22121(x)) = 1,\n\nwhere e1i = (1, 0,\u00b7\u00b7\u00b7 , 0)T \u2208 Rdi\u22121+1, and f\u2217i\u22121 is the augmented (i \u2212 1)th hidden layer as de\ufb01ned\nin Equation (2). Therefore, the new af\ufb01ne transformation could be parameterized by Vi de\ufb01ned in\nEquation (3), such that (cid:107)Vi(cid:107)1,\u221e\n= max(1, c). Then the result is the minimum of the bound of [10,\nTheorem 2] on DNNs without bias neurons and that of Proposition 2 when p = 1.\nProposition 2. Fix p, q \u2265 1, k \u2265 0, c, co > 0, di \u2208 N+ for i = 1,\u00b7\u00b7\u00b7 , k, then for any set\nS = {x1,\u00b7\u00b7\u00b7 , xn} \u2286 X , we have\n\nk(cid:89)\n\n(cid:32)k+1(cid:88)\n(cid:104)\n\ni=1\n\n(k + 1) log 16\n\nn\n\nq ]+\n\np\u2217 \u2212 1\n[ 1\ni\n\nd\n\n1\n\np\u2217\n1\n\nm\n\nck\u2212i+1\n\n(cid:16)\n\nl=i\n\n(cid:112)\np\u2217 \u2212 1,\n\n(cid:112)\n\nmin\n\n(\n\n[ 1\np\u2217 \u2212 1\nd\nl\n\nq ]+\n\n+\n\n2 log(2m1)\n\n(cid:17)\n\n(cid:112)\n\n+\n\n(k + 1) log 16\n\n(cid:105)\n\n,\n\n(cid:33)\n\n+\n\nck\u2212i+1\n\n[ 1\np\u2217 \u2212 1\nd\nl\n\nq ]+\n\n(cid:32)k+1(cid:88)\n(cid:16)(cid:112)\n\ni=1\n\n1\n\np\u2217\nm\n1\n\nk(cid:89)\n\nl=i\n\n2 log(2m1) +\n\n(k + 1) log 16\n\n(5)\n\n(6)\n\n(cid:17)\n\n.\n\n(cid:33)\n\n(cid:112)\n\n(a) for p \u2208 (1, 2],\n\n(cid:98)RS(N k,d\n\np,q,c,co) \u2264 co\n\ncock\u221a\nn\n\n(cid:114)\nk(cid:89)\n\ni=1\n\n(b) for p \u2208 1 \u222a (2,\u221e),\n\n(cid:98)RS(N k,d\n\np,q,c,co) \u2264 co\n\n(cid:114)\n\n1\u221a\nn\n\ncock\n\n(k + 1) log 16\n\nk(cid:89)\n\nn\n\np\u2217 \u2212 1\n[ 1\nd\ni\n\nq ]+\n\ni=1\n\n5\n\n\f(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n(cid:88)\n\ni=1\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)p\u2217\n\n,\n\n\u0001i\u03c3 \u25e6 fj(xi)\n\nZj =\n\nsup\nf\u2208N k,d\n\np,q,c,co\n\n(cid:32)\n\nE\u0001 exp(tZj) \u2264 4j exp\n\nt2ns2\nj\n\n2\n\n+ tcj\n\n[ 1\np\u2217 \u2212 1\nd\ni\n\nq ]+\n\nwhere\n\nsj =\n\nj(cid:88)\n\ni=2\n\nj(cid:89)\n\nl=i\n\ncj\u2212i+1\n\nq ]+\n\n[ 1\np\u2217 \u2212 1\nl\n\nd\n\n\u2217\n\n+ (m1/p\n\n1 + 1)cj\n\nj(cid:89)\n\ni=1\n\n(cid:33)\n\nAp\n\nm1,S\n\n,\n\nq ]+\n\n[ 1\np\u2217 \u2212 1\nl\n\n,\n\nd\n\nj(cid:89)\n\nl=1\n\n(cid:33)\uf8f6\uf8f8 .\n\n\u0001if (xi)\n\n(cid:32) n(cid:88)\n\ni=1\n\nProof sketch. The proof consists of two steps. In the \ufb01rst step, following the notations in Section 2,\nwe de\ufb01ne a series of random variables\n\nwhere {\u00011,\u00b7\u00b7\u00b7 , \u0001n} are n i.i.d Rademacher random variables, and fj is the jth hidden layer of the\nneural network f. We prove by induction that for any t \u2208 R,\n\nand Ap\nm1,S is some constant only depends on the sample. In addition, we relies on H\u00f6lder\u2019s inequality\nwith an optimal parameter to separate the bias neuron. Step 2 is motivated by the idea of [10]. By\nJensen\u2019s inequality\n\nn(cid:98)RS(N k,d\n\nsup\nf\u2208N k,d\nFinally we get the desired result by choosing the optimal \u03bb.\n\np,q,c,co\n\np,q,c,co) \u2264 1\n\u03bb\n\nlog E\u0001 exp\n\n\uf8eb\uf8ed\u03bb\n\nWhen d = d1, the upper bound of Rademacher complexity depends on the width by O(dk[ 1\nq ]+),\nwhich is similar to the case without bias neurons [19]. Furthermore, the dependence on widths\ndisappears as long as q \u2208 [1, p\u2217].\nIn order to investigate the tightness of the bound given in\nProposition 2, we consider the binary classi\ufb01cation as a speci\ufb01c case, indicating that when 1\nq < 1,\nthe dependence on width is unavoidable.\nProposition 3. [19, Theorem 3] For any p, q \u2265 1, d = d1 and any k \u2265 2, n {\u22121, +1} points could\nbe shattered with unit margin by N k,d\n\np + 1\n\np\u2217 \u2212 1\n\n, with\n\np,q,c,co\n\nckco \u2264 (log2 n)\n\n1\n\np n( 1\n\np + 1\n\nq )d\u2212(k\u22122)[ 1\n\np\u2217 \u2212 1\n\nq ]+.\n\nk\n\nk2\n\nIssues on Bias Neurons. Lp,q norm-constrained fully connected DNNs with no bias neuron were\ninvestigated in prior studies [4, 10, 19, 25]. First of all, the generalization bounds given by [4, 19, 25]\nhave explicit exponential dependence on the depth, thus it is not meaningful to compare these results\nwith ours. Secondly, [10] provides the up-to-date Rademacher complexity bounds of both L1,\u221e and\nL2,2 norm-constrained fully connected DNNs without bias neurons. However, it is not straightforward\nto extend their results to fully connected DNNs with a bias neuron in each hidden layer. For example,\nconsider the L2,2 WN-DNNs with c = 1. If we simply treat each bias neuron as a hidden neuron, as\n\u221a\nin the proof for Proposition 1, the complexity bounds [10] grows exponentially with respect to the\ndepth by O(\n2 ), while our Proposition 2 gives a much tighter bound O(k\nand L2,2 WN-DNNs.\nComparison with [10] on the Rademacher compexity bounds of L1,\u221e\n[10] is the most recent work on the Rademacher complexities of the L1,\u221e and L2,2 norm-constrained\nfully connected DNNs without bias neurons. Consider a speci\ufb01c case when log(m1) is small and\nco = 1 to shed light on the possible in\ufb02uence of the bias neurons on the generalization properties.\nAs summarized in Table 1, these comparisons suggest that the inclusion of a bias neuron in each\nhidden layer might lead to extra dependence of generalization bounds on the depth especially when c\nkck \u2192 0, as k \u2192 \u221e. For\nis small. Note that, when c < 1,\n\u221ak\u221an ) without bias\nL2,2 WN-DNNs, when c = 1, the bounds are O( k\n) if including bias neurons\nneurons. For L2,2 WN-DNNs, when c > 1, the bounds are O(\n\u221akck\nand O(\n\u221an ) if excluding bias neurons. Another interesting observation is that the complexity\nbounds remain the same no matter whether bias neurons are included or not, when c > 1 for L1,\u221e\nWNN-DNNs.\n\nk(1 \u2212 ck+1)/(1 \u2212 c) \u2192 \u221e, while\n\n2\u221an ) if with bias neurons and O(\n\n\u221ak(ck+1\u22121)\n\n\u221a\n\n\u221a\n\n2 ).\n\n\u221an\n\n3\n\n3\n\n6\n\n\fWith Bias Neurons Without Bias Neurons [10]\n\nc < 1\n\nO(\n\nc = 1, L1,\u221e\nc = 1, L2,2\nc > 1, L1,\u221e\nc > 1, L2,2\n\nO(\n\n\u221ak(1\u2212ck+1)\n(1\u2212c)\u221an )\n\u221ak\u221an )\nO(\nO( k3/2\n\u221an )\n\u221akck\n\u221an )\nO(\n\u221ak(ck+1\u22121)\n\u221an\n\n)\n\n\u221akck\n\u221an )\n\u221ak\u221an )\n\u221ak\u221an )\n\u221akck\n\u221an )\n\u221akck\n\u221an )\n\nO(\n\nO(\n\nO(\n\nO(\n\nO(\n\nTable 1: Rademacher complexity bounds for L1,\u221e/L2,2 WN-DNNs with/without bias neurons.\n\n4 Approximation Properties\n\nr(cid:88)\n\nIn this section, we analyze the approximation properties of Lp,q WN-DNNs and show the theoretical\nadvantage of L1,\u221e WN-DNN. We \ufb01rst introduce a technical lemma, demonstrating that any wide\none-hidden-layer neural network could be exactly represented by a deep but narrow normalized\nneural network. In addition, Lemma 1 indicates that N 1,(m1,r,1)\nfor\nany r > 1, k \u2208 N , and co > 0, where [x] is the smallest integer which is greater than or equal to x,\n1,\u221e,\u00b7,co\nand 1k = (1,\u00b7\u00b7\u00b7 , 1) \u2208 Rk.\nLemma 1. Assume that a function\n\n\u2286 N k,(m1,([r/k]+2m1+3)1k,1)\n\np,\u221e,1,2co\n\nr(cid:80)\n\ni=1\n\nsatis\ufb01es that\n\ng(x) : Rm1 \u2192 R =\n\n|ci| \u2264 co and(cid:13)(cid:13)(bi, wT\n\ni )(cid:13)(cid:13)1 = 1. Then for any integer k \u2208 [1, r],\n\ni=1\n\nci\u03c3(wT\n\ni x + bi)\n\nwhere widk = [r/k] + 2m1 + 3, dk\n\n0 = m1, dk\n\ni = widk for i = 1,\u00b7\u00b7\u00b7 , k, and dk\n\nk+1 = 1.\n\ng \u2208 N k,dk\n\np,q,wid1/q\n\nk ,2co\n\n,\n\nProof sketch. Note that the shallow neural network g could be decomposed as\n\ni , c\u2212i > 0 and r1 + r2 = r. We consider a simpli\ufb01ed case when g(x) =\n\ni\n\ni=1\n\nc+\n\n(cid:1) ,\n\ni )T x + b+\n\n(cid:1) \u2212 r2(cid:88)\n\nc\u2212i \u03c3(cid:0)(w\u2212i )T x + b\u2212i\n\nr1(cid:88)\ni \u03c3(cid:0)(w+\n(cid:1) to illustrate the main idea of our proof. Without loss of generality, as-\ni )(cid:13)(cid:13)1 = 1. First create a set\nC = {\u03c3(cid:0)(w+\n\n(cid:1) , i = 1,\u00b7\u00b7\u00b7 , r1}.\n\ni )T x + b+\n\ni=1\n\ni\n\nwhere c+\n\ni \u03c3(cid:0)(w+\nr1(cid:80)\nsume that(cid:13)(cid:13)(bi, 2wT\n\nc+\n\ni=1\n\ni )T x + b+\n\ni\n\nIn order to build a k + 1-layer WN-DNN to represent g, we partition C into k equally sized subsets:\nC1,\u00b7\u00b7\u00b7 ,Ck. The key idea is to get all elements of Cj in the jth hidden layer for j = 1,\u00b7\u00b7\u00b7 , k, while\nkeeping both \u03c3\u25e6x, and \u03c3\u25e6\u2212x. In addition, the normalized cumulative sum Sj of \u222ai\u2264jCi is computed\nin the j + 1th hidden layer. More speci\ufb01cally,\n\njr1/k(cid:80)\n\ni \u03c3(cid:0)(w+\njr1/k(cid:80)\n\nc+\n\nc+\ni\n\ni=1\n\nSj =\n\ni=1\n\n(cid:1)\n\n.\n\ni )T x + b+\n\ni\n\nNote that\n\n(w+\n\ni )T x + b+\n\ni = (w+\n\ni )T \u03c3 \u25e6 x \u2212 (w+\n\ni )T \u03c3 \u25e6 (\u2212x) + b+\ni ,\n\n7\n\n\fand\n\n(j\u22121)r1/k(cid:80)\njr1/k(cid:80)\n\ni=1\n\njr1/k(cid:88)\n\n\u03c3(cid:0)(w+\n\n(cid:1) .\n\ni\n\ni=1\n\ni=1\n\nc+\ni\n\nc+\ni\n\nc+\ni\n\nSj =\n\ni )T x + b+\n\ni=(j\u22121)r1/k+1\n\n\u03c3(Sj\u22121) +\n\njr1/k(cid:80)\nc+\ni\nThus the L1,\u221e norm of the corresponding transformation still \u2264 1.\nBased on Lemma 1, we establish that a WN-DNN is able to approximate any Lipschitz-continuous\nfunction arbitrarily well by loosing the constraint for the norm of the output layer and either widening\nor deepening the neural network at the same time. Especially, for Lp,\u221e WN-DNNs, the approximation\nerror could be purely controlled by the norm of the output layer, while the Lp,\u221e norm of each hidden\nlayer is \ufb01xed to be 1.\n\u2264 L, and |f (x) \u2212 f (y)| \u2264\nTheorem 2. f\nq \u2208 [1,\u221e], and any integer k \u2208\nL(cid:107)x \u2212 y(cid:107)\n\u221e\n], if co greater than a constant depending\n[1, Cr(m1)(log co\nonly on m1, there exists a function h \u2208 N k,dk\n\nL )\u22122(m1+1)/(m1+4)(cid:0) co\n\n: X \u2192 R,\nThen for any p \u2208 [1,\u221e),\n\nsatisfying that (cid:107)f(cid:107)\n\n, where\n\n\u221e\n\nL\n\n.\n\n(cid:1)2(m1+3)/(m1+4)\n(cid:16) co\n\np,q,wid1/q\n\nk ,2co\n\n)\u2212 2(m1+1)\n\nm1+4\n\nco\nL\n\n(cid:17) 2(m1+3)\n\nm1+4\n\nL\n\n] + 2m1 + 3,\n\nwidk = [k\u22121Cr(m1)(log\ndk = (m1, widk,\u00b7\u00b7\u00b7 , widk, 1), such that\n\n|f (x) \u2212 h(x)| \u2264 C(m1)L(\n\n)\u2212 2\n\nm1+1 log\n\nco\nL\n\nco\nL\n\n,\n\nsup\n\n(cid:107)x(cid:107)\u221e\u22641\n\nwhere Cr(m1) and C(m1) denotes some constant that depends only on m1.\n\nTheorem 2 shows that the approximation bounds could be controlled by co given a suf\ufb01ciently deep\nor wide Lp,q WN-DNN. Assume that the loss function is 1-Lipschitz continuous, then the dependence\nof the corresponding generalization bound on the architecture for each N k,dk\nde\ufb01ned above\nare summarized as follows:\n(a) p = 1, q = \u221e: O\n(b) p = 1, q < \u221e: O\n(c) p > 1, q \u2208 (p\u2217,\u221e]: O\n(d) p > 1, q \u2208 [1, p\u2217]: O\n\n(cid:17)\n(cid:16)\u221a\n(cid:18)\u221a\n(cid:16)\u221a\n(cid:16)\u221a\n\np\u2217 ]k(cid:17)\nq ]k(cid:17)\n\nkco[(1 + widk)\n\nk\nq\nkcowid\nk\n\nkco[(1 + widk)\n\np,q,wid1/q\n\n(cid:19)\n\nk ,2co\n\nkco\n\n;\n\n;\n\n;\n\n.\n\n1\n\n1\n\n5 Concluding Remarks\n\nWe present a general framework for capacity control on WN-DNNs. In particular, we provide a\nsatisfying answer for the central question: we obtain the generalization bounds for L1,\u221e WN-DNNs\nthat grows with depth by a square root term while getting the approximation error controlled. It will\nbe interesting to extend this work to mullticlass classi\ufb01cation. However, if handling via Radermacher\ncomplexity analysis, the generalization bound will depend on the square root of the number of classes\n[28]. Besides the extension to convolutional neural networks, we are also working on the design of\neffective algorithms for L1,\u221e WN-DNNs.\nAcknowledgments\n\nWe thank the anonymous reviewers for their careful reading of our manuscript and their insightful\ncomments that have greatly improved the paper.\n\n8\n\n\fReferences\n[1] Martin Anthony and Peter L Bartlett. Neural network learning: Theoretical foundations.\n\ncambridge university press, 2009.\n\n[2] Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep\nneural networks with recti\ufb01ed linear units. In International Conference on Learning Represen-\ntations, 2018.\n\n[3] Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of\n\nMachine Learning Research, 18(19):1\u201353, 2017.\n\n[4] Peter L Bartlett. The sample complexity of pattern classi\ufb01cation with neural networks: the size\nof the weights is more important than the size of the network. IEEE transactions on Information\nTheory, 44(2):525\u2013536, 1998.\n\n[5] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds\nfor neural networks. In Advances in Neural Information Processing Systems, pages 6241\u20136250,\n2017.\n\n[6] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds\n\nand structural results. Journal of Machine Learning Research, 3:463\u2013482, 2002.\n\n[7] St\u00e9phane Boucheron, G\u00e1bor Lugosi, and Olivier Bousquet. Concentration inequalities. In\n\nSummer School on Machine Learning, pages 208\u2013240, 2003.\n\n[8] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of\n\ncontrol, signals and systems, 2(4):303\u2013314, 1989.\n\n[9] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In\n\nConference on Learning Theory, pages 907\u2013940, 2016.\n\n[10] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of\n\nneural networks. In Proceedings of the 31st Conference On Learning Theory, 2018.\n\n[11] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.\n\nMIT press Cambridge, 2016.\n\n[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\nrecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 770\u2013778, 2016.\n\n[13] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks,\n\n4(2):251\u2013257, 1991.\n\n[14] Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction:\nRisk bounds, margin bounds, and regularization. In Advances in Neural Information Processing\nSystems, pages 793\u2013800, 2009.\n\n[15] Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and\n\nprocesses. Springer Science & Business Media, 2013.\n\n[16] Shiyu Liang and R Srikant. Why deep neural networks for function approximation?\n\nInternational Conference on Learning Representations, 2017.\n\nIn\n\n[17] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning.\n\nMIT press, 2012.\n\n[18] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to\nspectrally-normalized margin bounds for neural networks. In International Conference on\nLearning Representations, 2018.\n\n[19] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural\n\nnetworks. In Conference on Learning Theory, pages 1376\u20131401, 2015.\n\n9\n\n\f[20] Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica,\n\n8:143\u2013195, 1999.\n\n[21] Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with\nneural networks. In Proceedings of the 34th International Conference on Machine Learning,\nvolume 70 of Proceedings of Machine Learning Research, pages 2979\u20132987, International\nConvention Centre, Sydney, Australia, 2017. PMLR.\n\n[22] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to\naccelerate training of deep neural networks. In Advances in Neural Information Processing\nSystems, pages 901\u2013909, 2016.\n\n[23] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to\n\nalgorithms. Cambridge university press, 2014.\n\n[24] Shai Shalev-Shwartz and Yoram Singer. A primal-dual perspective of online learning algorithms.\n\nMachine Learning, 69(2-3):115\u2013142, 2007.\n\n[25] Shizhao Sun, Wei Chen, Liwei Wang, Xiaoguang Liu, and Tie-Yan Liu. On the depth of deep\n\nneural networks: A theoretical view. In AAAI, pages 2066\u20132072, 2016.\n\n[26] Matus Telgarsky. Bene\ufb01ts of depth in neural networks. In 29th Annual Conference on Learning\nTheory, volume 49 of Proceedings of Machine Learning Research, pages 1517\u20131539, Columbia\nUniversity, New York, New York, USA, 2016. PMLR.\n\n[27] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks,\n\n94:103\u2013114, 2017.\n\n[28] Tong Zhang. Statistical analysis of some multi-category large margin classi\ufb01cation methods.\n\nJournal of Machine Learning Research, 5:1225\u20131251, 2004.\n\n10\n\n\f", "award": [], "sourceid": 95, "authors": [{"given_name": "Yixi", "family_name": "Xu", "institution": "Purdue University"}, {"given_name": "Xiao", "family_name": "Wang", "institution": "Purdue University"}]}