{"title": "Computing Linear Restrictions of Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 14132, "page_last": 14143, "abstract": "A linear restriction of a function is the same function with its domain restricted to points on a given line. This paper addresses the problem of computing a succinct representation for a linear restriction of a piecewise-linear neural network. This primitive, which we call ExactLine, allows us to exactly characterize the result of applying the network to all of the infinitely many points on a line. In particular, ExactLine computes a partitioning of the given input line segment such that the network is affine on each partition. We present an efficient algorithm for computing ExactLine for networks that use ReLU, MaxPool, batch normalization, fully-connected, convolutional, and other layers, along with several applications. First, we show how to exactly determine decision boundaries of an ACAS Xu neural network, providing significantly improved confidence in the results compared to prior work that sampled finitely many points in the input space. Next, we demonstrate how to exactly compute integrated gradients, which are commonly used for neural network attributions, allowing us to show that the prior heuristic-based methods had relative errors of 25-45% and show that a better sampling method can achieve higher accuracy with less computation. Finally, we use ExactLine to empirically falsify the core assumption behind a well-known hypothesis about adversarial examples, and in the process identify interesting properties of adversarially-trained networks.", "full_text": "Computing Linear Restrictions of Neural Networks\n\nMatthew Sotoudeh\n\nDepartment of Computer Science\nUniversity of California, Davis\n\nDavis, CA 95616\n\nmasotoudeh@ucdavis.edu\n\nAditya V. Thakur\n\nDepartment of Computer Science\nUniversity of California, Davis\n\nDavis, CA 95616\n\navthakur@ucdavis.edu\n\nAbstract\n\nA linear restriction of a function is the same function with its domain restricted to\npoints on a given line. This paper addresses the problem of computing a succinct\nrepresentation for a linear restriction of a piecewise-linear neural network. This\nprimitive, which we call EXACTLINE, allows us to exactly characterize the result\nof applying the network to all of the in\ufb01nitely many points on a line. In particular,\nEXACTLINE computes a partitioning of the given input line segment such that the\nnetwork is af\ufb01ne on each partition. We present an ef\ufb01cient algorithm for computing\nEXACTLINE for networks that use ReLU, MaxPool, batch normalization, fully-\nconnected, convolutional, and other layers, along with several applications. First,\nwe show how to exactly determine decision boundaries of an ACAS Xu neural net-\nwork, providing signi\ufb01cantly improved con\ufb01dence in the results compared to prior\nwork that sampled \ufb01nitely many points in the input space. Next, we demonstrate\nhow to exactly compute integrated gradients, which are commonly used for neural\nnetwork attributions, allowing us to show that the prior heuristic-based methods\nhad relative errors of 25-45% and show that a better sampling method can achieve\nhigher accuracy with less computation. Finally, we use EXACTLINE to empirically\nfalsify the core assumption behind a well-known hypothesis about adversarial\nexamples, and in the process identify interesting properties of adversarially-trained\nnetworks.\n\n1\n\nIntroduction\n\nThe past decade has seen the rise of deep neural networks (DNNs) [1] to solve a variety of problems,\nincluding image recognition [2, 3], natural-language processing [4], and autonomous vehicle con-\ntrol [5]. However, such models are dif\ufb01cult to meaningfully interpret and check for correctness. Thus,\nresearchers have tried to understand the behavior of such networks. For instance, networks have\nbeen shown to be vulnerable to adversarial examples\u2014inputs changed in a way imperceptible to\nhumans but resulting in a misclassi\ufb01cation by the network [6\u20139]\u2013and fooling examples\u2014inputs that\nare completely unrecognizable by humans but classi\ufb01ed with high con\ufb01dence by DNNs [10]. The\npresence of such adversarial and fooling inputs as well as applications in safety-critical systems has\nled to efforts to verify and certify DNNs [11\u201319]. Orthogonal approaches help visualize the behavior\nof the network [20\u201322] and interpret its decisions [23\u201327]. Despite the tremendous progress, more\nneeds to be done to help understand DNNs and increase their adoption [28\u201331].\nIn this paper, we present algorithms for computing the EXACTLINE primitive: given a piecewise-\nlinear neural network (e.g. composed of convolutional and ReLU layers) and line in the input space\nQR, we partition QR such that the network is af\ufb01ne on each partition. Thus, EXACTLINE precisely\ncaptures the behavior of the network for the in\ufb01nite set of points lying on the line between two points.\nIn effect, EXACTLINE computes a succinct representation for a linear restriction of a piecewise-linear\nneural network; a linear restriction of a function is the same function with its domain restricted\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fto points on a given line. We present an ef\ufb01cient implementation of EXACTLINE (Section 2) for\npiecewise-linear neural networks, as well as examples of how EXACTLINE can be used to understand\nthe behavior of DNNs. In Section 3 we consider a problem posed by Wang et al. [32], viz., to\ndetermine the classi\ufb01cation regions of ACAS Xu [5], an aircraft collision avoidance network, when\nlinearly interpolating between two input situations. This characterization can, for instance, determine\nat which precise distance from the ownship a nearby plane causes the network to instruct a hard\nchange in direction. Section 4 describes how EXACTLINE can be used to exactly compute the\nintegrated gradients [25], a state-of-the-art network attribution method that until now has only\nbeen approximated. We quantify the error of previous heuristics-based methods, and \ufb01nd that they\nresult in attributions with a relative error of 25-45%. Finally, we show that a different heuristic\nusing trapezoidal rule can produce signi\ufb01cantly higher accuracy with fewer samples. Section 5 uses\nEXACTLINE to probe interesting properties of the neighborhoods around test images. We empirically\nreject a fundamental assumption behind the Linear Explanation of Adversarial Examples [7] on\nmultiple networks. Finally, our results suggest that DiffAI-protected [33] neural networks exhibit\nsigni\ufb01cantly less non-linearity in practice, which perhaps contributes to their adversarial robustness.\nWe have made our source code available at https://doi.org/10.5281/zenodo.3520097.\n\n2 The EXACTLINE Primitive\n\nis a linear partitioning of f(cid:22)QR, denoted P(cid:0)f(cid:22)QR\n\nGiven a piecewise-linear neural network f and two points Q, R in the input space of f, we consider\nthe restriction of f to QR, denoted f(cid:22)QR, which is identical to the function f except that its input\ndomain has been restricted to QR. We now want to \ufb01nd a succinct representation for f(cid:22)QR that we\ncan analyze more readily than the neural network corresponding to f. In this paper, we propose to use\nthe EXACTLINE representation, which corresponds to a linear partitioning of f(cid:22)QR, de\ufb01ned below.\nDe\ufb01nition 1. Given a function f : A \u2192 B and line segment QR \u2286 A, a tuple (P1, P2, P3, . . . , Pn)\nif: (1) {PiPi+1 | 1 \u2264 i < n} partitions QR (except for overlap at endpoints); (2) P1 = Q and\nPn = R; and (3) for all 1 \u2264 i < n, there exists an af\ufb01ne map Ai such that f (x) = Ai(x) for all\nx \u2208 PiPi+1.\nIn other words, we wish to partition QR into a set of pieces where the action of f on all points in\n\n(cid:1) and referred to as \u201cEXACTLINE of f over QR,\u201d\n\neach piece is af\ufb01ne. Note that, given P(cid:0)f(cid:22)QR\nf (x) = (1 \u2212 \u03b1)f (Pi) + \u03b1f (Pi+1). In this way, P(cid:0)f(cid:22)QR\n\nfor each partition PiPi+1 can be determined by recognizing that af\ufb01ne maps preserve ratios along\nlines. In other words, given point x = (1 \u2212 \u03b1)Pi + \u03b1Pi+1 on linear partition PiPi+1, we have\n\nrepresentation for the behavior of f on all points along QR.\nConsider an illustrative DNN taking as input the age and income of an individual and returning a\nloan-approval score and premium that should be charged over a baseline amount:\n\n(cid:1) = (P1, . . . , Pn), the corresponding af\ufb01ne function\n(cid:1) provides us a succinct and precise\n(cid:20)\u22121.7\n(cid:1) = (P1 = Q, P2 = (23.3, 36.6),\n\n1.0\n2.0 \u22121.3\n\n(cid:20)3\n(cid:21)\n\n(cid:21)\n\nX +\n\n(1)\n\n3\n\nSuppose an individual of 20 years old making $30k/year (Q = (20, 30)) predicts that their earnings\nwill increase linearly every year until they reach 30 years old and are making $50k/year (R =\n(30, 50)). We wish to understand how they will be classi\ufb01ed by this system over these 10 years.\n\nf (X = (x0, x1)) = ReLU (A(X)) , where A(X) =\n\nP3 = (26.6, 43.3), P4 = R), where f(cid:22)QR is exactly described by the following piecewise-linear\nfunction (Figure 1):\n\nx +\n\n, x \u2208 P2P3\n\n(2)\n\nWe can use EXACTLINE (De\ufb01nition 1) to compute P(cid:0)f(cid:22)QR\n(cid:35)\n(cid:34)\n(cid:35)\n\n(cid:34)\n(cid:34)\u22121.7\n(cid:34)\u22121.7\n\n0\n0\n2 \u22121.3\n\nf(cid:22)QR(x) =\n\n1\n\u22121.3\n\n(cid:35)\n\n(cid:34)\n\n(cid:35)\n\n(cid:35)\n\n(cid:34)\n\n(cid:35)\n\nx +\n\n0\n3\n\n3\n0\n\n,\n\n,\n\n3\n3\n\n\uf8f1\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f3\n\n2\n\n0\n\nx \u2208 QP2\n\nx \u2208 P3R\n\n1\n0\n\nx +\n\n2\n\n\fFigure 1: Computing the linear restriction of f (Equation 1) using EXACTLINE. The input line\nsegment QR is divided into three linear partitions such that the transformation from input space to\noutput space (left plot to right plot) is af\ufb01ne (Equation 2). Tick marks (on the left) are used in \ufb01gures\nthroughout this paper to indicate the partition endpoints (P1, P2, . . . , Pn).\n\nOther Network Analysis Techniques Compared to prior work, our solution to EXACTLINE\npresents an interesting and unique design point in the space of neural-network analysis. Approaches\nsuch as [12, 34, 16] are precise, but exponential time because they work over the entire input domain.\nOn another side of the design space, approaches such as those used in [15, 34, 35, 17, 36, 32] are\nsigni\ufb01cantly faster while still working over the full-dimensional input space, but accomplish this\nby trading analysis precision for speed. This trade-off between speed and precision is particularly\nwell-illuminated by [37], which monotonically re\ufb01nes its analysis when given more time. In con-\ntrast, the key observation underlying our work is that we can perform both an ef\ufb01cient (worst-case\npolynomial time for a \ufb01xed number of layers) and precise analysis by restricting the input domain\nto be one dimensional (a line). This insight opens a new dimension to the discussion of network\nanalysis tools, showing that dimensionality can be traded for signi\ufb01cant gains in both precision and\nef\ufb01ciency (as opposed to prior work which has explored the tradeoff primarily along the precision\nand ef\ufb01ciency axes under the assumption of high-dimensional input regions). ? ] similarly considers\none-dimensional input spaces, but the paper is focused on a number of theoretical properties and does\nnot focus on the algorithm used in their empirical results.\n\nAlgorithm We will \ufb01rst discuss computation of EXACTLINE for individual layers. Note that by\nde\ufb01nition, EXACTLINE for af\ufb01ne layers does not introduce any new linear partitions. This is captured\nby Theorem 1 (proved in Appendix D) below:\nTheorem 1. For any af\ufb01ne function A : X \u2192 Y and line segment QR \u2282 X, the following is a\n\nsuitable linear partitioning (De\ufb01nition 1): P(cid:0)A(cid:22)QR\nThe following theorem (proved in Appendix E) presents a method of computing P(cid:0)ReLU(cid:22)QR\n\n(cid:1) = (Q, R).\n\n(cid:1).\n\nP(cid:0)ReLU(cid:22)QR\n\nTheorem 2. Given a line segment QR in d dimensions and a recti\ufb01ed linear layer ReLU(x) =\n(max(x1, 0), . . . , max(xd, 0)), the following is a suitable linear partitioning (De\ufb01nition 1):\n\n(cid:1) = sorted(cid:0)({Q, R} \u222a {Q + \u03b1(R \u2212 Q) | \u03b1 \u2208 D}) \u2229 QR(cid:1) ,\n\n(3)\nwhere D = {\u2212Qi/(Ri \u2212 Qi) | 1 \u2264 i \u2264 d}, Vi is the ith component of vector V , and sorted returns\na tuple of the points sorted by distance from Q.\n\nThe essential insight is that we can \u201cfollow\u201d the line until an orthant boundary is reached, at which\npoint a new linear region begins. To that end, each number in D represents a ratio between Q and R\nat which QR crosses an orthant boundary. Notably, D actually computes such \u201ccrossing ratios\u201d for\nthe unbounded line QR, hence intersecting the generated endpoints with QR in Equation 3.\nAn analogous algorithm for MaxPool is presented in Appendix F; the intuition is to follow the line\nuntil the maximum in any window changes. When a ReLU layer is followed by a MaxPool layer (or\nvice-versa), the \u201cfused\u201d algorithm described in Appendix G can improve ef\ufb01ciency signi\ufb01cantly. More\ngenerally, the algorithm described in Appendix H can compute EXACTLINE for any piecewise-linear\nfunction.\n\n(cid:1) for entire neural networks (i.e. sequential com-\n\nFinally, in practice we want to compute P(cid:0)f(cid:22)QR\n\npositions of layers), not just individual layers (as we have demonstrated above). The next theorem\n\n3\n\nQ=(20, 30)R=(30, 50)Af(Q)f(P2)f(P3)f(R)Additional PremiumApproval scoreAgeIncomeP2=(23.3, 36.6)P3=(26.6, 43.3)ReLUA(Q)A(R)\f(cid:26)\n\nPi +\n\ni=1\n\nTheorem 3. Given any piecewise-linear functions f, g, h such that f = h \u25e6 g along with a line\n\nshows that, as long as one can compute P(cid:0)Li(cid:22)M N\n(cid:1) for each individual layer Li and arbitrary line\nsegment M N, then these algorithms can be composed to compute P(cid:0)f(cid:22)QR\n(cid:1) for the entire network.\nsegment QR where g(R) (cid:54)= g(Q) and P(cid:0)g(cid:22)QR\n(cid:1) = (P1, P2, . . . , Pn) is EXACTLINE applied to g\n(cid:1)(cid:27)(cid:33)\nP(cid:0)f(cid:22)QR\n\u00d7 (Pi+1 \u2212 Pi) | y \u2208 P(cid:0)h(cid:22)g(Pi)g(Pi+1)\nThe key insight is that we can \ufb01rst compute EXACTLINE for the \ufb01rst layer, i.e. P(cid:0)L1(cid:22)QR\n(cid:1) =\n\nwhere sorted returns a tuple of the points sorted by distance from Q.\n\n(cid:1) = sorted\n\nover QR, the following holds:\n\ng(Pi+1) \u2212 g(Pi)\n\n(cid:32)n\u22121(cid:91)\n\ny \u2212 g(Pi)\n\ni P 1\n\n1 , P 1\n\n2 , . . . , P 1\n\ni+1 individually.\n\nn), then we can continue computing EXACTLINE for the rest of the network within\n\n(P 1\neach of the partitions P 1\nIn Appendix C we show that, over arbitrarily many af\ufb01ne layers, l ReLU layers each with d units, and\nm MaxPool or MaxPool + ReLU layers with w windows each of size s, at most O((d + ws)l+m)\nsegments may be produced. If only ReLU and af\ufb01ne layers are used, at most O(dl) segments\nmay be produced. Notably, this is a signi\ufb01cant improvement over the O((2d)l) upper-bound and\n\u2126(l \u00b7 (2d)) lower-bound of Xiang et al. [34]. One major reason for our improvement is that we\nparticularly consider one-dimensional input lines as opposed to arbitrary polytopes. Lines represent a\nparticularly ef\ufb01cient special case as they are ef\ufb01ciently representable (by their endpoints) and, being\none-dimensional, are not subject to the combinatorial blow-up faced by transforming larger input\nregions. Furthermore, in practice, we have found that the majority of ReLU nodes are \u201cstable\u201d, and\nthe actual number of segments remains tractable; this algorithm for EXACTLINE often executes in a\nmatter of seconds for networks with over 60, 000 units (whereas the algorithm of Xiang et al. [34]\nruns in at least exponential O(l \u00b7 (2d)) time regardless of the input region as it relies on trivially\nconsidering all possible orthants).\n\n3 Characterizing Decision Boundaries for ACAS Xu\n\nThe \ufb01rst application of EXACTLINE we consider is that of understanding the decision boundaries of a\nneural network over some in\ufb01nite set of inputs. As a motivating example, we consider the ACAS Xu\nnetwork trained by Julian et al. [5] to determine what action an aircraft (the \u201cownship\u201d) should take in\norder to avoid a collision with an intruder. After training such a network, one usually wishes to probe\nand visualize the recommendations of the network. This is desirable, for example, to determine at\nwhat distance from the ownship an intruder causes the system to suggest a strong change in heading,\nor to ensure that distance is roughly the same regardless of which side the intruder approaches.\nThe simplest approach, shown in Figure 2f and currently the standard in prior work, is to consider\na (\ufb01nite) set of possible input situations (samples) and see how the network reacts to each of them.\nThis can help one get an overall idea of how the network behaves. For example, in Figure 2f, we\ncan see that the network has a mostly symmetric output, usually advising the plane to turn away\nfrom the intruder when suf\ufb01ciently close. Although sampling in this way gives human viewers an\nintuitive and meaningful way of understanding the network\u2019s behavior, it is severely limited because\nit relies on sampling \ufb01nitely many points from a (practically) in\ufb01nite input space. Thus, there is a\nsigni\ufb01cant chance that some interesting or dangerous behavior of the network may be exposed with\nmore samples.\nBy contrast, the EXACTLINE primitive can be used to exactly determine the output of the network at\nall of the in\ufb01nitely many points on a line in the input region. For example, in Figure 2a, we have used\nEXACTLINE to visualize a particular head-on collision scenario where we vary the distance of the\nintruder (speci\ufb01ed in polar coordinates (\u03c1, \u03b8)) with respect to the ownship (always at (0, 0)). Notably,\nthere is a region of \u201cStrong Left\u201d in a region of the line that is otherwise entirely \u201cWeak Left\u201c that\nshows up in Figure 2a (the EXACTLINE method) but not in Figure 2b (the sampling method). We\ncan do this for lines varying the \u03b8 parameter instead of \u03c1, result in Figure 2c and Figure 2d. Finally,\nrepeating this process for many lines and overlapping them on the same graph produces a detailed\n\u201cgrid\u201d as shown in Figure 2e.\n\n4\n\n\fLegend:\n\nClear-of-Con\ufb02ict, Weak Right,\n\nStrong Right,\n\nStrong Left, Weak Left.\n\n(a) Single line varying \u03c1\n\n(b) Sampling different \u03c1s\n\n(c) Single line varying \u03b8\n\n(d) Sampling different \u03b8s\n\n(e) Combination of lines varying\n\u03c1 and lines varying \u03b8\n\n(f) Sampling \ufb01nitely many points\n\nFigure 2: (a)\u2013(d) Understanding the decision boundary of an ACAS Xu aircraft avoidance network\nalong individual lines using EXACTLINE ((a), (c)) and \ufb01nite sampling ((b), (d)). In the EXACTLINE\nvisualizations there is a clear region of \u201cstrong left\u201d in a region that is otherwise \u201cweak left\u201d that\ndoes not show in the sampling plots due to the sampling density chosen. In practice, it is not clear\nwhat sampling density to choose, thus the resulting plots can be inaccurate and/or misleading. (e)\u2013(f)\nComputing the decision boundaries among multiple lines and plotting on the same graph. Using\nEXACTLINE to sample in\ufb01nitely many points provides more con\ufb01dence in the interpretation of the\ndecision boundaries. Compare to similar \ufb01gures in Julian et al. [5], Katz et al. [12].\n\nFigure 2e also shows a number of interesting and potentially dangerous behaviors. For example, there\nis a signi\ufb01cant region behind the plane where an intruder on the left may cause the ownship to make\na weak left turn towards the intruder, an unexpected and asymmetric behavior. Furthermore, there are\nclear regions of strong left/right where the network otherwise advises weak left/right. Meanwhile,\nin Figure 2f, we see that the sampling density used is too low to notice the majority of this behavior.\nIn practice, it is not clear what sampling density should be taken to ensure all potentially-dangerous\nbehaviors are caught, which is unacceptable for safety-critical systems such as aircraft collision\navoidance.\nTakeaways. EXACTLINE can be used to visualize the network\u2019s output on in\ufb01nite subsets of the\ninput space, signi\ufb01cantly improving the con\ufb01dence one can have in the resulting visualization and in\nthe safety and accuracy of the model being visualized.\nFuture Work. One particular area of future work in this direction is using EXACTLINE to assist in\nnetwork veri\ufb01cation tools such as Katz et al. [12] and Gehr et al. [15]. For example, the relatively-fast\nEXACTLINE could be used to check in\ufb01nite subsets of the input space for counter-examples (which\ncan then be returned immediately) before calling the more-expensive complete veri\ufb01cation tools.\n\n4 Exact Computation of Integrated Gradients\n\nIntegrated Gradients (IG) [25] is a method of attributing the prediction of a DNN to its input features.\nSuppose function F : Rn \u2192 [0, 1] de\ufb01nes the network. The integrated gradient along the ith\ndimension for an input x = (x1, . . . , xn) \u2208 Rn and baseline x(cid:48) \u2208 Rn is de\ufb01ned as:\n\n(cid:90) 1\n\nIGi(x)\n\ndef\n\n= (xi \u2212 x\ni) \u00d7\n(cid:48)\n\n\u2202F (x(cid:48) + \u03b1 \u00d7 (x \u2212 x(cid:48)))\n\n\u03b1=0\n\n\u2202xi\n\nd\u03b1\n\n(4)\n\nThus, the integrated gradient along all input dimensions is the integral of the gradient computed on all\npoints on the straightline path from the baseline x(cid:48) to the input x. In prior work it was not known how\n\n5\n\n5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)\fFigure 3: \u201cIntegrated Gradients\u201d is a powerful method of neural network attribution. IG relies on\ncomputing the integral of the gradients of the network at all points linearly interpolated between two\nimages (as shown above), however previous work has only been able to approximate the true IG,\ncasting uncertainty on the results. Within each partition identi\ufb01ed by EXACTLINE (delineated by\nvertical lines) the gradient is constant, so computing the exact IG is possible for the \ufb01rst time.\n\ndef\n\nm\ni\n\n(cid:48)\n\n(5)\n\n0\u2264k