{"title": "Computing Linear Restrictions of Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 14132, "page_last": 14143, "abstract": "A linear restriction of a function is the same function with its domain restricted to points on a given line. This paper addresses the problem of computing a succinct representation for a linear restriction of a piecewise-linear neural network. This primitive, which we call ExactLine, allows us to exactly characterize the result of applying the network to all of the infinitely many points on a line. In particular, ExactLine computes a partitioning of the given input line segment such that the network is affine on each partition. We present an efficient algorithm for computing ExactLine for networks that use ReLU, MaxPool, batch normalization, fully-connected, convolutional, and other layers, along with several applications. First, we show how to exactly determine decision boundaries of an ACAS Xu neural network, providing significantly improved confidence in the results compared to prior work that sampled finitely many points in the input space. Next, we demonstrate how to exactly compute integrated gradients, which are commonly used for neural network attributions, allowing us to show that the prior heuristic-based methods had relative errors of 25-45% and show that a better sampling method can achieve higher accuracy with less computation. Finally, we use ExactLine to empirically falsify the core assumption behind a well-known hypothesis about adversarial examples, and in the process identify interesting properties of adversarially-trained networks.", "full_text": "Computing Linear Restrictions of Neural Networks\n\nMatthew Sotoudeh\n\nDepartment of Computer Science\nUniversity of California, Davis\n\nDavis, CA 95616\n\nmasotoudeh@ucdavis.edu\n\nAditya V. Thakur\n\nDepartment of Computer Science\nUniversity of California, Davis\n\nDavis, CA 95616\n\navthakur@ucdavis.edu\n\nAbstract\n\nA linear restriction of a function is the same function with its domain restricted to\npoints on a given line. This paper addresses the problem of computing a succinct\nrepresentation for a linear restriction of a piecewise-linear neural network. This\nprimitive, which we call EXACTLINE, allows us to exactly characterize the result\nof applying the network to all of the in\ufb01nitely many points on a line. In particular,\nEXACTLINE computes a partitioning of the given input line segment such that the\nnetwork is af\ufb01ne on each partition. We present an ef\ufb01cient algorithm for computing\nEXACTLINE for networks that use ReLU, MaxPool, batch normalization, fully-\nconnected, convolutional, and other layers, along with several applications. First,\nwe show how to exactly determine decision boundaries of an ACAS Xu neural net-\nwork, providing signi\ufb01cantly improved con\ufb01dence in the results compared to prior\nwork that sampled \ufb01nitely many points in the input space. Next, we demonstrate\nhow to exactly compute integrated gradients, which are commonly used for neural\nnetwork attributions, allowing us to show that the prior heuristic-based methods\nhad relative errors of 25-45% and show that a better sampling method can achieve\nhigher accuracy with less computation. Finally, we use EXACTLINE to empirically\nfalsify the core assumption behind a well-known hypothesis about adversarial\nexamples, and in the process identify interesting properties of adversarially-trained\nnetworks.\n\n1\n\nIntroduction\n\nThe past decade has seen the rise of deep neural networks (DNNs) [1] to solve a variety of problems,\nincluding image recognition [2, 3], natural-language processing [4], and autonomous vehicle con-\ntrol [5]. However, such models are dif\ufb01cult to meaningfully interpret and check for correctness. Thus,\nresearchers have tried to understand the behavior of such networks. For instance, networks have\nbeen shown to be vulnerable to adversarial examples\u2014inputs changed in a way imperceptible to\nhumans but resulting in a misclassi\ufb01cation by the network [6\u20139]\u2013and fooling examples\u2014inputs that\nare completely unrecognizable by humans but classi\ufb01ed with high con\ufb01dence by DNNs [10]. The\npresence of such adversarial and fooling inputs as well as applications in safety-critical systems has\nled to efforts to verify and certify DNNs [11\u201319]. Orthogonal approaches help visualize the behavior\nof the network [20\u201322] and interpret its decisions [23\u201327]. Despite the tremendous progress, more\nneeds to be done to help understand DNNs and increase their adoption [28\u201331].\nIn this paper, we present algorithms for computing the EXACTLINE primitive: given a piecewise-\nlinear neural network (e.g. composed of convolutional and ReLU layers) and line in the input space\nQR, we partition QR such that the network is af\ufb01ne on each partition. Thus, EXACTLINE precisely\ncaptures the behavior of the network for the in\ufb01nite set of points lying on the line between two points.\nIn effect, EXACTLINE computes a succinct representation for a linear restriction of a piecewise-linear\nneural network; a linear restriction of a function is the same function with its domain restricted\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fto points on a given line. We present an ef\ufb01cient implementation of EXACTLINE (Section 2) for\npiecewise-linear neural networks, as well as examples of how EXACTLINE can be used to understand\nthe behavior of DNNs. In Section 3 we consider a problem posed by Wang et al. [32], viz., to\ndetermine the classi\ufb01cation regions of ACAS Xu [5], an aircraft collision avoidance network, when\nlinearly interpolating between two input situations. This characterization can, for instance, determine\nat which precise distance from the ownship a nearby plane causes the network to instruct a hard\nchange in direction. Section 4 describes how EXACTLINE can be used to exactly compute the\nintegrated gradients [25], a state-of-the-art network attribution method that until now has only\nbeen approximated. We quantify the error of previous heuristics-based methods, and \ufb01nd that they\nresult in attributions with a relative error of 25-45%. Finally, we show that a different heuristic\nusing trapezoidal rule can produce signi\ufb01cantly higher accuracy with fewer samples. Section 5 uses\nEXACTLINE to probe interesting properties of the neighborhoods around test images. We empirically\nreject a fundamental assumption behind the Linear Explanation of Adversarial Examples [7] on\nmultiple networks. Finally, our results suggest that DiffAI-protected [33] neural networks exhibit\nsigni\ufb01cantly less non-linearity in practice, which perhaps contributes to their adversarial robustness.\nWe have made our source code available at https://doi.org/10.5281/zenodo.3520097.\n\n2 The EXACTLINE Primitive\n\nis a linear partitioning of f(cid:22)QR, denoted P(cid:0)f(cid:22)QR\n\nGiven a piecewise-linear neural network f and two points Q, R in the input space of f, we consider\nthe restriction of f to QR, denoted f(cid:22)QR, which is identical to the function f except that its input\ndomain has been restricted to QR. We now want to \ufb01nd a succinct representation for f(cid:22)QR that we\ncan analyze more readily than the neural network corresponding to f. In this paper, we propose to use\nthe EXACTLINE representation, which corresponds to a linear partitioning of f(cid:22)QR, de\ufb01ned below.\nDe\ufb01nition 1. Given a function f : A \u2192 B and line segment QR \u2286 A, a tuple (P1, P2, P3, . . . , Pn)\nif: (1) {PiPi+1 | 1 \u2264 i < n} partitions QR (except for overlap at endpoints); (2) P1 = Q and\nPn = R; and (3) for all 1 \u2264 i < n, there exists an af\ufb01ne map Ai such that f (x) = Ai(x) for all\nx \u2208 PiPi+1.\nIn other words, we wish to partition QR into a set of pieces where the action of f on all points in\n\n(cid:1) and referred to as \u201cEXACTLINE of f over QR,\u201d\n\neach piece is af\ufb01ne. Note that, given P(cid:0)f(cid:22)QR\nf (x) = (1 \u2212 \u03b1)f (Pi) + \u03b1f (Pi+1). In this way, P(cid:0)f(cid:22)QR\n\nfor each partition PiPi+1 can be determined by recognizing that af\ufb01ne maps preserve ratios along\nlines. In other words, given point x = (1 \u2212 \u03b1)Pi + \u03b1Pi+1 on linear partition PiPi+1, we have\n\nrepresentation for the behavior of f on all points along QR.\nConsider an illustrative DNN taking as input the age and income of an individual and returning a\nloan-approval score and premium that should be charged over a baseline amount:\n\n(cid:1) = (P1, . . . , Pn), the corresponding af\ufb01ne function\n(cid:1) provides us a succinct and precise\n(cid:20)\u22121.7\n(cid:1) = (P1 = Q, P2 = (23.3, 36.6),\n\n1.0\n2.0 \u22121.3\n\n(cid:20)3\n(cid:21)\n\n(cid:21)\n\nX +\n\n(1)\n\n3\n\nSuppose an individual of 20 years old making $30k/year (Q = (20, 30)) predicts that their earnings\nwill increase linearly every year until they reach 30 years old and are making $50k/year (R =\n(30, 50)). We wish to understand how they will be classi\ufb01ed by this system over these 10 years.\n\nf (X = (x0, x1)) = ReLU (A(X)) , where A(X) =\n\nP3 = (26.6, 43.3), P4 = R), where f(cid:22)QR is exactly described by the following piecewise-linear\nfunction (Figure 1):\n\nx +\n\n, x \u2208 P2P3\n\n(2)\n\nWe can use EXACTLINE (De\ufb01nition 1) to compute P(cid:0)f(cid:22)QR\n(cid:35)\n(cid:34)\n(cid:35)\n\n(cid:34)\n(cid:34)\u22121.7\n(cid:34)\u22121.7\n\n0\n0\n2 \u22121.3\n\nf(cid:22)QR(x) =\n\n1\n\u22121.3\n\n(cid:35)\n\n(cid:34)\n\n(cid:35)\n\n(cid:35)\n\n(cid:34)\n\n(cid:35)\n\nx +\n\n0\n3\n\n3\n0\n\n,\n\n,\n\n3\n3\n\n\uf8f1\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f3\n\n2\n\n0\n\nx \u2208 QP2\n\nx \u2208 P3R\n\n1\n0\n\nx +\n\n2\n\n\fFigure 1: Computing the linear restriction of f (Equation 1) using EXACTLINE. The input line\nsegment QR is divided into three linear partitions such that the transformation from input space to\noutput space (left plot to right plot) is af\ufb01ne (Equation 2). Tick marks (on the left) are used in \ufb01gures\nthroughout this paper to indicate the partition endpoints (P1, P2, . . . , Pn).\n\nOther Network Analysis Techniques Compared to prior work, our solution to EXACTLINE\npresents an interesting and unique design point in the space of neural-network analysis. Approaches\nsuch as [12, 34, 16] are precise, but exponential time because they work over the entire input domain.\nOn another side of the design space, approaches such as those used in [15, 34, 35, 17, 36, 32] are\nsigni\ufb01cantly faster while still working over the full-dimensional input space, but accomplish this\nby trading analysis precision for speed. This trade-off between speed and precision is particularly\nwell-illuminated by [37], which monotonically re\ufb01nes its analysis when given more time. In con-\ntrast, the key observation underlying our work is that we can perform both an ef\ufb01cient (worst-case\npolynomial time for a \ufb01xed number of layers) and precise analysis by restricting the input domain\nto be one dimensional (a line). This insight opens a new dimension to the discussion of network\nanalysis tools, showing that dimensionality can be traded for signi\ufb01cant gains in both precision and\nef\ufb01ciency (as opposed to prior work which has explored the tradeoff primarily along the precision\nand ef\ufb01ciency axes under the assumption of high-dimensional input regions). ? ] similarly considers\none-dimensional input spaces, but the paper is focused on a number of theoretical properties and does\nnot focus on the algorithm used in their empirical results.\n\nAlgorithm We will \ufb01rst discuss computation of EXACTLINE for individual layers. Note that by\nde\ufb01nition, EXACTLINE for af\ufb01ne layers does not introduce any new linear partitions. This is captured\nby Theorem 1 (proved in Appendix D) below:\nTheorem 1. For any af\ufb01ne function A : X \u2192 Y and line segment QR \u2282 X, the following is a\n\nsuitable linear partitioning (De\ufb01nition 1): P(cid:0)A(cid:22)QR\nThe following theorem (proved in Appendix E) presents a method of computing P(cid:0)ReLU(cid:22)QR\n\n(cid:1) = (Q, R).\n\n(cid:1).\n\nP(cid:0)ReLU(cid:22)QR\n\nTheorem 2. Given a line segment QR in d dimensions and a recti\ufb01ed linear layer ReLU(x) =\n(max(x1, 0), . . . , max(xd, 0)), the following is a suitable linear partitioning (De\ufb01nition 1):\n\n(cid:1) = sorted(cid:0)({Q, R} \u222a {Q + \u03b1(R \u2212 Q) | \u03b1 \u2208 D}) \u2229 QR(cid:1) ,\n\n(3)\nwhere D = {\u2212Qi/(Ri \u2212 Qi) | 1 \u2264 i \u2264 d}, Vi is the ith component of vector V , and sorted returns\na tuple of the points sorted by distance from Q.\n\nThe essential insight is that we can \u201cfollow\u201d the line until an orthant boundary is reached, at which\npoint a new linear region begins. To that end, each number in D represents a ratio between Q and R\nat which QR crosses an orthant boundary. Notably, D actually computes such \u201ccrossing ratios\u201d for\nthe unbounded line QR, hence intersecting the generated endpoints with QR in Equation 3.\nAn analogous algorithm for MaxPool is presented in Appendix F; the intuition is to follow the line\nuntil the maximum in any window changes. When a ReLU layer is followed by a MaxPool layer (or\nvice-versa), the \u201cfused\u201d algorithm described in Appendix G can improve ef\ufb01ciency signi\ufb01cantly. More\ngenerally, the algorithm described in Appendix H can compute EXACTLINE for any piecewise-linear\nfunction.\n\n(cid:1) for entire neural networks (i.e. sequential com-\n\nFinally, in practice we want to compute P(cid:0)f(cid:22)QR\n\npositions of layers), not just individual layers (as we have demonstrated above). The next theorem\n\n3\n\nQ=(20, 30)R=(30, 50)Af(Q)f(P2)f(P3)f(R)Additional PremiumApproval scoreAgeIncomeP2=(23.3, 36.6)P3=(26.6, 43.3)ReLUA(Q)A(R)\f(cid:26)\n\nPi +\n\ni=1\n\nTheorem 3. Given any piecewise-linear functions f, g, h such that f = h \u25e6 g along with a line\n\nshows that, as long as one can compute P(cid:0)Li(cid:22)M N\n(cid:1) for each individual layer Li and arbitrary line\nsegment M N, then these algorithms can be composed to compute P(cid:0)f(cid:22)QR\n(cid:1) for the entire network.\nsegment QR where g(R) (cid:54)= g(Q) and P(cid:0)g(cid:22)QR\n(cid:1) = (P1, P2, . . . , Pn) is EXACTLINE applied to g\n(cid:1)(cid:27)(cid:33)\nP(cid:0)f(cid:22)QR\n\u00d7 (Pi+1 \u2212 Pi) | y \u2208 P(cid:0)h(cid:22)g(Pi)g(Pi+1)\nThe key insight is that we can \ufb01rst compute EXACTLINE for the \ufb01rst layer, i.e. P(cid:0)L1(cid:22)QR\n(cid:1) =\n\nwhere sorted returns a tuple of the points sorted by distance from Q.\n\n(cid:1) = sorted\n\nover QR, the following holds:\n\ng(Pi+1) \u2212 g(Pi)\n\n(cid:32)n\u22121(cid:91)\n\ny \u2212 g(Pi)\n\ni P 1\n\n1 , P 1\n\n2 , . . . , P 1\n\ni+1 individually.\n\nn), then we can continue computing EXACTLINE for the rest of the network within\n\n(P 1\neach of the partitions P 1\nIn Appendix C we show that, over arbitrarily many af\ufb01ne layers, l ReLU layers each with d units, and\nm MaxPool or MaxPool + ReLU layers with w windows each of size s, at most O((d + ws)l+m)\nsegments may be produced. If only ReLU and af\ufb01ne layers are used, at most O(dl) segments\nmay be produced. Notably, this is a signi\ufb01cant improvement over the O((2d)l) upper-bound and\n\u2126(l \u00b7 (2d)) lower-bound of Xiang et al. [34]. One major reason for our improvement is that we\nparticularly consider one-dimensional input lines as opposed to arbitrary polytopes. Lines represent a\nparticularly ef\ufb01cient special case as they are ef\ufb01ciently representable (by their endpoints) and, being\none-dimensional, are not subject to the combinatorial blow-up faced by transforming larger input\nregions. Furthermore, in practice, we have found that the majority of ReLU nodes are \u201cstable\u201d, and\nthe actual number of segments remains tractable; this algorithm for EXACTLINE often executes in a\nmatter of seconds for networks with over 60, 000 units (whereas the algorithm of Xiang et al. [34]\nruns in at least exponential O(l \u00b7 (2d)) time regardless of the input region as it relies on trivially\nconsidering all possible orthants).\n\n3 Characterizing Decision Boundaries for ACAS Xu\n\nThe \ufb01rst application of EXACTLINE we consider is that of understanding the decision boundaries of a\nneural network over some in\ufb01nite set of inputs. As a motivating example, we consider the ACAS Xu\nnetwork trained by Julian et al. [5] to determine what action an aircraft (the \u201cownship\u201d) should take in\norder to avoid a collision with an intruder. After training such a network, one usually wishes to probe\nand visualize the recommendations of the network. This is desirable, for example, to determine at\nwhat distance from the ownship an intruder causes the system to suggest a strong change in heading,\nor to ensure that distance is roughly the same regardless of which side the intruder approaches.\nThe simplest approach, shown in Figure 2f and currently the standard in prior work, is to consider\na (\ufb01nite) set of possible input situations (samples) and see how the network reacts to each of them.\nThis can help one get an overall idea of how the network behaves. For example, in Figure 2f, we\ncan see that the network has a mostly symmetric output, usually advising the plane to turn away\nfrom the intruder when suf\ufb01ciently close. Although sampling in this way gives human viewers an\nintuitive and meaningful way of understanding the network\u2019s behavior, it is severely limited because\nit relies on sampling \ufb01nitely many points from a (practically) in\ufb01nite input space. Thus, there is a\nsigni\ufb01cant chance that some interesting or dangerous behavior of the network may be exposed with\nmore samples.\nBy contrast, the EXACTLINE primitive can be used to exactly determine the output of the network at\nall of the in\ufb01nitely many points on a line in the input region. For example, in Figure 2a, we have used\nEXACTLINE to visualize a particular head-on collision scenario where we vary the distance of the\nintruder (speci\ufb01ed in polar coordinates (\u03c1, \u03b8)) with respect to the ownship (always at (0, 0)). Notably,\nthere is a region of \u201cStrong Left\u201d in a region of the line that is otherwise entirely \u201cWeak Left\u201c that\nshows up in Figure 2a (the EXACTLINE method) but not in Figure 2b (the sampling method). We\ncan do this for lines varying the \u03b8 parameter instead of \u03c1, result in Figure 2c and Figure 2d. Finally,\nrepeating this process for many lines and overlapping them on the same graph produces a detailed\n\u201cgrid\u201d as shown in Figure 2e.\n\n4\n\n\fLegend:\n\nClear-of-Con\ufb02ict, Weak Right,\n\nStrong Right,\n\nStrong Left, Weak Left.\n\n(a) Single line varying \u03c1\n\n(b) Sampling different \u03c1s\n\n(c) Single line varying \u03b8\n\n(d) Sampling different \u03b8s\n\n(e) Combination of lines varying\n\u03c1 and lines varying \u03b8\n\n(f) Sampling \ufb01nitely many points\n\nFigure 2: (a)\u2013(d) Understanding the decision boundary of an ACAS Xu aircraft avoidance network\nalong individual lines using EXACTLINE ((a), (c)) and \ufb01nite sampling ((b), (d)). In the EXACTLINE\nvisualizations there is a clear region of \u201cstrong left\u201d in a region that is otherwise \u201cweak left\u201d that\ndoes not show in the sampling plots due to the sampling density chosen. In practice, it is not clear\nwhat sampling density to choose, thus the resulting plots can be inaccurate and/or misleading. (e)\u2013(f)\nComputing the decision boundaries among multiple lines and plotting on the same graph. Using\nEXACTLINE to sample in\ufb01nitely many points provides more con\ufb01dence in the interpretation of the\ndecision boundaries. Compare to similar \ufb01gures in Julian et al. [5], Katz et al. [12].\n\nFigure 2e also shows a number of interesting and potentially dangerous behaviors. For example, there\nis a signi\ufb01cant region behind the plane where an intruder on the left may cause the ownship to make\na weak left turn towards the intruder, an unexpected and asymmetric behavior. Furthermore, there are\nclear regions of strong left/right where the network otherwise advises weak left/right. Meanwhile,\nin Figure 2f, we see that the sampling density used is too low to notice the majority of this behavior.\nIn practice, it is not clear what sampling density should be taken to ensure all potentially-dangerous\nbehaviors are caught, which is unacceptable for safety-critical systems such as aircraft collision\navoidance.\nTakeaways. EXACTLINE can be used to visualize the network\u2019s output on in\ufb01nite subsets of the\ninput space, signi\ufb01cantly improving the con\ufb01dence one can have in the resulting visualization and in\nthe safety and accuracy of the model being visualized.\nFuture Work. One particular area of future work in this direction is using EXACTLINE to assist in\nnetwork veri\ufb01cation tools such as Katz et al. [12] and Gehr et al. [15]. For example, the relatively-fast\nEXACTLINE could be used to check in\ufb01nite subsets of the input space for counter-examples (which\ncan then be returned immediately) before calling the more-expensive complete veri\ufb01cation tools.\n\n4 Exact Computation of Integrated Gradients\n\nIntegrated Gradients (IG) [25] is a method of attributing the prediction of a DNN to its input features.\nSuppose function F : Rn \u2192 [0, 1] de\ufb01nes the network. The integrated gradient along the ith\ndimension for an input x = (x1, . . . , xn) \u2208 Rn and baseline x(cid:48) \u2208 Rn is de\ufb01ned as:\n\n(cid:90) 1\n\nIGi(x)\n\ndef\n\n= (xi \u2212 x\ni) \u00d7\n(cid:48)\n\n\u2202F (x(cid:48) + \u03b1 \u00d7 (x \u2212 x(cid:48)))\n\n\u03b1=0\n\n\u2202xi\n\nd\u03b1\n\n(4)\n\nThus, the integrated gradient along all input dimensions is the integral of the gradient computed on all\npoints on the straightline path from the baseline x(cid:48) to the input x. In prior work it was not known how\n\n5\n\n5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)5000010000Downrange (ft)500006000Crossrange (ft)\fFigure 3: \u201cIntegrated Gradients\u201d is a powerful method of neural network attribution. IG relies on\ncomputing the integral of the gradients of the network at all points linearly interpolated between two\nimages (as shown above), however previous work has only been able to approximate the true IG,\ncasting uncertainty on the results. Within each partition identi\ufb01ed by EXACTLINE (delineated by\nvertical lines) the gradient is constant, so computing the exact IG is possible for the \ufb01rst time.\n\ndef\n\nm\ni\n\n(cid:48)\n\n(5)\n\n0\u2264k<m\n\n(cid:102)IG\n\nto exactly compute this integral for complex networks, so it was approximated using the left-Riemann\nsummation of the gradients at m uniformly sampled points along the straightline path from x(cid:48) to x:\n\ni) \u00d7 (cid:88)\nThe number of samples m determines the quality of this approximation. Let (cid:101)m denote the number of\nsamples that is large enough to ensure that(cid:80)n\n(cid:101)m\nnumber of samples suggested by Sundararajan et al. [25]. In practice, (cid:101)m can range between 20 and\ni \u2248 F (x) \u2212 F (x(cid:48)). This is the recommended\n\nm \u00d7 (x \u2212 x(cid:48)))\n\u2202xi\n\ni=1(cid:102)IG\n\n\u2202F (x(cid:48) + k\n\n= (xi \u2212 x\n\n\u00d7 1\nm\n\n1000 [25, 38].\nWhile the (exact) IG described by Equation 4 satis\ufb01es properties such as completeness [25, \u00a73] and\nsensitivity [25, \u00a72.1], the approximation computed in practice using Equation 5 need not.\nThe integral in Equation 4 can be computed exactly by adding an additional condition to the de\ufb01nition\nof EXACTLINE: that the gradient of the network within each partition is constant. It turns out that\nthis stricter de\ufb01nition is met by all of the algorithms we have discussed so far, a fact we discuss in\nmore detail in Appendix J. For ReLU layers, for example, because the network acts like a single af\ufb01ne\nmap within each orthant, and we split the line such that each partition is entirely contained within\nan orthant, the network\u2019s gradient is constant within each orthant (and thus along each EXACTLINE\npartition). This is stated formally by Theorem 4 and proved in Appendix J:\nTheorem 4. For any network f with nonlinearities introduced only by ReLU functions and\nits input vector x, i.e. \u2207f (x), is constant within each linear partition PiPi+1.\nThis allows us to exactly compute the IG of each individual partition r (RIGr\nat any point in that partition and multiplying by the width of the partition:\n= (P(r+1)i \u2212 Pri) \u00d7 \u2202F (0.5 \u00d7 (Pri + P(r+1)i))\n\n(cid:1) = (P1, P2, . . . , Pn) computed according to Equation 3, the gradient of f with respect to\n\nP(cid:0)f(cid:22)QR\n\ni ) by \ufb01nding the gradient\n\nRIGr\n\n(6)\n\ni (x)\n\ndef\n\n\u2202xi\n\nn(cid:88)\n\nCompared to Equation 4, Equation 6 computes the IG for partition for PrPr+1 by replacing the\nintegral with a single term (arbitrarily choosing \u03b1 = 0.5, i.e. the midpoint) because the gradient is\nuniform along PrPr+1. The exact IG of x(cid:48)x is the sum of the IGs of each partition:\n\nIGi(x) =\n\nRIGr\n\ni (x)\n\n(7)\n\nr=1\n\nEmpirical Results. A prime interest of ours was to determine the accuracy of the existing sampling\nmethod. On three different CIFAR10 networks [39], we took each image in the test set and computed\n\nthe exact IG against a black baseline using Equation 7. We then found (cid:101)m and computed the mean\n\nrelative error between the exact IG and the approximate one. As shown in Table 1, the approximate\nIG has an error of 25 \u2212 45%. This is a concerning result, indicating that the existing use of IG in\npractice may be misleading. Notably, without EXACTLINE, there would be no way of realizing this\nissue, as this analysis relies on computing the exact IG to compare with.\nFor many smaller networks considered, we have found that computing the exact IG is relatively\nfast (i.e., at most a few seconds) and would recommend doing so for situations where it is feasible.\nHowever, in some cases the number of gradients that would have to be taken to compute exact IG is\nhigh (see Column 2, Table 2). In such cases, we can use our exact computation to understand how\n\n6\n\n\fTable 1: Mean relative error\n\nfor approximate IG (using (cid:101)m)\n\ncompared to exact IG (using\nEXACTLINE) on CIFAR10. The\napproximation error is surpris-\ningly high.\n\nError (%)\nconvsmall\n24.95\nconvmedium 24.05\n45.34\nconvbig\n\nTable 2: Average number of samples needed by different\nIG approximations to reach 5% relative error w.r.t. exact IG\n(using EXACTLINE). Outliers requiring over 1, 000 samples\nwere not considered. Using trapezoidal rule instead of a\nleft-sum can cause large gains in accuracy and performance.\n\nExact\n\nApproximate\n\nconvsmall\n2582.74\nconvmedium 3578.89\n14064.65\nconvbig\n\nleft\n136.74\n150.31\n269.43\n\nright\n139.40\n147.59\n278.20\n\ntrap.\n91.67\n91.57\n222.79\n\nmany uniform samples one should use to compute the approximate IG to within 5% of the exact IG.\nTo that end, we performed a second experiment by increasing the number of samples taken until the\napproximate IG reached within 5% error of the exact IG. In practice, we found there were a number\nof \u201clucky guess\u201d situations where taking, for example, exactly 5 samples led to very high accuracy,\nbut taking 4 or 6 resulted in very poor accuracy. To minimize the impact of such outliers, we also\nrequire that the result is within 5% relative error when taking up to 5 more samples. In Table 2 the\nresults of this experiment are shown under the column \u201cleft,\u201d where it becomes clear that relatively\nfew samples (compared to the number of actual linear regions under \u201cexact\u201d) are actually needed.\nThus, one can use EXACTLINE on a test set to understand how many samples are needed on average\nto get within a desired accuracy, then use that many samples when approximating IG.\nFinally, the left Riemann sum used by existing work in IG [25] is only one of many possible sampling\nmethods; one could also use a right Riemann sum or the \u201cTrapezoidal Rule.\u201d With EXACTLINE\nwe can compute the exact IG and quantify how much better or worse each approximation method\nis. To that end, the columns \u201cright\u201d and \u201ctrap.\u201d in Table 2 show the number of samples one must\ntake to get consistently within 5% of the exact integrated gradients. Our results show the number of\nsamples needed to accurately approximate IG with trapezoidal rule is signi\ufb01cantly less than using left\nand right sums. Note that, because these networks are piecewise-linear functions, it is possible for\ntrapezoidal sampling to be worse than left or right sampling, and in fact for all tested models there\nwere images where trapezoidal sampling was less accurate than left and right sampling. This is why\nhaving the ability to compute the exact IG is important, to ensure that we can empirically quantify the\nerror involved in and justify choices between different heuristics. Furthermore, we \ufb01nd a per-image\nreduction in the number of samples needed of 20 \u2212 40% on average. Thus, we also recommend that\nusers of IG use a trapezoidal approximation instead of the currently-standard left sum.\nTakeaways. EXACTLINE is the \ufb01rst method for exactly computing integrated gradients, as all other\nuses of the technique have sampled according to Equation 5. From our results, we provide two\nsuggestions to practitioners looking to use IG: (1) Use EXACTLINE on a test set before deployment\nto better understand how the number of samples taken relates to approximation error, then choose\nsampling densities at runtime based on those statistics and the desired accuracy; (2) Use the trapezoidal\nrule approximation when approximating to get better accuracy with fewer samples.\nFuture Work. EXACTLINE can be used to design new IG approximation using non-uniform sam-\npling. EXACTLINE can similarly be used to exactly compute other measures involving path integrals\nsuch as neuron conductance [40], a re\ufb01nement of integrated gradients.\n\n5 Understanding Adversarial Examples\n\nWe use EXACTLINE to empirically investigate and falsify a fundamental assumption behind a well-\nknown hypothesis for the existence of adversarial examples. In the process, using EXACTLINE, we\ndiscover a strong association between robustness and the empirical linearity of a network, which we\nbelieve may help spur future work in understanding the source of adversarial examples.\nEmpirically Testing the Linear Explanation of Adversarial Examples. We consider in this sec-\ntion one of the dominant hypotheses for the problem of adversarial examples ([6\u20138]), \ufb01rst proposed\nby Goodfellow et al. [7] and termed the \u201cLinear Explanation of Adversarial Examples.\u201d It makes\na fundamental assumption that, when restricted to the region around a particular input point, the\n\n7\n\n\fFigure 4: Comparing the density of linear partitions from a test image to FGSM and random\nbaselines. The long blue line segment indicates the change in classi\ufb01cation. (1) Even within a small\nperturbation of the input point, there is signi\ufb01cant non-linear behavior in the network. (2) Although\nthe linear hypothesis for adversarial examples predicts that both normal inputs and their corresponding\nadversarial examples (\u201cFGSM + 3%\u201d) will lie on the same linear region, we \ufb01nd in practice that,\nnot only do they lie on different linear regions, but there is signi\ufb01cantly more non-linearity in the\nadversarial (FGSM) direction than a random direction. This falsi\ufb01es the fundamental assumption\nbehind the linear explanation of adversarial examples.\n\nFigure 5: Comparing the density of linear partitions from a test image to random baselines for normal\n(in black) and DiffAI-protected networks (in green). Networks trained with the DiffAI robust-training\nscheme tend to exhibit signi\ufb01cantly fewer non-linearities than their normally-trained counterparts.\n\noutput of the neural network network is linear, i.e. the tangent plane to the network\u2019s output at\nthat point exactly matches the network\u2019s true output. Starting from this assumption, Goodfellow\net al. [7] makes a theoretical argument concluding that the phenomenon of adversarial examples\nresults from the network being \u201ctoo linear.\u201d Although theoretical debate has ensued as to the merits\nof this argument [41], the effectiveness of adversarials generated according to this hypothesis (i.e.,\nusing the standard \u201cfast gradient sign method\u201d or \u201cFGSM\u201d) has sustained it as one of the most\nwell-subscribed-to theories as to the source of adversarial examples. However, to our knowledge, the\nfundamental assumption of the theory\u2014that real images and their adversarial counterparts lie on the\nsame linear region of a neural network\u2014has not been rigorously validated empirically.\nTo empirically validate this hypothesis, we looked at the line between an image and its FGSM-\nperturbed counterpart (which is classi\ufb01ed differently by the network), then used EXACTLINE to\nquantify the density of each line (the number of linear partitions \u2014 each delineated by tick marks\nin Figure 4 \u2014 divided by the distance between the endpoints). If the underlying linearity assumption\nholds, we would expect that both the real image and the perturbed image lie on the same linear\npartition, thus we would not expect to see the line between them pass through any other linear\npartitions. In fact (top of Figure 4), we \ufb01nd that the adversarial image seems to lie across many\nlinear partitions, directly contradicting the fundamental assumption of Goodfellow et al. [7]. We also\ncompared this line to the line between the real image and the real image randomly perturbed by the\nsame amount (bottom of Figure 4). As shown in Table 3, the FGSM direction seems to pass through\nsigni\ufb01cantly more linear partitions than a randomly chosen direction. This result shows that, not only\nis the linearity assumption not met, but in fact the opposite appears to be true: adversarial examples\nare associated with unusually non-linear directions of the network.\nWith these results in mind, we realized that the linearity assumption as initially presented in Good-\nfellow et al. [7] is stronger than necessary; it need only be the case that the gradient is reasonably\nconstant across the line, so that the tangent plane approximation used is still reasonably accurate. To\n\n8\n\nBirdTrainingMethod:NormalDirection:FGSM+3%Density:212.60Label:CatTrainingMethod:NormalDirection:Random+3%Density:129.46Label:BirdBirdTrainingMethod:NormalDirection:Random+3%Density:129.46Label:BirdTrainingMethod:DiffAIDirection:Random+3%Density:37.87Label:Bird\fTable 3: Density of FGSM line partitions divided by density of Random line partitions. FGSM\ndirections tend to be signi\ufb01cantly more dense than random directions, contradicting the well-known\nLinear Explanation of Adversarial Examples. Mdn: Median, 25%: 25% percentile, 75%: 75% percentile\n\n(a) MNIST\n\n(b) CIFAR10\n\nFGSM/Random\nTraining Method Mdn 25% 75%\n1.36 0.99 1.76\nNormal\nDiffAI\n0.98 0.92 1.38\n1.22 0.97 1.51\nPGD\n\nFGSM/Random\nTraining Method Mdn 25% 75%\n1.78 1.60 2.02\nNormal\nDiffAI\n1.67 1.47 2.03\n1.84 1.65 2.10\nPGD\n\nTable 4: Comparing density of lines when different training algorithms (normal, DiffAI, and PGD) are\nused. We report the mean of those ratios across all tested lines. These results indicate that networks\ntrained to be adversarially robust with DiffAI or PGD training methods tend to behave more linearly\nthan non-robust models. Mdn: Median, 25%: 25% percentile, 75%: 75% percentile\n\n(a) MNIST\n\n(b) CIFAR10\n\nNormal/DiffAI\n\nNormal/PGD\n\nNormal/DiffAI\n\nNormal/PGD\n\nDir. Mdn 25% 75% Mdn 25% 75%\n0.88 0.66 1.14\nFGSM 3.05 2.05 4.11\nRand. 2.50 1.67 3.00\n0.80 0.62 1.00\n\nDir. Mdn 25% 75% Mdn 25% 75%\n1.48 1.22 1.67\nFGSM 3.37 2.77 4.94\nRand. 3.43 2.78 4.42\n1.51 1.21 1.80\n\nmeasure how well the tangent plane approximates the function between normal and adversarial inputs,\nwe compared the gradient taken at the real image (i.e., used by FGSM) to the gradients at each of the\nintermediate linear partitions, \ufb01nding the mean error between each and averaging weighted by size\nof the partition. If this number is near 0, it implies that, although many theoretically distinct linear\npartitions exist between the two points, they have roughly the same \u201cslope\u201d and thus the tangent\nplane approximation would be accurate and the Linearity Hypothesis may still be a worthwhile\nexplanation of the phenomenon of adversarial examples. However, we \ufb01nd that this number is larger\nthan 250% when averaged over all images tested, implying that the tangent plane is not a particularly\ngood approximation of the underlying function in these regions of input space and providing further\nempirical evidence against the Linear Explanation of adversarial examples.\nCharacteristics of Adversarially-Trained Networks. We also noticed an unexpected trend in the\nprevious experiment: networks trained to be robust to adversarial perturbations (particularly DiffAI-\ntrained networks [33]) seemed to have signi\ufb01cantly fewer linear partitions in all of the lines that we\nevaluated them on (see Figure 5). Further experiments, summarized in Table 4, showed that networks\ntrained to be adversarially robust with PGD and especially DiffAI training methods exhibit up to 5\u00d7\nfewer linear partitions for the same change in input. This observation suggests that the neighborhoods\naround points in adversarially-trained networks are \u201c\ufb02at\u201d (more linear).\nTakeaways. Our results falsify the fundamental assumption behind the well-known Linear Explana-\ntion for adversarial examples. Adversarial training tends to make networks more linear.\nFuture Work. EXACTLINE can be used to investigate adversarial robustness. Further investigation\ninto why DiffAI-protected networks tend to be more linear will help resolve the question (raised\nin this work) of whether reduced density of linear partitions contributes to robustness, or increased\nrobustness results in fewer linear partitions (or if there is a third important variable impacting both).\n\n6 Conclusion\n\nWe address the problem of computing a succinct representation of a linear restriction of a neural\nnetwork. We presented EXACTLINE, a novel primitive for the analysis of piecewise-linear deep\nneural networks. Our algorithm runs in a matter of a few seconds on large convolutional and ReLU\nnetworks, including ACAS Xu, MNIST, and CIFAR10. This allows us to investigate questions about\nthese networks, both shedding new light and raising new questions about their behavior.\n\n9\n\n\fAcknowledgements\n\nWe thank Nina Amenta, Yong Jae Lee, Cindy Rubio-Gonz\u00e1lez, and Mukund Sundararajan for their\nfeedback and suggestions on this work. This material is based upon work supported by AWS Cloud\nCredits for Research.\n\nReferences\n[1] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.\n\n[2] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re-\nthinking the inception architecture for computer vision. In Proceedings of the IEEE conference\non computer vision and pattern recognition CVPR, 2016.\n\n[3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classi\ufb01cation with deep\n\nconvolutional neural networks. Commun. ACM, 60(6):84\u201390, 2017.\n\n[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of\n\ndeep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.\n\n[5] Kyle D Julian, Mykel J Kochenderfer, and Michael P Owen. Deep neural network compression\nfor aircraft collision avoidance systems. Journal of Guidance, Control, and Dynamics, 42(3):\n598\u2013608, 2018.\n\n[6] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J.\nIn International\n\nIntriguing properties of neural networks.\n\nGoodfellow, and Rob Fergus.\nConference on Learning Representations, ICLR, 2014.\n\n[7] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver-\n\nsarial examples. In International Conference on Learning Representations, ICLR, 2015.\n\n[8] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple\nand accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer\nVision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages\n2574\u20132582, 2016.\n\n[9] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-\n\nto-text. In 2018 IEEE Security and Privacy Workshops (SPW), pages 1\u20137. IEEE, 2018.\n\n[10] Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High\ncon\ufb01dence predictions for unrecognizable images. In IEEE Conference on Computer Vision and\nPattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 427\u2013436, 2015.\n\n[11] Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori,\nand Antonio Criminisi. Measuring neural net robustness with constraints. In Advances in Neural\nInformation Processing Systems, 2016.\n\n[12] Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex:\nAn ef\ufb01cient smt solver for verifying deep neural networks. In International Conference on\nComputer Aided Veri\ufb01cation, 2017.\n\n[13] Ruediger Ehlers. Formal veri\ufb01cation of piece-wise linear feed-forward neural networks. In\nInternational Symposium on Automated Technology for Veri\ufb01cation and Analysis ATVA, 2017.\n\n[14] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety veri\ufb01cation of deep\n\nneural networks. In International Conference on Computer Aided Veri\ufb01cation CAV, 2017.\n\n[15] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and\nMartin T. Vechev. AI2: safety and robustness certi\ufb01cation of neural networks with abstract\ninterpretation. In 2018 IEEE Symposium on Security and Privacy, SP 2018, Proceedings, 21-23\nMay 2018, San Francisco, California, USA, 2018.\n\n10\n\n\f[16] Rudy R. Bunel, Ilker Turkaslan, Philip H. S. Torr, Pushmeet Kohli, and Pawan Kumar\nMudigonda. A uni\ufb01ed view of piecewise linear neural network veri\ufb01cation. In Advances\nin Neural Information Processing Systems 31: Annual Conference on Neural Information\nProcessing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montr\u00e9al, Canada., pages\n4795\u20134804, 2018.\n\n[17] Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane S.\nBoning, and Inderjit S. Dhillon. Towards fast computation of certi\ufb01ed robustness for relu\nnetworks. In International Conference on Machine Learning, ICML, 2018.\n\n[18] Gagandeep Singh, Timon Gehr, Markus P\u00fcschel, and Martin T. Vechev. An abstract domain for\n\ncertifying neural networks. PACMPL, 3(POPL):41:1\u201341:30, 2019.\n\n[19] Greg Anderson, Shankara Pailoor, Isil Dillig, and Swarat Chaudhuri. Optimization and abstrac-\ntion: A synergistic approach for analyzing neural network robustness. CoRR, abs/1904.09959,\n2019.\n\n[20] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In\nComputer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September\n6-12, 2014, Proceedings, Part I, pages 818\u2013833, 2014.\n\n[21] Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding\n\nneural networks through deep visualization. CoRR, abs/1506.06579, 2015.\n\n[22] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection:\nIn 2017 IEEE Conference on\nQuantifying interpretability of deep visual representations.\nComputer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017,\n2017.\n\n[23] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and\nKlaus-Robert M\u00fcller. How to explain individual classi\ufb01cation decisions. Journal of Machine\nLearning Research, 11:1803\u20131831, 2010.\n\n[24] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a\nblack box: Learning important features through propagating activation differences. CoRR,\nabs/1605.01713, 2016.\n\n[25] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In\n\nInternational Conference on Machine Learning ICML, 2017.\n\n[26] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Vi\u00e9gas,\nand Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept\nactivation vectors (TCAV). In International Conference on Machine Learning, ICML, 2018.\n\n[27] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and\nDino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys\n(CSUR), 51(5):93, 2018.\n\n[28] Travers Ching, Daniel S. Himmelstein, Brett K. Beaulieu-Jones, Alexandr A. Kalinin, Brian T.\nDo, Gregory P. Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M. Hoffman,\nWei Xie, Gail L. Rosen, Benjamin J. Lengerich, Johnny Israeli, Jack Lanchantin, Stephen\nWoloszynek, Anne E. Carpenter, Avanti Shrikumar, Jinbo Xu, Evan M. Cofer, Christopher A.\nLavender, Srinivas C. Turaga, Amr M. Alexandari, Zhiyong Lu, David J. Harris, Dave DeCaprio,\nYanjun Qi, Anshul Kundaje, Yifan Peng, Laura K. Wiley, Marwin H. S. Segler, Simina M. Boca,\nS. Joshua Swamidass, Austin Huang, Anthony Gitter, and Casey S. Greene. Opportunities and\nobstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, 15\n(141):20170387, 2018. doi: 10.1098/rsif.2017.0387.\n\n[29] Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T. Dudley. Deep learning\nfor healthcare: review, opportunities and challenges. Brie\ufb01ngs in Bioinformatics, 19(6):1236\u2013\n1246, 2018.\n\n[30] Ahmed Hosny, Chintan Parmar, John Quackenbush, Lawrence H Schwartz, and Hugo JWL\n\nAerts. Arti\ufb01cial intelligence in radiology. Nature Reviews Cancer, page 1, 2018.\n\n11\n\n\f[31] Ellen B Mendelson. Arti\ufb01cial intelligence in breast imaging: potentials and limitations. Ameri-\n\ncan Journal of Roentgenology, 212(2):293\u2013299, 2019.\n\n[32] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Formal security\nanalysis of neural networks using symbolic intervals. In 27th USENIX Security Symposium,\nUSENIX Security 2018, Baltimore, MD, USA, August 15-17, 2018., pages 1599\u20131614, 2018.\n\n[33] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for\nprovably robust neural networks. In International Conference on Machine Learning ICML,\n2018.\n\n[34] Weiming Xiang, Hoang-Dung Tran, and Taylor T Johnson. Reachable set computation and\nsafety veri\ufb01cation for neural networks with relu activations. arXiv preprint arXiv:1712.08163,\n2017.\n\n[35] Weiming Xiang, Hoang-Dung Tran, Joel A. Rosenfeld, and Taylor T. Johnson. Reachable set\nestimation and safety veri\ufb01cation for piecewise linear systems with neural network controllers.\nIn 2018 Annual American Control Conference, ACC, 2018.\n\n[36] Souradeep Dutta, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. Output range\nanalysis for deep feedforward neural networks. In NASA Formal Methods Symposium, pages\n121\u2013138. Springer, 2018.\n\n[37] Matt Jordan, Justin Lewis, and Alexandros G Dimakis. Provable certi\ufb01cates for adversarial\n\nexamples: Fitting a ball in the union of polytopes. arXiv preprint arXiv:1903.08778, 2019.\n\n[38] Kristina Preuer, G\u00fcnter Klambauer, Friedrich Rippmann, Sepp Hochreiter, and Thomas Un-\n\nterthiner. Interpretable deep learning in drug discovery. CoRR, abs/1903.02788, 2019.\n\n[39] ETH robustness analyzer for neural networks (ERAN). https://github.com/eth-sri/\n\neran, 2019. Accessed: 2019-05-01.\n\n[40] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron? arXiv\n\npreprint arXiv:1805.12233, 2018.\n\n[41] Thomas Tanay and Lewis Grif\ufb01n. A boundary tilting persepective on the phenomenon of\n\nadversarial examples. arXiv preprint arXiv:1608.07690, 2016.\n\n[42] Dirk Beyer, Stefan L\u00f6we, and Philipp Wendler. Reliable benchmarking: Requirements and\nsolutions. International Journal on Software Tools for Technology Transfer, 21(1):1\u201329, 2019.\n\n[43] A collection of pre-trained, state-of-the-art models in the onnx format. https://github.com/\n\nonnx/models, 2019. Accessed: 2019-05-01.\n\n12\n\n\f", "award": [], "sourceid": 7897, "authors": [{"given_name": "Matthew", "family_name": "Sotoudeh", "institution": "University of California, Davis"}, {"given_name": "Aditya", "family_name": "Thakur", "institution": "University of California, Davis"}]}