{"title": "Fast and Effective Robustness Certification", "book": "Advances in Neural Information Processing Systems", "page_first": 10802, "page_last": 10813, "abstract": "We present a new method and system, called DeepZ, for certifying neural network\nrobustness based on abstract interpretation. Compared to state-of-the-art automated\nverifiers for neural networks, DeepZ: (i) handles ReLU, Tanh and Sigmoid activation functions, (ii) supports feedforward and convolutional architectures, (iii)\nis significantly more scalable and precise, and (iv) and is sound with respect to\nfloating point arithmetic. These benefits are due to carefully designed approximations tailored to the setting of neural networks. As an example, DeepZ achieves a\nverification accuracy of 97% on a large network with 88,500 hidden units under\n$L_{\\infty}$ attack with $\\epsilon = 0.1$ with an average runtime of 133 seconds.", "full_text": "Fast and Effective Robustness Certi\ufb01cation\n\nGagandeep Singh, Timon Gehr, Matthew Mirman, Markus P\u00fcschel, Martin Vechev\n\nDepartment of Computer Science\n\nETH Zurich, Switzerland\n\n{gsingh,timon.gehr,matthew.mirman,pueschel,martin.vechev}@inf.ethz.ch\n\nAbstract\n\nWe present a new method and system, called DeepZ, for certifying neural network\nrobustness based on abstract interpretation. Compared to state-of-the-art automated\nveri\ufb01ers for neural networks, DeepZ: (i) handles ReLU, Tanh and Sigmoid activa-\ntion functions, (ii) supports feedforward, convolutional, and residual architectures,\n(iii) is signi\ufb01cantly more scalable and precise, and (iv) and is sound with respect to\n\ufb02oating point arithmetic. These bene\ufb01ts are due to carefully designed approxima-\ntions tailored to the setting of neural networks. As an example, DeepZ achieves a\nveri\ufb01cation accuracy of 97% on a large network with 88, 500 hidden units under\nL\u221e attack with \u0001 = 0.1 with an average runtime of 133 seconds.\n\n1\n\nIntroduction\n\nNeural networks have become an integral part of many critical applications such as vehicle control,\npattern recognition, and medical diagnosis. However, it has been shown recently that neural networks\nare susceptible to adversarial attacks, where the network can be easily tricked into making wrong\nclassi\ufb01cation by only slightly modifying its inputs [26, 12, 17, 3, 2, 9, 23, 22]. As a result, there is\nconsiderable interest in formally ensuring robustness of neural networks against such attacks.\nRobustness veri\ufb01ers can be complete or incomplete. Complete veri\ufb01ers do not have false positives\nbut have limited scalability as they are based on computationally expensive methods such as SMT\nsolving [14, 8], mixed integer linear programming [27] or input re\ufb01nement [28]. On the other\nhand, incomplete veri\ufb01ers can produce false positives but they scale better than complete veri\ufb01ers.\nIncomplete veri\ufb01ers employ a variety of methods including duality [24, 7], abstract interpretation\n[10], and linear approximations [29, 15].\nOrthogonal to robustness certi\ufb01cation, adversarial training focuses on making neural networks robust\nby training against an adversarial model of attack. Gu and Rigazio [13] add concrete noise to the\ntraining set and remove it statistically for defending against adversarial examples. Goodfellow et\nal. [12] generate adversarial examples misclassi\ufb01ed by neural networks and then design a defense\nagainst this attack by explicit training against perturbations generated by the attack. Madry et al.\n[19] show that training against an optimal attack also guards against non-optimal attacks. While this\ntechnique was effective in experiments, Carlini et al. [4] demonstrated an attack for the safety-critical\nproblem of ground-truthing, where this defense occasionally exacerbated the problem. The work of\n[30] and [21] proposes scalable defenses against the standard L\u221e-based adversarial attacks.\nIn this paper we build on the work of Gehr et al. [10], which introduces the idea of using the classic\nframework of abstract interpretation [5] to soundly approximate the behavior of the network. Similar\nto [10], we also use the Zonotope abstraction for sound approximations. The Zonotope domain\nprovides a closed af\ufb01ne form for each variable and enables a precise and cheap way to capture\nthe effect of af\ufb01ne transformations inside neural networks, without requiring backpropagation as\nin [15, 29]. The key limitation of [10] however, is that it only provides a fairly generic abstract\ntransformer for the ReLU activation function, which is slow and imprecise. Further, the work does not\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fdiscuss approximations of other important functions (e.g., Sigmoid, Tanh). Indeed, de\ufb01ning sound,\nscalable and precise abstract transformers is the most dif\ufb01cult aspect of abstract-interpretation-based\nanalyses. While generic transformers tend to be easier to reason about and ensure soundness of, they\noften lack the scalability and precision of transformers that exploit the underlying properties of the\nabstract domain (e.g., Zonotope) and the function being approximated (e.g., ReLU).\nOur contributions. In this work we address these limitations and make the following contributions:\n\u2022 We introduce new, point-wise Zonotope abstract transformers speci\ufb01cally designed for the\nReLU, Sigmoid, and Tanh activations often used by neural networks. Our transformers\nminimize the area of the projection of the zonotope to the 2-D input-output plane. Further,\nour transformers are sound w.r.t. \ufb02oating point arithmetic.\n\n\u2022 We implemented both, sequential and parallel versions of our transformers in an end-to-end\n\nautomated veri\ufb01cation system called DeepZ.\n\n\u2022 We evaluated DeepZ on the task of verifying local robustness against L\u221e-norm based\nattacks on large MNIST and CIFAR10 feed forward, convolutional, and residual networks.\nIn our evaluation we considered both, undefended as well as defended versions of the same\nnetwork (defended against L\u221e attacks using state-of-the-art defenses).\n\n\u2022 Our experimental results demonstrate that DeepZ is more precise and faster than prior work.\nDeepZ precisely veri\ufb01es large networks with > 88, 000 hidden units under L\u221e-norm based\nperturbations within few minutes, while being sound w.r.t to \ufb02oating point arithmetic.\n\nTo our best knowledge, DeepZ is currently the most scalable system for certifying local robustness\nof neural networks while guaranteed soundness w.r.t to \ufb02oating point operations (used by all neural\nnetworks). All of our code, datasets and results are publicly available at http://safeai.ethz.ch/.\n\n2 Abstract Interpretation for Verifying Robustness of Neural Networks\n\nAbstract Interpretation [5] is a classic method for sound and precise over-approximation of potentially\nunbounded or in\ufb01nite set of program behaviors. The key idea behind this framework consists of\nde\ufb01ning so called abstract transformers for statements used by the program (e.g., af\ufb01ne arithmetic,\nReLU functions, etc). These transformers approximate (i.e., they are sound) the behavior of the\nstatement by de\ufb01ning its effect on an abstract domain. An abstract domain is simply a set of abstract\nelements (approximations) typically ordered in a lattice of some kind.\nA key challenge when de\ufb01ning abstract transformers is striking a balance between scalability (how\nfast the transformer computes the approximation) and precision (how much precision it loses). Once\ntransformers are de\ufb01ned, the analysis with abstract interpretation proceeds by executing them on the\nparticular program (e.g., a neural network) and computing a \ufb01nal approximation (a \ufb01xed point). The\nrelevant property can then be checked on this \ufb01nal approximation: if the property can be proved, then\nit holds for any concrete input to the program, otherwise, it may either hold but the abstraction was\ntoo coarse and unable to prove it (i.e., a false positive) or it may indeed not hold.\nVerifying robustness properties of neural networks exactly is computationally expensive as it usually\nrequires evaluating the network exhaustively on a prohibitively large set of inputs. Abstract inter-\npretation can be leveraged for this problem by designing abstract transformers speci\ufb01cally for the\ncomputations used in the network, e.g., af\ufb01ne arithmetic and activation functions. The network can\nthen be analyzed using these abstract transformers. For example, we can abstract a concrete input x\nand relevant perturbations to x (resulting in many different inputs) into one abstract element \u03b1R and\nthen analyze the network starting from \u03b1R, producing an abstract output \u03b1oR. We can then verify\nthe robustness property of interest over \u03b1oR: if successful, it means we veri\ufb01ed it over all concrete\noutputs corresponding to all perturbations of the concrete input.\nIn this paper, we consider local robustness properties (Rx,\u0001,CL) where Rx,\u0001 represents the set of\nperturbed inputs around the original input x \u2208 Rm based on a small constant \u0001 > 0. CL is a robustness\ncondition which de\ufb01nes the set of outputs that all have the same label L:\n\n(cid:110)\n\nCL =\n\ny \u2208 Rn | arg max\ni\u2208{1,...,n}\n\n(cid:111)\n\n(yi) = L\n\n.\n\n2\n\n\fA robustness property (Rx,\u0001,CL) holds iff the set of outputs OR corresponding to all inputs in Rx,\u0001\nis included in CL. (Rx,\u0001,CL) can be veri\ufb01ed using abstract interpretation by checking if the abstract\noutput \u03b1oR resulting from analyzing the network with an abstraction of Rx,\u0001 is included in CL.\nZonotope Abstraction. In this work, we build on the classic Zonotope numerical abstract domain,\nwhich we discuss below. This domain was already shown to be a suitable basis for analyzing neural\nnetworks by Gehr et al. [10]. In the next section, we introduce our new abstract transformers which\nleverage properties of the domain and are the novel contribution of this work.\nLet X be the set of n variables. The Zonotope abstraction [11] builds on af\ufb01ne arithmetic by\nassociating an af\ufb01ne expression \u02c6x for each variable x \u2208 X :\n\n\u02c6x := \u03b10 +\n\n\u03b1i \u00b7 \u0001i, where \u03b10, \u03b1i \u2208 R, \u0001i \u2208 [ai, bi] \u2286 [\u22121, 1]\n\n(1)\n\np(cid:88)\n\ni=1\n\np(cid:88)\n\nThis expression consists of a center coef\ufb01cient \u03b10, a set of noise symbols \u0001i, and coef\ufb01cients \u03b1i\nrepresenting partial deviations around the center. Crucially, the noise symbols \u0001i can be shared\nbetween af\ufb01ne forms for different variables which creates implicit dependencies and constraints\nbetween the af\ufb01ne forms. This makes the Zonotope abstraction more powerful than an Interval\nabstraction which only maintains ranges of a variable x. A range [lx, ux] can be simply derived from\nthe af\ufb01ne form by computing the minimal and maximal value possible.\nA zonotope Z \u2286 Rn is represented by a collection of af\ufb01ne forms for all variables x \u2208 X , and is the\nset of all possible (joint) values of the af\ufb01ne forms for an arbitrary instantiation of the noise symbols\n\u0001i. As in practice, it is impossible to compute with arbitrary real numbers, we instead use a slightly\nmodi\ufb01ed de\ufb01nition:\n\n\u02c6x := [\u03b10, \u03b20] +\n\n[\u03b1i, \u03b2i] \u00b7 \u0001i, where \u03b10, \u03b20, \u03b1i, \u03b2i \u2208 R, \u0001i \u2208 [ai, bi] \u2286 [\u22121, 1]\n\n(2)\n\ni=1\n\nIn this interval af\ufb01ne form, we have replaced all coef\ufb01cients by intervals. All computations on\nintervals are performed using standard interval arithmetic. To ensure soundness with respect to\ndifferent rounding modes and to account for the lack algebraic properties such as associativity and\ndistributivity in the \ufb02oating point world, the lower bounds and the upper bounds are rounded towards\n\u2212\u221e and +\u221e respectively and suitable error values are added as explained in [20].\nSince af\ufb01ne arithmetic is fast and exact for af\ufb01ne transformations, it is an attractive candidate for\nthe veri\ufb01cation of neural networks [10]. However, the Zonotope abstraction is inherently not exact\nfor non-linear activation functions such as ReLU, Sigmoid, and Tanh. Thus, approximation is\nneeded, which creates a tradeoff between the cost of computation and precision. As mentioned\nearlier, a generic approximation of the ReLU function was proposed by Gehr et al. [10], however,\nthis approximation is both imprecise and costly as it relies on the expensive Zonotope join operator.\nOverall, this results in suboptimal precision and performance of the analysis.\n\n3 Fast Zonotope Abstract Transformers\n\nWe now introduce our fast and precise pointwise Zonotope abstract transformers for the ReLU,\nSigmoid, and Tanh activations (Sigmoid and Tanh are not supported by Gehr et al. [10]) and show\ntheir optimality in terms of area in the input-output plane. Our evaluation in Section 4 shows that our\nproposed approximations strike a good balance between precision and performance.\n\n3.1 ReLU\nThe effect of applying the ReLU function on an input zonotope Z can be represented with the\nassignment y := max(0, x) where x, y \u2208 X . If x can only have positive (lx > 0) or non-positive\nvalues (ux \u2264 0) in Z, then \u02c6y = \u02c6x or \u02c6y = [0, 0] respectively. The af\ufb01ne forms for the remaining\nvariables are not affected and the resulting zonotope is exact. When x can have both positive and\nnegative values, then the output cannot be exactly captured by the zonotope abstraction and thus\napproximations are required. We de\ufb01ne such an approximation for this case. The approximation can\nalso be applied pointwise per layer, namely, only altering the af\ufb01ne form \u02c6y while keeping all other\naf\ufb01ne forms in Z unaltered.\n\n3\n\n\f\u02c6y\n\n\u02c6y = \u03bb \u00b7 \u02c6x + ux \u00b7 (1 \u2212 \u03bb)\n\n\u02c6y = \u03bb \u00b7 \u02c6x\n\nux\nlx\n(a) 0 \u2264 \u03bb \u2264 ux\nux\u2212lx\n\n\u02c6y\n\n\u02c6y = \u03bb \u00b7 (\u02c6x \u2212 lx)\n\u02c6y = \u03bb \u00b7 \u02c6x\n\n\u02c6x\n\nlx\n(b)\n\nux\n\nux\u2212lx\n\nux\n\u2264 \u03bb \u2264 1\n\n\u02c6x\n\nFigure 1: Two zonotope approximations for the ReLU function parameterized by the slope \u03bb.\n\nFig. 1 shows the projections into the xy-plane of two sets of sound zonotope approximations. The\nprojections have the shape of a parallelogram with two vertical lines and two parallel lines of slope\n\u03bb, which is a parameter. To ensure soundness for all approximations in Fig. 1 (a), we require\n\u2264 \u03bb \u2264 1 for Fig. 1 (b). Notice that the two sets have one element\n0 \u2264 \u03bb \u2264 ux\nux\u2212lx\nin common at \u03bb = ux\n. Among the different candidate approximations in Fig. 1, we choose the\none minimizing the area of the parallelogram in the xy-plane. The area A1(\u03bb) of the parallelogram\nin Fig. 1 (a) is:\n\n. Similarly,\nux\u2212lx\n\nux\u2212lx\n\nux\n\nA1(\u03bb) = (1 \u2212 \u03bb) \u00b7 ux \u00b7 (ux \u2212 lx).\n\nA1(\u03bb) is a decreasing function of \u03bb. Thus A1 is minimized at \u03bb = ux\nof the parallelogram in Fig. 1 (b) is:\n\nux\u2212lx\n\nA2(\u03bb) = \u03bb \u00b7 (\u2212lx) \u00b7 (ux \u2212 lx).\n\n(3)\n. Similarly, the area A2(\u03bb)\n\n(4)\n. In summary, we obtain the\n\nux\u2212lx\n\nA2(\u03bb) is an increasing function of \u03bb and also minimized at \u03bb = ux\nfollowing theorem:\nTheorem 3.1 Let Z be the input to a ReLU function y = ReLU (x). Consider the set of pointwise\nZonotope approximations O of the output that only alter the af\ufb01ne form \u02c6y of the variable y. The new\naf\ufb01ne form \u02c6y for the output with the minimal area in the xy-plane is given by:\nif lx > 0,\nif ux \u2264 0,\n[0, 0],\n[\u03bbl, \u03bbu] \u00b7 \u02c6x + [\u00b5l, \u00b5u] + [\u00b5l, \u00b5u] \u00b7 \u0001new, otherwise.\n\n\uf8f1\uf8f2\uf8f3\u02c6x,\n\n\u02c6y =\n\n(5)\n\nHere \u03bbl, \u03bbu are \ufb02oating point representations of \u03bbopt = ux\nrespectively. Similarly, \u00b5l, \u00b5u are \ufb02oating point representations of \u00b5 = \u2212 ux\u00b7lx\ntowards \u2212\u221e and +\u221e respectively, and \u0001new \u2208 [\u22121, 1] is a new noise symbol.\nThe running time of the optimal transformer in Theorem 3.1 is linear in the number p of noise\nsymbols. One can also de\ufb01ne an optimal Zonotope transformer minimizing the volume of the output\nzonotope, however this is too expensive and the resulting transformer cannot be applied pointwise.\n\nusing rounding towards \u2212\u221e and +\u221e\n2\u00b7(ux\u2212lx) using rounding\n\nux\u2212lx\n\n(cid:105)\n\n(cid:104) eux\n\n3.2 Sigmoid\nThe effect of applying the Sigmoid function on an input zonotope Z can be represented with the\nassignment y := \u03c3(x) where x, y \u2208 X and \u03c3(x) = ex\n1+ex . For the assigned variable y, we have\n[ly, uy] \u2286 [0, 1]. When lx = ux, then \u02c6y :=\neux\nand the resulting zonotope is exact,\notherwise the output cannot be exactly represented by a zonotope and thus approximations are required.\nWe de\ufb01ne pointwise approximations for the Sigmoid function such that ly = \u03c3(lx), uy = \u03c3(ux) and\nthen choose the one minimizing the area of its projection in the xy-plane.\nFig. 2 shows the projections into the xy-plane of a set of sound zonotope approximations for the output\nof the Sigmoid function which have ly = \u03c3(lx), uy = \u03c3(ux). As for ReLU, the projections have the\nshape of a parallelogram with two vertical lines and two parallel lines of slope \u03bb which parameterizes\nthe set. To ensure soundness, we have 0 \u2264 \u03bb \u2264 min(\u03c3(cid:48)(lx), \u03c3(cid:48)(ux)) where \u03c3(cid:48)\nThe area A(\u03bb) of the parallelogram with slope \u03bb in Fig. 2 is:\n\nx = ex\n\n(1+ex)2 .\n\n1+eux ,\n\n1+eux\n\nA(\u03bb) = (\u03c3(ux) \u2212 \u03c3(lx) \u2212 \u03bb \u00b7 (ux \u2212 lx)) \u00b7 (ux \u2212 lx)\n\n(6)\n\n4\n\n\f\u02c6y\n\n\u02c6y = f (ux) + \u03bb \u00b7 (\u02c6x \u2212 ux)\n\n\u02c6y = f (lx) + \u03bb \u00b7 (\u02c6x \u2212 lx)\n\nFigure 2: Zonotope approximation for the sigmoid function parameterized by slope \u03bb, where\n0 \u2264 \u03bb \u2264 min(f(cid:48)(lx), f(cid:48)(ux)).\n\nlx\n\nux\n\n\u02c6x\n\nA(\u03bb) is a decreasing function of \u03bb and thus A(\u03bb) is minimized at \u03bbopt = min(\u03c3(cid:48)(lx), \u03c3(cid:48)(ux)). This\nyields the following theorem:\nTheorem 3.2 Let Z be the input to a smooth S-shaped1 function y = f (x) (such as the Sigmoid\n1+ex ). Consider the set of pointwise Zonotope approximations O of the output\nfunction y = \u03c3(x) = ex\nthat only alter the af\ufb01ne form \u02c6y of the variable y and where the box concretization of \u02c6y satis\ufb01es\nly = \u03c3(lx), uy = \u03c3(ux). The new af\ufb01ne form \u02c6y for the output with the minimum area in the xy-plane\nis given by:\n\n(cid:26)[f (ux)l, f (ux)u],\n\n[\u03bbl, \u03bbu] \u00b7 \u02c6x + [\u00b51\n\nl , \u00b51\n\n\u02c6y =\n\nu] + [\u00b52\n\nl , \u00b52\n\nif lx = ux,\nu] \u00b7 \u0001new, otherwise,\n\n(7)\n\nHere, f (ux)l, f (ux)u are \ufb02oating point representations of f (ux) rounded towards \u2212\u221e and +\u221e re-\nspectively and \u03bbl, \u03bbu are \ufb02oating point representations of \u03bbopt = min(f(cid:48)(lx), f(cid:48)(ux)) using rounding\ntowards \u2212\u221e and +\u221e respectively. Similarly \u00b51\nu and \u00b52\nu are \ufb02oating point representations of\n2 f (ux)\u2212f (lx)\u2212\u03bbopt\u00b7(ux\u2212lx) computed using\n\u00b51 = 1\nrounding towards \u2212\u221e and +\u221e and adding the error due to the non-associativity of \ufb02oating point\naddition, and \u0001new \u2208 [\u22121, 1] is a new noise symbol. As with ReLU, the optimal Sigmoid transformer\nin Theorem 3.2 has linear running time in the number of noise symbols and can be applied pointwise.\n\nl , \u00b51\n2 (f (ux)+f (lx)\u2212\u03bbopt\u00b7(ux +lx)) and \u00b52 = 1\n\nl , \u00b52\n\n3.3 Tanh\n\nThe Tanh function is also S-shaped, like the Sigmoid function. A fast, optimal, and pointwise Tanh\ntransformer can be de\ufb01ned using Theorem 3.2 by setting f (x) = tanh(x) and f(cid:48)(x) = 1\u2212 tanh2(x).\n\n4 Experiments\n\nWe now evaluate the effectiveness of our new Zonotope transformers for verifying local robustness\nof neural networks. Our implementation is available as an end-to-end automated veri\ufb01er, called\nDeepZ. The veri\ufb01er is implemented in Python, however, the underlying abstract transformers are\nimplemented in C (for performance) in both the sequential and the parallel version, and are made\navailable as part of the public ELINA [1, 25] library.\n\n4.1 Experimental setup\n\nEvaluation datasets. We used the popular MNIST [18] and CIFAR10 [16] datasets for our experi-\nments. MNIST contains 60 000 grayscale images of size 28 \u00d7 28 pixels and CIFAR10 consists of\n60 000 RGB images of size 32 \u00d7 32 pixels.\nNeural networks. Table 1 shows the fully connected feedforward (FFNNs), convolutional (CNNs),\nand residual networks for the MNIST and CIFAR10 datasets used in our experiments. We used both\n1A smooth function f : R \u2192 R is said to be S-shaped if f(cid:48)(x) \u2265 0 and there exists a value x(cid:48) such that for\n\nall x \u2208 R, we have f(cid:48)(cid:48)(x) \u2264 0 \u21d4 x \u2264 x(cid:48).\n\n5\n\n\fTable 1: Neural network architectures used in our experiments.\n\nDataset\n\nMNIST\n\nCIFAR10\n\nModel\n\nType\n\n#Hidden units\n\nFFNNSmall\nFFNNBig\nConvSmall\nConvMed\nConvBig\nConvSuper\nSkip\n\nFFNNBig\nConvSmall\nConvBig\n\nfully connected\nfully connected\nconvolutional\nconvolutional\nconvolutional\nconvolutional\nresidual\n\nfully connected\nconvolutional\nconvolutional\n\n610\n3 010\n3 604\n4 804\n34 688\n88 500\n71,650\n\n3 010\n4 852\n62 464\n\nVeri\ufb01ed robustness\n\nTime (s)\n\n100%\n\n80%\n\n60%\n\n40%\n\n20%\n\n0%\n\nAI2\nDeepZ\nFast-Lin\n\n60\n\n40\n\n20\n\n0\n\nAI2\nDeepZ\nFast-Lin\n\n0.010\n\n0.005\n0.025\n(a) MNIST FFNNSmall ReLU\n\n0.015\n\n0.020\n\n0.030\n\n0.010\n\n0.005\n0.025\n(b) MNIST FFNNSmall ReLU\n\n0.015\n\n0.020\n\n0.030\n\nFigure 3: Comparing the performance and precision of DeepZ with the state of the art.\n\nundefended and defended training procedures for training our networks. For adversarial training, we\nused DiffAI [21] and projected gradient descent (PGD) [6] parameterized with \u0001. In our evaluation,\nwe refer to the undefended nets as Point, and to the defended networks with the name of the training\nprocedure (either DiffAI or PGD). More details on our neural networks and the training procedures\ncan be found in the appendix.\nRobustness property. We consider the standard L\u221e-norm-based perturbation regions Rx,\u0001 [3],\nwhere Rx,\u0001 contains all perturbed inputs x(cid:48) where each pixel x(cid:48)\ni has a distance of at most \u0001 from the\ncorresponding pixel xi in the original input x. Rx,\u0001 can be exactly represented by a single zonotope.\nBenchmarks. We selected the \ufb01rst 100 images from the test set of each data set. Then, we speci\ufb01ed\na robustness property for each image using a set of robustness bounds \u0001.\n\n4.2 Experimental results\n\nAll experiments for the FFNNs were carried out on a 3.3 GHz 10 core Intel i9-7900X Skylake CPU\nwith 64 GB main memory; the CNNs and the residual network were evaluated on a 2.6 GHz 14 core\nIntel Xeon CPU E5-2690 with 512 GB main memory. We used a time limit of 10 minutes per run for\nall our experiments.\nComparison with prior work. We compare the precision and performance of the sequential version\nof DeepZ against two state-of-the-art certi\ufb01ers Fast-Lin [29] and AI2 [10] on the FFNNSmall MNIST\nnetwork with ReLU activation. We note that both of these certi\ufb01ers support only a subset of the\nnetwork architectures that DeepZ can support. Speci\ufb01cally, Fast-Lin only supports FFNNs with\nReLU activations whereas AI2 supports FFNNs and CNNs with ReLU activations. We also note that\nFast-Lin is not sound under \ufb02oating point semantics.\n\n6\n\n\fVeri\ufb01ed robustness\n\nTime (s)\n\n100%\n\n50%\n\n0%\n\nP GD\u0001=0.3\nP GD\u0001=0.1\n\nPoint\n\n30\n\n20\n\n10\n\n0\n\nP GD\u0001=0.3\nP GD\u0001=0.1\n\nPoint\n\n0.015\n\n0.020\n\n0.035\n(a) MNIST FFNNBig ReLU\n\n0.025\n\n0.030\n\n0.040\n\n0.015\n\n0.025\n\n0.020\n\n0.035\n(b) MNIST FFNNBig ReLU\n\n0.030\n\nVeri\ufb01ed robustness\n\nTime (s)\n\n0.040\n\nDiffAI\nPGD\nPoint\n\n100%\n\n50%\n\n0%\n\n0.040\n\n0.100\n0.020\n(c) MNIST ConvSmall ReLU\n\n0.060\n\n0.080\n\nDiffAI\nPGD\nPoint\n\n0.120\n\n2\n\n1.5\n\n1\n\n0.5\n\n0\n\n0.020\n\n0.040\n\n0.100\n(d) MNIST ConvSmall ReLU\n\n0.060\n\n0.080\n\n0.120\n\nFigure 4: Veri\ufb01ed robustness by DeepZ on the MNIST networks with ReLU activations.\n\nTable 2: Veri\ufb01ed robustness by DeepZ on the large networks trained with DiffAI.\n\n\u0001 % veri\ufb01ed robustness\n\naverage runtime(s)\n\nDataset\n\nMNIST\n\nModel\n\nConvBig\nConvBig\nConvBig\nConvSuper\nSkip\n\n0.1\n0.2\n0.3\n0.1\n0.1\n\nCIFAR10 ConvBig\n\n0.006\n\n97\n79\n37\n97\n95\n\n50\n\n5\n7\n17\n133\n29\n\n39\n\nFig. 3 shows the percentage of veri\ufb01ed robustness and the average analysis time of all three certi\ufb01ers.\nThe values of \u0001 are shown on the x-axis. DeepZ has the same precision as Fast-Lin but is up to 2.5x\ntimes faster. We note that the runtime of DeepZ increases with increasing value of \u0001; this is because\nthe complexity of our analysis is determined by the maximum number of noise symbols in the af\ufb01ne\nform. Our ReLU transformer creates one noise symbol for any variable that can take both positive\nand negative values. The number of such cases rises with the increasing value of \u0001. On the other hand,\nAI2 is signi\ufb01cantly less precise and slower compared to both Fast-Lin and DeepZ. We also compared\nDeepZ against the duality-based certi\ufb01er from [7], however it always timed out in our experiments.\nDetailed experiments. Next, we evaluate DeepZ on the remaining networks using the parallelized\nversion of our Zonotope transformers. Fig. 4 shows the percentage of veri\ufb01ed robustness and the\naverage analysis time of DeepZ for the MNIST networks with ReLU activations. DeepZ analyzes all\nFFNNBig networks with average runtime \u2264 22 seconds and proves 95% of the robustness properties\nfor \u0001 = 0.04 for the defended P GD\u0001=0.3 network. DeepZ is able to analyze all ConvSmall networks\nwith average runtime \u2264 2 seconds. It proves 95% of the robustness properties for \u0001 = 0.1 on the\n\n7\n\n\fVeri\ufb01ed robustness\n\nVeri\ufb01ed robustness\n\nP GD\u0001=0.3\nP GD\u0001=0.1\n\nPoint\n\nP GD\u0001=0.3\nP GD\u0001=0.1\n\nPoint\n\n100%\n\n50%\n\n0%\n\n0.010\n\n0.005\n0.025\n(a) MNIST FFNNBig Sigmoid\n\n0.015\n\n0.020\n\n100%\n\n50%\n\n0%\n\n0.030\n\n0.005\n\n0.015\n\n0.010\n\n0.025\n(b) MNIST FFNNBig Tanh\n\n0.020\n\n0.030\n\nVeri\ufb01ed robustness\n\nVeri\ufb01ed robustness\n\nP GD\u0001=0.3\nP GD\u0001=0.1\n\nPoint\n\nP GD\u0001=0.3\nP GD\u0001=0.1\n\nPoint\n\n100%\n\n50%\n\n0%\n\n0.040\n\n0.100\n0.020\n(c) MNIST ConvMed Sigmoid\n\n0.060\n\n0.080\n\n100%\n\n50%\n\n0%\n\n0.120\n\n0.020\n\n0.060\n\n0.040\n\n0.100\n(d) MNIST ConvMed Tanh\n\n0.080\n\n0.120\n\nFigure 5: Veri\ufb01ed robustness by DeepZ on the MNIST networks with Sigmoid and Tanh activations.\n\nConvSmall network defended with DiffAI. Table 2 shows the precision and the performance of\nDeepZ on ConvBig, ConvSuper, and Skip networks trained with DiffAI. DeepZ proves 95% and 97%\nof robustness properties for the Skip and ConvSuper networks containing > 71, 000 and > 88 000\nhidden units respectively in 29 and 133 seconds on average.\nFig. 5 shows the precision and the performance of DeepZ on the MNIST FFNNBig and ConvMed\nnetworks with Sigmoid and Tanh activations. It can be seen that DeepZ veri\ufb01es 74% of the robustness\nproperties on the FFNNBig Sigmoid and Tanh networks trained with P GD\u0001=0.3 for \u0001 = 0.03. DeepZ\nveri\ufb01es 82% of the robustness properties on the ConvMed Sigmoid network for \u0001 = 0.1. The\ncorresponding number for the Tanh network is 33%. We note that unlike the ReLU transformer, both\nSigmoid and Tanh transformers always create a new noise symbol whenever lx (cid:54)= ux. Thus, the\nruntime does not increase signi\ufb01cantly with \u0001 and is not plotted. DeepZ has an average runtime of\n\u2264 35 and \u2264 22 seconds on all FFNNBig and ConvMed networks, respectively.\nFig. 6 shows that DeepZ has an average runtime of \u2264 50 seconds for the CIFAR10 FFNNBig ReLU\nnetworks. It can be seen that the defended FFNNBig CIFAR10 ReLU networks are not signi\ufb01cantly\nmore provable than the undefended network. However, DeepZ veri\ufb01es more properties on the\ndefended ConvSmall networks than the undefended one and proves 75% of robustness properties\non the DiffAI defended network for \u0001 = 0.01. DeepZ has an average runtime of \u2264 3 seconds on all\nConvSmall networks. DeepZ is able to verify 50% of robustness properties for ConvBig network\ndefended with DiffAI with an average runtime of 39 seconds as shown in Table 2.\nDeepZ veri\ufb01es 82% of robustness properties on the FFNNBig Sigmoid network defended with\nP GD\u0001=0.0078 for \u0001 = 0.012 in Fig. 7. It veri\ufb01es 46% of the robustness properties on the FFNNBig\nnetwork with Tanh activation trained using P GD\u0001=0.0313 for the same \u0001. The average runtime of\nDeepZ on all networks is \u2264 90 seconds.\n\n8\n\n\fVeri\ufb01ed robustness\n\nTime (s)\n\n100%\n\n50%\n\n0%\n\nP GD\u0001=0.0313\nP GD\u0001=0.0078\n\nPoint\n\n80\n\n60\n\n40\n\n20\n\n0\n\nP GD\u0001=0.0313\nP GD\u0001=0.0078\n\nPoint\n\n0.0005\n\n0.0010\n\n0.0025\n(a) CIFAR10 FFNNBig ReLU\n\n0.0020\n\n0.0015\n\n0.0030\n\n0.0005\n\n0.0010\n\n0.0015\n\n0.0025\n(b) CIFAR10 FFNNBig ReLU\n\n0.0020\n\n0.0030\n\nVeri\ufb01ed robustness\n\nTime (s)\n\n100%\n\n50%\n\n0%\n\nDiffAI\nPGD\nPoint\n\n6\n\n4\n\n2\n\n0\n\nDiffAI\nPGD\nPoint\n\n0.004\n\n0.010\n0.002\n(c) CIFAR10 ConvSmall ReLU\n\n0.008\n\n0.006\n\n0.012\n\n0.004\n\n0.002\n0.010\n(d) CIFAR10 ConvSmall ReLU\n\n0.008\n\n0.006\n\n0.012\n\nFigure 6: Veri\ufb01ed robustness by DeepZ on the CIFAR10 networks with ReLU activations.\n\nVeri\ufb01ed robustness\n\nVeri\ufb01ed robustness\n\nP GD\u0001=0.0313\nP GD\u0001=0.0078\n\nPoint\n\nP GD\u0001=0.0313\nP GD\u0001=0.0078\n\nPoint\n\n100%\n\n50%\n\n0%\n\n0.0004\n\n0.0002\n0.0010\n(a) CIFAR10 FFNNBig Sigmoid\n\n0.0008\n\n0.0006\n\n100%\n\n50%\n\n0%\n\n0.0012\n\n0.0002\n\n0.0004\n\n0.0006\n\n0.0010\n(b) CIFAR10 FFNNBig Tanh\n\n0.0008\n\n0.0012\n\nFigure 7: Veri\ufb01ed robustness by DeepZ on the CIFAR10 networks with Sigmoid and Tanh activations.\n\n5 Conclusion\n\nWe introduced fast and precise Zonotope abstract transformers for key non-linear activations used\nin modern neural networks. We used these transformers to build DeepZ, an automated veri\ufb01er for\nproving the robustness of neural networks against adversarial attacks. We evaluated the effectiveness\nof DeepZ on verifying robustness of large feedforward, convolutional, and residual networks against\nchallenging L\u221e-norm attacks. Our results show that DeepZ is more precise and faster than prior\nwork, while also ensuring soundness with respect to \ufb02oating point operations.\n\n9\n\n\fReferences\n[1] ELINA: ETH Library for Numerical Analysis. http://elina.ethz.ch.\n\n[2] Anish Athalye and Ilya Sutskever. Synthesizing robust adversarial examples. In Proc. Interna-\n\ntional Conference on Machine Learning (ICML), 2018.\n\n[3] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks.\n\nIn Proc. IEEE Symposium on Security and Privacy (SP), pages 39\u201357, 2017.\n\n[4] Nicholas Carlini, Guy Katz, Clark Barrett, and David L. Dill. Ground-truth adversarial examples.\n\nCoRR, abs/1709.10207, 2017.\n\n[5] Patrick Cousot and Radhia Cousot. Abstract interpretation: A uni\ufb01ed lattice model for static\nanalysis of programs by construction or approximation of \ufb01xpoints. In Proc. Symposium on\nPrinciples of Programming Languages (POPL), pages 238\u2013252, 1977.\n\n[6] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li.\nBoosting adversarial attacks with momentum. In The IEEE Conference on Computer Vision\nand Pattern Recognition (CVPR), 2018.\n\n[7] Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, and Pushmeet Kohli.\nA dual approach to scalable veri\ufb01cation of deep networks. In Proc. Uncertainty in Arti\ufb01cial\nIntelligence (UAI), pages 162\u2013171, 2018.\n\n[8] R\u00fcdiger Ehlers. Formal veri\ufb01cation of piece-wise linear feed-forward neural networks. In\n\nAutomated Technology for Veri\ufb01cation and Analysis (ATVA), pages 269\u2013286, 2017.\n\n[9] Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash,\nAmir Rahmati, and Dawn Song. Robust physical-world attacks on deep learning models. arXiv\npreprint arXiv:1707.08945, 1, 2017.\n\n[10] T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev. Ai2:\nSafety and robustness certi\ufb01cation of neural networks with abstract interpretation. In Proc.\nIEEE Symposium on Security and Privacy (SP), pages 948\u2013963, 2018.\n\n[11] Khalil Ghorbal, Eric Goubault, and Sylvie Putot. The zonotope abstract domain taylor1+. In\n\nProc. Computer Aided Veri\ufb01cation (CAV), pages 627\u2013633, 2009.\n\n[12] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial\n\nexamples. In Proc. International Conference on Learning Representations (ICLR), 2015.\n\n[13] Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial\n\nexamples. arXiv preprint arXiv:1412.5068, 2014.\n\n[14] Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Relu-\nplex: An ef\ufb01cient SMT solver for verifying deep neural networks. In Proc. Computer Aided\nVeri\ufb01cation (CAV), pages 97\u2013117, 2017.\n\n[15] J Zico Kolter and Eric Wong. Provable defenses against adversarial examples via the convex\nouter adversarial polytope. In Proc. International Conference on Machine Learning (ICML),\n2018.\n\n[16] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report,\n\nUniversity of Toronto, 2009.\n\n[17] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical\n\nworld. CoRR, abs/1607.02533, 2016.\n\n[18] Yann Lecun, L\u00e9on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied\n\nto document recognition. In Proc. of the IEEE, pages 2278\u20132324, 1998.\n\n[19] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.\nTowards deep learning models resistant to adversarial attacks. In Proc. International Conference\non Learning Representations (ICLR), 2018.\n\n10\n\n\f[20] Antoine Min\u00e9. Relational abstract domains for the detection of \ufb02oating-point run-time errors.\n\nIn Proc. European Symposium on Programming (ESOP), pages 3\u201317, 2004.\n\n[21] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for\nprovably robust neural networks. In Proc. International Conference on Machine Learning\n(ICML), 2018.\n\n[22] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learn-\narXiv preprint\n\nfrom phenomena to black-box attacks using adversarial samples.\n\ning:\narXiv:1605.07277, 2016.\n\n[23] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Anan-\nthram Swami. Practical black-box attacks against machine learning. In Proc. Asia Conference\non Computer and Communications Security, pages 506\u2013519, 2017.\n\n[24] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certi\ufb01ed defenses against adversarial\n\nexamples. In Proc. International Conference on Machine Learning (ICML), 2018.\n\n[25] Gagandeep Singh, Markus P\u00fcschel, and Martin Vechev. Fast polyhedra abstract domain. In\n\nProc. Principles of Programming Languages (POPL), pages 46\u201359, 2017.\n\n[26] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J.\nGoodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199,\n2013.\n\n[27] Vincent Tjeng and Russ Tedrake. Verifying neural networks with mixed integer programming.\n\nCoRR, abs/1711.07356, 2017.\n\n[28] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Formal security\nanalysis of neural networks using symbolic intervals. In USENIX Security Symposium (USENIX\nSecurity 18), pages 1599\u20131614, 2018.\n\n[29] Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane\nBoning, and Inderjit Dhillon. Towards fast computation of certi\ufb01ed robustness for ReLU\nnetworks. In Proc. International Conference on Machine Learning (ICML), pages 5273\u20135282,\n2018.\n\n[30] Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J. Zico Kolter. Scaling provable adversarial\n\ndefenses. CoRR, abs/1805.12514, 2018. URL http://arxiv.org/abs/1805.12514.\n\n11\n\n\fA Dataset Normalization\n\nFor each dataset we include a normalization layer (which gets applied after the \u0001-sized box has been\ncalculated) using an approximated mean \u00b5 and standard deviation \u03c3 per channel as X\u2212\u00b5\n\u03c3 .\nMNIST: \u00b5 = 0.1307, \u03c3 = 0.3081.\nCIFAR10: \u00b5 = [0.4914, 0.4822, 0.4465], \u03c3 = [0.2023, 0.1994, 0.2010].\n\nB Neural Networks Evaluated\n\nWe test with six networks: one feed forward, four convolutional (without maxpool), and one with\na residual connection. These are trained in various ways. In the following descriptions, we use\nConvsC \u00d7 W \u00d7 H to mean a convolutional layer that outputs C channels, with a kernel width of W\npixels and height of H, with a stride of s which then applies ReLU to every output. FC n is a fully\nconnected layer which outputs n neurons without automatically applying ReLU.\nFor each architecture we test three versions: (i) an undefended network; (ii) a network defended with\nMI-FGSM (a PGD variant which we refer to as PGD in the graphs) [6] with \u00b5 = 1, 22 iterations and\ntwo restarts, where the step size is \u0001 = 5.5\u22121 for the \u0001 used for training; (iii) a network defended with\na system based on DiffAI [21].\nFFNN. A 6 layer feed forward net with 500 nodes in each and an activation (ReLU, Sigmoid, Tanh)\nafter each layer except the last.\nConvSmall. Our smallest convolutional network with no convolutional padding.\n\nx \u2192 Conv216 \u00d7 4 \u00d7 4 \u2192 ReLU \u2192 Conv232 \u00d7 4 \u00d7 4 \u2192 ReLU \u2192 FC 100 \u2192 z.\n\nConvMed. Similar to ConvSmall, but with a convolutional padding of 1. Here we test with the three\nactivations Act = ReLU, Sigmoid, and Tanh.\n\nx \u2192 Conv216 \u00d7 4 \u00d7 4 \u2192 Act \u2192 Conv232 \u00d7 4 \u00d7 4 \u2192 Act \u2192 FC 1000 \u2192 z.\n\nConvBig. A signi\ufb01cantly larger convolutional network with a convolutional padding of 1.\n\nx \u2192 Conv132 \u00d7 3 \u00d7 3 \u2192 ReLU \u2192 Conv232 \u00d7 4 \u00d7 4 \u2192 ReLU\n\u2192 Conv164 \u00d7 3 \u00d7 3 \u2192 ReLU \u2192 Conv264 \u00d7 4 \u00d7 4 \u2192 ReLU\n\u2192 FC 512 \u2192 ReLU \u2192 FC 512 \u2192 z.\n\nConvSuper Our largest convolutional network with no padding.\n\nx \u2192 Conv132 \u00d7 3 \u00d7 3 \u2192 Conv132 \u00d7 4 \u00d7 4\n\u2192 Conv164 \u00d7 3 \u00d7 3 \u2192 Conv164 \u00d7 4 \u00d7 4\n\u2192 FC 512 \u2192 ReLU \u2192 FC 512 \u2192 z.\n\nSkip Two convolutional networks of different sizes, which are then concatenated together. This\nnetwork uses no convolutional padding.\n\nx \u2192 Conv116 \u00d7 3 \u00d7 3 \u2192 ReLU\n\u2192 Conv116 \u00d7 3 \u00d7 3 \u2192 ReLU\n\u2192 Conv132 \u00d7 3 \u00d7 3 \u2192 ReLU \u2192 FC 200 \u2192 o1,\nx \u2192 Conv132 \u00d7 4 \u00d7 4 \u2192 ReLU\n\u2192 Conv132 \u00d7 4 \u00d7 4 \u2192 ReLU \u2192 FC 200 \u2192 o2,\nCAT(o1, o2) \u2192 ReLU \u2192 FC 200 \u2192 ReLU\n\u2192 z.\n\n12\n\n\f", "award": [], "sourceid": 6892, "authors": [{"given_name": "Gagandeep", "family_name": "Singh", "institution": "ETH Zurich"}, {"given_name": "Timon", "family_name": "Gehr", "institution": "ETH Zurich"}, {"given_name": "Matthew", "family_name": "Mirman", "institution": "ETH Zurich"}, {"given_name": "Markus", "family_name": "P\u00fcschel", "institution": "ETH Zurich"}, {"given_name": "Martin", "family_name": "Vechev", "institution": "DeepCode and ETH Zurich, Switzerland"}]}