{"title": "Demystifying Black-box Models with Symbolic Metamodels", "book": "Advances in Neural Information Processing Systems", "page_first": 11304, "page_last": 11314, "abstract": "Understanding the predictions of a machine learning model can be as crucial as the model's accuracy in many application domains. However, the black-box nature of most highly-accurate (complex) models is a major hindrance to their interpretability. To address this issue, we introduce the symbolic metamodeling framework \u2014 a general methodology for interpreting predictions by converting \"black-box\" models into \"white-box\" functions that are understandable to human subjects. A symbolic metamodel is a model of a model, i.e., a surrogate model of a trained (machine learning) model expressed through a succinct symbolic expression that comprises familiar mathematical functions and can be subjected to symbolic manipulation. We parameterize symbolic metamodels using Meijer G-functions \u2014 a class of complex-valued contour integrals that depend on scalar parameters, and whose solutions reduce to familiar elementary, algebraic, analytic and closed-form functions for different parameter settings. This parameterization enables efficient optimization of metamodels via gradient descent, and allows discovering the functional forms learned by a machine learning model with minimal a priori assumptions. We show that symbolic metamodeling provides an all-encompassing framework for model interpretation \u2014 all common forms of global and local explanations of a model can be analytically derived from its symbolic metamodel.", "full_text": "Demystifying Black-box Models with\n\nSymbolic Metamodels\n\nAhmed M. Alaa\nECE Department\n\nUCLA\n\nMihaela van der Schaar\n\nUCLA, University of Cambridge, and\n\nahmedmalaa@ucla.edu\n\n{mv472@cam.ac.uk,mihaela@ee.ucla.edu}\n\nAlan Turing Institute\n\nAbstract\n\nUnderstanding the predictions of a machine learning model can be as crucial as the\nmodel\u2019s accuracy in many application domains. However, the black-box nature of\nmost highly-accurate (complex) models is a major hindrance to their interpretability.\nTo address this issue, we introduce the symbolic metamodeling framework \u2014 a\ngeneral methodology for interpreting predictions by converting \u201cblack-box\u201d models\ninto \u201cwhite-box\u201d functions that are understandable to human subjects. A symbolic\nmetamodel is a model of a model, i.e., a surrogate model of a trained (machine\nlearning) model expressed through a succinct symbolic expression that comprises\nfamiliar mathematical functions and can be subjected to symbolic manipulation. We\nparameterize metamodels using Meijer G-functions \u2014 a class of complex-valued\ncontour integrals that depend on real-valued parameters, and whose solutions reduce\nto familiar algebraic, analytic and closed-form functions for different parameter\nsettings. This parameterization enables ef\ufb01cient optimization of metamodels via\ngradient descent, and allows discovering the functional forms learned by a model\nwith minimal a priori assumptions. We show that symbolic metamodeling provides\na generalized framework for model interpretation \u2014 many common forms of model\nexplanation can be analytically derived from a symbolic metamodel.\n\n1\n\nIntroduction\n\nThe ability to interpret the predictions of a machine learning model brings about user trust and supports\nunderstanding of the underlying processes being modeled. In many application domains, such as the\nmedical and legislative domains [1\u20133], model interpretability can be a crucial requirement for the\ndeployment of machine learning, since a model\u2019s predictions would inform critical decision-making.\nModel explanations can also be central in other domains, such as social and natural sciences [4, 5],\nwhere the primary utility of a model is to help understand an underlying phenomenon, rather than\nmerely making predictions about it. Unfortunately, most state-of-the-art models \u2014 such as ensemble\nmodels, kernel methods, and neural networks \u2014 are perceived as being complex \u201cblack-boxes\u201d, the\npredictions of which are too hard to be interpreted by human subjects [1, 6\u201316].\nSymbolic metamodeling. In this paper, we approach the problem of model interpretation by introduc-\ning the symbolic metamodeling framework for expressing black-box models in terms of transparent\nmathematical equations that can be easily understood and analyzed by human subjects (Section 2).\nThe proposed metamodeling procedure takes as an input a (trained) model \u2014 represented by a black-\nbox function f (x) that maps a feature x to a prediction y \u2014 and retrieves a symbolic metamodel\ng(x), which is meant to be an interpretable mathematical abstraction of f (x). The metamodel g(x)\nis a tractable symbolic expression comprising a \ufb01nite number of familiar functions (e.g., polynomial,\nanalytic, algebraic, or closed-form expressions) that are combined via elementary arithmetic opera-\ntions (i.e., addition and multiplication), which makes it easily understood by inspection, and can be\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Pictorial depiction of the symbolic metamodeling framework. Here, the model f (x) is a deep\nneural network (left), and the metamodel g(x) is a closed-form expression x1 x2 (1 \u2212 x2 exp(\u2212x1)) (right).\nanalytically manipulated via symbolic computation engines such as Mathematica [17], Wolfram\nalpha [18], or Sympy [19]. Our approach is appropriate for models with small to moderate number\nof features, where the physical interpretation of these features are of primary interest.\nA high-level illustration of the proposed metamodeling approach is shown in Figure 1. In this Figure,\nwe consider an example of using a neural network to predict the risk of cardiovascular disease based\non a (normalized) feature vector x = (x1, x2), where x1 is a person\u2019s age and x2 is their blood\npressure. For a clinician using this model in their daily practice or in the context of an epidemiological\nstudy, the model f (x) is completely obscure \u2014 it is hard to explain or draw insights into the model\u2019s\npredictions, even with a background knowledge of neural networks. On the other hand, the metamodel\ng(x) = x1 x2 (1\u2212 x2 exp(\u2212x1)) is a fully transparent abstraction of the neural network model, from\nwhich one can derive explanations for the model\u2019s predictions through simple analytic manipulation,\nwithout the need to know anything about the model structure and its inner workings1. Interestingly\nenough, having such an explicit (simulatable) equation for predicting risks is already required by\nvarious clinical guidelines to ensure the transparency of prognostic models [21].\nMetamodeling with Meijer G-functions. In order to \ufb01nd the symbolic metamodel g(x) that best\napproximates the original model f (x), we need to search a space of mathematical expressions and\n\ufb01nd the expression that minimizes a \u201cmetamodeling loss\u201d (cid:96)(g(x), f (x)). But how can we construct a\nspace of symbolic expressions without predetermining its functional from? In other words, how do\nwe know that the metamodel g(x) = x1 x2 (1 \u2212 x2 exp(\u2212x1)) in Figure 1 takes on an exponential\nform and not, say, a trigonometric or a polynomial functional form?\nTo answer this question, we introduce a novel parameterized representation of symbolic expressions\n(Section 3), G(x; \u03b8), which reduces to most familiar functional forms \u2014 e.g., arithmetic, polynomial,\nalgebraic, closed-form, and analytic expressions, in addition to special functions, such as Bessel\nfunctions and Hypergeometric functions \u2014 for different settings of a real-valued parameter \u03b8. The\nrepresentation G(x; \u03b8) is based on Meijer G-functions [22\u201324], a class of contour integrals used in\nthe mathematics community to \ufb01nd closed-form solutions for otherwise intractable integrals. The\nproposed Meijer G-function parameterization enables minimizing the metamodeling loss ef\ufb01ciently\nvia gradient descent \u2014 this is a major departure from existing approaches to symbolic regression,\nwhich use genetic programming to select among symbolic expressions that comprise a small number\nof predetermined functional forms [25\u201327].\nSymbolic metamodeling as a gateway to all explanations. Existing methods for model interpre-\ntation focus on crafting explanation models that support only one \u201cmode\u201d of model interpretation.\nFor instance, methods such as DeepLIFT [8] and LIME [16], can explain the predictions of a model\nin terms of the contributions of individual features to the prediction, but cannot tell us whether the\nmodel is nonlinear, or whether statistical interactions between features exist. Other methods such\nas GA2M [9] and NIT [13], focus exclusively on uncovering the statistical interactions captured by\nthe model, which may not be the most relevant mode of explanation in many application domains.\nMoreover, none of the existing methods can uncover the functional forms by which a model captures\nnonlinearities in the data \u2014 such type of interpretation is important in applications such as applied\nphysics and material sciences, since researchers in these \ufb01elds focus on distilling an analytic law that\ndescribes how the model \ufb01ts experimental data [4, 5].\n\n1Note that here we are concerned with explaining the predictions of a trained model, i.e., its response surface.\n\nOther works, such as [20], focus on explaining the model\u2019s loss surface in order to understand how it learns.\n\n2\n\nModel Metamodel Symbolic Metamodeling Age Blood pressure Age Blood pressure \fOur perspective on model interpretation departs from previous works in that, a symbolic metamodel\ng(x) is not hardwired to provide any speci\ufb01c type of explanation, but is rather designed to provide\na full mathematical description of the original model f (x). In this sense, symbolic metamodeling\nshould be understood as a tabula rasa upon which different forms of explanations can be derived \u2014\nas we will show in Section 4, most forms of model explanation covered in previous literature can be\narrived at through simple analytic manipulation of a symbolic metamodel.\n\n2 Symbolic Metamodeling\nLet f : X \u2192 Y be a machine learning model trained to predict a target outcome y \u2208 Y on the basis of\na d-dimensional feature instance x = (x1, . . . , xd) \u2208 X . We assume that f (.) is a black-box model\nto which we only have query access, i.e., we can evaluate the model\u2019s output y = f (x) for any given\nfeature instance x, but we do not have any knowledge of the model\u2019s internal structure. Without loss\nof generality, we assume that the feature space X is the unit hypercube, i.e., X = [0, 1]d.\nThe metamodeling problem. A symbolic metamodel g \u2208 G\nis a \u201cmodel of the model\u201d f that approximates f (x) for all\nx \u2208 X , where G is a class of succinct mathematical expres-\nsions that are understandable to users and can be analytically\nmanipulated. Typically, G would be set as the class of all\narithmetic, polynomial, algebraic, closed-form, or analytic\nexpressions. Choice of G will depend on the desired com-\nplexity of the metamodel, which in turn depends on the appli-\ncation domain. For instance, in experimental physics, special\nfunctions \u2014 such as Bessel functions \u2014 would be considered\ninterpretable [4], and hence we can take G to be the set of all\nanalytic functions. On the contrary, in medical applications,\nwe might opt to restrict G to algebraic expressions. Given G,\nthe metamodeling problem consists in \ufb01nding the function g\nin G that bests approximates the model f.\nFigure 2 shows a pictorial depiction of the metamodeling problem as a mapping from the modeling\nspace F \u2014 i.e., the function class that the model f inhabits2 \u2014 to the interpretable metamodeling\nspace G. Metamodling is only relevant when F spans functions that are considered uninterpretable to\nusers. For models that are deemed interpretable, such as linear regression, F will already coincide\nwith G, because the linear model is already an algebraic expression (and a \ufb01rst-order polynomial). In\nthis case, the best metamodel for f is the model f itself, i.e., g = f.\nFormally, the metamodeling problem can be de\ufb01ned through the following optimization problem:\n\nFigure 2: The metamodeling problem.\n\n(cid:90)\n\nX\n\ng\u2217 = arg min\n\ng\u2208G (cid:96)(g, f ),\n\n(cid:96)(g, f ) = (cid:107) f \u2212 g (cid:107)2\n\n2 =\n\n(g(x) \u2212 f (x))2 dx,\n\n(1)\n\nwhere (cid:96)(.) is the metamodeling loss, which we set to be the mean squared error (MSE) between f and g.\nIn the following Section, we will focus on solving the optimization problem in (1).\n\n3 Metamodeling via Meijer G-functions\n\nIn order to solve the optimization problem in (1), we need to induce some structure into the meta-\nmodeling space G. This is obviously very challenging since G encompasses in\ufb01nitely many possible\nmathematical expressions with very diverse functional forms. For instance, consider the exemplary\nmetamodel in Figure 1, where g(x) = x1 x2 (1 \u2212 x2 exp(\u2212x1)). If G is set to be the space of all\nclosed-form expressions, then it would include all polynomial, hyperbolic, trigonometric, logarithmic\nfunctions, rational and irrational exponents, and any combination thereof [28, 29]. Expressions such\n2) and g(cid:48)(cid:48)(x) = sin(x1) \u00b7 cos(x2) are both valid metamodels, i.e., g(cid:48), g(cid:48)(cid:48) \u2208 G, yet\nas g(cid:48)(x) = (x2\nthey each have functional forms that are very different from g. Thus, we need to parameterize G in\nsuch a way that it encodes all such functional forms, and enables an ef\ufb01cient solution to (1).\n\n1 + x2\n\n2For instance, for an L-layer neural network, F is the space of compositions of L nested activation functions.\n\nFor a random forest with L trees, F is the space of summations of L piece-wise functions.\n\n3\n\nModel space Algebraic Closed-form Analytic Metamodel space Increasing complexity \fTo this end, we envision a parameterized metamodel g(x) = G(x; \u03b8), \u03b8 \u2208 \u0398, where \u0398 = RM is a\nparameter space that fully speci\ufb01es the metamodeling space G, i.e., G = {G(.; \u03b8) : \u03b8 \u2208 \u0398}. Such\nparameterization should let G(x; \u03b8) reduce to different functions for different settings of \u03b8 \u2014 for the\n2) and G(x; \u03b8(cid:48)(cid:48)) = sin(x1) \u00b7 cos(x2)\naforementioned example, we should have G(x; \u03b8(cid:48)) = (x2\nfor some \u03b8(cid:48), \u03b8(cid:48)(cid:48) \u2208 \u0398. Given the parameterization G(x; \u03b8), the problem in (1) reduces to\n\n1 + x2\n\ng\u2217(x) = G(x; \u03b8\u2217), where \u03b8\u2217 = arg min\n\u03b8\u2208\u0398\n\n(cid:96)(G(x; \u03b8), f (x)).\n\n(2)\n\nThus, if we have a parameterized symbolic expression G(x; \u03b8), then the metamodeling problem boils\ndown to a straightforward parameter optimization problem. We construct G(x; \u03b8) in Section 3.1.\n\n3.1 Parameterizing symbolic metamodels with Meijer G-functions\n\nWe propose a parameterization of G(x; \u03b8) that includes two steps. The \ufb01rst step involves decomposing\nthe metamodel G(x; \u03b8) into a combination of univariate functions. The second step involves modeling\nthese univariate functions through a very general class of special functions that includes most known\nfamiliar functions as particular cases. Both steps are explained in detail in what follows.\nStep 1: Decomposing the metamodel. We breakdown the multivariate function g(x) into simpler,\nunivariate functions. From the Kolmogorov superposition theorem [30], we know that every mul-\ntivariate continuous function g(x) can be written as a \ufb01nite composition of univariate continuous\nfunctions and the addition operation as follows3:\n\nr(cid:88)\n\n\uf8eb\uf8ed d(cid:88)\n\ni=0\n\nj=1\n\n\uf8f6\uf8f8 ,\n\ng(x) = g(x1, . . . , xn) =\n\ngout\ni\n\ngin\nij (xj)\n\n(3)\n\ni\n\ni and gout\n\n: R \u2192 R, and gin\n\nij are continuous univariate basis functions, and r \u2208 N+. The exact decomposition\nwhere gin\nij : [0, 1] \u2192 R [36].\nin (3) always exists for r = 2d, and for some basis functions gout\nWhen r = 1, (3) reduces to the generalized additive model [37]. While we proceed our analysis with\nthe general formula in (3), in our practical implementation we set r = 1, gout as the identify function,\nij of the interactions {xi xj}i,j to account for the complexity of g(x).\nand include extra functions gin\nStep 2: Meijer G-functions as basis functions. Based on the decomposition in (3), we can now\nij }i,j),\nparameterize metamodels in terms of their univariate bases, i.e., G(x; \u03b8) = G(x;{gout\nwhere every selection of a different set of bases would lead to a different corresponding metamodel.\nHowever, in order to fully specify the parameterization G(x; \u03b8), we still need to parameterize the\nbasis functions themselves in terms of real-valued parameters that we can practically optimize, while\nensuring that the corresponding parameter space spans a wide range of symbolic expressions.\nTo fully specify G(x; \u03b8), we model the basis functions in (3) as instances of a Meijer G-function \u2014 a\nunivariate special function given by the following line integral in the complex plane s [22, 23]:\n\ni }i,{gin\n\n(cid:0) a1,...,ap\n\nb1,...,bq\n\n(cid:12)(cid:12)x(cid:1) =\n\nGm,n\np,q\n\n(cid:90)\n\n(cid:81)m\nj=1 \u0393(bj \u2212 s)(cid:81)n\n(cid:81)q\nj=m+1 \u0393(1 \u2212 bj + s)(cid:81)p\n\n1\n2\u03c0i\n\nL\n\nj=1 \u0393(1 \u2212 aj + s)\n\nj=n+1 \u0393(aj + s)\n\nxs ds,\n\n(4)\n\nwhere \u0393(.) is the Gamma function and L is the integration path in the complex plane. (In Appendix A,\nwe provide conditions for the convergence of the integral in (4), and the detailed construction of\nthe integration path L.) The contour integral in (4) is known as Mellin-Barnes representation [24].\nAn instance of a Meijer G-function is speci\ufb01ed by the real-valued parameters ap = (a1, . . ., ap),\nbq = (b1, . . ., bq), and indexes n and m, which de\ufb01ne the poles and zeros of the integrand in (4) on the\ncomplex plane4. In the rest of the paper, we refer to Meijer G-functions as G functions for brevity.\nFor each setting of ap and bq, the integrand in (4) is con\ufb01gured with different poles and zeros, and\nthe resulting integral converges to a different function of x. A powerful feature of the G function is\nthat it encompasses most familiar functions as special cases [24] \u2014 for different settings of ap and\nbq, it reduces to almost all known elementary, algebraic, analytic, closed-form and special functions.\n\n3The Kolmogorov decomposition in (3) is a universal function approximator [31]. In fact, (3) can be thought\n4Since \u0393(x) = (x\u22121)!, the zeros of factors \u0393(bj\u2212s) and \u0393(1\u2212aj +s) are (bj\u2212k) and (1\u2212aj\u2212k), k \u2208 N0,\n\nof as a 2-layer neural network with generalized activation functions [32\u201334, 31, 35].\nrespectively, whereas the poles of \u0393(1 \u2212 bj + s) and \u0393(aj + s) are (\u2212aj \u2212 k) and (1 \u2212 bj \u2212 k), k \u2208 N0.\n\n4\n\n\fExamples for special values of the poles and zeros for\nwhich the G function reduces to familiar functions are\nshown in Table 1. (A more elaborate Table of equivalen-\ncies is provided in Appendix A.) Perturbing the poles\nand zeros around their values in Table 1 gives rise to\nvariants of these functional forms, e.g., x log(x), sin(x),\nx2e\u2212x, etc. A detailed illustrative example for the dif-\nferent symbolic expressions that G functions take on a\n2D parameter space is provided in Appendix A. Tables\nof equivalence between G functions and familiar func-\ntions can be found in [38], or computed using programs\nsuch as Mathematica [17] and Sympy [19].\n\nG-function\n\n1\n\nG0,1\n3,1\n0,1 ( \u2212\nG1,0\nG1,2\n2,2\n\n(cid:12)(cid:12)x(cid:1)\n(cid:0) 2,2,2\n(cid:12)(cid:12)x(cid:1)\n(cid:0) 1,1\n0 |x)\n(cid:12)(cid:12)(cid:12) x2\n(cid:17)\n(cid:16) \u2212\n(cid:12)(cid:12)(cid:12)x\n(cid:16) 1\n(cid:17)\n\n0, 1\n2\n2 ,1\n1\n2 ,0\n\n1,0\n\n4\n\nG1,0\n0,2\n\nG1,2\n2,2\n\nEquivalent form\n\nx\ne\u2212x\n\nlog(1 + x)\n\n1\u221a\n\n\u03c0 cos(x)\n\n2 arctan(x)\n\nTable 1: Representation of familiar elementary\nfunctions in terms of the G function.\n\nBy using G functions as univariate basis functions (gin\narrive at the following parameterization for G(x; \u03b8):\n\ni and gout\n\nij ) for the decomposition in (3), we\n\nr(cid:88)\n\ni=0\n\n(cid:16)\n\n(cid:12)(cid:12)(cid:12)\n\nd(cid:88)\n\nj=1\n\n(cid:0)\u03b8in\n\nij | xj\n\n(cid:1)(cid:17)\n\nG(x; \u03b8) =\n\nGm,n\np,q\n\n\u03b8out\ni\n\nGm,n\np,q\n\n,\n\n(5)\n\nr\n\n0\n\n, . . ., \u03b8out\n\ni1 , . . ., \u03b8in\n\n) and \u03b8in = {(\u03b8in\n\nid )}i are the G function parameters.\np,q (ap, bq | x), \u03b8 = (ap, bq), as a shortened notation for the G function\n\nwhere \u03b8 = (\u03b8out, \u03b8in), \u03b8out = (\u03b8out\np,q (\u03b8 | x) = Gm,n\nHere, we use Gm,n\nfor convenience. The indexes (m, n, p, q, r) are viewed as hyperparameters of the metamodel.\nSymbolic metamodeling in action. To demonstrate how the parameterization G(x; \u03b8) in (5) captures\nsymbolic expressions, we revisit the stylized example in Figure 1. Recall that in Figure 1, we had a\nneural network model with two features, x1 and x2, and a metamodel g(x) = x1 x2 (1\u2212 x2 e\u2212x1). In\nwhat follows, we show how the metamodel g(x) can be arrived at from the parameterization G(x; \u03b8).\nFigure 3 shows a schematic illustration for\nthe parameterization G(x; \u03b8) in (5) \u2014 with\nr = 2 \u2014 put in the format of a \u201ccomputa-\ntion graph\u201d. Each box in this graph corre-\ni }i\nsponds to one of the basis functions {gin\nand {gout\nij }i,j, and inside each box, we show\nthe corresponding instance of G function\nthat is needed to give rise to the symbolic\nexpression g(x) = x1 x2 (1\u2212 x2e\u2212x1 ). To\ntune the poles and zeros of each of the 6 G\nfunctions in Figure 3 to the correct values,\nwe need to solve the optimization problem\nin (2). In Section 3.2, we show that this can\nbe done ef\ufb01ciently via gradient descent.\n\nFigure 3: Schematic for the metamodel in Figure 1.\n\n3.2 Optimizing symbolic metamodels via gradient descent\n\nAnother advantage of the parameterization in (5) is that the gradients of the G function with respect\nto its parameters can be approximated in analytic form as follows [24]:\n\n(cid:16) ap\n(cid:16) ap\n\nbq\n\nbq\n\n(cid:12)(cid:12)(cid:12)x\n(cid:12)(cid:12)(cid:12)x\n\n(cid:17) \u2248 xak\u22121 \u00b7 Gm,n+1\n(cid:17) \u2248 x1\u2212bk \u00b7 Gm,n\n(cid:0)\n\np+1,q\n\np,q+1\n\n(cid:16) \u22121,a1\u22121,. . .,an\u22121,an+1\u22121,. . .,ap\u22121\n\nb1,. . .,bm,bm+1,. . .,bq\n\na1,. . .,an,an+1,. . .,ap\n\nb1\u22121,. . .,bm\u22121,0,bm+1\u22121,. . .,bq\u22121\n\n(cid:12)(cid:12)(cid:12)x\n(cid:17)\n(cid:12)(cid:12)x(cid:1) 1 \u2264 k \u2264 q.\n\n, 1 \u2264 k \u2264 p,\n\nd\ndak\nd\ndbk\n\nGm,n\np,q\n\nGm,n\np,q\n\n(6)\n\nFrom (6), we see that the approximate gradient of a G function is also a G function, and hence the\noptimization problem in (2) can be solved ef\ufb01ciently via standard gradient descent algorithms.\nThe solution to the metamodel optimization problem in (2) must be con\ufb01ned to a prede\ufb01ned space of\nmathematical expressions G. In particular, we consider the following classes of expressions:\nPolynomial expressions \u2282 Algebraic expressions \u2282 Closed-form expressions \u2282 Analytic expressions,\nwhere the different classes of mathematical expressions correspond to different levels of metamodel\ncomplexity, with polynomial metamodels being the least complex (See Figure 2).\n\n5\n\n\fAlgorithm 1 Symbolic Metamodeling\n\nAlgorithm 1 summarizes all the steps involved\nin solving the metamodel optimization prob-\nlem. The algorithm starts by drawing n fea-\nture points uniformly at random from the fea-\nture space [0, 1]d \u2014 these feature points are\nused to evaluate the predictions of both the\nmodel and the metamodel in order to esti-\nmate the metamodeling loss in (1). Gradient\ndescent is then executed using the gradient\nestimates in (6) until convergence. (Any vari-\nant of gradient descent can be used.) We then\ncheck if every basis function in the resulting\nmetamodel g(x) lies in G. If g(x) /\u2208 G, we\nsearch for an approximate version of the meta-\nmodel \u02dcg(x) \u2248 g(x), such that \u02dcg(x) \u2208 G.\nThe approximate metamodel \u02dcg(x) \u2208 G is obtained by randomly perturbing the optimized parameter\n\u03b8 (within a Euclidean ball of radius \u03b4) and searching for a valid \u02dcg(x) \u2208 G. If no solution is found,\nwe resort to a Chebyshev polynomial approximation of g(x) \u2014 we can also use the Taylor or Pad\u00e9\napproximations \u2014 since polynomials are valid algebraic, closed-form and analytic expressions.\n\n(cid:4) Input: Model f (x), hyperparameters (m, n, p, q, r)\n(cid:4) Output: Metamodel g(x) \u2208 G\n\u2022 Xi \u223c Unif([0, 1]d), i = {1, . . ., n}.\ni (cid:96)(G(Xi; \u03b8), f (Xi))(cid:12)(cid:12)\u03b8=\u03b8k\n(cid:80)\n\u2022 Repeat until convergence:\n......\u03b8k+1 := \u03b8k \u2212 \u03b3 \u2207\u03b8\n\u2022 g(x) \u2190 G(Xi; \u03b8k)\n\u2022 If g(x) /\u2208 G:\n......\u02dcg(x) = G(x; \u00af\u03b8), G(x; \u00af\u03b8) \u2208 G,(cid:107)\u00af\u03b8 \u2212 \u03b8k(cid:107) < \u03b4, or\n......\u02dcg(x) = Chebyshev(g(x))\n\n4 Related Work: Symbolic Metamodels as Gateways to Interpretation\n\nThe strand of literature most relevant to our work is the work on symbolic regression [25\u201327]. This is a\nregression model that searches a space of mathematical expressions using genetic programming. The\nmain difference between this method and ours is that symbolic regression requires prede\ufb01ning the\nfunctional forms to be searched over, hence the number of its parameters increases with the number of\nfunctions that it can \ufb01t. On the contrary, our Meijer G-function parameterization enables recovering\nin\ufb01nitely many functional forms through a \ufb01xed-dimensional parameter space, and allows optimizing\nmetamodels via gradient descent. We compare our method with symbolic regression in Section 5.\nSymbolic metamodeling as a unifying framework for interpretation. We now demonstrate how\nsymbolic metamodeling can serve as a gateway to the different forms of model explanation covrered in\nthe literature. To vivify this view, we go through common types of model explanation, and show that\ngiven a metamodel g(x) we can recover these explanations via analytic manipulation of g(x).\nThe most common form of model explanation involves computing importance scores of each feature\ndimension in x on the prediction of a given instance. Examples for methods that provide this type of\nexplanation include SHAP [1], INVASE [6], DeepLIFT [8], L2X [15], LIME [10, 16], GAM [37],\nand Saliency maps [39]. Each of these methods follows one of two approaches. The \ufb01rst approach,\nadopted by saliency maps, use the gradients of the model output with respect to the input as a measure\nof feature importance. The second approach, followed by LIME, DeepLIFT, GAM and SHAP, uses\nlocal additive approximations to explicitly quantify the additive contribution of each feature.\nSymbolic metamodeling enables a uni\ufb01ed framework for (instancewise) feature importance scoring\nthat encapsulates the two main approaches in the literature. To show how this is possible, consider the\nfollowing Taylor expansion of the metamodel g(x) around a feature point x0:\n\ng(x) = g(x0) + (x \u2212 x0) \u00b7 \u2207x g(x0) + (x \u2212 x0) \u00b7 H(x) \u00b7 (x \u2212 x0) + . . . .,\n\n(7)\nwhere H(x) = [\u22022g/\u2202xi\u2202xj]i,j is the Hessian matrix. Now consider \u2014 for simplicity of exposition\n\u2014 a second-order approximation of (7) with a two-dimensional feature space x = (x1, x2), i.e.,\n2 (x1 \u2212 x0,1)2 gx1x1(x0)\n2 (x2 \u2212 x0,2)2 gx2x2(x0)\n\ng(x) \u2248 g(x0) + (x1 \u2212 x0,1) \u00b7 gx1 (x0) \u2212 x0,2 \u00b7 x1 \u00b7 gx1x2(x0) + 1\n+ (x2 \u2212 x0,2) \u00b7 gx2 (x0) \u2212 x0,1 \u00b7 x2 \u00b7 gx1x2(x0) + 1\n+ x1 \u00b7 x2 \u00b7 gx1x2(x0),\n\n(8)\nwhere gx = \u2207x g and x0 = (x0,1, x0,2). In (8), the term in blue (\ufb01rst line) re\ufb02ects the importance\nof feature x1, the term in red (second line) re\ufb02ects the importance of feature x2, whereas the last\nterm (third line) is the interaction between the two features. The \ufb01rst two terms are what generalized\nadditive models, such as GAM and SHAP, compute. LIME is a special case of (8) that corresponds\n\n6\n\n\fto a \ufb01rst-order Taylor approximation. Similar to saliency methods, the feature contributions in (8)\nare computed using the gradients of the model with respect to the input, but (8) is more general as it\ninvolves higher order gradients to capture the feature contributions more accurately. All the gradients\nin (8) can be computed ef\ufb01ciently since the exact gradient of the G function with respect to its input\ncan be represented analytically in terms of another G function (see Appendix A).\nStatistical interactions between features are another form of model interpretation that has been recently\naddressed in [9, 13]. As we have seen in (8), feature interactions can be analytically derived from a\nsymbolic metamodel. The series in (8) resembles the structure of the pairwise interaction model GA2M\nin [9] and the NIT disentanglement method in [13]. Unlike both methods, a symbolic metamodel\ncan analytically quantify the strength of higher-order (beyond pairwise) interactions with no extra\nalgorithmic complexity. Moreover, unlike the NIT model in [13], which is tailored to neural networks,\na symbolic metamodel can quantify the interactions in any machine learning model (7).\n\nTable 2: Comparison between SM and SR.\n\nf1(x) = e\u22123x\n\nSMp \u2212x3 + 5\n\n2 (x2 \u2212 x) + 1\nR2: 0.995\n\nSMc\n\nSR\n\nx4\u00d710\u22126\n\ne\u22122.99x\n\nR2: 0.999\n\nx2 \u2212 1.9x + 0.9\n\nR2: 0.970\n\n3\n\nx3\n\n(x+1)2\n5 + 2x\n\nf2(x) = x\n3 \u2212 4x2\nR2: 0.985\nx (x + 1)\u22122\nR2: 0.999\n\nf3(x) = sin(x)\n\n\u22121\n4 x2 + x\nR2 : 0.999\n\n1.4 x1.12\nR2 : 0.999\n\n0.7x\n\nx2+0.9x+0.75\n\nR2 0.981\n\n\u22120.17 x2 + x + 0.016\n\nR2 : 0.998\n\n\u221a\n\nf4(x) = J0(10\nx)\n\u22127 (x2 \u2212 x) \u2212 1.4\n\n(cid:16)\n\nR2 : \u22124.75\n\u221a\n\n(cid:17)\n\nI0.0003\n\n10 e\n\nj\u03c0\n2\n\nx\n\nR2 : 0.999\n\n\u2212x (x \u2212 0.773)\n\nR2 : 0.116\n\n5 Experiments and Use Cases\n\nBuilding on the discussions in Section 4, we demonstrate the use cases of symbolic metamodeling\nthrough experiments on synthetic and real data. In all experiments, we used Sympy [19] (a symbolic\ncomputation library in Python) to carry out computations involving Meijer G-functions5.\n\n5.1 Synthetic experiments\n\nCan we learn complex symbolic expressions? We start off with four synthetic experiments with the\naim of evaluating the richness of symbolic expressions discovered by our metamodeling algorithm.\nIn each experiment, we apply Algorithm 1 (Section 3.2) on a ground-truth univariate function f (x)\nto \ufb01t a metamodel g(x) \u2248 f (x), and compare the resulting mathematical expression for g(x) with\nthat obtained by Symbolic regression [25], which we implement using the gplearn library [40].\nIn Table 2, we compare symbolic metamodeling (SM) and symbolic regression (SR) in terms of the\nexpressions they discover and their R2 coef\ufb01cient with respect to the true functions. We consider four\nfunctions: an exponential e\u22123x, a rational x/(x + 12), a sinusoid sin(x) and a Bessel function of the\n\u221a\nx). We consider two versions of SM: SMp for which G = Polynomial expressions,\n\ufb01rst kind J0(10\nand SMc for which G = Closed-form expressions. As we can see, SM is generally more accurate\nand more expressive than SR. For f1(x), f2(x) and f4(x), SM managed to \ufb01gure out the functional\n2 x), where I0(x) is the Bessel function of the second\nforms of the true functions (J0(x) = I0(e\nkind. For f3(x), SMc recovered a parsimonious approximation g3(x) since sin(x) \u2248 x for x \u2208 [0, 1].\nMoreover, SMp managed to retrieve more accurate polynomial expressions than SR.\nInstancewise feature importance. Now we evaluate the ability of symbolic metamodels to explain\npredictions in terms of instancewise feature importance (Section 4). To this end, we replicate the\nexperiments in [15] with the following synthetic data sets: XOR, Nonlinear additive features, and\nFeature switching. (See Section 4.1 in [15] or Appendix B for a detailed description of the data sets.)\nEach data set has a 10-dimensional feature space and 1000 data samples.\nFor each of the three data sets above, we \ufb01t a 2-layer neural network f (x) (with 200 hidden units) to\npredict the labels based on the 10 features, and then \ufb01t a symbolic metamodel g(x) for the trained\n\nj\u03c0\n\n5The code is provided at https://bitbucket.org/mvdschaar/mlforhealthlabpub.\n\n7\n\n\fFigure 4: Box-plots for the median ranks of features by their estimated importance per sample over the 1000\nsamples of each data set. The red line is the median. Lower median ranks are better.\n\nnetwork f (x) using the algorithm in Section 3.2. Instancewise feature importance is derived using\nthe (\ufb01rst-order) Taylor approximation in (8). Since the underlying true features are known for each\nsample, we use the median feature importance ranking of each algorithm as a measure of the accuracy\nof its feature ranks as in [15]. Lower median ranks correspond to more accurate algorithms.\nIn Figure 4, we compare the performance of metamodeling (SM) with DeepLIFT, SHAP, LIME, and\nL2X. We also use the Taylor approximation in (8) to derive feature importance scores from a symbolic\nregression (SR) model as an additional benchmark. For all data sets, SM performs competitively\ncompared to L2X, which is optimized speci\ufb01cally to estimate instancewise feature importance. Unlike\nLIME and SHAP, SM captures the strengths of feature interactions, and consequently it provides\nmore modes of explanation even in the instances where it does not outperform the additive methods\nin terms of feature ranking. Moreover, because SM recovers more accurate symbolic expressions\nthan SR, it provides a more accurate feature ranking as a result.\n\n5.2 Predicting prognosis for breast cancer\n\nWe demonstrate the utility of symbolic metamodeling in a real-world setup for which model inter-\npretability and transparency are of immense importance. In particular, we consider the problem of\npredicting the risk of mortality for breast cancer patients based on clinical features. For this setup, the\nACCJ guidelines require prognostic models to be formulated as transparent equations [21] \u2014 sym-\nbolic metamodeling can enable machine learning models to meet these requirements by converting\nblack-box prognostic models into risk equations that can be written on a piece of paper.\nUsing data for 2,000 breast cancer patients extracted from the UK cancer registry (data description is\nin Appendix B), we \ufb01t an XGBoost model f (x) to predict the patients\u2019 5 year mortality risk based on\n5 features: age, number of nodes, tumor size, tumor grade and Estrogen-receptor (ER) status. Using\n5-fold cross-validation, we compare the area under receiver operating characteristic (AUC-ROC) accu-\nracy of the XGBoost model with that of the PREDICT risk calculator (https://breast.predict.nhs.uk/),\nwhich is the risk equation most commonly used in current practice [41]. The results in Table 3 show\nthat the XGBoost model provides a statistically signi\ufb01cant improvement over the PREDICT score.\nUsing our metamodeling algorithm (with\nG set to be the space of closed-form ex-\npressions), we obtained the symbolic\nmetamodel for both the XGBoost and\nPREDICT models. As we can see in Fig-\nure 5, by inspecting the median instance-\nwise feature ranks, we can see that PRE-\nDICT overestimates the importance of\nsome features and underestimates that of\nothers. This gives us an indication as to\nwhy XGBoost was able to achieve a gain\nin predictive accuracy.\nThrough a symbolic equation, clinicians can transparently use the accurate prognostic model learned\nby XGBoost, without worrying about its original black-box nature. The transparent nature of the\nmetamodel not only ensures its trustworthiness, but also helps us understand the sources of perfor-\nmance gain achieved by the original XGBoost model. Moreover, using the metamodel, we were able\nto draw insights into the impact of the interactions between ER, number of nodes and tumor size on a\npatient\u2019s risk. Such insights would be very hard to distill from the original XGBoost model.\n\nFigure 5: Feature importance for PREDICT and XGBoost.\n\n8\n\nDeepLIFTSHAPLIMEL2XSRSM2468Median rankXORDeepLIFTSHAPLIMEL2XSRSM345678Median rankNonlinear additiveDeepLIFTSHAPLIMEL2XSRSM345678Median rankSwitchAGEScreeningvsClinicalTUMOURSIZEGRADEG1GRADEG2GRADEG3GRADEG4NODESINVOLVEDER_STATUSNHER2_STATUSP05101520Median Feature RankPREDICT: AUC-ROC = 0.762 +/- 0.02XGBoost: AUC-ROC = 0.833 +/- 0.02\fAcknowledgments\n\nThis work was supported by the National Science Foundation (NSF grants 1462245 and 1533983),\nand the US Of\ufb01ce of Naval Research (ONR). The data for our experiments was provided by the UK\nnational cancer registration and analysis service (NCRAS).\n\nReferences\n[1] Scott M Lundberg and Su-In Lee. A uni\ufb01ed approach to interpreting model predictions. In\n\nAdvances in Neural Information Processing Systems (NeurIPS), pages 4765\u20134774, 2017.\n\n[2] Zachary C Lipton. The mythos of model interpretability. arXiv preprint arXiv:1606.03490,\n\n2016.\n\n[3] Ahmed M. Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease\n\nprogression. In Advances in Neural Information Processing Systems (NeurIPS), 2019.\n\n[4] Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data.\n\nScience, 324(5923):81\u201385, 2009.\n\n[5] Yiqun Wang, Nicholas Wagner, and James M Rondinelli. Symbolic regression in materials\n\nscience. arXiv preprint arXiv:1901.04136, 2019.\n\n[6] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable\nselection using neural networks. International Conference on Representation Learning (ICLR),\n2018.\n\n[7] James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Knockoffgan: Generating knockoffs\nInternational Conference on\n\nfor feature selection using generative adversarial networks.\nRepresentation Learning (ICLR), 2018.\n\n[8] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through\npropagating activation differences. In International Conference on Machine Learning (ICML),\npages 3145\u20133153. JMLR. org, 2017.\n\n[9] Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with\npairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on\nKnowledge discovery and data mining, pages 623\u2013631. ACM, 2013.\n\n[10] Ethan Elenberg, Alexandros G Dimakis, Moran Feldman, and Amin Karbasi. Streaming weak\nsubmodularity: Interpreting neural networks on the \ufb02y. In Advances in Neural Information\nProcessing Systems (NeurIPS), pages 4044\u20134054, 2017.\n\n[11] Kristofer Bouchard, Alejandro Bujan, Farbod Roosta-Khorasani, Shashanka Ubaru, Mr Prabhat,\nAntoine Snijders, Jian-Hua Mao, Edward Chang, Michael W Mahoney, and Sharmodeep\nBhattacharya. Union of intersections (uoi) for interpretable data driven discovery and prediction.\nIn Advances in Neural Information Processing Systems (NeurIPS), pages 1078\u20131086, 2017.\n\n[12] David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining\nneural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages\n7775\u20137784, 2018.\n\n[13] Michael Tsang, Hanpeng Liu, Sanjay Purushotham, Pavankumar Murali, and Yan Liu. Neural\ninteraction transparency (nit): Disentangling learned interactions for improved interpretability.\nIn Advances in Neural Information Processing Systems (NeurIPS), pages 5804\u20135813, 2018.\n\n[14] Xin Zhang, Armando Solar-Lezama, and Rishabh Singh. Interpreting neural network judgments\nvia minimal, stable, and symbolic corrections. In Advances in Neural Information Processing\nSystems (NeurIPS), pages 4874\u20134885, 2018.\n\n[15] Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. Learning to explain: An\ninformation-theoretic perspective on model interpretation. International Conference on Machine\nLearning (ICML), 2018.\n\n9\n\n\f[16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining\nthe predictions of any classi\ufb01er. In Proceedings of the 22nd ACM SIGKDD international\nconference on knowledge discovery and data mining, pages 1135\u20131144. ACM, 2016.\n\n[17] Martha LL Abell and James P Braselton. Mathematica by example. Academic Press, 2017.\n\n[18] Stephen Wolfram. Wolfram research. Inc., Mathematica, Version, 8:23, 2013.\n[19] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ond\u02c7rej \u02c7Cert\u00edk, Sergey B. Kirpichev,\nMatthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath-\nnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta,\nShivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, \u0160t\u02c7ep\u00e1n\nRou\u02c7cka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony\nScopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017.\n\n[20] Pang Wei Koh and Percy Liang. Understanding black-box predictions via in\ufb02uence functions.\nIn Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages\n1885\u20131894. JMLR. org, 2017.\n\n[21] Michael W Kattan, Kenneth R Hess, Mahul B Amin, Ying Lu, Karl GM Moons, Jeffrey E\nGershenwald, Phyllis A Gimotty, Justin H Guinney, Susan Halabi, Alexander J Lazar, et al.\nAmerican joint committee on cancer acceptance criteria for inclusion of risk models for individ-\nualized prognosis in the practice of precision medicine. CA: a cancer journal for clinicians,\n66(5):370\u2013374, 2016.\n\n[22] CS Meijer. On the g-function. North-Holland, 1946.\n\n[23] CS Meijer. Uber whittakersche bezw. besselsche funktionen und deren produkte (english\ntranslation: About whittaker and bessel functions and their products). Nieuw Archief voor\nWiskunde, 18(2):10\u201329, 1936.\n\n[24] Richard Beals and Jacek Szmigielski. Meijer g-functions: a gentle introduction. Notices of the\n\nAMS, 60(7):866\u2013872, 2013.\n\n[25] Patryk Orzechowski, William La Cava, and Jason H Moore. Where are we now?: a large\nbenchmark study of recent symbolic regression methods. In Proceedings of the Genetic and\nEvolutionary Computation Conference, pages 1183\u20131190. ACM, 2018.\n\n[26] Telmo Menezes and Camille Roth. Symbolic regression of generative network models. Scienti\ufb01c\n\nreports, 4:6284, 2014.\n\n[27] Ekaterina J Vladislavleva, Guido F Smits, and Dick Den Hertog. Order of nonlinearity as a com-\nplexity measure for models generated by symbolic regression via pareto genetic programming.\nIEEE Transactions on Evolutionary Computation, 13(2):333\u2013349, 2009.\n\n[28] Timothy Y Chow. What is a closed-form number? The American mathematical monthly,\n\n106(5):440\u2013448, 1999.\n\n[29] Jonathan M Borwein, Richard E Crandall, et al. Closed forms: what they are and why we care.\n\nNotices of the AMS, 60(1):50\u201365, 2013.\n\n[30] Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of many\nvariables by superposition of continuous functions of one variable and addition. In Doklady\nAkademii Nauk, volume 114, pages 953\u2013956. Russian Academy of Sciences, 1957.\n\n[31] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are\n\nuniversal approximators. Neural networks, 2(5):359\u2013366, 1989.\n\n[32] Vera Kurkova. Kolmogorov\u2019s theorem and multilayer neural networks. Neural networks,\n\n5(3):501\u2013506, 1992.\n\n[33] Federico Girosi and Tomaso Poggio. Representation properties of networks: Kolmogorov\u2019s\n\ntheorem is irrelevant. Neural Computation, 1(4):465\u2013469, 1989.\n\n10\n\n\f[34] Boris Igelnik and Neel Parikh. Kolmogorov\u2019s spline network. IEEE transactions on neural\n\nnetworks, 14(4):725\u2013733, 2003.\n\n[35] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of\n\ncontrol, signals and systems, 2(4):303\u2013314, 1989.\n\n[36] David A Sprecher. A universal mapping for kolmogorov\u2019s superposition theorem. Neural\n\nNetworks, 6(8):1089\u20131094, 1993.\n\n[37] Trevor J Hastie. Generalized additive models. In Statistical models, pages 249\u2013307. 2017.\n\n[38] I. Gradshteyn and I. Ryzhik. Table of integrals, series, and products. Academic press, 2014.\n\n[39] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks:\nVisualising image classi\ufb01cation models and saliency maps. arXiv preprint arXiv:1312.6034,\n2013.\n\n[40] T Stephens. Gplearn model, genetic programming, 2015.\n\n[41] Francisco J Candido dos Reis, Gordon C Wishart, Ed M Dicks, David Greenberg, Jem Rashbass,\nMarjanka K Schmidt, Alexandra J van den Broek, Ian O Ellis, Andrew Green, Emad Rakha,\net al. An updated predict breast cancer prognostication and treatment bene\ufb01t prediction model\nwith independent validation. Breast Cancer Research, 19(1):58, 2017.\n\n[42] Jianbo Chen, Mitchell Stern, Martin J Wainwright, and Michael I Jordan. Kernel feature selec-\ntion via conditional covariance minimization. In Advances in Neural Information Processing\nSystems (NeurIPS), pages 6946\u20136955, 2017.\n\n11\n\n\f", "award": [], "sourceid": 6036, "authors": [{"given_name": "Ahmed", "family_name": "Alaa", "institution": "UCLA"}, {"given_name": "Mihaela", "family_name": "van der Schaar", "institution": "University of Cambridge, Alan Turing Institute and UCLA"}]}