{"title": "Accelerating Bayesian Structural Inference for Non-Decomposable Gaussian Graphical Models", "book": "Advances in Neural Information Processing Systems", "page_first": 1285, "page_last": 1293, "abstract": "In this paper we make several contributions towards accelerating approximate Bayesian structural inference for non-decomposable GGMs. Our first contribution is to show how to efficiently compute a BIC or Laplace approximation to the marginal likelihood of non-decomposable graphs using convex methods for precision matrix estimation. This optimization technique can be used as a fast scoring function inside standard Stochastic Local Search (SLS) for generating posterior samples. Our second contribution is a novel framework for efficiently generating large sets of high-quality graph topologies without performing local search. This graph proposal method, which we call Neighborhood Fusion\" (NF), samples candidate Markov blankets at each node using sparse regression techniques. Our final contribution is a hybrid method combining the complementary strengths of NF and SLS. Experimental results in structural recovery and prediction tasks demonstrate that NF and hybrid NF/SLS out-perform state-of-the-art local search methods, on both synthetic and real-world datasets, when realistic computational limits are imposed.\"", "full_text": "Accelerating Bayesian Structural Inference for\nNon-Decomposable Gaussian Graphical Models\n\nBaback Moghaddam\nJet Propulsion Laboratory\n\nCalifornia Institute of Technology\n\nbaback@jpl.nasa.gov\n\nMohammad Emtiyaz Khan\n\nDepartment of Computer Science\nUniversity of British Columbia\n\nemtiyaz@cs.ubc.ca\n\nBenjamin M. Marlin\n\nDepartment of Computer Science\nUniversity of British Columbia\n\nbmarlin@cs.ubc.ca\n\nKevin P. Murphy\n\nDepartment of Computer Science\nUniversity of British Columbia\n\nmurphyk@cs.ubc.ca\n\nAbstract\n\nWe make several contributions in accelerating approximate Bayesian structural\ninference for non-decomposable GGMs. Our (cid:2)rst contribution is to show how to\nef(cid:2)ciently compute a BIC or Laplace approximation to the marginal likelihood of\nnon-decomposable graphs using convex methods for precision matrix estimation.\nThis optimization technique can be used as a fast scoring function inside standard\nStochastic Local Search (SLS) for generating posterior samples. Our second con-\ntribution is a novel framework for ef(cid:2)ciently generating large sets of high-quality\ngraph topologies without performing local search. This graph proposal method,\nwhich we call (cid:147)Neighborhood Fusion(cid:148) (NF), samples candidate Markov blankets\nat each node using sparse regression techniques. Our third contribution is a hybrid\nmethod combining the complementary strengths of NF and SLS. Experimental\nresults in structural recovery and prediction tasks demonstrate that NF and hybrid\nNF/SLS out-perform state-of-the-art local search methods, on both synthetic and\nreal-world datasets, when realistic computational limits are imposed.\n\nIntroduction\n\n1\nThere are two main reasons to learn the structure of graphical models: knowledge discovery (to\ninterpret the learned topology) and density estimation (to compute log-likelihoods and make pre-\ndictions). The main dif(cid:2)culty in graphical model structure learning is that the hypothesis space is\nextremely large, containing up to 2d(d(cid:0)1)=2 graphs on d nodes. When the sample size n is small,\nthere can be signi(cid:2)cant uncertainty with respect to the graph structure. It is therefore advantageous\nto adopt a Bayesian approach and maintain an approximate posterior over graphs instead of using a\nsingle (cid:147)best(cid:148) graph, especially since Bayesian model averaging (BMA) can improve predictions.\nThere has been much work on Bayesian inference for directed acyclic graphical model (DAG)\nstructure, mostly based on Markov chain Monte Carlo (MCMC) or stochastic local search (SLS)\n[22, 19, 16, 14]. MCMC and SLS methods for DAGs exploit the important fact that the marginal\nlikelihood of a DAG, or an approximation such as the Bayesian Information Criterion (BIC) score,\ncan be computed very ef(cid:2)ciently under standard assumptions including independent conjugate\npriors, and complete data. An equally important property in the DAG setting is that the score can be\nquickly updated when small local changes are made to the graph. This conveniently allows one to\nmove rapidly through the very large graph space of DAGs.\nHowever, for knowledge discovery, a DAG may be an unsuitable representation for several reasons.\nFirst, it does not allow directed cycles, which may be an unnatural restriction in certain domains.\n\n\fSecond, DAGs can only be identi(cid:2)ed up to Markov equivalence in the general case. In contrast,\nundirected graphs (UGs) avoid these issues and may be a more natural representation for some\nproblems. Also, for UGs there are fast methods available for identifying the local connectivity at\neach node (the node\u2019s Markov blanket). We note that while the UG and DAG representations have\ndifferent properties and enable different inference and structure learning algorithms, the distinction\nbetween UGs and DAGs from a density estimation perspective may be less important [12].\nMost prior work on Bayesian inference for Gaussian Graphical Models (GGMs) has focused on the\nspecial case of decomposable graphs (e.g., [17, 2, 29]). The popularity of decomposable GGMs is\nmostly due to the fact that one can compute the marginal likelihood in closed form using similar\nassumptions to the DAG case. In addition, one can update the marginal likelihood in constant time\nafter single-edge moves in graph space [17]. However, the space of decomposable graphs is much\nsmaller than the space of general undirected graphs. For example, the number of decomposable\ngraphs on d nodes for d = 2; : : : ; 8 is 2, 8, 61, 822, 18154, 617675, 30888596 [1, p.158]. If we\ndivide the number of decomposable graphs by the number of general undirected graphs, we get the\n(cid:147)volume(cid:148) ratios: 1, 1, 0:95, 0:80, 0:55, 0:29, 0:12. This means that decomposability signi(cid:2)cantly\nlimits the subclass of UGs available for modeling purposes, even for small d. Several authors\nhave studied Bayesian inference for GGM structure in the general case using approximations to the\nmarginal likelihood based on Monte Carlo methods (e.g., [8, 31, 20, 3]). However, these methods\ncannot scale to large graphs because of the high computational cost of Monte Carlo approximation.\nIn this paper, we propose several techniques to help accelerate approximate Bayesian structural\ninference for non-decomposable GGMs. In Section 2, we show how to ef(cid:2)ciently compute BIC\nand Laplace approximations to the marginal likelihood p(DjG) by using recent convex optimization\nmethods for estimating the precision matrix of a GGM. In Section 3, we present a novel framework\nfor generating large sets of high-quality graphs which we call (cid:147)Neighborhood Fusion(cid:148) (NF). This\nframework is quite general in scope and can use any Markov blanket (cid:2)nding method to devise a\nset of probability distributions (proposal densities) over the local topology at each node. It then\nspeci(cid:2)es rules for (cid:147)fusing(cid:148) these local densities (via sampling) into an approximate posterior over\nwhole graphs p(GjD). In Section 4, we combine the complementary strengths of NF and existing\nSLS methods to obtain even higher quality posterior distributions in certain cases. In Section 5,\nwe present an empirical evaluation of both knowledge discovery and predictive performance of our\nmethods. For knowledge discovery, we measure structural recovery in terms of accuracy of (cid:2)nding\ntrue edges in synthetic GGMs (with known structure). For predictive performance, we evaluate\ntest set log-likelihood as well as missing-data imputation on real data (with unknown structure).\nWe show that the proposed NF and hybrid NF/SLS methods for general graphs outperform current\napproaches to GGM learning for both decomposable and general (non-decomposable) graphs.\nThroughout this paper we will view the marginal likelihood p(DjG) as the key to structural inference\nand as being equivalent to the graph posterior p(GjD) by adopting a (cid:3)at structural prior p(G) w.l.o.g.\n\n2 Marginal Likelihood for General Graphs\nIn this section we will review the G-Wishart distribution and discuss approximations to the marginal\nlikelihood of a non-decomposable GGM under the G-Wishart prior. Unlike the decomposable case,\nhere the marginal likelihood can not be found in closed form. Our main contribution is the insight\nthat recently proposed convex optimization methods for precision matrix estimation can be used\nto ef(cid:2)ciently (cid:2)nd the mode of a G-Wishart distribution, which in turn allows for more ef(cid:2)cient\ncomputation of BIC and Laplace modal approximations to the marginal likelihood.\nWe begin with some notation. We de(cid:2)ne n to be the number of data cases and d to be the number of\ndata dimensions. We denote the ith data case by xi and a complete data set D with the n (cid:2) d matrix\nX, with the corresponding scatter matrix S = X T X (we assume centered data). We use G to denote\nan undirected graph, or more precisely its adjacency matrix. Graph edges are denoted by unordered\npairs (i; j) and the edge (i; j) is in the graph G if Gij = 1. The space of all positive de(cid:2)nite matrices\nhaving the same zero-pattern as G is denoted by S ++\nG . The covariance matrix is denoted by (cid:6) and\nits inverse or the precision matrix by (cid:10) = (cid:6)(cid:0)1. We also de(cid:2)ne hA; Bi = Trace(AB).\nThe Gaussian likelihood p(Dj(cid:10)) is expressed in terms of the data scatter matrix S in Equation 1.\nWe denote the prior distribution over precision matrices given a graph G by p((cid:10)jG). The standard\n\n\fmeasure of model quality in the Bayesian model selection setting is the marginal likelihood p(DjG)\nwhich is obtained by integrating p(Dj(cid:10))p((cid:10)jG) over the space S ++\n\nG as shown in Equation 2.\n\nn\n\nN (xij 0; (cid:10)(cid:0)1) / j(cid:10)jn=2 exp((cid:0)\n\np(Dj(cid:10)) =\n\nYi=1\np(DjG) = ZS++\n\nG\n\np(Dj(cid:10)) p((cid:10)jG) d(cid:10)\n\n1\n2\n\nh(cid:10); Si)\n\n(1)\n\n(2)\n\nThe G-Wishart density in Equation 3 is the Diaconis-Ylvisaker conjugate form [10] for the GGM\nlikelihood as shown in [27]. The indicator function I[(cid:10) 2 S ++\nG ] in Equation 3 restricts the density\u2019s\nsupport to S ++\nG . The G-Wishart generalizes the hyper inverse Wishart (HIW) distribution to general\nnon-decomposable graphs. The G-Wishart normalization constant Z is shown in Equation 4.\n\nW ((cid:10)jG; (cid:14)0; S0) =\n\nI[(cid:10) 2 S ++\nG ]\nZ(G; (cid:14)0; S0)\n\nj(cid:10)j((cid:14)0(cid:0)2)=2 exp((cid:0)\n\n1\n2\n\nh(cid:10); S0i)\n\nZ(G; (cid:14)0; S0) = ZS++\np(DjG) = ZS++\n\nG\n\nG\n\nj(cid:10)j((cid:14)0(cid:0)2)=2 exp((cid:0)\n\n1\n2\n\nh(cid:10); S0i) d(cid:10)\n\np(Dj(cid:10)) W ((cid:10)jG; (cid:14)0; S0) d(cid:10) /\n\nZ(G; (cid:14)n; Sn)\nZ(G; (cid:14)0; S0)\n\n(3)\n\n(4)\n\n(5)\n\nBecause of the conjugate prior in Equation 3, the (cid:10) posterior has a similar form W ((cid:10)jG; (cid:14)n; Sn)\nwhere (cid:14)n = (cid:14)0 + n is the posterior degrees of freedom and the posterior scatter matrix Sn = S + S0.\nThe resulting marginal likelihood is then the ratio of the two normalizing terms shown in Equation 5\n(which we refer to as Zn and Z0 for short).\nThe main drawback of the G-Wishart for general graphs, compared to the HIW for decomposable\ngraphs, is that one cannot compute the normalization terms Zn and Z0 in closed form. As a\nresult, Bayesian model selection for non-decomposable GGMs relies on approximating the marginal\nlikelihood p(DjG). The existing literature focuses on Monte Carlo and Laplace approximations.\nOne strategy that makes use of Monte Carlo estimates of both Zn and Z0 is given by [3]. However,\nthe computation time required to (cid:2)nd accurate estimates can be extremely high [20] (see Section 6).\nAn effective approximation strategy based on using a Laplace approximation to Zn and a Monte\nCarlo approximation to Z0 is given in [21]. This requires (cid:2)nding the mode of the G-Wishart, with\nwhich a closed-form expression for the Hessian is derived [21]. We consider a simpler method which\napplies the Laplace approximation to both Zn and Z0 for greater speed, which we call full-Laplace.\nNevertheless, computing the Hessian determinant has a computational complexity of O(E 3), where\nE is the number of edges in G. Since E = O(d2) in the worst-case scenario, computing a full\nHessian determinant becomes infeasible for large d in all but the sparsest of graphs.\nDue to the high computational cost of Monte Carlo and Laplace approximation in high dimensions,\nwe consider two alternative marginal likelihood approximations that are signi(cid:2)cantly more ef(cid:2)cient.\nThe (cid:2)rst alternative is to approximate Zn and Z0 by Laplace computations in which the Hessian\nmatrix is replaced by its diagonal (by setting off-diagonal elements to zero). We refer to this method\nas the diagonal-Laplace score. The other alternative is the Bayesian Information Criterion (BIC)\nscore shown in Equation 6, which is another large-sample Laplace approximation\n\nBIC(G) = log p(Dj ^(cid:10)G) (cid:0)\n\ndof(G) log n ;\n\n1\n2\n\ndof(G) = d +Xi<j\n\nGij\n\n(6)\n\nwhere, by analogy to [34], we de(cid:2)ne the GGM\u2019s degrees-of-freedom (dof) to be the number of free\nparameters in the precision matrix. For BIC we use the G-Wishart posterior mode ^(cid:10)G as the plug-in\nestimate, since the MLE is unde(cid:2)ned for n < d. But we use a vague and proper prior ((cid:14)0 = 3).\nTherefore, all three approximations will require (cid:2)nding the mode of a G-Wishart (for the posterior\nand/or the prior). In [21] an Iterative Proportional Scaling (IPS) algorithm [30] is proposed to (cid:2)nd\nthe G-Wishart mode. However, IPS requires (cid:2)nding the maximal cliques of the graph, which is an\nNP-hard problem. We will now derive a much more ef(cid:2)cient G-Wishart mode-(cid:2)nder using convex\noptimization techniques. We apply this method to (cid:2)nd ^(cid:10)G when computing BIC scores, as well as\nthe prior and posterior G-Wishart modes when computing Laplace approximations to Z0 and Zn.\n\n\fObserve that we can express the mode of any G-Wishart distribution with the optimization problem\nin Equation 7, where the density is parameterized by graph G, degree (cid:14) and the scatter matrix S.\n\n^(cid:10)G = arg max\n(cid:10)2S++\n\nG\n\nlog W ((cid:10)jG; (cid:14); S) = arg min\n(cid:10)2S++\n\nG\n\n(cid:0) log j(cid:10)j + (cid:28)(cid:10);\n\nS\n\n(cid:14) (cid:0) 2(cid:29)\n\n(7)\n\ni (Pj6=i Gij)3) complexity per iteration [18, p.634].\n\nThis (cid:147)COVSEL(cid:148) type problem [9] is equivalent to (cid:2)nding the maximum likelihood precision matrix\nof a GGM with known structure G, and is a convex optimization problem. Several new methods for\nsolving this precision estimation problem have been recently proposed, and unlike IPS they do not\nrequire computing the clique structure of the underlying graph. Hastie et al. [18] present one such\nmethod which consists of iteratively solving a series of least square problems on the free elements\nof the precision matrix, which has O(Pd\nThe G-Wishart mode in Equation 7 can also be found more directly with a gradient-based optimizer\nsuch as L-BFGS [6], by using the implementation convention that the objective function is 1 for a\nnon-positive de(cid:2)nite matrix. This technique has been used previously by Duchi et al. for the more\ndif(cid:2)cult problem of \u20181 penalized precision matrix estimation [13]. The gradient of the objective\nfunction is simply set to ((cid:0)(cid:10)(cid:0)1 + S) (cid:12) G, where (cid:12) indicates element-wise multiplication. The\nelements of the precision matrix corresponding to absent edges in G are (cid:2)xed to zero, and we\noptimize over the remaining elements. The complexity per iteration is O(d3). In practice, initializing\nthe above optimization with the output of few iterations of the block coordinate descent method\nof [18] (Glasso with known G) is quite effective, as it requires fewer subsequent L-BFGS steps.\nIn Section 5 we explore the speed vs. accuracy trade-off of the various marginal likelihood approx-\nimation schemes discussed above; comparing full-Laplace, diagonal-Laplace and the BIC score\nfunctions to the marginal likelihood values obtained with the Monte Carlo method of [3].\n\n3 Neighborhood Fusion\nIn this section we describe a novel framework we call (cid:147)Neighborhood Fusion(cid:148) (NF) for generating an\napproximate posterior distribution p(GjD) over general graphs. An important advantage of working\nwith general graphs, instead of decomposable graphs, is that we can leverage simple and stable\nmethods for quickly exploring Markov blankets. One popular method for structural recovery is\nGlasso which imposes an l1 penalty on (cid:10) [4, 15, 32]. Finding the corresponding graph takes O(d3)\ntime per iteration for each setting of the regularization parameter (cid:21). However, the choice of the (cid:21)\nparameter is critical, and in practice we often (cid:2)nd that no setting of this parameter leads to good\nrecovery. A related approach, proposed in [23], uses l1-regularized linear regression or Lasso to\nidentify the Markov blanket (MB) of each node. These Markov blankets are then combined using\nintersection or union (AND/OR) to give the global graph G.\nThese methods essentially produce a single (cid:147)best(cid:148) graph, but our main interest is in approximating\nthe full posterior p(GjD). Our NF framework uses a Markov blanket (cid:2)nding method to derive a set\nof probability distributions over the local topology at each node, and speci(cid:2)es a rule for combining\nthese into an approximate posterior over graphs. The detailed steps of the generic NF algorithm are:\n\n1. Regress each node i on all others to (cid:2)nd neighborhoods of all cardinalities k = 0 : d (cid:0) 1\n\nusing a sparse regression method. Denote the set of Markov blankets for node i by Ni\n\n2. Compute the linear regression scores s(b) for each Markov blanket b in Ni, and de(cid:2)ne\n\n3. Independently sample a Markov blanket for each node i from its proposal density pi(b),\n\npi(b) = exp(s(b))=(Pb02Ni\nand then combine all d sampled Markov blankets to assemble a single graph G\n\nexp(s(b0))) as the node\u2019s Markov blanket proposal density\n\n4. Find G\u2019s precision matrix using Equation 7 and compute the graph score as in Section 2\n5. Repeat sampling step 3 and 4 to produce a large ensemble of posterior-weighted graphs\n\nThe design choices in the NF framework are the choice of a sparse linear regression method (and its\nscore function), the choice of a method for combining Markov blankets, and the choice of a graph\nscore function (for marginal likelihood). In all the results that follow we use the linear regression\nBIC score induced by regressing node i on Ni, and generate whole graphs by intersecting the\n\n\fMarkov blankets using the AND operator. This essentially constitutes sampling from the (cid:147)AND-\ncensored(cid:148) pseudo marginal likelihood and is therefore likely to produce good candidate MBs that\ncan be fused into high-quality graphs. Note that the uncertainty modeled by the MB proposal density\nis critical, as it promotes ef(cid:2)cient exploration of model space to generate a large variety of high-\nscoring models. Indeed, the best NF-sampled graphs typically have higher scores than the pseudo\n(cid:147)MAP(cid:148) graph obtained by simply intersecting the best MBs [23], due to the inherent noise in the\nlinear regression BIC scores and the possibility of over-(cid:2)tting. Moreover, our MB proposals can be\n(cid:147)(cid:3)attened(cid:148) with a temperature parameter to trade-off exploration vs. (cid:2)delity of the sampled graphs,\nthough we generally (cid:2)nd it unnecessary to go to such extremes and use a default temperature of one.\nWe next consider two further specialized instances of the NF framework using different sparse linear\nregression methods. The (cid:2)rst method uses the full Lasso/LARS regularization path and is called\nL1MB ((cid:147)L1 Markov Blanket(cid:148)) which we adapted from the DAG-learning method of [28]. NF based\non these l1-derived MBs we call NF-L1MB (or NF-L1 for short). In light of recent theoretical results\non the superiority of greedy forward/backward search over Lasso [33] we also use the l0-based\nmethod of [24] which we call L0MB ((cid:147)L0 Markov blanket(cid:148)). And NF based on L0MB we will call\nNF-L0MB (or NF-L0 for short). Our experimental results show that the improvement of the l0-based\ngreedy search of [24] over Lasso/LARS translates directly to obtaining improved MB proposals with\nNF-L0MB compared to NF-L1MB. Similar forward/backward greedy variable selection techniques\nwere put to good use in the (cid:147)compositional network(cid:148) DAG-to-UG method of [11], however not for\nderiving proposal distributions for parents/MBs as we do here for NF.\nOur overall computational scheme is quite fast by design: (cid:2)nding MB proposals is at most O(d4)\nwith L1MB/L0MB (although L0MB has a smaller constant for both the forward and backward\npasses). Thereafter, we sample full graphs in O(d2) time (since we are sampling a discrete p.m.f.\nfor d MB candidates at each node) and computing a G-Wishart mode ^(cid:10)G is just O(d3) per iteration.\n\n4 Stochastic Local Search\nStochastic Local Search (SLS) can also be viewed as a mechanism for generating an approximate\nposterior distribution over graphs. Like MCMC methods, SLS explores high probability regions of\ngraph space, but unlike MCMC it computes approximate model probabilities directly for each graph\nit visits. This is sensible for large discrete hypothesis spaces like the space of UGs since the chance of\nvisiting the same graph multiple times is extremely small. We note that SLS represents an orthogonal\nand complementary approach to structural inference relative to the NF framework presented in\nSection 3. In this section we discuss SLS for both decomposable and general (non-decomposable)\nGGMs. Speci(cid:2)cally, we describe new initialization and edge-marginal updating methods for non-\ndecomposable GGMs, and also introduce a highly effective hybrid NF/SLS method.\nSLS with decomposable graphs has the advantage that its natural scoring function, the marginal\nlikelihood, can be computed exactly under the conjugate Hyper Inverse Wishart prior. The marginal\nlikelihood can also be updated ef(cid:2)ciently when local changes are made to the underlying graph. A\nstate-of-the-art SLS method for decomposable GGMs is given in [29], which can be used with an\narbitrary score function over the space of general graphs. Here we consider SLS for general graphs\nusing the Laplace score described in Section 2. In the SLS in [29], at iteration t, an edge (i; j) from\nGt is chosen at random and (cid:3)ipped with probability qij. If the resulting graph is admissible and has\nnot been visited before, this graph becomes Gt+1, and we evaluate its score. In the general case,\nevery new graph generated is admissible. In the decomposable case, only decomposable graphs are\nadmissible. We should note that unlike exhaustive search methods, this method avoids evaluating\nthe score of all O(d2) neighboring graphs at each iteration, and instead picks one at random.\nThere are two key modi(cid:2)cations used in [29] which help this method work well in practice. First,\nthe marginal edge probabilities qij are updated online, so edges that have proved useful in the past\nare more likely to be proposed in the future. Second, on each iteration the algorithm chooses to\nperform a resampling step with probability pr or a global move with probability pg. In a resampling\nstep we set Gt+1 to Gv, where v (cid:20) t, with probability proportional to the score (or exponentiated\nscore) of Gv. In a global move we sample a completely new graph (based on the edge marginals qij)\nfor Gt+1. We note that a similar idea of using edge-marginals to propose moves in DAG space was\nsuggested in [14]. In this paper, we set pr = 0:02 and pg = 0 (i.e., we do not use global moves).\n\n\fBIC\nLaplace\nDiag Laplace\n\n \n\n \n\nBIC\nLaplace\nDiag Laplace\nMC\n\n1  \n\n10\u22121 \n\n)\nc\ne\ns\n(\n \n\ne\nm\nT\n\ni\n\n10\u22122 \n\n0.6\n\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\n)\ns\nt\ni\n\nb\n(\n \n\nL\nK\n\ne\nr\no\nc\nS\n\n9550\n\n9500\n\n9450\n\n9400\n\n9350\n\n9300\n\n9250\n\n9200\n\n9150\n\nGLS\u2212NF\n\nNF\u2212L0 (5k)\nGLS\u2212T\nNF\u2212L0 (100)\n\nDLS\u2212T\n\nC\u2212L Tree\n\nNF\u2212L1 (5k)\n\n0\n\n \n\n6\n\n8\n\n10\n\n12\n\nDimension (d)\n\n14\n\n16\n\n10\u22123 \n \n\n6\n\n8\n\n10\n\n12\n\nDimension (d)\n\n14\n\n16\n\n1000 2000 3000 4000 5000\n\nIteration\n\n(a) Posterior Error\n\n(b) Score CPU Time\n\n(c) MF Score Trace\n\nij =\n\nPt\n\nl=1 Gl\nPt\n\nl=1 p(DjGl)\n\n0; :::; wk\n\n0; :::; Gk\n\n0; :::; Gk\n\nl=1 wl\n\n0Gl\n\nl=1 wl\nij ; (cid:12)ij +t(1(cid:0)f t\n\n0(1 (cid:0) Gl\nij)), where f t\nij)=((cid:11)ij + (cid:12)ij + t).\n\nFigure 1: Score trade-offs: (a) average KL error of posterior approximations and (b) the average time to score\na single graph as a function of data dimensionality. (c) Results on the MF dataset: scores for various methods.\nWe now propose a new initialization and updating scheme for non-decomposable SLS based on\na set of k initial graphs G1\n0 de(cid:2)ned by normalized scores)\n0 (with positive weights w1\nobtained from our NF graph-sampling framework. Our approach views qij as a Beta random variable\n0;ij). We update this distri-\n0;ij and (cid:12)ij = Pk\nwith prior parameters (cid:11)ij = Pk\nij p(DjGl)\nbution online using p(qijjG1:t) = Beta((cid:11)ij +tf t\n.\nWe then (cid:3)ip an edge with probability E[qij] = ((cid:11)ij + tf t\nSLS\u2019s main drawback is that, if started from the empty graph as in [29], it will necessarily take at\nleast E steps to (cid:2)nd the highest scoring graph, where E is the number of true edges. This means that\nit will likely require a very large number of iterations even in moderately large and dense graphs.\nAn improved initialization strategy is to start the search from the optimal tree, which can be found\nin O(d2) time using the Chow-Liu algorithm [7]. An even better initialization strategy, for non-\ndecomposable graphs, is to (cid:147)seed(cid:148) SLS with a batch of NF-sampled graphs for G1\n0 and then\nstart the search by executing a resampling step. In this way, a limited number of SLS steps can\neffectively explore the space around these initial high-quality graphs. We refer to this new method,\nwhere NF is used to both initialize the edge-marginals and seed the graph history, as hybrid NF/SLS.\n5 Experiments\nWe begin our experimental analysis by (cid:2)rst assessing the speed vs. accuracy trade-off of the different\nmarginal likelihood approximations in Section 2. For this evaluation we use the Monte Carlo method\nof [3] as a proxy for the ground truth marginal likelihood. For data dimensions d = 6; :::; 16,\nwe sample 100 random, sparse precision matrices with an average edge density of 0:5. For each\nsampled precision matrix (cid:10) we generate 10d observations from the corresponding GGM. Using\neach approximation method, we score all d(d (cid:0) 1)=2 neighbors of G obtained from G by single edge\n(cid:3)ips. We then compute a posterior distribution over this set of graphs by normalizing the scores (or\nexponentiated scores as appropriate). We then compute the Kullback-Leibler (KL) divergence from\nthe Monte Carlo based posterior to each approximate posterior. We also record the time required to\nscore each graph. The scoring methods we use are BIC, full-Laplace and diagonal-Laplace for Zn\nand Z0. We use a G-Wishart prior with parameters (cid:14)0 = 3 and S0 = I. In Figure 1(a) we show the\naverage error of these posterior approximations as a function of data dimensionality d, as measured\nby KL divergence. In Figure 1(b) we show the average time required to score a single graph as\na function of graph size d. As expected, full-Laplace is the most accurate and most costly of the\napproximations next to Monte Carlo.\nInterestingly, diagonal-Laplace appears to be signi(cid:2)cantly\nmore accurate than BIC (for this test) and is in fact only twice as costly. Moreover, diagonal-\nLaplace is already more than 20 times faster than Monte Carlo and full-Laplace at d = 16. On the\nbasis of the speed vs. accuracy trade-off seen in Figure 1(a) and Figure 1(b), we will report only the\ndiagonal-Laplace score in the remainder of our experiments.\nWe next evaluate the NF-L1MB and NF-L0MB methods described in Section 3 (note that we will\nuse the short labels NF-L1 and NF-L0 in the Figures), and SLS for decomposable and general graphs\n\n\f9550\n\n9500\n\n9450\n\n9400\n\n9350\n\n9300\n\n9250\n\n9200\n\n9150\n\n9100\n\n4700\n\n4650\n\n4600\n\n4550\n\n4500\n\n256\n\n254\n\n252\n\n250\n\n248\n\n246\n\n244\n\n242\n\nDLS\u2212T GLS\u2212T GLS\u2212NF NF\u2212L0 NF\u2212L1\n\nDLS\u2212T GLS\u2212T GLS\u2212NF NF\u2212L0 NF\u2212L1\n\nDLS\u2212T GLS\u2212T GLS\u2212NF NF\u2212L0 NF\u2212L1\n\n(a) Diagonal-Laplace Score\n\n(b) Test log-likelihood\n\n(c) Imputed log-likelihood\n\nFigure 2: Mutual Fund results: box plots of the (a) scores, (b) test set log-likelihoods and (c) test set imputation\nlog-likelihoods (averaged over all possible missing 3-nodes). The BMA performance is indicated with a circle.\n\ninitialized from the optimal tree as described in Section 4 (denoted as DLS-T and GLS-T, respec-\ntively), and a L0MB-based hybrid NF/SLS method as described in Section 4 (denoted as GLS-NF).\nWe sample 5000 graphs for each of the NF methods and run each of the SLS methods for 5000 steps,\nalso producing 5000 graphs. The hybrid NF/SLS method is initialized with a sample of 100 NF\ngraphs, and then run for 5000 steps. We compute the score for each set of graphs (diagonal-Laplace\nfor non-decomposable and exact marginal likelihood for decomposables). We extract the 100 best\ngraphs by score, and produce an approximation to p(GjD) by normalizing the exponentiated scores.\nWe report results for individual graphs in the best 100, but our main focus is on performance statistics\nunder Bayesian model averaging (BMA) with approximate scores of each method. In the following\nexperiments we use a G-Wishart prior degree (cid:14)0 = 3 (the smallest integer yielding a proper prior)\nand unless otherwise noted, a default prior scatter matrix of S0 = mean(diag(cov(X))) (cid:1) Id.\nWe examine the two main inferential tasks of prediction and knowledge discovery. We (cid:2)rst measure\nthe predictive ability of each method by computing both test set log-likelihoods and test set impu-\ntation log-likelihoods. For this task we use the (cid:147)Mutual Funds(cid:148) (MF) dataset used by [29] for SLS\nwith decomposable GGMs, with d = 59, which they split into 60 months of training data and 26\nmonths of test data. But due to the resulting critical sampling (n (cid:25) d), here we use a more stable\nS0 = (cid:26) (cid:1) Diag(X T X) with (cid:26) = 0:055 (a Ledoit-Wolf shrinkage). In Figure 1(c) we show a trace plot\nof scores for the SLS methods and best scores for the NF and tree methods. Box plots of diagonal-\nLaplace scores for each method on the MF data are shown in Figure 2(a). The corresponding test\nset log-likelihoods are shown in Figure 2(b). For the imputation experiment, we impute (cid:147)missing(cid:148)\ntriplets of variables given the values of the remaining variables. We compute the log-likelihood of\nthis predictive (imputed) distribution by averaging it over all 59-choose-3 = 32509 possible missing\npatterns and all 26 test cases. The imputation log-likelihoods are shown in Figure 2(c). We can see\nthat NF-L0MB out-performs NF-L1MB on both predictive tasks (full and missing). Interestingly, on\nthis small data set SLS for general graphs (GLS-T) performs rather well. But our hybrid NF-L0MB\n(cid:147)seeding(cid:148) approach for SLS (GLS-NF) has the best overall BMA performance.\nIn the second set of tasks, we evaluate the structural recovery of each method by measuring the\ntrue positive and false positive rates for edge inclusion w.r.t. a ground-truth GGM. The synthetic\ndata sets contain d = 100 nodes, E = 300 edges and n=d ratios of 5:0 (Synth-1) and 0:5 (Synth-2).\nSynth-1 is thus generously oversampled while Synth-2 is undersampled. Both synthetic GGMs were\ngenerated by moralizing a random DAG. Figures 3(a) and 3(b) show plots of TPR vs. FPR for edge\nrecovery. The rates for individual graphs are shown as small grey symbols while the BMA rate is\nshown with a large bold colored symbol. The results show that NF-L0MB and GLS-NF (based on\nseeding GLS with 100 NF-L0MB graphs) are the best methods on both data sets. We also see that\nNF-L0MB dominates NF-L1MB, while the hybrid GLS-NF dominates both GLS-T and DLS-T.\nFor the d = 59 MF dataset in Figure 1(c), NF-sampling 5000 graphs and doing the G-Wishart mode-\n(cid:2)ts and diagonal-Laplace scoring, takes a total of 13 mins, and likewise 30 mins for the synthetic\nd = 100 dataset in Figure 3. Generating and scoring 5000 graphs with non-decomposable SLS takes\n37 mins on the MF dataset and 59 mins on the synthetic one. Decomposable SLS takes 31 mins on\nMF and 43 mins on the synthetic. All times quoted are for Matlab code running on a 3.16 GHz PC.\n\n\f1\n\n0.8\n\n0.6\n\n0.4\n\n0.2\n\nR\nP\nT\n\n \n\n0\n10\u22124\n\n \n\nDLS\u2212T\nGLS\u2212T\nGLS\u2212NF\nNF\u2212L0\nNF\u2212L1\n\n10\u22123\n\n10\u22122\nFPR\n\n10\u22121\n\n1\n\n0.55\n\n0.5\n\n0.45\n\nR\nP\nT\n\n0.4\n\n0.35\n\n0.3\n\n \n\n \n\n1\n\nDLS\u2212T\nGLS\u2212T\nGLS\u2212NF\nNF\u2212L0\nNF\u2212L1\n\n10\u22122\n\n10\u22121\n\nFPR\n\n(a) Synth-1: n=d = 5:0 (d = 100)\n\n(b) Synth-2: n=d = 0:5 (d = 100)\n\nFigure 3: True Positive vs. False Positive rates for (a) Synth-1 and (b) Synth-2 datasets for each recovery\nmethod. The top 100 graphs are shown with a grey symbol and the bold colored symbol is the BMA graph.\n6 Discussion\nWe offer a practical framework for fast inference in non-decomposable GGMs providing reasonable\naccuracy for marginal likelihoods. While Monte Carlo methods are the (cid:147)gold standard(cid:148) (modulo the\nusual convergence issues) they are exorbitantly costly for even moderately large d. For example,\nscoring all the neighbors of a 150-node graph via SLS required over 40 days of computation in [20].\nA similar size task would take less than 40 mins with our diagonal-Laplace approximation method.\nAs pointed out by [21] there may not always be suf(cid:2)cient concentration for a Laplace approximation\nto Z0 to be very accurate, which is why they use MC for this quantity. We chose Laplace for both Zn\nand Z0 solely for speed (to avoid MC altogether) and found good agreement between full-Laplace\nand BIC for much larger graphs than in Figure 1(a). Our Laplace scores also roughly matched the\nMC values for the Fisher Iris data in [3], selecting essentially the same top-ranked 16 graphs (see\nFigure 5 in [3]). Using a diagonal instead of a full Hessian was yet another compromise for speed.\nAn issue that should be explored further is the sensitivity of these approximations to different priors.\nWe experimentally validated NF on nearly 104 synthetic cases ranging in size from d = 10; :::; 500,\nwith various edge densities and n=d ratios, with consistently good results, typi(cid:2)ed by the two test\ncases shown in Figure 3. Note that the sub-par performance of NF-L1 is not a failing of NF but due\nto l1-based MBs, and superiority of l0-based F/B greedy search is not without precedent [25, 24, 33].\nWe note that NF can be partially justi(cid:2)ed as a pseudo marginal likelihood (PML), but whereas most\nauthors rely only on its maximizer [23] we exploit the full (pseudo) density. Without the AND (cid:2)lter,\nNF-drawn MBs are sampled from a set of (cid:147)consistent(cid:148) full-conditionals in the sense of Besag [5],\nand their max-BIC MBs are collectively the PML mode (note that here we mean the node regression\nBIC, not graph BIC). Enforcing AND is a necessary domain truncation for a valid UG which alters\nthe mode. This symmetrized (cid:147)pseudo-MAP(cid:148) G is often an average-scoring one compared to the best\nand worst found by NF, which motivates BMA and justi(cid:2)es NF. We can also view NF as an over-\ndispersed proposal density; its weighted graphs a rough proxy for p(GjD). This approximation may\nbe biased but our results show it is quite useful for prediction and imputation (and seeding SLS with\nhigh-quality graphs). Finally, while use of BIC/Laplace for hypothesis testing is often criticized, it\ncan still be useful for estimation [26], and nowhere in our framework are these scores being used to\nselect a single (cid:147)best(cid:148) model (whether it be a MB or a G) due to our reliance on sampling and BMA.\n\nAcknowledgments\nWe like to thank the reviewers for their helpful and encouraging feedback. BMM was supported\nby the Killam Trusts at UBC and KPM would like to thank NSERC and CIFAR. This work was in\npart carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract\nwith the National Aeronautics and Space Administration.\n\n\fReferences\n[1] H. Armstrong. Bayesian Estimation of Decomposable GGMs. PhD thesis, UNSW, 2005.\n[2] H. Armstrong, C. Carter, K. Wong, and R. Kohn. Bayesian covariance matrix estimation using a mixture\n\nof decomposable graphical models. Statistics and Computing, 2008.\n\n[3] A. Atay-Kayis and H. Massam. A Monte Carlo method for computing the marginal likelihood in non-\n\ndecomposable Gaussian graphical models. Biometrika, 92, 2005.\n\n[4] O. Banerjee, L. El Ghaoui, A. d\u2019Aspremont, and G. Natsoulis. Convex optimization techniques for (cid:2)tting\n\nsparse Gaussian graphical models. In Intl. Conf. on Machine Learning, 2006.\n\n[5] J. Besag. Ef(cid:2)ciency of pseudo-likelihood estimation for simple Gaussian (cid:2)elds. Biometrika, 1977.\n[6] R. Byrd, P. Lu, J. Nocedal, and C. Zhu. A limited memory algorithm for bound constrained optimization.\n\nSIAM J. of Scienti(cid:2)c & Statistical Computing, 16(5), 1995.\n\n[7] C. Chow and C. Liu. Approximating discrete probability distributions with dependence trees. IEEE Trans.\n\non Info. Theory, 14, 1968.\n\n[8] P. Dellaportas, P. Giudici, and G. Roberts. Bayesian inference for nondecomposable graphical Gaussian\n\nmodels. Sankhya, Ser. A, 65, 2003.\n\n[9] A. Dempster. Covariance selection. Biometrics, 28(1), 1972.\n[10] P. Diaconis and D. Ylvisaker. Conjugate priors for exponential families. Annals of statistics, 7(2), 1979.\n[11] D. Dobra, C. Hans, B. Jones, J. Nevins, G. Yao, and M. West. Sparse graphical models for exploring gene\n\nexpression data. J. Multivariate analysis, 90, 2004.\n\n[12] J. Domke, A. Karapurkar, and Y. Aloimonos. Who killed the directed model? In CVPR, 2008.\n[13] J. Duchi, S. Gould, and D. Koller. Projected subgradients for learning sparse Gaussians. In UAI, 2008.\n[14] D. Eaton and K. Murphy. Bayesian structure learning using DP and MCMC. In UAI, 2007.\n[15] J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse covariance estimation in Glasso. Biostats, 2007.\n[16] N. Friedman and D. Koller. Being Bayesian about network structure: A Bayesian approach to structure\n\ndiscovery in Bayesian networks. Machine Learning, 50, 2003.\n\n[17] P. Giudici and P. Green. Decomposable graphical Gaussian model determination. Biometrika, 1999.\n[18] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2009.\n[19] D. Heckerman, D. Geiger, and M. Chickering. Learning Bayesian networks: the combination of knowl-\n\nedge and statistical data. Machine Learning, 20(3), 1995.\n\n[20] B. Jones, C. Carvalho, A. Dobra, C. Hans, C. Carter, and M. West. Experiments in stochastic computation\n\nfor high-dimensional graphical models. Statistical Science, 20, 2005.\n\n[21] A. Lenkoski and A. Dobra. Bayesian structural learning and estimation in Gaussian graphical models.\n\nTechnical Report 545, Department of Statistics, University of Washington, 2008.\n\n[22] D. Madigan and A. Raftery. Model selection and accounting for model uncertainty in graphical models\n\nusing Occam\u2019s window. J. of the Am. Stat. Assoc., 89, 1994.\n\n[23] N. Meinshausen and P. Buhlmann. High dimensional graphs and variable selection with the Lasso. The\n\nAnnals of Statistics, 2006.\n\n[24] B. Moghaddam, A. Gruber, Y. Weiss, and S. Avidan. Sparse regression as a sparse eigenvalue problem.\n\nIn Information Theory & Applications Workshop (ITA\u201908), 2008.\n\n[25] B. Moghaddam, Y. Weiss, and S. Avidan. Spectral bounds for sparse PCA: Exact & greedy algorithms.\n\nIn NIPS, 2006.\n\n[26] A. Raftery. Bayesian model selection in social research. Sociological Methodology, 25, 1995.\n[27] A. Roverato. Hyper inverse Wishart distribution for non-decomposable graphs and its application to\n\nBayesian inference for Gaussian graphical models. Scand. J. Statistics, 29, 2002.\n\n[28] M. Schmidt, A Niculescu-Mizil, and K Murphy. Learning graphical model structure using l1 regulariza-\n\ntion paths. In AAAI, 2007.\n\n[29] J. Scott and C. Carvalho. Feature-inclusion stochastic search for Gaussian graphical models.\n\nComputational and Graphical Statistics, 17(4), 2008.\n\nJ. of\n\n[30] T. Speed and H. Kiiveri. Gaussian Markov distributions over (cid:2)nite graphs. Annals of Statistics, 1986.\n[31] F. Wong, C. Carter, and R. Kohn. Ef(cid:2)cient estimation of covariance selection models. Biometrika, 2003.\n[32] M. Yuan and Yi Lin. Model selection and estimation in the GGM. Biometrika, 94(1), 2007.\n[33] T. Zhang. Adaptive forward-backward greedy algorithm for sparse learning. In NIPS, 2008.\n[34] H. Zou, T. Hastie, and R. Tibshirani. On the (cid:148)degrees of freedom(cid:148) of Lasso. Annals of Statistics, 2007.\n\n\f", "award": [], "sourceid": 1072, "authors": [{"given_name": "Baback", "family_name": "Moghaddam", "institution": null}, {"given_name": "Emtiyaz", "family_name": "Khan", "institution": null}, {"given_name": "Kevin", "family_name": "Murphy", "institution": null}, {"given_name": "Benjamin", "family_name": "Marlin", "institution": null}]}