{"title": "Information Theoretic Properties of Markov Random Fields, and their Algorithmic Applications", "book": "Advances in Neural Information Processing Systems", "page_first": 2463, "page_last": 2472, "abstract": "Markov random fields are a popular model for high-dimensional probability distributions. Over the years, many mathematical, statistical and algorithmic problems on them have been studied. Until recently, the only known algorithms for provably learning them relied on exhaustive search, correlation decay or various incoherence assumptions. Bresler gave an algorithm for learning general Ising models on bounded degree graphs. His approach was based on a structural result about mutual information in Ising models.   Here we take a more conceptual approach to proving lower bounds on the mutual information. Our proof generalizes well beyond Ising models, to arbitrary Markov random fields with higher order interactions. As an application, we obtain algorithms for learning Markov random fields on bounded degree graphs on $n$ nodes with $r$-order interactions in $n^r$ time and $\\log n$ sample complexity. Our algorithms also extend to various partial observation models.", "full_text": "Information Theoretic Properties of Markov Random\n\nFields, and their Algorithmic Applications\n\nLinus Hamilton\u2217\n\nFrederic Koehler \u2020\n\nAnkur Moitra \u2021\n\nAbstract\n\nMarkov random \ufb01elds are a popular model for high-dimensional probability distri-\nbutions. Over the years, many mathematical, statistical and algorithmic problems\non them have been studied. Until recently, the only known algorithms for provably\nlearning them relied on exhaustive search, correlation decay or various incoher-\nence assumptions. Bresler [4] gave an algorithm for learning general Ising models\non bounded degree graphs. His approach was based on a structural result about\nmutual information in Ising models.\nHere we take a more conceptual approach to proving lower bounds on the mutual\ninformation. Our proof generalizes well beyond Ising models, to arbitrary Markov\nrandom \ufb01elds with higher order interactions. As an application, we obtain algo-\nrithms for learning Markov random \ufb01elds on bounded degree graphs on n nodes\nwith r-order interactions in nr time and log n sample complexity. Our algorithms\nalso extend to various partial observation models.\n\n1\n\nIntroduction\n\n1.1 Background\n\nMarkov random \ufb01elds are a popular model for de\ufb01ning high-dimensional distributions by using a\ngraph to encode conditional dependencies among a collection of random variables. More precisely,\nthe distribution is described by an undirected graph G = (V, E) where to each of the n nodes u \u2208 V\nwe associate a random variable Xu which takes on one of ku different states. The crucial property\nis that the conditional distribution of Xu should only depend on the states of u\u2019s neighbors. It turns\nout that as long as every con\ufb01guration has positive probability, the distribution can be written as\n\nPr(a1,\u00b7\u00b7\u00b7 an) = exp\n\n\u03b8i1\u00b7\u00b7\u00b7i(cid:96)(a1,\u00b7\u00b7\u00b7 an) \u2212 C\n\n(1)\n\n(cid:32) r(cid:88)\n\n(cid:88)\n\n(cid:96)=1\n\ni1<i2<\u00b7\u00b7\u00b7<i(cid:96)\n\n(cid:33)\n\nHere \u03b8i1\u00b7\u00b7\u00b7i(cid:96) : [ki1] \u00d7 . . . \u00d7 [ki(cid:96) ] \u2192 R is a function that takes as input the con\ufb01guration of states on\nthe nodes i1, i2,\u00b7\u00b7\u00b7 i(cid:96) and is assumed to be zero on non-cliques. These functions are referred to as\nclique potentials. In the equation above, C is a constant that ensures the distribution is normalized\nand is called the log-partition function. Such distributions are also called Gibbs measures and arise\nfrequently in statistical physics and have numerous applications in computer vision, computational\nbiology, social networks and signal processing. The Ising model corresponds to the special case\n\u2217Massachusetts Institute of Technology. Department of Mathematics. Email: luh@mit.edu. This work\n\u2020Massachusetts Institute of Technology. Department of Mathematics. Email: fkoehler@mit.edu.\n\u2021Massachusetts Institute of Technology. Department of Mathematics and the Computer Science and Arti-\n\ufb01cial Intelligence Lab. Email: moitra@mit.edu. This work was supported in part by NSF CAREER Award\nCCF-1453261, NSF Large CCF-1565235, a David and Lucile Packard Fellowship and an Alfred P. Sloan Fel-\nlowship.\n\nwas supported in part by Hertz Fellowship.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fwhere every node has two possible states and the only non-zero clique potentials correspond to\nsingle nodes or to pairs of nodes.\nOver the years, many sorts of mathematical, statistical and algorithmic problems have been stud-\nied on Markov random \ufb01elds. Such models \ufb01rst arose in the context of statistical physics where\nthey were used to model systems of interacting particles and predict temperatures at which phase\ntransitions occur [6]. A rich body of work in mathematical physics aims to rigorously understand\nsuch phenomena. It is also natural to seek algorithms for sampling from the Gibbs distribution when\ngiven its clique potentials. There is a natural Markov chain to do so, and a number of works have\nidenti\ufb01ed a critical temperature (in our model this is a part of the clique potentials) above which the\nMarkov chain mixes rapidly and below which it mixes slowly [14, 15]. Remarkably in some cases\nthese critical temperatures also demarcate where approximate sampling goes from being easy to\nbeing computationally hard [19, 20]. Finally, various inference problems on Markov random \ufb01elds\nlead to graph partitioning problems such as the metric labelling problem [12].\nIn this paper, we will be primarily concerned with the structure learning problem. Given samples\nfrom a Markov random \ufb01eld, our goal is to learn the underlying graph G with high probability.\nThe problem of structure learning was initiated by Chow and Liu [7] who gave an algorithm for\nlearning Markov random \ufb01elds whose underlying graph is a tree by computing the maximum-weight\nspanning tree where the weight of each edge is equal to the mutual information of the variables at its\nendpoints. The running time and sample complexity are on the order of n2 and log n respectively.\nSince then, a number of works have sought algorithms for more general families of Markov random\n\ufb01elds. There have been generalizations to polytrees [10], hypertrees [21] and tree mixtures [2].\nOthers works construct the neighborhood by exhaustive search [1, 8, 5], impose certain incoherence\nconditions [13, 17, 11] or require that there are no long range correlations (e.g. between nodes at\nlarge distance in the underlying graph) [3, 5].\nIn a breakthrough work, Bresler [4] gave a simple greedy algorithm that provably works for any\nbounded degree Ising model, even if it has long-range correlations. This work used mutual informa-\ntion as its underlying progress measure and for each node it constructed its neighborhood. For a set\nS of nodes, let XS denote the random variable representing their joint state. Then the key fact is the\nfollowing:\nFact 1.1. For any node u, for any S \u2286 V \\ {u} that does not contain all of u\u2019s neighbors, there is a\nnode v (cid:54)= u which has non-negligible conditional mutual information (conditioned on XS) with u.\nThis fact is simultaneously surprising and not surprising. When S contains all the neighbors of u,\nthen Xu has zero conditional mutual information (again conditioned on XS) with any other node\nbecause Xu only depends on XS. Conversely shouldn\u2019t we expect that if S does not contain the en-\ntire neighborhood of u, that there is some neighbor that has nonzero conditional mutual information\nwith u? The dif\ufb01culty is that the in\ufb02uence of a neighbor on u can be cancelled out indirectly by the\nother neighbors of u. The key fact above tells us that it is impossible for the in\ufb02uences to all cancel\nout. But is this fact only true for Ising models or is it an instance of a more general phenomenon\nthat holds over any Markov random \ufb01eld?\n\n1.2 Our Techniques\n\nIn this work, we give a vast generalization of Bresler\u2019s [4] lower bound on the conditional mutual\ninformation. We prove that it holds in general Markov random \ufb01elds with higher order interactions\nprovided that we look at sets of nodes. More precisely we prove, in a Markov random \ufb01eld with\nnon-binary states and order up to r interactions, the following fundamental fact:\nFact 1.2. For any node u, for any S \u2286 V \\ {u} that does not contain all of u\u2019s neighbors, there\nis a set I of at most r \u2212 1 nodes which does not contain u where Xu and XI have non-negligible\nconditional mutual information (conditioned on XS).\n\nOur approach goes through a two-player game that we call the GUESSINGGAME between Alice and\nBob. Alice samples a con\ufb01guration X1, X2, . . . Xn and reveals I and XI for a randomly chosen\nset of u\u2019s neighbors with |I| \u2264 r \u2212 1. Bob\u2019s goal is to guess Xu with non-trivial advantage over\nits marginal distribution. We give an explicit strategy for Bob that achieves positive expected value.\nOur approach is quite general because we base Bob\u2019s guess on the contribution of XI to the overall\nclique potentials that Xu participates in, in a way that the expectation over I yields an unbiased\n\n2\n\n\festimator of the total clique potential. The fact that the strategy has positive expected value is then\nimmediate, and all that remains is to prove a quantitative lower bound on it using the law of total\nvariance. From here, the intuition is that if the mutual information I(Xu; XI ) were zero for all sets\nI then Bob could not have positive expected value in the GUESSINGGAME.\n\n1.3 Our Results\n\nLet \u0393(u) denote the neighbors of u. We require certain conditions (De\ufb01nition 2.3) on the clique\npotentials to hold, which we call \u03b1, \u03b2-non-degeneracy, to ensure that the presence or absence of\neach hyperedge can be information-theoretically determined from few samples (essentially that no\nclique potential is too large and no non-zero clique potential is too small). Under this condition, we\nprove:\nTheorem 1.3. Fix any node u in an \u03b1, \u03b2-non-degenerate Markov random \ufb01eld of bounded degree\nand a subset of the vertices S which does not contain the entire neighborhood of u. Then taking I\nuniformly at random from the subsets of the neighbors of u not contained in S of size s = min(r \u2212\n1,|\u0393(u) \\ S|), we have EI [I(Xu; XI|XS)] \u2265 C.\n\nSee Theorem 4.3 which gives the precise dependence of C on all of the constants, including \u03b1, \u03b2,\nthe maximum degree D, the order of the interactions r and the upper bound K on the number of\nstates of each node. We remark that C is exponentially small in D, r and \u03b2 and there are examples\nwhere this dependence is necessary [18].\nNext we apply our structural result within Bresler\u2019s [4] greedy framework for structure learning to\nobtain our main algorithmic result. We obtain an algorithm for learning Markov random \ufb01elds on\nbounded degree graphs with a logarithmic number of samples, which is information-theoretically\noptimal [18]. More precisely we prove:\nTheorem 1.4. Fix any \u03b1, \u03b2-non-degenerate Markov random \ufb01eld on n nodes with r-order interac-\ntions and bounded degree. There is an algorithm for learning G that succeeds with high probability\ngiven C(cid:48) log n samples and runs in time polynomial in nr.\nRemark 1.5. It is easy to encode an r \u2212 1-sparse parity with noise as a Markov random \ufb01eld with\norder r interactions. This means if we could improve the running time to no(r) this would yield\nthe \ufb01rst no(k) algorithm for learning k-sparse parities with noise, which is a long-standing open\nquestion. The best known algorithm of Valiant [22] runs in time n0.8k.\nSee Theorem 5.1 for a more precise statement. The constant C(cid:48) depends doubly exponentially on\nD. In the special case of Ising models with no external \ufb01eld, Vuffray et al. [23] gave an algorithm\nbased on convex programming that reduces the dependence on D to singly exponential. In greedy\napproaches based on mutual information like the one we consider here, doubly-exponential depen-\ndence on D seems intrinsic. As in Bresler\u2019s [4] work, we construct a superset of the neighborhood\nthat contains roughly 1/C nodes where C comes from Theorem 1.3. Recall that C is exponentially\nsmall in D. Then to accurately estimate conditional mutual information when conditioning on the\nstates of this many nodes, we need doubly exponential in D many samples.\nOur results extend to a model where we are only allowed partial observations. More precisely, for\neach sample we are allowed to specify a set J of size at most C(cid:48)(cid:48) and all we observe is XJ. We\nprove:\nTheorem 1.6. Fix any \u03b1, \u03b2-non-degenerate Markov random \ufb01eld on n nodes with r-order inter-\nactions and bounded degree. There is an algorithm for learning G with C(cid:48)(cid:48)-bounded queries that\nsucceeds with high probability given C(cid:48) log n samples and runs in time polynomial in nr.\n\nSee Theorem 5.3 for a more precise statement. This is a natural scenario that arises when it is\ntoo expensive to obtain a sample where the states of all nodes are known. We also consider a model\nwhere each node\u2019s state is erased (and unobserved) independently with some \ufb01xed probability p. See\nthe supplementary material for a precise statement. The fact that we can straightforwardly obtain\nalgorithms for these alternative settings demonstrates the \ufb02exibility of greedy, information-theoretic\napproaches to learning.\n\n3\n\n\f2 Preliminaries\n\nFor reference, all fundamental parameters of the graphical model (max degree, etc.) are de\ufb01ned in\nthe next two subsections. In terms of these fundamental parameters, we de\ufb01ne additional parameters\n\u03b3 and \u03b4 in (3), C(cid:48)(\u03b3, K, \u03b1) in Theorem 4.3, and \u03c4 in (5) and L in (6).\n\n2.1 Markov Random Fields and the Canonical Form\n\n(cid:96)(a(cid:48)\n\n1, . . . , a(cid:48)\n\n(cid:96)) where the i(cid:48)\n\n1\u00b7\u00b7\u00b7i(cid:48)\n\n1, . . . , a(cid:48)\n\n(cid:96) are the corresponding copies of a1, . . . , a(cid:96).\n\nLet K be an upper bound on the maximum number of states of any node. Recall the joint probability\ndistribution of the model, given in (1). For notational convenience, even when i1, . . . , i(cid:96) are not\nsorted in increasing order, we de\ufb01ne \u03b8i1\u00b7\u00b7\u00b7i(cid:96)(a1, . . . , a(cid:96)) = \u03b8i(cid:48)\n1, . . . , i(cid:48)\nare the sorted version of i1, . . . , i(cid:96) and the a(cid:48)\nThe parameterization in (1) is not unique. It will be helpful to put it in a normal form as below. A\ntensor \ufb01ber is the vector given by \ufb01xing all of the indices of the tensor except for one; this generalizes\nthe notion of row/column in matrices. For example for any 1 \u2264 m \u2264 (cid:96), i1 < . . . < im <\n. . . i(cid:96) and a1, . . . , am\u22121, am+1, . . . a(cid:96) \ufb01xed, the corresponding tensor \ufb01ber is the set of elements\n\u03b8i1\u00b7\u00b7\u00b7i(cid:96)(a1, . . . , am, . . . , a(cid:96)) where am ranges from 1 to kim.\nDe\ufb01nition 2.1. We say that the weights \u03b8 are in canonical form4 if for every tensor \u03b8i1\u00b7\u00b7\u00b7i(cid:96), the sum\nover all of the tensor \ufb01bers of \u03b8i1\u00b7\u00b7\u00b7i(cid:96) is zero.\nMoreover we say that a tensor with the property that the sum over all tensor \ufb01bers is zero is a\ncentered tensor. Hence having a Markov random \ufb01eld in canonical form just means that all of the\ntensors corresponding to its clique potentials are centered. We observe that every Markov random\n\ufb01eld can be put in canonical form:\nClaim 2.2. Every Markov random \ufb01eld can be put in canonical form\n\n(cid:96)\n\n2.2 Non-Degeneracy\nLet H = (V, H) denote a hypergraph obtained from the Markov random \ufb01eld as follows. For every\nnon-zero tensor \u03b8i1\u00b7\u00b7\u00b7i(cid:96) we associate a hyperedge (i1 \u00b7\u00b7\u00b7 i(cid:96)). We say that a hyperedge h is maximal if\nno other hyperedge of strictly larger size contains h. Now G = (V, E) can be obtained by replacing\nevery hyperedge with a clique. Let D be a bound on the maximum degree. Recall that \u0393(u) denotes\nthe neighbors of u. We will require the following conditions in order to ensure that the presence and\nabsence of every maximal hyperedge is information-theoretically determined:\nDe\ufb01nition 2.3. We say that a Markov random \ufb01eld is \u03b1,\u03b2-non-degenerate if\n\n(a) Every edge (i, j) in the graph G is contained in some hyperedge h \u2208 H where the corre-\n\nsponding tensor is non-zero.\n\n(b) Every maximal hyperedge h \u2208 H has at least one entry lower bounded by \u03b1 in absolute\n\nvalue.\n\n(c) Every entry of \u03b8i1i2\u00b7\u00b7\u00b7i(cid:96) is upper bounded by a constant \u03b2 in absolute value.\n\nWe will refer to a hyperedge h with an entry lower bounded by \u03b1 in absolute value as \u03b1-\nnonvanishing.\n\n2.3 Bounds on Conditional Probabilities\n\nFirst we review properties of the conditional probabilities in a Markov random \ufb01eld as well as\nintroduce some convenient notation which we will use later on. Fix a node u and its neighborhood\nU = \u0393(u). Then for any R \u2208 [ku] we have\n\nP (Xu = R|XU ) =\n\n(cid:80)ku\nexp(E X\nu,R)\nB=1 exp(E X\n\nu,B)\n\n(2)\n\n4This is the same as writing the log of the probability mass function according to the Efron-Stein decom-\nposition with respect to the uniform measure on colors; this decomposition is known to be unique. See e.g.\nChapter 8 of [16]\n\n4\n\n\fwhere we de\ufb01ne\n\nr(cid:88)\n\n(cid:88)\n\n(cid:96)=1\n\ni2<\u00b7\u00b7\u00b7<i(cid:96)\n\nE X\nu,R =\n\n\u03b8ui2\u00b7\u00b7\u00b7i(cid:96)(R, Xi2,\u00b7\u00b7\u00b7 , Xi(cid:96))\n\nand i2, . . . , i(cid:96) range over elements of the neighborhood U; when (cid:96) = 1 the inner sum is just \u03b8u(R).\nLet X\u223cu = X[n]\\{u}. To see that the above is true, \ufb01rst condition on X\u223cu, and observe that the\nprobability for a certain Xu is proportional to exp(E X\nu,R), which gives the right hand side of (2).\nThen apply the tower property for conditional probabilities.\nTherefore if we de\ufb01ne (where |T|max denotes the maximum entry of a tensor T )\n\n(cid:18) D\n\nr(cid:88)\n\n(cid:19)\n\n(cid:96) \u2212 1\n\n,\n\n\u03b4 :=\n\nexp(\u22122\u03b3)\n\n1\nK\n\n\u03b3 := sup\n\nu\n\n(cid:96)=1\n\nthen for any R\n\nr(cid:88)\n\n(cid:88)\n\n|\u03b8ui2\u00b7\u00b7\u00b7i(cid:96)|max \u2264 \u03b2\n\ni2<\u00b7\u00b7\u00b7<i(cid:96)\nP (Xu = R|XU ) \u2265 exp(\u2212\u03b3)\n\n(cid:96)=1\n\nK exp(\u03b3)\n\n=\n\n1\nK\n\nexp(\u22122\u03b3) = \u03b4\n\n(3)\n\n(4)\n\nObserve that if we pick any node i and consider the new Markov random \ufb01eld given by conditioning\non a \ufb01xed value of Xi, then the value of \u03b3 for the new Markov random \ufb01eld is non-increasing.\n\n3 The Guessing Game\n\nHere we introduce a game-theoretic framework for understanding mutual information in general\nMarkov random \ufb01elds. The GUESSINGGAME is de\ufb01ned as follows:\n\n1. Alice samples X = (X1, . . . , Xn) and X(cid:48) = (X(cid:48)\n\n1, . . . , X(cid:48)\n\nn) independently from the Markov random\n\n\ufb01eld\n\n2. Alice samples R uniformly at random from [ku]\n3. Alice samples a set I of size s = min(r \u2212 1, du) uniformly at random from the neighbors of u\n4. Alice tells Bob I, XI and R\n\n5. Bob wagers w with |w| \u2264 \u03b3K(cid:0) D\n\nr\u22121\n6. Bob gets \u2206 = w1Xu=R \u2212 w1X(cid:48)\n\n(cid:1)\n\nu=R\n\nBob\u2019s goal is to guess Xu given knowledge of the states of some of u\u2019s neighbors. The Markov\nrandom \ufb01eld (including all of its parameters) are common knowledge. The intuition is that if Bob\ncan obtain a positive expected value, then there must be some set I of neighbors of u which have\nnon-zero mutual information. In this section, will show that there is a simple, explicit strategy for\nBob that yields positive expected value.\n\n3.1 A Good Strategy for Bob\n\nHere we will show an explicit strategy for Bob that has positive expected value. Our analysis will\nrest on the following key lemma:\n\nLemma 3.1. There is a strategy for Bob that wagers at most \u03b3K(cid:0) D\nu,R \u2212 (cid:88)\n\n[w|X\u223cu, R] = E X\n\n(cid:1) in absolute value that satis\ufb01es\n\nr\u22121\nE X\n\nu,B\n\nB(cid:54)=R\n\nProof. First we explicitly de\ufb01ne Bob\u2019s strategy. Let\n\n\u03a6(R, I, XI ) =\n\nCu,(cid:96),s\n\n1{i1\u00b7\u00b7\u00b7i(cid:96)}\u2286I \u03b8ui1\u00b7\u00b7\u00b7i(cid:96) (R, Xi1, . . . , Xi(cid:96))\n\nE\nI,XI\n\ns(cid:88)\n\n(cid:96)=1\n\n(cid:88)\n\ni1<i2<\u00b7\u00b7\u00b7<i(cid:96)\n\n5\n\n\fwhere Cu,(cid:96),s =\n\n(du\ns )\n(du\u2212(cid:96)\ns\u2212(cid:96) )\n\n. Then Bob wagers\n\nw = \u03a6(R, I, XI ) \u2212 (cid:88)\n\nB(cid:54)=R\n\n\u03a6(B, I, XI )\n\nNotice that the strategy only depends on XI because all terms in the summation where {i1 \u00b7\u00b7\u00b7 i(cid:96)}\nare not a subset of I have zero contribution.\nThe intuition behind this strategy is that the weighting term satisifes\n\nCu,(cid:96),s =\n\n1\n\nPr[{i1, . . . i(cid:96)} \u2282 I]\n\nE\nI,XI\n\n[\u03a6(R, I, XI )|X\u223cu, R] =\n\nThus when we take the expectation over I and XI we get\n\nr(cid:88)\n(cid:88)\nu,R \u2212(cid:80)\ni2<\u00b7\u00b7\u00b7<i(cid:96)\nu,B. To complete the proof, notice that Cu,(cid:96),s \u2264\nB(cid:54)=R E X\n\n\u03b8ui2\u00b7\u00b7\u00b7i(cid:96)(R, Xi2,\u00b7\u00b7\u00b7 , Xi(cid:96) ) = E X\n\nu,R\n\n(cid:96)=1\n\n(cid:0) D\n(cid:1) which using the de\ufb01nition of \u03b3 implies that |\u03a6(R, I, XI )| \u2264 \u03b3(cid:0) D\nand hence EI,XI [w|X\u223cu, R] = E X\nr\u22121\nBob wagers at most the desired amount (in absolute value).\nTheorem 3.2. There is a strategy for Bob that wagers at most \u03b3K(cid:0) D\n\nNow we are ready to analyze the strategy:\n\nr\u22121\n\n(cid:1) for any state B, and thus\n(cid:1) in absolute value which\n\nr\u22121\n\nsatis\ufb01es\n\nE[\u2206] \u2265 4\u03b12\u03b4r\u22121\n\nr2re2\u03b3\n\nProof. We will use the strategy from Lemma 3.1. First we \ufb01x X\u223cu, X(cid:48)\n\nE\nI,XI\n\n[\u2206|X\u223cu, X(cid:48)\n\n[w|X\u223cu, R]\nwhich follows because \u2206 = r1Xu=R \u2212 r1X(cid:48)\nsimilarly X(cid:48)\n\n\u223cu, R] = E\nI,XI\n\nu does not depend on X\u223cu . Now using (2) we calculate:\n\n\u223cu and R. Then we have\n\n(cid:16)\n\u223cu, R]\nu=R and because r and Xu do not depend on X(cid:48)\n\nPr[Xu = R|X\u223cu, R] \u2212 Pr[X(cid:48)\n\nu = R|X(cid:48)\n\n(cid:17)\n\n\u223cu and\n\n(cid:80)\nexp(E X\nu,R)\n(cid:16) (cid:88)\nB exp(E X\n\nu,B)\nexp(E X\n\n(cid:80)\n\u2212 exp(E X(cid:48)\nu,R)\nB exp(E X(cid:48)\nu,B)\nu,R + E X(cid:48)\nu,B) \u2212 exp(E X\n\n. Thus putting it all together we have\n\nexp(E X\n\nu,R + E X(cid:48)\n\nu,B) \u2212 exp(E X\n\nu,B + E X(cid:48)\nu,R)\n\n(cid:17)\n\nu,B + E X(cid:48)\nu,R)\n(cid:17)\n\nPr[Xu = R|X\u223cu, R] \u2212 Pr[X(cid:48)\n\nu = R|X(cid:48)\n\n\u223cu, R] =\n\nB(cid:54)=R\n\n=\n\n1\nD\n\n(cid:17)\n(cid:17)(cid:16) (cid:88)\n\nB(cid:54)=R\n\n(cid:16)(cid:80)\n\nwhere D =\n\nB exp(E X\n\nu,B)\n\n[\u2206|X\u223cu, X(cid:48)\n\n\u223cu, R] =\n\n1\nD\n\nE\nI,XI\n\nNow it is easy to see that(cid:88)\n\ndistinct R,G,B\n\nB exp(E X(cid:48)\nu,B)\nE X\n\nu,B\n\n(cid:17)(cid:16)(cid:80)\n(cid:16)E X\nu,R \u2212 (cid:88)\n\uf8eb\uf8ed(cid:88)\n\nB(cid:54)=R\n\nG(cid:54)=R\n\nE X\n\nu,B\n\nexp(E X\n\nu,R + E X(cid:48)\n\nu,G) \u2212 exp(E X\n\nu,G + E X(cid:48)\nu,R)\n\n\uf8f6\uf8f8 = 0\n\nwhich follows because when we interchange R and G the entire term multiplies by a negative one\nand so we can pair up the terms in the summation so that they exactly cancel. Using this identity we\nget\n\n[\u2206|X\u223cu, X(cid:48)\n\n\u223cu] =\n\nE\nI,XI\n\n1\n\nkuD\n\nu,R \u2212 E X\n\nu,B\n\nexp(E X\n\nu,R + E X(cid:48)\n\nu,B) \u2212 exp(E X\n\nu,B + E X(cid:48)\nu,R)\n\n(cid:16)E X\n\n(cid:88)\n\n(cid:88)\n\nB(cid:54)=R\n\nR\n\n(cid:17)\n\n(cid:17)(cid:16)\n\n6\n\n\fwhere we have also used the fact that R is uniform on ku. And \ufb01nally using the fact that X\u223cu and\nX(cid:48)\n\u223cu are identically distributed we can sample Y\u223cu and Z\u223cu and \ufb02ip a coin to decide whether we\nset X\u223cu = Y\u223cu and X(cid:48)\n[\u2206|Y\u223cu, Z\u223cu] =\n\n\u223cu = Z\u223cu or vice-versa. Now we have\n\n(cid:16)E Y\n\nu,B \u2212 eE Y\n\n(cid:88)\n\n(cid:88)\n\nu,B \u2212 E Z\n\nu,R + E Z\n\nu,R \u2212 E Y\n\n(cid:17)(cid:16)\n\nu,B +E Z\n\nu,R+E Z\n\neE Y\n\n(cid:17)\n\nu,B\n\nu,R\n\n1\n\nE\nI,XI\n\n2kuD\n\nB(cid:54)=R\n\nR\n\nWith the appropriate notation it is easy to see that the above sum is strictly positive. Let aR,B =\nE Y\nu,R + E Z\n\nu,B. With this notation:\n\nu,B and bR,B = E Z\n[\u2206|Y\u223cu, Z\u223cu] =\n\nu,R + E Y\n1\n\nE\nI,XI\n\n(cid:88)\n\n(cid:88)\n\n(cid:16)\n\n2Dku\n\nB(cid:54)=R\n\nR\n\n(cid:17)(cid:16)\n\naR,B \u2212 bR,B\n\nexp(aR,B) \u2212 exp(bR,B)\n\n(cid:17)\n\nSince exp(x) is a strictly increasing function it follows that as long as aR,B (cid:54)= bR,B for some term\nin the sum, the sum is positive. In Lemma 3.3 we prove that the expectation over Y and Z of this\nsum is at least 4\u03b12\u03b4r\u22121\n\nr2re2\u03b3 , which completes the proof.\n\nIn the supplementary material we show how to use the law of total variance to give a quantitative\nlower bound on the sum that arose in the proof of Theorem 3.2. More precisely we show:\nLemma 3.3.\n\nu,R\u2212E Y\n\nu,B \u2212E Z\n\nu,R +E Z\n\nu,B\n\nexp(E Y\n\nu,R +E Z\n\nu,B)\u2212 exp(E Y\n\nu,B +E Z\n\nu,R)\n\n(cid:17)(cid:105) \u2265 4\u03b12\u03b4r\u22121\n\nr2re2\u03b3\n\n(cid:104)(cid:88)\n\n(cid:88)\n\n(cid:16)E Y\n\nB(cid:54)=R\n\nR\n\nE\nY,Z\n\n(cid:17)(cid:16)\n\n4\n\nImplications for Mutual Information\n\nIn this section we show that Bob\u2019s strategy implies a lower bound on the mutual information between\nnode u and a subset I of its neighbors of size at most r \u2212 1. We then extend the argument to work\nwith conditional mutual information as well.\n\n4.1 Mutual Information in Markov Random Fields\n\nRecall that the goal of the GUESSINGGAME is for Bob to use information about the states of nodes\nI to guess the state of node u. Intuitively, if XI conveys no information about Xu then it should\ncontradict the fact that Bob has a strategy with positive expected value. We make this precise below.\nOur argument proceeds in two steps. First we upper bound the expected value of any strategy.\nLemma 4.1. For any strategy,\nE[\u2206] \u2264 \u03b3K\n\n(cid:104)| Pr[Xu = R|XI ] \u2212 Pr[Xu = R]|(cid:105)\n\n(cid:18) D\n\n(cid:19)\n\nE\n\nr \u2212 1\n\nI,XI ,R\n\nIntuitively this follows because Bob\u2019s optimal strategy given I, XI and R is to guess\n\nw = sgn(Pr[Xu = R|XI ] \u2212 Pr[Xu = R])\u03b3K\n\nNext we lower bound the mutual information using (essentially) the same quantity. We prove\n\nLemma 4.2. (cid:114) 1\n\nI(Xu; XI ) \u2265 1\n\nK r E\n\nXI ,R\n\n2\n\n(cid:104)| Pr(Xu = R|XI ) \u2212 Pr(Xu = R)|(cid:105)\n\nThese bounds together yield a lower bound on the mutual information. In the supplementary ma-\nterial, we show how to extend the lower bound for mutual information to conditional mutual infor-\nmation. The main idea is to show there is a setting of XS where the hyperedges do not completely\ncancel out in the Markov random \ufb01eld we obtain by conditioning on XS.\nTheorem 4.3. Fix a vertex u such that all of the maximal hyperedges containing u are \u03b1-\nnonvanishing, and a subset of the vertices S which does not contain the entire neighborhood of\n\n7\n\n\fu. Then taking I uniformly at random from the subsets of the neighbors of u not contained in S of\nsize s = min(r \u2212 1,|\u0393(u) \\ S|),\n\n(cid:34)(cid:114) 1\n\n2\n\n(cid:35)\n\nI(Xu; XI|XS)\n\n\u2265 C(cid:48)(\u03b3, K, \u03b1)\n\nE\nI\n\nwhere explicitly\n\n5 Applications\n\nC(cid:48)(\u03b3, K, \u03b1) :=\n\nr2rK r+1(cid:0) D\n\n4\u03b12\u03b4r+d\u22121\nr\u22121\n\n(cid:1)\u03b3e2\u03b3\n\nWe now employ the greedy approach of Bresler [4] which was previously used to learn Ising mod-\nels on bounded degree graphs. Suppose we are given m independent samples from the Markov\n\nrandom \ufb01eld. Let (cid:99)Pr denote the empirical distribution and let (cid:98)E denote the expectation under this\n\ndistribution.\nWe compute empirical estimates for a certain information theoretic quantity \u03bdu,I|S (de\ufb01ned in the\nsupplementary material) as follows\n\n(cid:98)EXS [|(cid:99)Pr(Xu = R, XI = G|XS) \u2212(cid:99)Pr(Xu = R|XS)(cid:99)Pr(XI = G|XS)|]\n\n(cid:98)\u03bdu,I|S := E\n\nR,G\n\nwhere R is a state drawn uniformly at random from [ku], and G is an |I|-tuple of states drawn\nindependently uniformly at random from [ki1 ]\u00d7 [ki2]\u00d7 . . .\u00d7 [ki|I|] where I = (i1, i2, . . . i|I|). Also\nwe de\ufb01ne \u03c4 (which will be used as a thresholding constant) as\n\n\u03c4 := C(cid:48)(\u03b3, k, \u03b1)/2\n\n(5)\n\nand L, which is an upper bound on the size of the superset of a neighborhood of u that the algorithm\nwill construct,\n\nL := (8/\u03c4 2) log K = (32/C(cid:48)(\u03b3, k, \u03b1)2) log K.\n\n(6)\n\nThen the algorithm MRFNBHD at node u is:\n\n1. Fix input vertex u. Set S := \u2205.\n2. While |S| \u2264 L and there exists a set of vertices I \u2282 [n] \\ S of size at most r \u2212 1 such that\n\n(cid:98)\u03bdu,I|S > \u03c4, set S := S \u222a I.\n3. For each i \u2208 S, if(cid:98)\u03bdu,i|S\\i < \u03c4 then remove i from S.\n\n4. Return set S as our estimate of the neighborhood of u.\n\n(cid:16)\n\nm \u2265 60K 2L\n\u03c4 2\u03b42L\n\n(cid:17)\n\nTheorem 5.1. Fix \u03c9 > 0. Suppose we are given m samples from an \u03b1, \u03b2-non-degenerate Markov\nrandom \ufb01eld with r-order interactions where the underlying graph has maximum degree at most D\nand each node takes on at most K states. Suppose that\n\nlog(1/\u03c9) + log(L + r) + (L + r) log(nK) + log 2\n\n.\n\nThen with probability at least 1 \u2212 \u03c9, MRFNBHD when run starting from each node u recovers the\ncorrect neighborhood of u, and thus recovers the underlying graph G. Furthermore, each run of the\nalgorithm takes O(mLnr) time.\n\nIn many situations, it is too expensive to obtain full samples from a Markov random \ufb01eld (e.g. this\ncould involve needing to measure every potential symptom of a patient). Here we consider a model\nwhere we are allowed only partial observations in the form of a C-bounded query:\nDe\ufb01nition 5.2. A C-bounded query to a Markov random \ufb01eld is speci\ufb01ed by a set S with |S| \u2264 C\nand we observe XS\n\n8\n\n\fOur algorithm MRFNBHD can be made to work with C-bounded queries instead of full observations.\nWe prove:\nTheorem 5.3. Fix an \u03b1, \u03b2-non-degenerate Markov random \ufb01eld with r-order interactions where the\nunderlying graph has maximum degree at most D and each node takes on at most K states. The\nbounded queries modi\ufb01cation to the algorithm returns the correct neighborhood of every vertex u\nusing m(cid:48)Lrnr-bounded queries of size at most L + r where\n\n(cid:17)\n\n(cid:16)\n\nm(cid:48) =\n\n60K 2L\n\u03c4 2\u03b42L\n\nlog(Lrnr/\u03c9) + log(L + r) + (L + r) log(nK) + log 2\n\n,\n\nwith probability at least 1 \u2212 \u03c9.\nIn the supplementary material, we extend our results to the setting where we observe partial samples\nwhere the state of each node is revealed independently with probability p, and the choice of which\nnodes to reveal is independent of the sample.\n\nAcknowledgements: We thank Guy Bresler for valuable discussions and feedback.\n\nReferences\n[1] Pieter Abbeel, Daphne Koller, and Andrew Y Ng. Learning factor graphs in polynomial time and sample\n\ncomplexity. Journal of Machine Learning Research, 7(Aug):1743\u20131788, 2006.\n\n[2] Anima Anandkumar, Daniel J Hsu, Furong Huang, and Sham M Kakade. Learning mixtures of tree\n\ngraphical models. In Advances in Neural Information Processing Systems, pages 1052\u20131060, 2012.\n\n[3] Animashree Anandkumar, Vincent YF Tan, Furong Huang, and Alan S Willsky. High-dimensional struc-\nture estimation in ising models: Local separation criterion. The Annals of Statistics, pages 1346\u20131375,\n2012.\n\n[4] Guy Bresler. Ef\ufb01ciently learning ising models on arbitrary graphs. In Proceedings of the Forty-Seventh\n\nAnnual ACM on Symposium on Theory of Computing, pages 771\u2013782. ACM, 2015.\n\n[5] Guy Bresler, Elchanan Mossel, and Allan Sly. Reconstruction of markov random \ufb01elds from samples:\nSome observations and algorithms. In Approximation, Randomization and Combinatorial Optimization.\nAlgorithms and Techniques, pages 343\u2013356. Springer, 2008.\n\n[6] Stephen G Brush. History of the lenz-ising model. Reviews of modern physics, 39(4):883, 1967.\n\n[7] C Chow and Cong Liu. Approximating discrete probability distributions with dependence trees. IEEE\n\ntransactions on Information Theory, 14(3):462\u2013467, 1968.\n\n[8] Imre Csisz\u00b4ar and Zsolt Talata. Consistent estimation of the basic neighborhood of markov random \ufb01elds.\nIn Information Theory, 2004. ISIT 2004. Proceedings. International Symposium on, page 170. IEEE,\n2004.\n\n[9] Gautam Dasarathy, Aarti Singh, Maria-Florina Balcan, and Jong Hyuk Park. Active learning algorithms\n\nfor graphical model selection. J. Mach. Learn. Res, page 199207, 2016.\n\n[10] Sanjoy Dasgupta. Learning polytrees.\n\nIn Proceedings of the Fifteenth conference on Uncertainty in\n\narti\ufb01cial intelligence, pages 134\u2013141. Morgan Kaufmann Publishers Inc., 1999.\n\n[11] Ali Jalali, Pradeep Ravikumar, Vishvas Vasuki, and Sujay Sanghavi. On learning discrete graphical\nmodels using group-sparse regularization. In Proceedings of the Fourteenth International Conference on\nArti\ufb01cial Intelligence and Statistics, pages 378\u2013387, 2011.\n\n[12] Jon Kleinberg and Eva Tardos. Approximation algorithms for classi\ufb01cation problems with pairwise re-\nlationships: Metric labeling and markov random \ufb01elds. Journal of the ACM (JACM), 49(5):616\u2013639,\n2002.\n\n[13] Su-In Lee, Varun Ganapathi, and Daphne Koller. Ef\ufb01cient structure learning of markov networks using l\n1-regularization. In Proceedings of the 19th International Conference on Neural Information Processing\nSystems, pages 817\u2013824. MIT Press, 2006.\n\n[14] Fabio Martinelli and Enzo Olivieri. Approach to equilibrium of glauber dynamics in the one phase region.\n\nCommunications in Mathematical Physics, 161(3):447\u2013486, 1994.\n\n9\n\n\f[15] Elchanan Mossel, Dror Weitz, and Nicholas Wormald. On the hardness of sampling independent sets\n\nbeyond the tree threshold. Probability Theory and Related Fields, 143(3):401\u2013439, 2009.\n\n[16] Ryan O\u2019Donnell. Analysis of Boolean Functions. Cambridge University Press, New York, NY, USA,\n\n2014.\n\n[17] Pradeep Ravikumar, Martin J Wainwright, John D Lafferty, et al. High-dimensional ising model selection\n\nusing ?1-regularized logistic regression. The Annals of Statistics, 38(3):1287\u20131319, 2010.\n\n[18] Narayana P Santhanam and Martin J Wainwright. Information-theoretic limits of selecting binary graph-\n\nical models in high dimensions. IEEE Transactions on Information Theory, 58(7):4117\u20134134, 2012.\n\n[19] Allan Sly. Computational transition at the uniqueness threshold. In Foundations of Computer Science\n\n(FOCS), 2010 51st Annual IEEE Symposium on, pages 287\u2013296. IEEE, 2010.\n\n[20] Allan Sly and Nike Sun. The computational hardness of counting in two-spin models on d-regular graphs.\nIn Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on, pages 361\u2013369.\nIEEE, 2012.\n\n[21] Nathan Srebro. Maximum likelihood bounded tree-width markov networks. In Proceedings of the Seven-\nteenth conference on Uncertainty in arti\ufb01cial intelligence, pages 504\u2013511. Morgan Kaufmann Publishers\nInc., 2001.\n\n[22] Gregory Valiant. Finding correlations in subquadratic time, with applications to learning parities and\nIn Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on, pages\n\njuntas.\n11\u201320. IEEE, 2012.\n\n[23] Marc Vuffray, Sidhant Misra, Andrey Lokhov, and Michael Chertkov. Interaction screening: Ef\ufb01cient and\nsample-optimal learning of ising models. In Advances in Neural Information Processing Systems, pages\n2595\u20132603, 2016.\n\n10\n\n\f", "award": [], "sourceid": 1443, "authors": [{"given_name": "Linus", "family_name": "Hamilton", "institution": "MIT"}, {"given_name": "Frederic", "family_name": "Koehler", "institution": "MIT"}, {"given_name": "Ankur", "family_name": "Moitra", "institution": null}]}