{"title": "Multi-dynamic Bayesian Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 409, "page_last": 416, "abstract": null, "full_text": "Multi-dynamic Bayesian Networks\n\nKarim Filali and Jeff A. Bilmes Departments of Computer Science & Engineering and Electric al Engineering University of Washington Seattle, WA 98195 {karim@cs,bilmes@ee}.washington.edu\n\nAbstract\nWe present a generalization of dynamic Bayesian networks to concisely describe complex probability distributions such as in problems with multiple interacting variable-length streams of random variables. Our framewor k incorporates recent graphical model constructs to account for existence uncert ainty, value-specific independence, aggregation relationships, and local and global constraints, while still retaining a Bayesian network interpretation and effic ient inference and learning techniques. We introduce one such general technique, which is an extension of Value Elimination, a backtracking search inference algo rithm. Multi-dynamic Bayesian networks are motivated by our work on Statistical Machine Translation (MT). We present results on MT word alignment in support of our claim that MDBNs are a promising framework for the rapid prototyping of new MT systems.\n\n1\n\nINTRODUCTION\n\nThe description of factorization properties of families of probabilities using graphs (i.e., graphical models, or GMs), has proven very useful in modeling a wide variety of statistical and machine learning domains such as expert systems, medical diagnosis, decision making, speech recognition, and natural language processing. There are many different t ypes of graphical model, each with its own properties and benefits, including Bayesian networks, undirected Markov random fields, and factor graphs. Moreover, for different types of scientific m odeling, different types of graphs are more or less appropriate. For example, static Bayesian netw orks are quite useful when the size of set of random variables in the domain does not grow or shrink f or all data instances and queries of interest. Hidden Markov models (HMMs), on the other hand, are such that the number of underlying random variables changes depending on the desired length (which can be a random variable), and HMMs are applicable even without knowing this length as they can be extended indefinitely using online inference. HMMs have been generalized to dynamic Bayesian networks (DBNs) and temporal conditional random fields (CRFs), where an underlying set of variables gets repeated as needed to fill any finite but unbounded length. Probabilistic relational models (PRMs) [5] allow for a more complex template that can be expanded in multiple dimensions simultaneously. An attribute common to all of the above cases is that the specification of rules for expanding any particular instance of a model is finite. In other words, these forms of GM allow the specification of models with an unlimited number of random variables (RVs) using a finite descript ion. This is achieved using parameter tying, so while the number of RVs increases without bound, the number of parameters does not. In this paper, we introduce a new class of model we call multi-dynamic Bayesian networks. MDBNs are motivated by our research into the application of graphi cal models to the domain of statistical machine translation (MT) and they have two key attributes from the graphical modeling perspective. First, an MDBN generalizes a DBN in that there are multiple \"streams\" of variables that can get unrolled, but where each stream may be unrolled by a differin g amount. In the most general case, connecting these different streams together would require the specification of conditional probabil-\n\n\f\nity tables with a varying and potentially unlimited number of parents. To avoid this problem and retain the template's finite description length, we utilize a switching parent functionality (also called value-specific independence). Second, in order to capture t he notion of fertility in MT-systems (defined later in the text), we employ a form of existence uncertainty [7] (that we call switching existence), whereby the existence of a given random variable might dep end on the value of other random variables in the network. Being fully propositional, MDBNs lie between DBNs and PRMs in terms of expressiveness. While PRMs are capable of describing any MDBN, there are, in general, advantages to restricting ourselves to a more specific class of model. For example, in the DBN case, it is possible to provide a bound on inference costs just by looking at attributes of the DBN template only (e.g., the left or right interfaces [12, 2]). Restricting the model can also make it simpler to use in practice. MDBNs are still relatively simple, while at the same time making possible the easy expression of MT systems, and opening doors to novel forms of probabilistic inference as we show be low. In section 2, we introduce MDBNs, and describe their application to machine translation showing how it is possible to represent even complex MT systems. In section 3, we describe MDBN learning and decoding algorithms. In section 4, we present experimental results in the area of statistical machine translation, and future work is discussed in sectio n 5.\n\n2\n\nMDBNs\n\nA standard DBN [4] template consists of a directed acyclic graph G = (V , E ) = (V1 V2 , E1 E2 E2 ) with node set V and edge set E . For t {1, 2}, the sets Vt are the nodes at slice t, Et are the intra-slice edges between nodes in Vt , and Et are the inter-slice edges between nodes in V1 and V2 . To unroll a DBN to length T , the nodes V2 along with the edges adjacent to any node in V2 are cloned T - 1 times (where parameters of cloned variables are constraine d to be the same as the template) and re-connected at the corresponding places. An MDBN with K streams consists of the union of K DBN templates along with a template structure specifying rules to connect the various streams togeth er. An MDBN template is a directed graph k k (k ) G = (V , E ) = ( V (k) , E (k) E ) where (V (k) , E (k) ) is the k th DBN, and the edges E are rules specifying how to connect stream k to the other streams. These rules are general in that they spe cify the set of edges for all values of Tk . There can be arbitrary nesting of the streams such as, for ex ample, it is possible to specify a model that can grow along several dimensions simultaneously. An MDBN also utilizes \"switching existence\", meaning some subset of the variables in V bestow existence onto other variables in the network. We call these variables existence bestowing (or ebnodes). The idea of bestowing existence is well defined over a discrete space, and is not dissimilar to a variable length DBN. For example, we may have a joint distribution over lengths as follows: p(X1 , . . . , XN , N ) = p(X1 , . . . , Xn |N = n)p(N = n) where here N is an eb-node that determines the number of other random vari ables in the DGM. Our notion of eb-nodes allows us to model certain characteri stics found within machine translation systems, such as \"fertility\" [3], where a given English word is cloned a random number of times in the generative process that explains a translation from F rench into English. This random cloning might happen simultaneously at all points along a given MDBN stream. This means that even for a given fixed stream length Ti = ti , each stream could have a randomly varying number of random variables. Our graphical notation for eb-nodes consists of the eb-node as a square box containing variables whose existence is determined by the eb-node. We start by providing a simple example of an expanded MDBN for three well known MT systems, namely the IBM models 1 and 2 [3], and the \"HMM\" model [15].1 We adopt the convention in [3] that our goal is to translate from a string of French words F = f of length M = m into a string of English words E = e of length L = l -- of course these can be any two languages. The basic generative (noisy channel) approach when translating from French to English is to represent the joint\n1\n\n(k )\n\nWe will refer to it as M-HMM to avoid confusion with regular HMMs.\n\n\f\ndistribution P (f , e) = P (f |e)P (e). P (e) is a language model specifying the prior over the word string e. The key goal is to produce a finite-description length repre sentation for P (f |e) where f and e are of arbitrary length. A hidden alia nment string, a, specifies how the English words align to g the French word, leading to P (f |e) = P (f , a|e). Figure 1(a) is a 2-stream MDBN expanded representation of the three models, in this case = 4 and m = 3. As shown, it appears that the fan-in to node fi will be and thus will grow without bound. However, a switching mechanism whereby P (fi |e, ai ) = P (fi |eai ) limits the number of parameters regardless of L. This means that the alignment variable ai indicates the English word eai that should be aligned to French word fi . The variable e0 is a null word that connects to French words not explained by any of e1 , . . . , e . The graph expresses all three models -- the difference is that, in Models 1 and 2, there are no edges between aj and aj +1 . In Model 1, p(aj = ) is uniform on the set {1, . . . , L}; in Model 2, the distribution over aj is a function only of its position j , and on the English and French lengths and m respectively. In the M-HMM model, the ai variables form a first order Markov chain.\n\nl\ne0 e1 e2 e3 e4\nm'\n0\n\n\ne1\n1\n\ne2\n2\n\ne3\n3\n\n01\n 01\n\n02\n\n11\n 11\n\n12\n 12\n\n13\n 13\n\n21\n 21\n\nf1 a1\nm\n\nf2 a2\n\nf3 a3\n\nuv\n\nw\n\n 02\n\ny f1\na1\n\nf2\na2\n\nf3\na3\n\nf4\na4\n\nf5\na5\n\nf6\na6\n\nx\n\nm\n\n(a) Models 1,2 and M-HMM\n\n(b) Expanded M3 graph\n\nFigure 1: Expanded 2-stream MDBN description of IBM Models 1 and 2, and the M-HMM model for MT; and the expanded MDBN description of IBM Model 3 with fertility assignment 0 = 2, 1 = 3, 2 = 1, 3 = 0. From the above, we see that it would be difficult to express this model graphically using a standard DBN since L and M are unequal random variables. Indeed, there are two DBNs in o peration, one consisting of the English string, and the other consisting of the French string and its alignment. Moreover, the fully connected structure of the graph in the fi gure can represent the appropriate family of model, but it also represents models whose paramet er space grows without bound -- the switching function allows the model template to stay finite regardless of L and M . With our MDBN descriptive abilities complete, it is now possible to describe the more complex IBM models 3, and 4[3] (an MDBN for Model3 is depicted in fig. 1(b)). The top most random variable, , is a hidden switching existence variable corresponding to t he length of the English string. The box abutting includes all the nodes whose existence depends on the value of . In the figure, = 3, thus resulting in three English words e1 , e2 , and e3 connected using a second-order Markov chain. To each English word ei corresponds a conditionally dependent fertility eb-node i , which indicates how many times ei is used by words in the French string. Each i in turn controls the existence of a set of variables under it. Given the fertilities (the figure d epicts the case 1 = 3, 2 = 1, 3 = 0), for each word ei , i French word variables are granted existence and are denoted by i1 , i2 , . . . , ii , what is called the tablet [3] of ei . The values taken by the variables need to match the actual observed French sequence f1 , . . . , fm . This is represented as a shared constraint between all the f , , and variables which have incoming edges into the observed varia ble v . v 's conditional probability table is such that it is one only when the associated constraint is satisfied2 . The variable\n2 This type of encoding of constraints corresponds to the standard mecha nism used by Pearl [14]. A naive implementation, however, would enumerate a number of configurations exponential in the number of constrained variables, while typically only a small fraction of the configurations would have positive probability.\n\n\f\ni,k {1, . . . , m} is a switching dependency parent with respect to the constra int variable v and determines which fj participates in an equality constraint with i,k . The bottom variable m is a switching existence node (observed to be 6 in the figure) with corresponding French word sequence and alignment variables. The French sequence participates in the v constraint described above, while the alignment variables aj {1, . . . , }, j 1, . . . , m constrain the fertilities to take their unique allowable values (for t he given alignment). Alignments also restrict the domain of permutation variables, , using the constraint variable x. Finally, the domain size of each aj has to lie in the interval [0, ] and that is enforced by the variable u. The dashed edges connecting the alignment a variables represent an extension to implement an M3/M-HMM hybrid. The null submodel involving the deterministic node m (= i=1 i ) and eb-node 0 accounts for French words that are not explained by any of the English word s e1 , . . . , e . In this submodel, successive permutation variables are ordered and this cons traint is implemented using the observed child w of 0i and 0(i+1) . Model 4 [3] is similar to Model 3 except that the former is based on a more elaborate distortion model that uses relative instead of absolute positions both within and between tablets.\n\n3\n\nInference, Parameter Estimation and MPE\n\nMulti-dynamic Bayesian Networks are amenable to any type of inference that is applicable to regular Bayesian networks as long as switching existence relationships are respected and all the constraints (aggregation for example) are satisfied. Unfortunately DBN inference procedures that take advantage of the repeatable template and can preprocess it offline , are not easy to apply to MDBNs. A case in point is the Junction Tree algorithm [11]. Triangula tion algorithms exist that create an offline triangulated version of the input graph and do not re-triang ulate it for each different instance of the input data [12, 2]. In MDBNs, due to the flexibility to unroll templates in several dimensions and to specify dependencies and constraints spanning the entire unrolled graph, it is not obvious how we can exploit any repetitive patterns in a Junction Tree-styl e offline triangulation of the graph template. In section 4, we discuss sampling inference methods we have used. Here we discuss our extension to a backtracking search algorithm with the same performance g uarantees as the JT algorithm, but with the advantage of easily handling determinism, existence uncertainty, and constraints, both learned and explicitly stated. Value Elimination (VE) ([1]), is a backtracking Bayesian network inference technique that caches factors associated with portions of the search tree and uses them to avoid iterating again over the same subtrees. We follow the notation introduced in [1] and refer the reader to that paper for details about VE inference. We have extended the VE inference approa ch to handle explicitly encoded constraints, existence uncertainty, and to perform approximate local domain pruning (see section 4). We omit these details as well as others in the original paper a nd briefly describe the main data structure required by VE and sketch the algorithm we refer to as FirstPass (fig. 1) since it constitutes the first step of the learning procedure, our main contributi on in this section. A VE factor, F , is such that we can write the following marginal of the joint distribution X P (X = x, Y = y, Z) = F.v al f (Z)\n=x\n\nsuch that (X Y) Z = , F.v al is a constant, and f (Z) a function of Z only. Y is a set of variables previously instantiated in the current branch of search tre e to the value vector y. The pair (Y, y) is referred to as a dependency set (F.Dset). X is referred to as a subsumed set (F.Sset). By caching the tuple (F.Dset, F.S set, F.v al), we avoid recomputing the marginal again whenever (1) F.Dset is active, meaning all nodes stored in F.Dset are assigned their cached values in the current branch of the search tree; and (2) none of the variables in F.S set are assigned yet. FirstPass (alg. 1) visits nodes in the graph in Depth First fashion. In line 7, we get the values of all Newly Single-valued (NSV) CPTs i.e., CPTs that involve the current node, V , and in which all\nWe use a general directed domain pruning constraint. Deterministic relation ships then become a special case of our constraint whereby the domain of the child variable is constrained to a single value with probability one.\n\n\f\nVariable traversal order: A, B, C, and D. Factors are numbered by order of creation. *Fi denotes the activation of factor i.\nF7: Dset={} Sset={A,B,C,D} val=P(E=e)\n\nTau values propagated recursively\nF7.tau = 1.0 = P(Evidence)/F7.val\n\nA\nF5: Dset={A=0} Sset={B,C,D}\n\nF6 0 1\n\nF5.tau = F7.tau * P(A=0)\n\nF6.tau = F7.tau * P(A=1)\n\nF3: Dset={B=0} Sset={C,D}\n\nF4\n\nC\nF1 F2\n\nC D\n*F1 *F2\n\n*F3 *F4\n\nFirst pass\n\nB\n\nB\n\nSecond pass\n\nF3.tau = F5.tau * P(B=0|A=0) + F6.tau * P(B=0|A=1) = P(B=0) F4.tau = F5.tau * P(B=1|A=0) + F6.tau * P(B=1|A=1) = P(B=1)\n\nD\n\nF1.tau = F3.tau * P(C=0|B=0) + F4.tau * P(C=0|B=1) = P(C=0) F2.tau = F3.tau * P(C=1|B=0) + F4.tau * P(C=1|B=1) = P(C=1)\n\nFactor values needed for c(A=0) and c(C=0,B=0) computation:\nF5.val=P(B=0|A=0)*F3.val+P(B=1|A=0)*F4.val F3.val=P(C=0|B=0)*F1.val+P(C=1|B=0)*F2.val F4.val=P(C=0|B=1)*F1.val+P(C=1|B=1)*F2.val F1.val=P(D=0|C=0)P(E=e|D=0)+P(D=1|C=0)P(E=e|D=1) F2.val=P(D=0|C=1)P(E=e|D=0)+P(D=1|C=1)P(E=e|D=1)\n\nc(A=0)=(1/P(e))*(F7.tau*P(A=0)*F5.val)=(1/P(e))(P(A=0)*P(E=e|A=0))=P(A=0|E=e) c(C=0,B=0)=(1/P(e))*F3.tau*P(C=0|B=0)*F1.val =(1/P(e) * (P(A=0,B=0)+P(A=1,B=0)) * P(C=0|B=0) * F1.val =(1/P(e)) * P(B=0) * P(C=0|B=0) * F1.val =(1/P(e)) * P(B=0) * P(C=0|B=0) * F1.val =(1/P(e)) * P(C=0,B=0) * F1.val =P(C=0,B=0,E=e)/P(e)=P(C=0,B=0|E=e)\n\nFigure 2: Learning example using the Markov chain A B C D E , where E is observed.\nIn the first pass, factors (Dset, Sset and val) are learned in a bottom up fashion. Also, the normalization constant P (E = e) (probability of evidence) is obtained. In the second pass, tau values are updated in a top-down fashion and used to calculate expected counts c(F.head, pa(F.head)) corresponding to each F.head (the figure shows the derivations for (A=0) and (C=0,B=0), but all counts are updated in the same pass).\n\nother variables are already assigned (these variables and t heir values are accumulated into Dset). We also check for factors that are active, multiply their value s in, and accumulate subsumed vars in Sset (to avoid branching on them). In line 10, we add V to the Sset. In line 11, we cache a new factor F with value F.v al = sum. We store V into F.head, a pointer to the last variable to be inserted into F.S set, and needed for parameter estimation described below. F.Dset consists of all the variables, except V , that appeared in any NSV CPT or the Dset of an activated factor at line 6. Regular Value Elimination is query-based, similar to varia ble elimination and recursive conditioning--what this means is that to answer a query of the type P (Q|E = e), where Q is query variable and E a set of evidence nodes, we force Q to be at the top of the search tree, run the backtracking algorithm and then read the answers to the quer ies P (Q = q |E = e), q Dom[Q], along each of the outgoing edges of Q. Parameter estimation would require running a number of queries on the order of the number of parameters to estimate. We extend VE into an algorithm that allows us to obtain Expectation Maximization sufficient statistics in a single run of Value Elimination plus a second pass, which can never take longer than the first one (and in practice is much faster). This two-pass procedur e is analogous to the collect-distribute evidence procedure in the Junction Tree algorithm, but here we do this via a search tree. Let X =x|pa(X)=y be a parameter associated with variable X with value x and parents Y = pa(X) when they have value y. Assuming a maximum likelihood learning scenario3 , to estimate X =x|pa(X)=y , we need to compute W f (X = x, pa(X) = y, E = e) = P (W, X = x, pa(X) = y, E = e)\n\\{X,pa(X)}\n\nwhich is a sum of joint probabilities of all configurations that are consistent with the assignment {X = x, pa(X) = y}. If we were to turn off factor caching, we would enumerate all such variable configurations and could compute the sum. When standard VE factors are used, however, this is no longer possible whenever X or any of its parents becomes subsumed. Fig. 2 illustrates an example of a VE tree and the factors that are learned in the case of a Mar kov chain with an evidence node at the end. We can readily estimate the parameters associate d with variables A and B as they are not subsumed along any branch. C and D become subsumed, however, and we cannot obtain the correct counts along all the branches that would lead to C and D in the full enumeration case. To address this issue, we store a special value, F.tau, in each factor. F.tau holds the sum over all path probabilities from the first level of the search tree to the level at which the factor F was\nFor Bayesian networks the likelihood function decomposes such that maximizing the expectation of the complete likelihood is equivalent to maximizing the \"local likelihood\" of each variable in the network.\n3\n\n\f\neither created or activated. For example, F 6.tau in fig. 2 is simply P (A = 1). Although we can compute F 3.tau directly, we can also compute it recursively using F 5.tau and F 6.tau as shown in the figure. This is because both F 5 and F 6 subsume F 3: in the context {F 5.Dset}, there exists a (unique) value dsub of F 5.head4 s.t. F 3 becomes activable. Likewise for F 6. We cannot compute F 1.tau directly, but we can, recursively, from F 3.tau and F 4.tau by taking advantage of a similar subsumption relationship. In general, we can show that the following recursive relationship holds: F F Fact .v al pa act Fact F .tau N S VF pa .head=dsub F.tau (1) F.v al pa pa\nF\n\nwhere F pa is the set of factors that subsume F , Fact is the set of all factors (including F ) that become active in the context of {F pa .Dset, F pa .head = dsub } and N S VF pa .head=dsub is the product of all newly single valued CPTs under the same context. For top-level factors (not subsumed by any factor), F.tau = Pevidence /F.v al, which is 1.0 when there is a unique top-level factor. Alg. 2 is a simple recursive computation of eq. 1 for each fact or. We visit learned factors in the reverse order in which they were learned to ensure that, for a ny factor F , F .tau is incremented (line 13) by any F that might have activated F (line 12). For example, in fig. 2, F 4 uses F 1 and F 2, so F 4.tau needs to be updated before F 1.tau and F 2.tau. In line 11, we can increment the counts for any NSV CPT entries since F.tau will account for the possible ways of reaching the configuration {F.Dset, F.head = d} in an equivalent full enumeration tree. Algorithm 1: FirstPass(level)\nInput: Graph G Output: A list of learned factors and Pevidence Select var V to branch on if V ==NONE then return Sset={}, Dset={} for d Dom[V ] do V d prod = productOfAllNSVsAndActiveFactors(Dset, Sset) if prod != 0 then FirstPass(level+1) sum += prod S set = S set {V } cacheNewFactor(F.head V ,F.v al sum, F.S set S set, F.Dset Dset);\n\n1 2 3 4 5 6 7 8 9 10\n\nAlgorithm 2: SecondPass()\nInput: F : List of factors in the reverse order learned in the first pass and Pevidence . Result: Updated counts foreach F F do if F.Dset = {} then F.tau Pevidence /F.v al else F.tau 0.0 Assign vars in F.Dset to their values V F.head (last node to have been subsumed in this factor) foreach d Dom[V ] do prod = productOfAllNSVsAndActiveFactors() prod = F.tau foreach newly single-valued CPT C do count(C.child,C.parents)+=prod/Pevidence F =getListOfActiveFactors() for F F do F .tau+ = prod/F .v al\n\n1 2 3 4 5 6 7 8 9 10 11 12 13\n\nMost Probable Explanation We compute MPE using a very similar two-pass algorithm. In the first pass, factors are used to store a maximum instead of a sum mation over variables in the Sset. We also keep track of the value of F.head at which the maximum is achieved. In the second pass, we recursively find the optimal variable configuration by following the trail of factors that are activated when we assign each F.head variable to its maximum value starting from the last learned factor.\n4\n\nRecall, F.head is the last variable to be added to a newly created factor in line 10 of alg. 1\n\n\f\n4\n\nMACHINE TRANSLATION WORD ALIGNMENT EXPERIMENTS\n\nA major motivation for pursuing the type of representation a nd inference described above is to make it possible to solve computationally-intensive real-world problems using large amounts of data, while retaining the full generality and expressiveness aff orded by the MDBN modeling language. In the experiments below we compare running times of MDBNs to GIZA++ on IBM Models 1 through 4 and the M-HMM model. GIZA++ is a special-purpose optimized MT word alignment C++ tool that is widely used in current state-of-the-art phrase-bas ed MT systems [10] and at the time of this writing is the only publicly available software that implements all of the IBM Models. We test on French-English 107 hand-aligned sentences5 from a corpus of the European parliament proceedings (Europarl [9]) and train on 10000 sentence pairs from the sam e corpus and of maximum number of words 40. The Alignment Error Rate (AER) [13] evaluation met ric quantifies how well the MPE assignment to the hidden alignment variables matches human-generated alignments. Several pruning and smoothing techniques are used by GIZA and MDBNs. GIZA prunes low lexical (P (f |e)) probability values and uses a default small value for unseen (or pruned) probability table entries. For models 3 and 4, for which there is no known polyno mial time algorithm to perform the full E-step or compute MPE, GIZA generates a set of high probability alignments using an MHMM and hill-climbing and collects EM counts over these alignments using M3 or M4. For MDBN models we use the following pruning strategy: at each level of the search tree we prune values which, together, account for the lowest specified percentage of the total probability mass of the product of all newly active CPTs in line 6 of alg. 1. This is a more effective pruning than simply removing low-probability values of each CPD because it factors in the joint contribution of multiple active variables. Table 1 shows a comparison of timing numbers obtained GIZA++ and MDBNs. The runtime numbers shown are for the combined tasks of training and decodin g; however, training time dominates given the difference in size between train and test sets. For models 1 and 2 neither GIZA nor MDBNs perform any pruning. For the M-HMM, we prune 60% of probabili ty mass at each level and use a Dirichlet prior over the alignment variables such that long -range transitions are exponentially less likely than shorter ones.6 This model achieves similar times and AER to GIZA's. Interestingly, without any pruning, the MDBN M-HMM takes 160 minutes to complete while only marginally improving upon the pruned model. Experimenting with severa l pruning thresholds, we found that AER would worsen much more slowly than runtime decreases. Models 3 and 4 have treewidth equal to the number of alignment variables (because of the global constraints tying them) and therefore require approximate inference. Using Model 3, and a drastic pruning threshold that only keeps the value with the top prob ability at each level, we were able to achieve an AER not much higher than GIZA's. For M4, it achieves a best AER of 31.7% while we do not improve upon Model3, most likely because a too restrictive pruning. Nevertheless, a simple variation on Model3 in the MDBN framework achieves a lower AER than our regular M3 (with pruning still the same). The M3-HMM hybrid model combines the Markov alignment dependencies from the M-HMM model with the fertility model of M3. MCMC Inference Sampling is widely used for inference in high-treewidth mod els. Although MDBNs support Likelihood Weighing, it is very inefficient when the probability of evidence is very small, as is the case in our MT models. Besides being slow, Markov chain Monte Carlo can be problematic when the joint distribution is not positive everywhere, in particular in the presence of determinism and hard constraints. Techniques such as block ing Gibbs sampling [8] try to address the problem. Often, however, one has to carefully choose a probl em-dependent proposal distribution. We used MCMC to improve training of the M3-HMM model. We were able to achieve an AER of 32.8% (down from 39.1%) but using 400 minutes of uniprocessor time.\n\n5\n\nCONCLUSION\n\nThe existing classes of graphical models are not ideally suited for representing SMT models because \"natural\" semantics for specifying the latter combine flavo rs of different GM types on top of standard directed Bayesian network semantics: switching parents found in Bayesian Multinets [6], aggregation relationships such as in Probabilistic Relational Models [5], and existence uncertainty [7]. We\nAvailable at http://www.cs.washington.edu/homes/karim French and English have similar word orders. On a different language pair, a different prior might be more appropriate. With a uniform prior, the MDBN M-HMM has 36.0% AER.\n6 5\n\n\f\nModel Init M1 M2 M -H M M M3 M4 M3-HMM\n\nGIZA++ M1 M -H M M 1m45s (47.7%) N/A 2m02s (41.3%) N/A 4m05s (35.0%) N/A 2m50 (45%) 5m20s (38.5%) 5m20s (34.8%) 7m45s (31.7%) N/A\n\nM DB N M1 3m20s (48.0%) 5m30s (41.0%) 4m15s (33.0%) 12m (43.6%) 25m (43.6%) 9m30 (41.0%) M -H M M N/A N/A N/A 9m (42.5%) 23m (42.6%) 9m15s (39.1%) MCMC 400m (32.8%)\n\nTable 1: MDBN VE-based learning versus GIZA++ timings and %AER using 5 EM iterations. The columns M1 and M-HMM correspond to the model that is used to initialize the model in the corresponding row. The last row is a hybrid Model3-HMM model that we implemented using MDBNs and is not expressible using GIZA. have introduced a generalization of dynamic Bayesian networks to easily and concisely build models consisting of varying-length parallel asynchronous and in teracting data streams. We have shown that our framework is useful for expressing various statistical machine translation models. We have also introduced new parameter estimation and decoding algorith ms using exact and approximate searchbased probability computation. While our timing results are not yet as fast as a hand-optimized C++ program on the equivalent model, we have shown that even in this general-purpose framework of MDBNs, our timing numbers are competitive and usable. Our framework can of course do much more than the IBM and HMM models. One of our goals is to use this framework to rapidly prototype novel MT systems and develop methods to statistically induce an interlingua. We also intend to use MDBNs in other domains such as multi-party social interaction analysis.\n\nReferences\n[1] F. Bacchus, S. Dalmao, and T. Pitassi. Value elimination: Bayesian inference via backtracking search. In UAI-03, pages 2028, San Francisco, CA, 2003. Morgan Kaufmann. [2] J. Bilmes and C. Bartels. On triangulating dynamic graphical models. In Uncertainty in Artificial Intelligence: Proceedings of the 19th Conference, pages 4756. Morgan Kaufmann, 2003. [3] P. F. Brown, J. Cocke, S. A. Della Piettra, V. J. Della Piettra, F. Jelinek, J. D. Lafferty, R. L. Mercer, and P. S. Roossin. A statistical approach to machine translation. Computational Linguistics, 16(2):7985, June 1990. [4] T. Dean and K. Kanazawa. Probabilistic temporal reasoning. AAAI, pages 524528, 1988. [5] N. Friedman, L. Getoor, D. Koller, and A. Pfeffer. Learning pro babilistic relational models. In IJCAI, pages 13001309, 1999. [6] D. Geiger and D. Heckerman. Knowledge representation and inference in similarity networks and Bayesian multinets. Artif. Intell., 82(1-2):4574, 1996. [7] L. Getoor, N. Friedman, D. Koller, and B. Taskar. Learning prob abilistic models of link structure. Journal of Machine Learning Research, 3(4-5):697707, May 2003. [8] C. Jensen, A. Kong, and U. Kjaerulff. Blocking Gibbs sampling in very large probabilistic expert systems. In International Journal of Human Computer Studies. Special Issue on Real-World Applications of Uncertain Reasoning., 1995. [9] P. Koehn. Europarl: A multilingual corpus for evaluation of machine http://www.isi.edu/koehn/publications/europarl, 2002. [11] S. Lauritzen. Graphical Models. Oxford Science Publications, 1996. [12] K. Murphy. Dynamic Bayesian Networks: Representation, Inference and Learning. PhD thesis, U.C. Berkeley, Dept. of EECS, CS Division, 2002. [13] F. J. Och and H. Ney. Improved statistical alignment models. In ACL, pages 440447, Oct 2000. [14] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 2nd printing edition, 1988. [15] S. Vogel, H. Ney, and C. Tillmann. HMM-based word alignment in statistical translation. In Proceedings of the 16th conference on Computational linguistics, pages 836841, Morristown, NJ, USA, 1996. translation.\n\n[10] P. Koehn, F. Och, and D. Marcu. Statistical phrase-based translation. In NAACL/HLT 2003, 2003.\n\n\f\n", "award": [], "sourceid": 3020, "authors": [{"given_name": "Karim", "family_name": "Filali", "institution": null}, {"given_name": "Jeff", "family_name": "Bilmes", "institution": null}]}