{"title": "Discriminative Log-Linear Grammars with Latent Variables", "book": "Advances in Neural Information Processing Systems", "page_first": 1153, "page_last": 1160, "abstract": null, "full_text": "Discriminative Log-Linear Grammars\n\nwith Latent Variables\n\nSlav Petrov and Dan Klein\n\nComputer Science Department, EECS Division\n\nUniversity of California at Berkeley, Berkeley, CA, 94720\n\n{petrov, klein}@cs.berkeley.edu\n\nAbstract\n\nWe demonstrate that log-linear grammars with latent variables can be practically\ntrained using discriminative methods. Central to ef\ufb01cient discriminative training\nis a hierarchical pruning procedure which allows feature expectations to be ef\ufb01-\nciently approximated in a gradient-based procedure. We compare L1 and L2 reg-\nularization and show that L1 regularization is superior, requiring fewer iterations\nto converge, and yielding sparser solutions. On full-scale treebank parsing exper-\niments, the discriminative latent models outperform both the comparable genera-\ntive latent models as well as the discriminative non-latent baselines.\n\n1 Introduction\n\nIn recent years, latent annotation of PCFG has been shown to perform as well as or better than stan-\ndard lexicalized methods for treebank parsing [1, 2]. In the latent annotation scenario, we imagine\nthat the observed treebank is a coarse trace of a \ufb01ner, unobserved grammar. For example, the single\ntreebank category NP (noun phrase) may be better modeled by several \ufb01ner categories representing\nsubject NPs, object NPs, and so on. At the same time, discriminative methods have consistently\nprovided advantages over their generative counterparts, including less restriction on features and\ngreater accuracy [3, 4, 5]. In this work, we therefore investigate discriminative learning of latent\nPCFGs, hoping to gain the best from both lines of work.\n\nDiscriminative methods for parsing are not new. However, most discriminative methods, at least\nthose which globally trade off feature weights, require repeated parsing of the training set, which is\ngenerally impractical. Previous work on end-to-end discriminative parsing has therefore resorted to\n\u201ctoy setups,\u201d considering only sentences of length 15 [6, 7, 8] or extremely small corpora [9]. To get\nthe bene\ufb01ts of discriminative methods, it has therefore become common practice to extract n-best\ncandidate lists from a generative parser and then use a discriminative component to rerank this list.\nIn such an approach, repeated parsing of the training set can be avoided because the discriminative\ncomponent only needs to select the best tree from a \ufb01xed candidate list. While most state-of-the-art\nparsing systems apply this hybrid approach [10, 11, 12], it has the limitation that the candidate list\noften does not contain the correct parse tree. For example 41% of the correct parses were not in the\ncandidate pool of \u224830-best parses in [10].\nIn this paper we present a hierarchical pruning procedure that exploits the structure of the model\nand allows feature expectations to be ef\ufb01ciently approximated, making discriminative training of\nfull-scale grammars practical. We present a gradient-based procedure for training a discriminative\ngrammar on the entire WSJ section of the Penn Treebank (roughly 40,000 sentences containing\n1 million words). We then compare L1 and L2 regularization and show that L1 regularization is\nsuperior, requiring fewer iterations to converge and yielding sparser solutions. Independent of the\nregularization, discriminative grammars signi\ufb01cantly outperform their generative counterparts in our\nexperiments.\n\n1\n\n\fRB\n\nNot\n\nFRAG\n\nNP\n\nDT\n\nthis\n\nNN\n\nyear\n\n(a)\n\n.\n\n.\n\nROOT\n\nFRAG\n\n.\n\n.\n\nFRAG\n\nRB\n\nNot\n\nNP\n\nDT\n\nthis\n\nNN\n\nyear\n\n(b)\n\nROOT\n\nFRAG-x\n\n.-x\n\n.\n\nFRAG-x\n\nRB-x\n\nNot\n\nNP-x\n\nDT-x\n\nthis\n\nNN-x\n\nyear\n\n(c)\n\nFigure 1: (a) The original tree. (b) The (binarized) X-bar tree. (c) The annotated tree.\n\n2 Grammars with latent annotations\n\nContext-free grammars (CFGs) underlie most high-performance parsers in one way or another [13,\n12, 14]. However, a CFG which simply takes the empirical productions and probabilities off of\na treebank does not perform well. This naive grammar is a poor one because its context-freedom\nassumptions are too strong in some places and too weak in others. Therefore, a variety of techniques\nhave been developed to both enrich and generalize the naive grammar. Recently an automatic state-\nsplitting approach was shown to produce state-of-the art performance [2, 14]. We extend this line of\nwork by investigating discriminative estimation techniques for automatically re\ufb01ned grammars.\n\nWe consider grammars that are automatically derived from a raw treebank. Our experiments are\nbased on a completely unsplit X-bar grammar, obtained directly from the Penn Treebank by the\nbinarization procedure shown in Figure 1. For each local tree rooted at an evaluation category X,\nwe introduce a cascade of new nodes labeled X so that each has two children in a right branching\nfashion. Each node is then re\ufb01ned with a latent variable, splitting each observed category into k\nunobserved subcategories. We refer to trees over unsplit categories as parse trees and trees over\nsplit categories as derivations.\n\nOur log-linear grammars are parametrized by a vector \u03b8 which is indexed by productions X \u2192 \u03b3.\nThe conditional probability of a derivation tree t given a sentence w can be written as:\n\nP\u03b8(t|w) =\n\n1\n\nZ(\u03b8, w) YX\u2192\u03b3\u2208t\n\ne\u03b8X\u2192\u03b3 =\n\n1\n\nZ(\u03b8, w)\n\ne\u03b8Tf (t)\n\n(1)\n\nwhere Z(\u03b8, w) is the partition function and f (t) is a vector indicating how many times each pro-\nduction occurs in the derivation t. The inside/outside algorithm [15] gives us an ef\ufb01cient way of\nsumming over an exponential number of derivations. Given a sentence w spanning the words\nw1, w2, . . . , wn = w1:n, the inside and outside scores of a (split) category A spanning (i, j) are\ncomputed by summing over all possible children B and C spanning (i, k) and (k, j) respectively:1\n\nSIN(A, i, j) = XA\u2192BC Xi<k<j\nSOUT(A, i, j) = XB\u2192CA X1\u2264k<i\nXB\u2192AC Xj<k\u2264n\n\n\u03c6A\u2192BC \u00d7 SIN(B, i, k) \u00d7 SIN(C, k, j)\n\n\u03c6B\u2192CA \u00d7 SOUT(B, k, j) \u00d7 SIN(C, k, i) +\n\n\u03c6B\u2192AC \u00d7 SOUT(B, i, k) \u00d7 SIN(C, j, k),\n\n(2)\n\nwhere we use \u03c6A\u2192BC = e\u03b8A\u2192BC . In the generative case these scores correspond to the inside and\ndef\noutside probabilities SIN(A, i, j) = PIN(A, i, j)\n=\nP(w1:iAwj:n) [15]. The scores lack this probabilistic interpretation in the discriminative case, but\nthey can nonetheless be normalized in the same way as probabilities to produce the expected counts\nof productions needed at training time. The posterior probability of a production A \u2192 BC spanning\n(i, j) with split point k in a sentence is easily expressed as:\n\ndef\n= P(wi:j |A) and SOUT(A, i, j) = POUT(A, i, j)\n\nhA \u2192 BC, i, j, ki \u221d SOUT(A, i, j) \u00d7 \u03c6A\u2192BC \u00d7 SIN(B, i, k) \u00d7 SIN(C, k, j)\n\n(3)\n\nTo obtain a grammar from the training trees, we want to learn a set of grammar parameters \u03b8 on\nlatent annotations despite the fact that the original trees lack the latent annotations. We will consider\n\n1Although we show only the binary component, of course both binary and unary productions are included.\n\n2\n\n\fgenerative grammars, where the parameters \u03b8 are set to maximize the joint likelihood of the train-\ning sentences and their parse trees, and discriminative grammars, where the parameters \u03b8 are set to\nmaximize the likelihood of the correct parse tree (vs. all possible trees) given a sentence. Previous\nwork on automatic grammar re\ufb01nement has focused on different estimation techniques for learning\ngenerative grammars with latent labels (training with basic EM [1], an EM-based split and merge\napproach [2], a non-parametric variational approach [16]). In the following, we review how gener-\native grammars are learned and present an algorithm for estimating discriminative grammars with\nlatent variables.\n\n2.1 Generative Grammars\n\nGenerative grammars with latent variables can be seen as tree structured hidden Markov models. A\nsimple EM algorithm [1] allows us to learn parameters for generative grammars which maximize\nthe log joint likelihood of the training sentences w and parse trees T :\n\nLjoint(\u03b8) = logYi\n\nP\u03b8(wi, Ti) = logYi Xt:Ti\n\nP\u03b8(wi, t),\n\n(4)\n\nwhere t are derivations (over split categories) corresponding to the observed parse tree (over unsplit\ncategories). In the E-Step we compute inside/outside scores over the set of derivations corresponding\nto the observed gold tree by restricting the sums in Eqn. 2 to produce only such derivations. 2\nWe then use Eqn. 3 to compute expectations which are normalized in the M-Step to update the\nproduction probabilities \u03c6X\u2192\u03b3 = e\u03b8X\u2192\u03b3 to their maximum likelihood estimates:\n\n\u03c6X\u2192\u03b3 = PT\nP\u03b3 \u2032 PT\n\nE\u03b8[fX\u2192\u03b3(t)|T ]\n\nE\u03b8[fX\u2192\u03b3 \u2032(t)|T ]\n\n(5)\n\nHere, E\u03b8 [fX\u2192\u03b3(t)|T ] denotes the expected count of the production (or feature) X \u2192 \u03b3 with respect\nto P\u03b8 in the set of derivations t, which are consistent with the observed parse tree T . Similarly, we\nwill write E\u03b8 [fX\u2192\u03b3(t)|w] for the expectation over all derivations of the sentence w.\nOur generative grammars with latent variables are probabilistic context-free grammars (CFGs),\n\nwhere P\u03b3 \u2032 \u03c6X\u2192\u03b3 \u2032 = 1 and Z(\u03b8) = 1. Note, however, that this normalization constraint poses\n\nno restriction on the model class, as probabilistic and weighted CFGs are equivalent [18].\n\n2.2 Discriminative Grammars\n\nDiscriminative grammars with latent variables can be seen as conditional random \ufb01elds [4] over\ntrees. For discriminative grammars, we maximize the log conditional likelihood:\n\nLcond(\u03b8) = logYi\n\nP\u03b8(Ti|wi) = logYi Xt:Ti\n\ne\u03b8Tf (t)\nZ(\u03b8, wi)\n\n(6)\n\nWe directly optimize this non-convex objective function using a numerical gradient based method\n(LBFGS [19] in our implementation).3 Fitting the log-linear model involves the following deriva-\ntives:\n\n\u2202Lcond(\u03b8)\n\n\u2202\u03b8X\u2192\u03b3\n\n= Xi\n\n(cid:18)E\u03b8 [fX\u2192\u03b3(t)|Ti] \u2212 E\u03b8[fX\u2192\u03b3(t)|wi](cid:19),\n\n(7)\n\nwhere the \ufb01rst term is the expected count of a production in derivations corresponding to the correct\nparse tree and the second term is the expected count of the production in all parses.\n\nThe challenge in estimating discriminative grammars is that the computation of some quantities\nrequires repeatedly taking expectations over all parses of all sentences in the training set. We will\ndiscuss ways to make their computation on large data sets practical in the next section.\n\n2Since the tree structure is observed this can be done in linear time [17].\n3Alternatively, maximum conditional likelihood estimation can also be seen as a special case of maximum\nlikelihood estimation, where P(w) is assumed to be the empirical one and not learned. The conditional like-\nlihood optimization can therefore be addressed by an EM algorithm which is similar to the generative case.\nHowever, while the E-Step remains the same, the M-Step involves \ufb01tting a log-linear model, which requires\noptimization, unlike the joint case, which can be done analytically using relative frequency estimators. This EM\nalgorithm typically converges to a comparable local maximum as direct optimization of the objective function\nbut requires 3-4 times more iterations.\n\n3\n\n\f3 Ef\ufb01cient Discriminative Estimation\n\nComputing the partition function in Eqn. 6 requires parsing of the entire training corpus. Even with\nrecent advances in parsing ef\ufb01ciency and fast CPUs, parsing the entire corpus repeatedly remains\nprohibitive. Fast parsers like [12, 14] can parse several sentences per second, but parsing the 40,000\ntraining sentences still requires more than 5 hours on a fast machine. Even in a parallel implemen-\ntation, parsing the training corpus several hundred times, as necessary for discriminative training,\nwould and, in fact, did in the case of maximum margin training [6], require weeks. Generally speak-\ning, there are two ways of speeding up the training process: reducing the total number of training\niterations and reducing the time required per iteration.\n\n3.1 Hierarchical Estimation\n\nThe number of training iterations can be reduced by training models of increasing complexity in a\nhierarchical fashion. For example in mixture modeling [20] and machine translation [21], a sequence\nof increasingly more complex models is constructed and each model is initialized with its (simpler)\npredecessor. In our case, we begin with the unsplit X-Bar grammar and iteratively split each category\nin two and re-train the grammar.\nIn each iteration, we initialize with the results of the smaller\ngrammar, splitting each annotation category in two and adding a small amount of randomness to\nbreak symmetry. In addition to reducing the number of training iterations, hierarchical training has\nbeen shown to lead to better parameter estimates [2]. However, even with hierarchical training,\nlarge-scale discriminative training will remain impractical, unless we can reduce the time required\nto parse the training corpus.\n\n3.2 Feature-Count Approximation\n\nHigh-performance parsers have employed coarse-to-\ufb01ne pruning schemes, where the sentence is\nrapidly pre-parsed with increasingly more complex grammars [22, 14]. Any constituent with suf-\n\ufb01ciently low posterior probability triggers the pruning of its re\ufb01ned variants in subsequent passes.\nWhile this method has no theoretical guarantees, it has been empirically shown to lead to a 100-fold\nspeed-up without producing search errors [14].\n\nInstead of parsing each sentence exhaustively with the most complex grammar in each iteration,\nwe can approximate the expected feature counts by parsing in a hierarchical coarse-to-\ufb01ne scheme.\nWe start by parsing exhaustively with the X-Bar grammar and then prune constituents with low\nposterior probability (e\u221210 in our experiments).4 We then continue to parse with the next more\nre\ufb01ned grammar, skipping over constituents whose less re\ufb01ned predecessor has been pruned. After\nparsing with the most re\ufb01ned grammar, we extract expected counts from the \ufb01nal (sparse) chart.\nThe expected counts will be approximations because many small counts have been set to zero by the\npruning procedure.\n\nEven though this procedure speeds-up each training iteration tremendously, training remains pro-\nhibitively slow. We can make repeated parsing of the same sentences signi\ufb01cantly more ef\ufb01cient\nby caching the pruning history from one training iteration to the next. Instead of computing each\nstage in the coarse-to-\ufb01ne scheme for every pass, we can compute it once when we start training a\ngrammar and update only the \ufb01nal, most re\ufb01ned scores in every iteration. Cached pruning has the\npositive side effect of constraining subcategories to re\ufb01ne their predecessors, so that we do not need\nto worry about issues like subcategory drift and projections [14].\n\nAs only extremely unlikely items are removed from the chart, pruning has virtually no effect on\nthe conditional likelihood. Pruning more aggressively leads to a training procedure reminiscent of\ncontrastive estimation [23], where the denominator is restricted to a neighborhood of the correct\nparse tree (rather than containing all possible parse trees). In our experiments, pruning more aggres-\nsively did not hurt performance for grammars with few subcategories, but limited the performance\nof grammars with many subcategories.\n\n4Even a tighter threshold produced no search errors on a held out set in [14]. We enforce that the gold parse\n\nis always reachable.\n\n4\n\n\fs\nt\nn\ne\nu\n\nt\ni\nt\ns\nn\no\nc\n \nd\ne\n\ne\nc\nn\ne\n\nt\n\nn\ne\ns\n \nr\ne\np\n\nt\nc\nu\nr\nt\ns\nn\no\nC\n\n 20000\n\n 15000\n\n 10000\n\n 5000\n\n 0\n\nNo pruning\nCoarse-to-fine pruning\nPrecomputed pruning\n\n21\n\n4\n\n8\n\n16\n\nNumber of latent subcategories\n\n(a)\n\nPARSING TIME\n1 subcategory\n2 subcategories\n4 subcategories\n8 subcategories\n16 subcategories\n\ncoarse-to-\ufb01ne\n\ncached pruning\n\n30 min\n40 min\n44 min\n47 min\n52 min\n\n350 min\n390 min\n434 min\n481 min\n533 min\n\n(b)\n\nFigure 2: Average number of constructed constituents per sentence (a) and time to parse the training\ncorpus for different pruning regimes and grammar sizes (b).\n\n4 Results\n\nWe ran our experiments on the Wall Street Journal (WSJ) portion of the English Penn Treebank\nusing the standard setup: we trained on sections 2 to 21. Section 22 was used as development set for\nintermediate results. All of section 23 was reserved for the \ufb01nal test. We used the EVALB parseval\nreference implementation for scoring. We will report F1-scores5 and exact match percentages. For\nthe \ufb01nal test, we selected the grammar that performed best on the development set.\n\nFor our lexicon, we used a simple approach where rare words (seen \ufb01ve times or less during training)\nare replaced by one of 50 unknown word tokens based on a small number of word-form features.\nTo parse new sentences with a grammar, we compute the posterior distribution over productions at\neach span and extract the tree with the maximum expected number of correct productions [14].\n\n4.1 Ef\ufb01ciency\n\nThe average number of constituents that are constructed while parsing a sentence is a good indicator\nfor the ef\ufb01ciency of our cached pruning scheme.6 Figure 2(a) shows the average number of chart\nitems that are constructed per sentence. Coarse-to-\ufb01ne pruning refers to hierarchical pruning without\ncaching [14] and while it is better than no-pruning, it still constructs a large number of constituents\nfor heavily re\ufb01ned grammars. In contrast, with cached pruning the number of constructed chart\nitems stays roughly constant (or even decreases) when the number of subcategories increases. The\nreduced number of constructed constituents results in a 10-fold reduction of parsing time, see Figure\n2(b), and makes discriminative training on a large scale corpus computationally feasible.\n\nWe found that roughly 100-150 training iterations were needed for LBFGS to converge after each\nsplit. Distributing the training over several machines is straightforward as each sentence can be\nparsed independently of all other sentences. Starting from an unsplit X-Bar grammar we were able\nto hierarchically train a 16 substate grammar in three days using eight CPUs in parallel.7\nIt should be also noted that we can expedite training further by training in an interleaved mode, where\nafter splitting a grammar we \ufb01rst run generative training for some time (which is very fast) and then\nuse the resulting grammar to initialize the discriminative training. In such a training regime, we only\nneeded around 50 iterations of discriminative training until convergence, signi\ufb01cantly speeding up\nthe training, while maintaining the same \ufb01nal performance.\n\n4.2 Regularization\n\nRegularization is often necessary to prevent discriminative models from over\ufb01tting on the training\nset. Surprisingly enough, we found that no regularization was necessary when training on the en-\ntire training set, even in the presence of an abundance of features. During development we trained\non subsets of the training corpus and found that regularization was crucial for preventing over\ufb01t-\n\n5The harmonic mean of precision P and recall R: 2P R\n6The other main factor determining the parsing time is the grammar size.\n7Memory limitations prevent us from learning grammars with more subcategories, a problem that could be\n\nP +R\n\n.\n\nalleviated by merging back the least usefull splits as in [2].\n\n5\n\n\fEXACT MATCH\n\ngenerative\n\ndiscriminative\n\nF1-SCORE\n\ngenerative\n\ndiscriminative\n\n1 subcategory\n2 subcategories\n4 subcategories\n8 subcategories\n16 subcategories\n\n7.6\n14.6\n24.6\n31.4\n35.8\n\n7.8\n20.1\n31.3\n37.0\n39.4\n\n64.8\n76.4\n83.7\n86.6\n88.7\n\n67.3\n80.8\n85.6\n87.8\n89.3\n\nTable 1: Discriminative training is superior to generative training for exact match and for F1-score.\n\n1 subcategory\n2 subcategories\n4 subcategories\n8 subcategories\n16 subcategories\n\nF1-score Exact\n7.8\n20.1\n31.3\n37.0\n39.4\n\nL1 regularization\n# Feat.\n23 K\n74 K\n147 K\n318 K\n698 K\n\n67.3\n80.8\n85.6\n87.8\n89.3\n\n# Iter.\n\n44\n108\n99\n82\n75\n\nF1-score Exact\n7.9\n19.5\n31.5\n36.9\n38.7\n\nL2 regularization\n# Feat.\n35 K\n123 K\n547 K\n2,983 K\n11,489 K\n\n67.4\n80.3\n85.7\n87.6\n89.1\n\n# Iter.\n\n67\n132\n148\n111\n102\n\nTable 2: L1 regularization produces sparser solutions and requires fewer training iterations than L2\nregularization.\n\nting. This result is in accordance with [16] where a variational Bayesian approach was found to be\nbene\ufb01cial for small training sets but performed on par with EM for large amounts of training data.\n\nRegularization is achieved by adding a penalty term to the conditional log likelihood function\nLcond(\u03b8). This penalty term is often a weighted norm of the parameter vector and thereby penalizes\nlarge parameter values. We investigated L1 and L2 regularization:\n\nL\u2032\n\ncond(\u03b8) = Lcond(\u03b8) \u2212\n\n1\n\n2 XX\u2192\u03b3\n\n|\u03b8X\u2192\u03b3|\n\n\u03c3\n\nL\u2032\u2032\n\ncond(\u03b8) = Lcond(\u03b8) \u2212 XX\u2192\u03b3\n\n(cid:18) \u03b8X\u2192\u03b3\n\n\u03c3 (cid:19)2\n\n(8)\n\nwhere the regularization parameter \u03c3 is tuned on a held out set. In the L2 case, the penalty term is\na convex and differentiable function of the parameters and hence can be easily intergrated into our\ntraining procedure. In the L1 case, however, the penalty term is discontinuous whenever some pa-\nrameter equals zero. To handle the discontinuinty of the gradient, we used the orthant-wise limited-\nmemory quasi-Newton algorithm of [24].\nTable 2 shows that while there is no signi\ufb01cant performance difference in models trained with L1 or\nL2 regularization, there is signi\ufb01cant difference in the number of training iterations and the sparsity\nof the parameter vector. L1 regularization leads to extremely sparse parameter vectors (96% of the\nparameters are zero in the 16 subcategory case), while no parameter value becomes exactly zero with\nL2 regularization. It remains to be seen how this sparsity can be exploited, as these zeros become\nones when exponentiated in order to be used in the computation of inside and outside scores.\n\n4.3 Final Test Set Results\n\nTable 1 shows a comparison of generative and discriminative grammars for different numbers of\nsubcategories. Discriminative training is superior to generative training for exact match as well as for\nF1-score for all numbers of subcategories. For our largest grammars, we see absolute improvements\nof 3.63% and 0.61% in exact match and F1 score respectively. The better performance is due to\nbetter parameter estimates, as the model classes de\ufb01ned by the generative and discriminative model\n(probabilistic vs. weighted CFGs) are equivalent [18] and the same feature sets were used in all\nexperiments.\nOur \ufb01nal test set parsing F1-score of 88.8/88.3 (40 word sentences/all sentences) is better than most\nother systems, including basic generative latent variable grammars [1] (F1-score of 86.7/86.1) and\neven fully lexicalized systems [13] (F1-score of 88.6/88.2), but falls short of the very best systems\n[12, 14], which achieve accuracies above 90%. However, many of the techniques used in [12, 14]\nare orthogonal to what was presented here (additional non-local/overlapping features, merging of\nunnecessary splits) and could be incorporated into the discriminative model.\n\n6\n\n\fe\nr\no\nc\ns\n \n1\nF\nn\n\n \n\ni\n \ns\ns\no\nL\n\n 0.2\n 0\n-0.2\n-0.4\n-0.6\n-0.8\n-1\n-1.2\n-1.4\n-1.6\n\ngenerative\ndiscriminative\n\n 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1\n\nMerging Percentage\n\n(a)\n\n80% MERGING\nonly in\ngenerative\n\ncommon\n\nonly in\ndiscriminative\n\nGrammar\n\n-\n\nS, S, SBAR,\n\nNP, NP,\nVP, VP\n\nADJP, SINV\n\n(b)\n\nLexicon\nNNP, NNS,\nCD, WP$\nDT, CC, IN,\n\nVBD, VB, VBZ,\n\nNN, RB, JJ\n\nVBG\n\nFigure 3: (a) Loss in F1 score for different amounts of merging. (b) Categories with two subcate-\ngories after merging 80% of the subcategories according to the merging criterion in [2].\n\n4.4 Analysis\n\nGeneratively trained grammars with latent variables have been shown to exhibit many linguistically\ninterpretable phenomena [2]. Space does not permit a thorough exposition, and post hoc analysis of\nlearned structures is prone to seeing what one expects, but nonetheless it can be helpful to illustrate\nthe broad patterns that are learned. Not surprisingly, many comparable trends can be observed in\ngeneratively and discriminatively trained grammars. For example, the same subdivisions of the\ndeterminer category (DT) into de\ufb01nite (the), inde\ufb01nite (a), demonstrative (this) and quanti\ufb01cational\n(some) elements emerge under both training regimes. Another example is the preposition category\n(IN) where subcategories for subordinating conjunctions like (that) and different types of proper\nprepositions are learned. Typically the divisions in the discriminative grammars are much more\npronounced, putting the majority of the weight on a few dominant words.\n\nWhile many similarities can be found, it is especially interesting to examine how generative and\ndiscriminative grammars differ. The nominal categories in generative grammars exhibit many clus-\nters of semantic nature (e.g. subcategories for dates, monetary units, capitalized words, etc.). For\nexample, the following two subcategories of the proper noun (NNP) category {New, San, Wall} and\n{York, Francisco, Street} (here represented by the three most likely words) are learned by the gen-\nerative grammars. These subcategories are very useful for modeling correlations when generating\nwords and many clusters with such semantic patterns appear in the generative grammars. How-\never, these clusters do not interact strongly with disambiguation and are therefore not learned by\nthe discriminative grammars. Similar observations hold for plural proper nouns (NNPS), superlative\nadjectives (JJS), and cardinal numbers (CD), which are heavily split into semantic subcategories in\nthe generative grammars but are split very little or not at all in the discriminative grammars.\n\nExamining the phrasal splits is much more intricate. We therefore give just one example from\ngrammars with two subcategories, which illustrates the main difference between generative and dis-\ncriminative grammars. Simple declarative clauses (S) are the most common sentences in the Penn\nTreebank, and in the generative case the most likely expansion of the ROOT category is ROOT\u2192S1,\nbeing chosen 91% of the time. In the discriminative case this production is only the third likeliest\nwith a weight of 13.2. The highest weighted expansion of the ROOT in the discriminative grammar\nis ROOT\u2192SBARQ1, with a weight of 46.5, a production that has a probability of 0.3% in the gener-\native grammar. While generative grammars model the empirical distributions of productions in the\ntraining set, discriminative grammars maximize the discriminative power of the model. This can for\nexample result in putting the majority of the weight on underrepresented productions.\n\nWe applied the merging criterion suggested in [2] to two grammars with two subcategories in order\nto quantitatively examine how many subcategories are learned. This criterion approximates the loss\nin joint likelihood incurred from merging two subcategories and we extended it to approximate the\nloss in conditional likelihood from merging two subcategories at a given node. Figure 3(a) shows\nthe loss in F1-score when the least useful fraction of the subcategories are merged. Our observation\nthat the discriminative grammars learn far fewer clusters are con\ufb01rmed, as one can merge back 80%\nof the subcategories at almost no loss in F1 (while one can merge only 50% in the generative case).\nThis suggest that one can learn discriminative grammars which are signi\ufb01cantly more compact and\naccurate than their generative counterparts. Figure 3(b) shows which categories remain split when\n80% of the splits are merged. While there is a substantial overlap between the learned splits, one\ncan see that joint likelihood can be better maximized by re\ufb01ning the lexicon, while conditional\nlikelihood is better maximized by re\ufb01ning the grammar.\n\n7\n\n\f5 Conclusions and Future Work\n\nWe have presented a hierarchical pruning procedure that allows ef\ufb01cient discriminative training of\nlog-linear grammars with latent variables. We avoid repeated computation of similar quantities by\ncaching information between training iterations and approximating feature expectations. We pre-\nsented a direct gradient-based procedure for optimizing the conditional likelihood function which\nin our experiments on full-scale treebank parsing lead to discriminative latent models which outper-\nform both the comparable generative latent models, as well as the discriminative non-latent base-\nlines. We furthemore investigated different regularization penalties and showed that L1 regulariza-\ntion leads to extremely sparse solutions\n\nWhile our results are encouraging, this is merely a \ufb01rst investigation into large-scale discriminative\ntraining of latent variable grammars and opens the door for many future experiments: discrimina-\ntive grammars allow the seamless integration of non-local and overlapping features and it will be\ninteresting to see how proven features from reranking systems [10, 11, 12] and other orthogonal\nimprovements like merging and smoothing [2] will perform in an end-to-end discriminative system.\n\nReferences\n\n[1] T. Matsuzaki, Y. Miyao, and J. Tsujii. Probabilistic CFG with latent annotations. In ACL \u201905, 2005.\n[2] S. Petrov, L. Barrett, R. Thibaux, and D. Klein. Learning accurate, compact, and interpretable tree anno-\n\ntation. In ACL \u201906, 2006.\n\n[3] A. Y. Ng and M. I. Jordan. On discriminative vs. generative classi\ufb01ers: A comparison of logistic regression\n\nand naive Bayes. In NIPS \u201902, 2002.\n\n[4] J. Lafferty, A. McCallum, and F. Pereira. Conditional Random Fields: Probabilistic models for segment-\n\ning and labeling sequence data. In ICML \u201901, 2001.\n\n[5] D. Klein and C. Manning. Conditional structure vs conditional estimation in NLP models. In EMNLP\n\n\u201902, 2002.\n\n[6] B. Taskar, D. Klein, M. Collins, D. Koller, and C. Manning. Max-margin parsing. In EMNLP \u201904, 2004.\n[7] J. Henderson. Discriminative training of a neural network statistical parser. In ACL \u201904, 2004.\n[8] J. Turian, B. Wellington, and I. D. Melamed. Scalable discriminative learning for natural language parsing\n\nand translation. In NIPS \u201907, 2007.\n\n[9] M. Johnson. Joint and conditional estimation of tagging and parsing models. In ACL \u201901, 2001.\n[10] M. Collins. Discriminative reranking for natural language parsing. In ICML \u201900, 2000.\n[11] T. Koo and M. Collins. Hidden-variable models for discriminative reranking. In EMNLP \u201905, 2005.\n[12] E. Charniak and M. Johnson. Coarse-to-Fine N-Best Parsing and MaxEnt Discriminative Reranking. In\n\nACL\u201905, 2005.\n\n[13] M. Collins. Head-Driven Statistical Models for Natural Language Parsing. PhD thesis, UPenn., 1999.\n[14] S. Petrov and D. Klein. Improved inference for unlexicalized parsing. In HLT-NAACL \u201907, 2007.\n[15] K. Lari and S. Young. The estimation of stochastic context-free grammars using the inside-outside algo-\n\nrithm. Computer Speech and Language, 1990.\n\n[16] P. Liang, S. Petrov, M. I. Jordan, and D. Klein. The in\ufb01nite PCFG using hierarchical Dirichlet processes.\n\nIn EMNLP \u201907, 2007.\n\n[17] F. Pereira and Y. Schabes. Inside-outside reestimation from partially bracketed corpora. In ACL, 1992.\n[18] N. A. Smith and M. Johnson. Weighted and probabilistic context-free grammars are equally expressive.\n\nTo appear in Computational Lingusitics, 2007.\n\n[19] J. Nocedal and S. J. Wright. Numerical Optimization. Springer, 1999.\n[20] N. Ueda, R. Nakano, Z. Ghahramani, and G. E. Hinton. Split and merge EM algorithm for mixture\n\nmodels. Neural Computation, 12(9):2109\u20132128, 2000.\n\n[21] P. F. Brown, S. A. D. Pietra, V. J. D. Pietra, and R. L. Mercer. The mathematics of statistical machine\n\ntranslation. Computational Lingusitics, 19(2), 1993.\n\n[22] E. Charniak, M. Johnson, D. McClosky, et al. Multi-level coarse-to-\ufb01ne PCFG Parsing. In HLT-NAACL\n\n\u201906, 2006.\n\n[23] N. A. Smith and J. Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In ACL\n\n\u201905, 2005.\n\n[24] G. Andrew and J. Gao. Scalable training of L1-regularized log-linear models. In ICML \u201907, 2007.\n\n8\n\n\f", "award": [], "sourceid": 630, "authors": [{"given_name": "Slav", "family_name": "Petrov", "institution": null}, {"given_name": "Dan", "family_name": "Klein", "institution": null}]}