{"title": "Dependent Multinomial Models Made Easy: Stick-Breaking with the Polya-gamma Augmentation", "book": "Advances in Neural Information Processing Systems", "page_first": 3456, "page_last": 3464, "abstract": "Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and year, and text documents are all commonly modeled with multinomial distributions.  In all of these cases, we expect some form of dependency between the draws: the nucleotide at one position in the DNA strand may depend on the preceding nucleotides, children's names are highly correlated from year to year, and topics in text may be correlated and dynamic.  These dependencies are not naturally captured by the typical Dirichlet-multinomial formulation.  Here, we leverage a logistic stick-breaking representation and recent innovations in P\\'{o}lya-gamma augmentation to reformulate the multinomial distribution in terms of latent variables with jointly Gaussian likelihoods, enabling us to take advantage of a host of Bayesian inference techniques for Gaussian models with minimal overhead.", "full_text": "Dependent Multinomial Models Made Easy:\n\nStick Breaking with the P\u00b4olya-Gamma Augmentation\n\nScott W. Linderman\u2217\nHarvard University\n\nCambridge, MA 02138\n\nswl@seas.harvard.edu\n\nMatthew J. Johnson\u2217\nHarvard University\n\nCambridge, MA 02138\n\nmattjj@csail.mit.edu\n\nRyan P. Adams\n\nrpa@seas.harvard.edu\n\nTwitter & Harvard University\n\nCambridge, MA 02138\n\nAbstract\n\nMany practical modeling problems involve discrete data that are best represented\nas draws from multinomial or categorical distributions. For example, nucleotides\nin a DNA sequence, children\u2019s names in a given state and year, and text documents\nare all commonly modeled with multinomial distributions. In all of these cases,\nwe expect some form of dependency between the draws: the nucleotide at one\nposition in the DNA strand may depend on the preceding nucleotides, children\u2019s\nnames are highly correlated from year to year, and topics in text may be corre-\nlated and dynamic. These dependencies are not naturally captured by the typical\nDirichlet-multinomial formulation. Here, we leverage a logistic stick-breaking\nrepresentation and recent innovations in P\u00b4olya-gamma augmentation to reformu-\nlate the multinomial distribution in terms of latent variables with jointly Gaussian\nlikelihoods, enabling us to take advantage of a host of Bayesian inference tech-\nniques for Gaussian models with minimal overhead.\n\n1\n\nIntroduction\n\nIt is often desirable to model discrete data in terms of continuous latent structure. In applications in-\nvolving text corpora, discrete-valued time series, or polling and purchasing decisions, we may want\nto learn correlations or spatiotemporal dynamics and leverage these structures to improve inferences\nand predictions. However, adding these continuous latent dependence structures often comes at the\ncost of signi\ufb01cantly complicating inference: such models may require specialized, one-off inference\nalgorithms, such as a non-conjugate variational optimization, or they may only admit very general\ninference tools like particle MCMC [1] or elliptical slice sampling [2], which can be inef\ufb01cient and\ndif\ufb01cult to scale. Developing, extending, and applying these models has remained a challenge.\nIn this paper we aim to provide a class of such models that are easy and ef\ufb01cient. We develop models\nfor categorical and multinomial data in which dependencies among the multinomial parameters are\nmodeled via latent Gaussian distributions or Gaussian processes, and we show that this \ufb02exible class\nof models admits a simple auxiliary variable method that makes inference easy, fast, and modular.\nThis construction not only makes these models simple to develop and apply, but also allows the\nresulting inference methods to use off-the-shelf algorithms and software for Gaussian processes and\nlinear Gaussian dynamical systems.\nThe paper is organized as follows. After providing background material and de\ufb01ning our general\nmodels and inference methods, we demonstrate the utility of this class of models by applying it to\nthree domains as case studies. First, we develop a correlated topic model for text corpora. Second,\nwe study an application to modeling the spatial and temporal patterns in birth names given only\nsparse data. Finally, we provide a new continuous state-space model for discrete-valued sequences,\n\n\u2217These authors contributed equally.\n\n1\n\n\fincluding text and human DNA. In each case, given our model construction and auxiliary variable\nmethod, inference algorithms are easy to develop and very effective in experiments.\nCode to use these models, write new models that leverage these inference methods, and reproduce\nthe \ufb01gures in this paper is available at github.com/HIPS/pgmult.\n\n2 Modeling correlations in multinomial parameters\n\nIn this section, we discuss an auxiliary variable scheme that allows multinomial observations to\nappear as Gaussian likelihoods within a larger probabilistic model. The key trick discussed in the\nproceeding sections is to introduce P\u00b4olya-gamma random variables into the joint distribution over\ndata and parameters in such a way that the resulting marginal leaves the original model intact.\nThe integral identity underlying the P\u00b4olya-gamma augmentation scheme [3] is\n\n(e\u03c8)a\n\n(1 + e\u03c8)b = 2\u2212be\u03ba\u03c8\n\n(1)\nwhere \u03ba = a \u2212 b/2 and p(\u03c9 | b, 0) is the density of the P\u00b4olya-gamma distribution PG(b, 0), which\ndoes not depend on \u03c8. Consider a likelihood function of the form\n\n0\n\ne\u2212\u03c9\u03c82/2p(\u03c9 | b, 0) d\u03c9,\n\n(cid:90) \u221e\n\np(x| \u03c8) = c(x)\n\n(e\u03c8)a(x)\n\n(1 + e\u03c8)b(x)\n\n(2)\n\n(cid:90) \u221e\n\n0\n\nfor some functions a, b, and c. Such likelihoods arise, e.g., in logistic regression and in binomial and\nnegative binomial regression [3]. Using (1) along with a prior p(\u03c8), we can write the joint density\nof (\u03c8, x) as\n\np(\u03c8, x) = p(\u03c8) c(x)\n\n(e\u03c8)a(x)\n\n(1 + e\u03c8)b(x)\n\n=\n\np(\u03c8) c(x) 2\u2212b(x)e\u03ba(x)\u03c8e\u2212\u03c9\u03c82/2p(\u03c9 | b(x), 0) d\u03c9.\n\n(3)\n\nThe integrand of (3) de\ufb01nes a joint density on (\u03c8, x, \u03c9) which admits p(\u03c8, x) as a marginal density.\nConditioned on these auxiliary variables \u03c9, we have\n\np(\u03c8 | x, \u03c9) \u221d p(\u03c8)e\u03ba(x)\u03c8e\u2212\u03c9\u03c82/2\n\n(4)\nwhich is Gaussian when p(\u03c8) is Gaussian. Furthermore, by the exponential tilting property of the\nP\u00b4olya-gamma distribution, we have \u03c9 | \u03c8, x \u223c PG(b(x), \u03c8). Thus the identity (1) gives rise to a\nconditionally conjugate augmentation scheme for Gaussian priors and likelihoods of the form (2).\nThis augmentation scheme has been used to develop Gibbs sampling and variational inference al-\ngorithms for Bernoulli, binomial [3], and negative binomial [4] regression models with logit link\nfunctions, and to the multinomial distribution with a multi-class logistic link function [3, 5].\nThe multi-class logistic \u201csoftmax\u201d function, \u03c0LN(\u03c8), maps a real-valued vector \u03c8 \u2208 RK to a proba-\nj=1 e\u03c8j . It is commonly used in multi-class regres-\nsion [6] and correlated topic modeling [7]. Correlated multinomial parameters can be modeled with\na Gaussian prior on the vector \u03c8, though the resulting models are not conjugate. The P\u00b4olya-gamma\naugmentation can be applied to such models [3, 5], but it only provides single-site Gibbs updating\nof \u03c8. This paper develops a joint augmentation in the sense that, given the auxiliary variables, the\nentire vector \u03c8 is resampled as a block in a single Gibbs update.\n\nbility vector \u03c0 \u2208 [0, 1]K by setting \u03c0k = e\u03c8k /(cid:80)K\n\n2.1 A new P\u00b4olya-gamma augmentation for the multinomial distribution\nFirst, rewrite the K-dimensional multinomial recursively in terms of K \u2212 1 binomial densities:\n\nK\u22121(cid:89)\n1 \u2212(cid:80)\n\nk=1\n\n2\n\nMult(x| N, \u03c0) =\n\nNk = N \u2212(cid:88)\n\nxj,\n\nj<k\n\n(cid:101)\u03c0k =\n\nBin(xk | Nk,(cid:101)\u03c0k),\n\n\u03c0k\nj<k \u03c0j\n\n,\n\nk = 2, 3, . . . , K,\n\n(5)\n\n(6)\n\n\fFigure 1: Correlated 2D Gaussian priors on \u03c8 and their implied densities on \u03c0SB(\u03c8). See text for details.\n\nThis decomposition of the multinomial density is a \u201cstick-breaking\u201d representation where\n\nwhere N1 = N =(cid:80)\nk xk and (cid:101)\u03c01 = \u03c01. For convenience, we de\ufb01ne N (x) \u2261 [N1, . . . , NK\u22121].\neach (cid:101)\u03c0k represents the fraction of the remaining probability mass assigned to the k-th com-\nponent. We let (cid:101)\u03c0k = \u03c3(\u03c8k), where \u03c3(\u00b7) denotes the logistic function, and de\ufb01ne the func-\nNext, we rewrite the density into the form required by (1) by substituting \u03c3(\u03c8k) for(cid:101)\u03c0k:\n\ntion, \u03c0SB : RK\u22121 \u2192 [0, 1]K, which maps a vector \u03c8 to a normalized probability vector \u03c0.\n\nMult(x| N, \u03c8) =\n\nK\u22121(cid:89)\n\nk=1\n\nBin(xk | Nk, \u03c3(\u03c8k)) =\n\n=\n\n\u03c3(\u03c8k)xk (1 \u2212 \u03c3(\u03c8k))Nk\u2212xk\n\nK\u22121(cid:89)\nK\u22121(cid:89)\n\nk=1\n\n(cid:18)Nk\n(cid:18)Nk\n\nxk\n\n(cid:19)\n(cid:19) (e\u03c8k )xk\n\nxk\n\n(1 + e\u03c8k )Nk\n\nk=1\n\n.\n\n(7)\n\n(8)\n\nChoosing ak(x) = xk and bk(x) = Nk for each k = 1, 2, . . . , K \u2212 1, we can then introduce P\u00b4olya-\ngamma auxiliary variables \u03c9k corresponding to each coordinate \u03c8k; dropping terms that do not\ndepend on \u03c8 and completing the square yields\n\ne(xk\u2212Nk/2)\u03c8k\u2212\u03c9k\u03c82\n\nk/2 \u221d N\n\n\u2126\u22121\u03ba(x)\n\n,\n\n(9)\n\n(cid:18)\n\n(cid:19)\n\n(cid:12)(cid:12)(cid:12)(cid:12) \u03c8, \u2126\u22121\n\np(x, \u03c9 | \u03c8) \u221d K\u22121(cid:89)\n\nk=1\n\nwhere \u2126 \u2261 diag(\u03c9) and \u03ba(x) \u2261 x \u2212 N (x)/2. That is, conditioned on \u03c9, the likelihood of \u03c8 under\nthe augmented multinomial model is proportional to a diagonal Gaussian distribution.\nFigure 1 shows how several Gaussian densities map to probability densities on the simplex. Cor-\nrelated Gaussians (left) put most probability mass near the \u03c01 = \u03c02 axis of the simplex, and anti-\ncorrelated Gaussians (center) put mass along the sides where \u03c01 is large when \u03c02 is small and\nvice-versa. Finally, a nearly isotropic Gaussian approximates a symmetric Dirichlet. Appendix A\ngives a closed-form expression for the density on \u03c0 induced by a Gaussian distribution on \u03c8, and\nalso an expression for a diagonal Gaussian that approximates a Dirichlet by matching moments.\n\n3 Correlated topic models\n\nThe Latent Dirichlet Allocation (LDA) [8] is a popular model for learning topics from text corpora.\nThe Correlated Topic Model (CTM) [7] extends LDA by including a Gaussian correlation structure\namong topics. This correlation model is powerful not only because it reveals correlations among\n\n3\n\n\fFigure 2: A comparison of correlated topic model performance. The left panel shows a subset of the inferred\ntopic correlations for the AP News corpus. Two examples are highlighted: a) positive correlation between topics\n(house, committee, congress, law) and (Bush, Dukakis, president, campaign), and b) anticorrelation between\n(percent, year, billion, rate) and (court, case, attorney, judge). The middle and right panels demonstrate the\nef\ufb01cacy of our SB-CTM relative to competing models on the AP News corpus and the 20 Newsgroup corpus,\nrespectively.\n\ntopics but also because inferring such correlations can signi\ufb01cantly improve predictions, especially\nwhen inferring the remaining words in a document after only a few have been revealed [7]. How-\never, the addition of this Gaussian correlation structure breaks the Dirichlet-Multinomial conjugacy\nof LDA, making estimation and particularly Bayesian inference and model-averaged predictions\nmore challenging. An approximate maximum likelihood approach using variational EM [7] is often\neffective, but a fully Bayesian approach which integrates out parameters may be preferable, espe-\ncially when making predictions based on a small number of revealed words in a document. A recent\nBayesian approach based on a P\u00b4olya-Gamma augmentation to the logistic normal CTM (LN-CTM)\n[5] provides a Gibbs sampling algorithm with conjugate updates, but the Gibbs updates are limited\nto single-site resampling of one scalar at a time, which can lead to slow mixing in correlated models.\nIn this section we show that MCMC sampling in a correlated topic model based on the stick breaking\nconstruction (SB-CTM) can be signi\ufb01cantly more ef\ufb01cient than sampling in the LN-CTM while\nmaintaining the same integration advantage over EM.\nIn the standard LDA model, each topic \u03b2t (t = 1, 2, . . . , T ) is a distribution over a vocabulary\nof V possible words, and each document d has a distribution over topics \u03b8d (d = 1, 2, . . . , D).\nThe n-th word in document d is denoted wn,d for d = 1, 2, . . . , Nd. When each \u03b2t and \u03b8d is given\na symmetric Dirichlet prior with parameters \u03b1\u03b2 and \u03b1\u03b8, respectively, the generative model is\n\u03b2t \u223c Dir(\u03b1\u03b2),\nzn,d | \u03b8d \u223c Cat(\u03b8d), wn,d | zn,d,{\u03b2t} \u223c Cat(\u03b2zn,d\n). (10)\nThe CTM replaces the Dirichlet prior on each \u03b8d with a correlated prior induced by \ufb01rst\nsampling a correlated Gaussian vector \u03c8d \u223c N (\u00b5, \u03a3) and then applying the logistic normal\nmap: \u03b8d = \u03c0LN(\u03c8d) Analogously, our SB-CTM generates the correlation structure by instead ap-\nplying the stick-breaking logistic map, \u03b8d = \u03c0SB(\u03c8d). The goal is then to infer the posterior dis-\ntribution over the topics \u03b2t, the documents\u2019 topic allocations \u03c8d, and their mean and correlation\nstructure (\u00b5, \u03a3), where the parameters (\u00b5, \u03a3) are given a conjugate normal-inverse Wishart (NIW)\nprior. Modeling correlation structure within the topics \u03b2 can be done analogously.\nFor fully Bayesian inference in the SB-CTM, we develop a Gibbs sampler that exploits the block\nconditional Gaussian structure provided by the stick-breaking construction. The Gibbs sampler\niteratively samples z | w, \u03b2, \u03c8; \u03b2 | z, w; \u03c8 | z, \u00b5, \u03a3, \u03c9;\nand \u00b5, \u03a3| \u03c8 as well as the auxiliary\nvariables \u03c9 | \u03c8, z. The \ufb01rst two are standard updates for LDA models, so we focus on the latter\nthree. Using the identities derived in Section 2.1, the conditional density of each \u03c8d | zd, \u00b5, \u03a3, \u03c9\ncan be written\n\n\u03b8d \u223c Dir(\u03b1\u03b8),\n\np(\u03c8d | zd, \u03c9d) \u221d N (\u2126\u22121\n\nd \u03ba(cd)| \u03c8d, \u2126\u22121\n\nd ) N (\u03c8d | \u00b5, \u03a3) \u221d N (\u03c8d |(cid:101)\u00b5,(cid:101)\u03a3),\n\n(11)\n\n4\n\n\fwhere we have de\ufb01ned\n\n(cid:101)\u00b5 = (cid:101)\u03a3(cid:2)\u03ba(cd) + \u03a3\u22121\u00b5(cid:3) ,\n\n(cid:101)\u03a3 =(cid:2)\u2126d + \u03a3\u22121(cid:3)\u22121\n\n,\n\ncd,t =\n\n(cid:88)\n\nn\n\nI[zn,d = t], \u2126d = diag(\u03c9d),\n\nand so it is resampled as a joint Gaussian. The correlation structure parameters \u00b5 and \u03a3 are sampled\nfrom their conditional NIW distribution. Finally, the auxiliary variables \u03c9 are sampled as P\u00b4olya-\nGamma random variables, with \u03c9d | zd, \u03c8d \u223c PG(N (cd), \u03c8d). A feature of the stick-breaking\nconstruction is that the the auxiliary variable update is embarrassingly parallel.\nWe compare the performance of this Gibbs sampling algorithm for the SB-CTM to the Gibbs sam-\npling algorithm of the LN-CTM [5], which uses a different P\u00b4olya-gamma augmentation, as well as\nthe original variational EM algorithm for the CTM and collapsed Gibbs sampling in standard LDA.\nFigure 2 shows results on both the AP News dataset and the 20 Newsgroups dataset, where models\nwere trained on a random subset of 95% of the complete documents and tested on the remaining 5%\nby estimating held-out likelihoods of half the words given the other half. The collapsed Gibbs sam-\npler for LDA is fast but because it does not model correlations its ability to predict is signi\ufb01cantly\nconstrained. The variational EM algorithm for the CTM is reasonably fast but its point estimate\ndoesn\u2019t quite match the performance from integrating out parameters via MCMC in this setting.\nThe LN-CTM Gibbs sampler continues to improve slowly but is limited by its single-site updates,\nwhile the SB-CTM sampler seems to both mix effectively and execute ef\ufb01ciently due to its block\nGaussian updating.\nThe SB-CTM demonstrates that the stick-breaking construction and corresponding P\u00b4olya-Gamma\naugmentation makes inference in correlated topic models both easy to implement and computa-\ntionally ef\ufb01cient. The block conditional Gaussianity also makes inference algorithms modular and\ncompositional: the construction immediately extends to dynamic topic models (DTMs) [9], in which\nthe latent \u03c8d evolve according to linear Gaussian dynamics, and inference can be implemented sim-\nply by applying off-the-shelf code for Gaussian linear dynamical systems (see Section 5). Finally,\nbecause LDA is so commonly used as a component of other models (e.g. for images [10]), easy,\neffective, modular inference for CTMs and DTMs is a promising general tool.\n\n4 Gaussian processes with multinomial observations\n\nConsider the United States census data, which lists the \ufb01rst names of children born in each state for\nthe years 1910-2013. Suppose we wish to predict the probability of a particular name in New York\nState in the years 2012 and 2013 given observed names in earlier years. We might reasonably expect\nthat name probabilities vary smoothly over time as names rise and fall in popularity, and that name\nprobability would be similar in neighboring states. A Gaussian process naturally captures these\nprior intuitions about spatiotemporal correlations, but the observed name counts are most naturally\nmodeled as multinomial draws from latent probability distributions over names for each combination\nof state and year. We show how ef\ufb01cient inference can be performed in this otherwise dif\ufb01cult model\nby leveraging the P\u00b4olya-gamma augmentation.\nLet Z \u2208 RM\u00d7D denote the matrix of D dimensional inputs and X \u2208 NM\u00d7K denote the observed K\ndimensional count vectors for each input. In our example, each row zm of Z corresponds to the\nyear, latitude, and longitude of an observation, and K is the number of names. Underlying these\nobservations we introduce a set of latent variables, \u03c8m,k such that the probability vector at input zm\nis \u03c0m = \u03c0SB(\u03c8m,:). The auxiliary variables for the k-th name, \u03c8:,k, are linked via a Gaussian\nprocess with covariance matrix, C, whose entry Ci,j is the covariance between input zi and zj\nunder the GP prior, and mean vector \u00b5k. The covariance matrix is shared by all names, and the\nmean is empirically set to match the measured name probability. The full model is then,\n\nxm \u223c Mult(Nm, \u03c0SB(\u03c8m,:)).\n\n\u03c8:,k \u223c GP(\u00b5k, C),\n(cid:18)\n\n(cid:101)\u03a3k =(cid:0)C\u22121 + \u2126k\n\n(cid:1)\u22121\n\nTo perform inference, introduce auxiliary P\u00b4olya-gamma variables, \u03c9m,k for each \u03c8m,k. Conditioned\non these variables, the conditional distribution of \u03c8:,k is,\np(\u03c8:,k | Z, X, \u03c9, \u00b5, C) \u221d N\n\n\u03c8:,k |(cid:101)\u00b5k,(cid:101)\u03a3k\n\n\u2126\u22121\nk \u03ba(X :,k)\n\n(cid:17)\n\n(cid:12)(cid:12)(cid:12)(cid:12) \u03c8:,k, \u2126\u22121\n(cid:101)\u00b5k = (cid:101)\u03a3k\n\nk\n\n(cid:19)\nN (\u03c8:,k | \u00b5k, C) \u221d N(cid:16)\n(cid:0)\u03ba(X :,k) + C\u22121\u00b5k\n(cid:1) ,\n\n5\n\n\f2012\n\n2013\n\nModel\n\nStatic 2011\nRaw GP\nLNM GP\nSBM GP\n\nTop 10\n4.2 (1.3)\n4.9 (1.1)\n6.7 (1.4)\n7.3 (1.0)\n\nBot. 10\n0.7 (1.2)\n0.7 (0.9)\n4.8 (1.7)\n4.0 (1.8)\n\nTop 10\n4.2 (1.4)\n5.0 (1.0)\n6.8 (1.4)\n7.0 (1.0)\n\nBot. 10\n0.8 (1.0)\n0.8 (0.9)\n4.6 (1.7)\n3.9 (1.4)\n\nAverage number of names correctly predicted\n\nFigure 3: A spatiotemporal Gaussian process applied to the names of children born in the United States from\n1960-2013. With a limited dataset of only 50 observations per state/year, the stick breaking and logistic normal\nmultinomial GPs (SBM GP and LNM GP) outperform na\u00a8\u0131ve approaches in predicting the top and bottom 10\nnames (bottom left, parentheses: std. error). Our SBM GP, which leverages the P\u00b4olya-gamma augmentation, is\nconsiderably more ef\ufb01cient than the non-conjugate LNM GP (bottom right).\n\ntion: \u03c9m,k | xm, \u03c8m,k \u223c PG(Nm,k, \u03c8m,k), where Nm,k = Nm \u2212(cid:80)\n\nwhere \u2126k = diag(\u03c9:,k). The auxiliary variables are updated according to their conditional distribu-\n\nj<k xm,j.\n\nFigure 3 illustrates the power of this approach on U.S. census data. The top two plots show the\ninferred probabilities under our stick-breaking multinomial GP model for the full dataset. Interest-\ning spatiotemporal correlations in name probability are uncovered. In this large-count regime, the\nposterior uncertainty is negligible since we observe thousands of names per state and year, and sim-\nply modeling the transformed empirical probabilities with a GP works well. However, in the sparse\ndata regime with only Nm = 50 observations per input, it greatly improves performance to model\nuncertainty in the latent probabilities using a Gaussian process with multinomial observations.\nThe bottom panels compare four methods of predicting future names in the years 2012 and 2013\nfor a down-sampled dataset with Nm = 50: predicting based on the empirical probability measured\nin 2011; a standard GP to the empirical probabilities transformed by \u03c0\u22121\nSB (Raw GP); a GP whose\noutputs are transformed by the logistic normal function, \u03c0LN, to obtain multinomial probabilities\n(LNM GP) \ufb01t using elliptical slice sampling [2]; and our stick-breaking multinomial GP (SBM\nGP). In terms of ability to predict the top and bottom 10 names, the multinomial models are both\ncomparable and vastly superior to the naive approaches.\nThe SBM GP model is considerably faster than the logistic normal version, as shown in the bottom\nright panel. The augmented Gibbs sampler is more ef\ufb01cient than the elliptical slice sampling algo-\nrithm used to handle the non-conjugacy in the LNM GP. Moreover, we are able to make collapsed\npredictions in which we compute the predictive distribution test \u03c8\u2019s given \u03c9, integrating out the\ntraining \u03c8. In contrast, the LNM GP must condition on the training GP values in order to make pre-\ndictions, and effectively integrate over training samples using MCMC. Appendix B goes into greater\ndetail on how marginal predictions are computed and why they are more ef\ufb01cient than predicting\nconditioned on a single value of \u03c8.\n\n6\n\n\fFigure 4: Predictive log likelihood comparison of time series models with multinomial observations.\n\n5 Multinomial linear dynamical systems\n\nWhile discrete-state hidden Markov models (HMMs) are ubiquitous for modeling time series and\nsequence data, it can be preferable to use a continuous state space model.\nIn particular, while\ndiscrete states have no intrinsic geometry, continuous states can correspond to natural Euclidean\nembeddings [11]. These considerations are particularly relevant to text, where word embeddings\n[12] have proven to be a powerful tool.\nGaussian linear dynamical systems (LDS) provide very ef\ufb01cient learning and inference algorithms,\nbut they can typically only be applied when the observations are themselves linear with Gaussian\nnoise. While it is possible to apply a Gaussian LDS to count vectors [11], the resulting model is\nmisspeci\ufb01ed in the sense that, as a continuous density, the model assigns zero probability to training\nand test data. However, Belanger and Kakade [11] show that this model can still be used for several\nmachine learning tasks with compelling performance, and that the ef\ufb01cient algorithms afforded by\nthe misspeci\ufb01ed Gaussian assumptions confer a signi\ufb01cant computational advantage. Indeed, the au-\nthors have observed that such a Gaussian model is \u201cworth exploring, since multinomial models with\nsoftmax link functions prevent closed-form M step updates and require expensive\u201d computations\n[13]; this paper aims to bridge precisely this gap and enable ef\ufb01cient Gaussian LDS computational\nmethods to be applied while maintaining multinomial emissions and an asymptotically unbiased\nrepresentation of the posterior. While there are other approximation schemes that effectively extend\nsome of the bene\ufb01ts of LDSs to nonlinear, non-Gaussian settings, such as the extended Kalman \ufb01lter\n(EKF) and unscented Kalman \ufb01lter (UKF) [14, 15], these methods do not allow for asymptotically\nunbiased Bayesian inference, can have complex behavior, and can make model learning a challenge.\nAlternatively, particle MCMC (pMCMC) [1] is a very powerful algorithm that provides unbiased\nBayesian inference for very general state space models, but it does not enjoy the ef\ufb01cient block\nupdates or conjugacy of LDSs or HMMs.\nThe stick-breaking multinomial linear dynamical system (SBM-LDS) generates states via a linear\nGaussian dynamical system but generates multinomial observations via the stick-breaking map:\nz0|\u00b50, \u03a30 \u223c N (\u00b50, \u03a30),\nzt|zt\u22121, A, B \u223c N (Azt\u22121, B), xt|zt, C \u223c Mult(Nt, \u03c0SB(Czt)),\nwhere zt \u2208 RD is the system state at time t and xt \u2208 NK are the multinomial observations.\nWe suppress notation for conditioning on A, B, C, \u00b50, and \u03a30, which are system parameters of\nappropriate sizes that are given conjugate priors. The logistic normal multinomial LDS (LNM-LDS)\nis de\ufb01ned analogously but uses \u03c0LN in place of \u03c0SB.\nTo produce a Gibbs sampler with fully conjugate updates, we augment the observations with\nP\u00b4olya-gamma random variables \u03c9t,k. As a result, the conditional state sequence z1:T|\u03c91:T , x1:T\nis jointly distributed according to a Gaussian LDS in which the diagonal observation potential at\ntime t is N (\u2126\u22121\nt ). Thus the state sequence can be jointly sampled using off-\n\nt \u03ba(xt)|Czt, \u2126\u22121\n\n7\n\n\fthe-shelf LDS software, and the system parameters can similarly be updated using standard al-\ngorithms. The only remaining update is to the auxiliary variables, which are sampled according to\n\u03c9t|zt, C, x \u223c PG(N (xt), Czt).\nWe compare the SBM-LDS and the Gibbs sampling inference algorithm to three baseline methods:\nan LNM-LDS using pMCMC and ancestor resampling [16] for inference, an HMM using Gibbs\nsampling, and a \u201craw\u201d LDS which treats the multinomial observation vectors as observations in\nRK as in [11]. We examine each method\u2019s performance on each of three experiments: in modeling\na sequence of 682 amino acids from human DNA with 22 dimensional observations, a set of 20\nrandom AP news articles with an average of 77 words per article and a vocabulary size of 200\nwords, and an excerpt of 4000 words from Lewis Carroll\u2019s Alice\u2019s Adventures in Wonderland with\na vocabulary of 1000 words. We reserved the \ufb01nal 10 amino acids, 10 words per news article, and\n100 words from Alice for computing predictive likelihoods. Each linear dynamical model had a 10-\ndimensional state space, while the HMM had 10 discrete states (HMMs with 20, 30, and 40 states\nall performed worse on these tasks).\nFigure 4 (left panels) shows the predictive log likelihood for each method on each experiment, nor-\nmalized by the number of counts in the test dataset and relative to the likelihood under a multinomial\nmodel \ufb01t to the training data mean. For the DNA data, which has the smallest \u201cvocabulary\u201d size, the\nHMM achieves the highest predictive likelihood, but the SBM-LDS edges out the other LDS meth-\nods. On the two text datasets, the SBM-LDS outperforms the other methods, particularly in Alice\nwhere the vocabulary is larger and the document is longer. In terms of run time, the SBM-LDS is\norders of magnitude faster than the LNM-LDS with pMCMC (right panel) because it mixes much\nmore ef\ufb01ciently over the latent trajectories.\n\n6 Related Work\n\nThe stick-breaking transformation used herein was applied to categorical models by Khan et al. [17],\nbut they used local variational bound instead of the P\u00b4olya-gamma augmentation. Their promising\nresults corroborate our \ufb01ndings of improved performance using this transformation. Their general-\nized expectation-maximization algorithm is not fully Bayesian, and does not integrate into existing\nGaussian modeling and inference code as easily as our augmentation.\nConversely, Chen et al. [5] used the P\u00b4olya-gamma augmentation in conjunction with the logistic\nnormal transformation for correlated topic modeling, exploiting the conditional conjugacy of a single\nentry \u03c8k | \u03c9k, \u03c8\u00ack with a Gaussian prior. Unlike our stick-breaking transformation, which admits\nblock Gibbs sampling over the entire vector \u03c8 simultaneously, their approach is limited to single-\nsite Gibbs sampling. As shown in our correlated topic model experiments, this has dramatic effects\non inferential performance. Moreover, it precludes analytical marginalization and integration with\nexisting Gaussian modeling algorithms. For example, it is not immediately applicable to inference\nin linear dynamical systems with multinomial observations.\n\n7 Conclusion\n\nThese case studies demonstrate that the stick-breaking multinomial model construction paired with\nthe P\u00b4olya-gamma augmentation yields a \ufb02exible class of models with easy, ef\ufb01cient, and compo-\nsitional inference. In addition to making these models easy, the methods developed here can also\nenable new models for multinomial and mixed data: the latent continuous structures used here to\nmodel correlations and state-space structure can be leveraged to explore new models for interpretable\nfeature embeddings, interacting time series, and dependence with other covariates.\n\n8 Acknowledgements\n\nS.W.L. is supported by a Siebel Scholarship and the Center for Brains, Minds and Machines\n(CBMM), funded by NSF STC award CCF-1231216. M.J.J. is supported by the Harvard/MIT Joint\nResearch Grants Program. R.P.A. is supported by NSF IIS-1421780 as well as the Alfred P. Sloan\nFoundation.\n\n8\n\n\fReferences\n[1] Christophe Andrieu, Arnaud Doucet, and Roman Holenstein. Particle Markov chain Monte\nCarlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72\n(3):269\u2013342, 2010.\n\n[2] Iain Murray, Ryan P. Adams, and David J.C. MacKay. Elliptical slice sampling. Journal of\nMachine Learning Research: Workshop and Conference Proceedings (AISTATS), 9:541\u2013548,\n05/2010 2010.\n\n[3] Nicholas G Polson, James G Scott, and Jesse Windle. Bayesian inference for logistic models\nusing P\u00b4olya\u2013gamma latent variables. Journal of the American Statistical Association, 108\n(504):1339\u20131349, 2013.\n\n[4] Mingyuan Zhou, Lingbo Li, David Dunson, and Lawrence Carin. Lognormal and gamma\nmixed negative binomial regression. In Proceedings of the International Conference on Ma-\nchine Learning, volume 2012, page 1343, 2012.\n\n[5] Jianfei Chen, Jun Zhu, Zi Wang, Xun Zheng, and Bo Zhang. Scalable inference for logistic-\nnormal topic models. In Advances in Neural Information Processing Systems, pages 2445\u2013\n2453, 2013.\n\n[6] Chris C Holmes, Leonhard Held, et al. Bayesian auxiliary variable models for binary and\n\nmultinomial regression. Bayesian Analysis, 1(1):145\u2013168, 2006.\n\n[7] David Blei and John Lafferty. Correlated topic models. Advances in Neural Information\n\nProcessing Systems, 18:147, 2006.\n\n[8] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent Dirichlet allocation. the Journal\n\nof machine Learning research, 3:993\u20131022, 2003.\n\n[9] David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the International\n\nConference on Machine Learning, pages 113\u2013120. ACM, 2006.\n\n[10] Xiaogang Wang and Eric Grimson. Spatial latent Dirichlet allocation. In Advances in Neural\n\nInformation Processing Systems, pages 1577\u20131584, 2008.\n\n[11] David Belanger and Sham Kakade. A linear dynamical system model for text. In Proceedings\n\nof the International Conference on Machine Learning, 2015.\n\n[12] Ronan Collobert and Jason Weston. A uni\ufb01ed architecture for natural language processing:\nDeep neural networks with multitask learning. In Proceedings of the International Conference\non Machine Learning, pages 160\u2013167. ACM, 2008.\n\n[13] David Belanger and Sham Kakade. Embedding word tokens using a linear dynamical system.\n\nIn NIPS 2014 Modern ML+NLP Workshop, 2014.\n\n[14] Eric A Wan and Rudolph Van Der Merwe. The unscented Kalman \ufb01lter for nonlinear estima-\ntion. In Adaptive Systems for Signal Processing, Communications, and Control Symposium\n2000. AS-SPCC. The IEEE 2000, pages 153\u2013158. IEEE, 2000.\n\n[15] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic robotics. MIT press, 2005.\n\n[16] Fredrik Lindsten, Thomas Sch\u00a8on, and Michael I Jordan. Ancestor sampling for particle Gibbs.\n\nIn Advances in Neural Information Processing Systems, pages 2591\u20132599, 2012.\n\n[17] Mohammad E Khan, Shakir Mohamed, Benjamin M Marlin, and Kevin P Murphy. A stick-\nbreaking likelihood for categorical data analysis with latent Gaussian models. In International\nConference on Arti\ufb01cial Intelligence and Statistics, pages 610\u2013618, 2012.\n\n9\n\n\f", "award": [], "sourceid": 1914, "authors": [{"given_name": "Scott", "family_name": "Linderman", "institution": "Harvard University"}, {"given_name": "Matthew", "family_name": "Johnson", "institution": "MIT"}, {"given_name": "Ryan", "family_name": "Adams", "institution": "Harvard"}]}