{"title": "Generative Image Modeling Using Spatial LSTMs", "book": "Advances in Neural Information Processing Systems", "page_first": 1927, "page_last": 1935, "abstract": "Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multi-dimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.", "full_text": "Generative Image Modeling Using Spatial LSTMs\n\nLucas Theis\n\nUniversity of T\u00a8ubingen\n\n72076 T\u00a8ubingen, Germany\nlucas@bethgelab.org\n\nMatthias Bethge\n\nUniversity of T\u00a8ubingen\n\n72076 T\u00a8ubingen, Germany\n\nmatthias@bethgelab.org\n\nAbstract\n\nModeling the distribution of natural images is challenging, partly because of\nstrong statistical dependencies which can extend over hundreds of pixels. Re-\ncurrent neural networks have been successful in capturing long-range dependen-\ncies in a number of problems but only recently have found their way into gener-\native image models. We here introduce a recurrent image model based on multi-\ndimensional long short-term memory units which are particularly suited for image\nmodeling due to their spatial structure. Our model scales to images of arbitrary\nsize and its likelihood is computationally tractable. We \ufb01nd that it outperforms the\nstate of the art in quantitative comparisons on several image datasets and produces\npromising results when used for texture synthesis and inpainting.\n\n1\n\nIntroduction\n\nThe last few years have seen tremendous progress in learning useful image representations [6].\nWhile early successes were often achieved through the use of generative models [e.g., 13, 23, 30],\nrecent breakthroughs were mainly driven by improvements in supervised techniques [e.g., 20, 34].\nYet unsupervised learning has the potential to tap into the much larger source of unlabeled data,\nwhich may be important for training bigger systems capable of a more general scene understand-\ning. For example, multimodal data is abundant but often unlabeled, yet can still greatly bene\ufb01t\nunsupervised approaches [36].\nGenerative models provide a principled approach to unsupervised learning. A perfect model of\nnatural images would be able to optimally predict parts of an image given other parts of an image and\nthereby clearly demonstrate a form of scene understanding. When extended by labels, the Bayesian\nframework can be used to perform semi-supervised learning in the generative model [19, 28] while it\nis less clear how to combine other unsupervised approaches with discriminative learning. Generative\nimage models are also useful in more traditional applications such as image reconstruction [33, 35,\n49] or compression [47].\nRecently there has been a renewed strong interest in the development of generative image models\n[e.g., 4, 8, 10, 11, 18, 24, 31, 35, 45, 47]. Most of this work has tried to bring to bear the \ufb02exibility of\ndeep neural networks on the problem of modeling the distribution of natural images. One challenge\nin this endeavor is to \ufb01nd the right balance between tractability and \ufb02exibility. The present article\ncontributes to this line of research by introducing a fully tractable yet highly \ufb02exible image model.\nOur model combines multi-dimensional recurrent neural networks [9] with mixtures of experts.\nMore speci\ufb01cally, the backbone of our model is formed by a spatial variant of long short-term\nmemory (LSTM) [14]. One-dimensional LSTMs have been particularly successful in modeling text\nand speech [e.g., 38, 39], but have also been used to model the progression of frames in video [36]\nand very recently to model single images [11]. In contrast to earlier work on modeling images,\nhere we use multi-dimensional LSTMs [9] which naturally lend themselves to the task of generative\nimage modeling due to their spatial structure and ability to capture long-range correlations.\n\n1\n\n\fA\n\nB\n\nMCGSM\n\nxij\nx<ij\n\nxij\nx<ij\n\nRIDE\n\nC\n\nPixels\n\nSLSTM units\n\nSLSTM units\n\nPixels\n\nFigure 1: (A) We factorize the distribution of images such that the prediction of a pixel (black)\nmay depend on any pixel in the upper-left green region. (B) A graphical model representation of an\nMCGSM with a causal neighborhood limited to a small region. (C) A visualization of our recurrent\nimage model with two layers of spatial LSTMs. The pixels of the image are represented twice and\nsome arrows are omitted for clarity. Through feedforward connections, the prediction of a pixel\ndepends directly on its neighborhood (green), but through recurrent connections it has access to the\ninformation in a much larger region (red).\n\nTo model the distribution of pixels conditioned on the hidden states of the neural network, we use\nmixtures of conditional Gaussian scale mixtures (MCGSMs) [41]. This class of models can be\nviewed as a generalization of Gaussian mixture models, but their parametrization makes them much\nmore suitable for natural images. By treating images as instances of a stationary stochastic process,\nthis model allows us to sample and capture the correlations of arbitrarily large images.\n\n2 A recurrent model of natural images\n\nIn the following, we \ufb01rst review and extend the MCGSM [41] and multi-dimensional LSTMs [9]\nbefore explaining how to combine them into a recurrent image model. Section 3 will demonstrate\nthe validity of our approach by evaluating and comparing the model on a number of image datasets.\n\n2.1 Factorized mixtures of conditional Gaussian scale mixtures\n\nOne successful approach to building \ufb02exible yet tractable generative models has been to use fully-\nvisible belief networks [21, 27]. To apply such a model to images, we have to give the pixels\nan ordering and specify the distribution of each pixel conditioned on its parent pixels. Several\nparametrizations have been suggested for the conditional distributions in the context of natural im-\nages [5, 15, 41, 44, 45]. We here review and extend the work of Theis et al. [41] who proposed to\nuse mixtures of conditional Gaussian scale mixtures (MCGSMs).\nLet x be a grayscale image patch and xij be the intensity of the pixel at location ij. Further, let x<ij\ndesignate the set of pixels xmn such that m < i or m = i and n < j (Figure 1A). Then\n\np(x; \u03b8) =(cid:81)\n\ni,j p(xij | x<ij; \u03b8)\n\n(1)\n\n(2)\n\nfor the distribution of any parametric model with parameters \u03b8. Note that this factorization does\nnot make any independence assumptions but is simply an application of the probability chain rule.\nFurther note that the conditional distributions all share the same set of parameters. One way to\nimprove the representational power of a model is thus to endow each conditional distribution with\nits own set of parameters,\n\np(x;{\u03b8ij}) =(cid:81)\n\ni,j p(xij | x<ij; \u03b8ij).\n\nApplying this trick to mixtures of Gaussian scale mixtures (MoGSMs) yields the MCGSM [40].\nUntying shared parameters can drastically increase the number of parameters. For images, it can\neasily be reduced again by adding assumptions. For example, we can limit x<ij to a smaller neigh-\nborhood surrounding the pixel by making a Markov assumption. We will refer to the resulting set of\nparents as the pixel\u2019s causal neighborhood (Figure 1B). Another reasonable assumption is stationar-\nity or shift invariance, in which case we only have to learn one set of parameters \u03b8ij which can then\n\n2\n\n\fbe used at every pixel location. Similar to convolutions in neural networks, this allows the model\nto easily scale to images of arbitrary size. While this assumption reintroduces parameter sharing\nconstraints into the model, the constraints are different from the ones induced by the joint mixture\nmodel.\nThe conditional distribution in an MCGSM takes the form of a mixture of experts,\np(xij | x<ij, c, s, \u03b8ij)\n,\n\np(xij | x<ij, \u03b8ij) =\n\np(c, s | x<ij, \u03b8ij)\n\n(cid:88)\n\n(3)\n\n(cid:125)\n\n(cid:124)\n\nc,s\n\n(cid:124)\n\n(cid:123)(cid:122)\np(c, s | x<ij) \u221d exp(cid:0)\u03b7cs \u2212 1\n\ngate\n\n2 e\u03b1csx(cid:62)\n\n(cid:123)(cid:122)\n\nexpert\n\n(cid:125)\n\n(cid:1) ,\n\nwhere the sum is over mixture component indices c corresponding to different covariances and scales\ns corresponding to different variances. The gates and experts in an MCGSM are given by\n\n<ijKcx<ij\n\np(xij | x<ij, c, s) = N (xij; a(cid:62)\n\n(4)\n(5)\nwhere Kc is positive de\ufb01nite. The number of parameters of an MCGSM still grows quadratically\nwith the dimensionality of the causal neighborhood. To further reduce the number of parameters, we\nintroduce a factorized form of the MCGSM with additional parameter sharing by replacing Kc with\nn . This factorized MCGSM allows us to use larger neighborhoods and more mixture\ncomponents. A detailed derivation of a more general version which also allows for multivariate\npixels is given in Supplementary Section 1.\n\nc x<ij, e\u2212\u03b1cs),\n\ncnbnb(cid:62)\n\n(cid:80)\n\nn \u03b22\n\n2.2 Spatial long short-term memory\n\nIn the following we brie\ufb02y describe the spatial LSTM (SLSTM), a special case of the multi-\ndimensional LSTM \ufb01rst described by Graves & Schmidhuber [9]. At the core of the model are\nmemory units cij and hidden units hij. For each location ij on a two-dimensional grid, the\noperations performed by the spatial LSTM are given by\n\n\uf8f6\uf8f7\uf8f7\uf8f7\uf8f8 =\n\n\uf8eb\uf8ec\uf8ec\uf8ec\uf8ed\n\n\uf8eb\uf8ec\uf8ec\uf8ec\uf8ed\n\ngij\noij\niij\nf r\nij\nf c\nij\n\ntanh\n\n\u03c3\n\u03c3\n\u03c3\n\u03c3\n\n\uf8f6\uf8f7\uf8f7\uf8f7\uf8f8 TA,b\n\n(cid:32) x<ij\n\n(cid:33)\n\nhi,j\u22121\nhi\u22121,j\n\n,\n\n(6)\n\ncij = gij (cid:12) iij + ci,j\u22121 (cid:12) f c\nhij = tanh (cij (cid:12) oij) ,\n\nij + ci\u22121,j (cid:12) f r\nij,\n\nwhere \u03c3 is the logistic sigmoid function, (cid:12) indicates a pointwise product, and TA,b is an af\ufb01ne\ntransformation which depends on the only parameters of the network A and b. The gating units iij\nand oij determine which memory units are affected by the inputs through gij, and which memory\nstates are written to the hidden units hij. In contrast to a regular LSTM de\ufb01ned over time, each\nmemory unit of a spatial LSTM has two preceding states ci,j\u22121 and ci\u22121,j and two corresponding\nforget gates f c\n\nij and f r\nij.\n\n2.3 Recurrent image density estimator\n\nWe use a grid of SLSTM units to sequentially read relatively small neighborhoods of pixels from\nthe image, producing a hidden vector at every pixel. The hidden states are then fed into a factorized\nMCGSM to predict the state of the corresponding pixel, that is, p(xij | x<ij) = p(xij | hij). Impor-\ntantly, the state of the hidden vector only depends on pixels in x<ij and does not violate the factor-\nization given in Equation 1. Nevertheless, the recurrent network allows this recurrent image density\nestimator (RIDE) to use pixels of a much larger region for prediction, and to nonlinearly transform\nthe pixels before applying the MCGSM. We can further increase the representational power of the\nmodel by stacking spatial LSTMs to obtain a deep yet still completely tractable recurrent image\nmodel (Figure 1C).\n\n2.4 Related work\n\nLarochelle & Murray [21] derived a tractable density estimator (NADE) in a manner similar to how\nthe MCGSM was derived [41], but using restricted Boltzmann machines (RBM) instead of mixture\nmodels as a starting point.\nIn contrast to the MCGSM, NADE tries to keep the weight sharing\n\n3\n\n\fconstraints induced by the RBM (Equation 1). Uria et al. extended NADE to real values [44] and\nintroduced hidden layers to the model [45]. Gregor et al. [10] describe a related autoregressive\nnetwork for binary data which additionally allows for stochastic hidden units.\nGregor et al. [11] used one-dimensional LSTMs to generate images in a sequential manner (DRAW).\nBecause the model was de\ufb01ned over Bernoulli variables, normalized RGB values had to be treated\nas probabilities, making a direct comparison with other image models dif\ufb01cult. In contrast to our\nmodel, the presence of stochastic latent variables in DRAW means that its likelihood cannot be\nevaluated but has to be approximated.\nRanzato et al. [31] and Srivastava et al. [37] use one-dimensional recurrent neural networks to model\nvideos, but recurrency is not used to describe the distribution over individual frames. Srivastava et\nal. [37] optimize a squared error corresponding to a Gaussian assumption, while Ranzato et al. [31]\ntry to side-step having to model pixel intensities by quantizing image patches. In contrast, here we\nalso try to solve the problem of modeling pixel intensities by using an MCGSM, which is equipped\nto model heavy-tailed as well as multi-modal distributions.\n\n3 Experiments\n\nRIDE was trained using stochastic gradient descent with a batch size of 50, momentum of 0.9, and\na decreasing learning rate varying between 1 and 10\u22124. After each pass through the training set,\nthe MCGSM of RIDE was \ufb01netuned using L-BFGS for up to 500 iterations before decreasing the\nlearning rate. No regularization was used except for early stopping based on a validation set. Except\nwhere indicated otherwise, the recurrent model used a 5 pixel wide neighborhood and an MCGSM\nwith 32 components and 32 quadratic features (bn in Section 2.1). Spatial LSTMs were imple-\nmented using the Caffe framework [17]. Where appropriate, we augmented the data by horizontal\nor vertical \ufb02ipping of images.\nWe found that conditionally whitening the data greatly sped up the training process of both models.\nLetting y represent a pixel and x its causal neighborhood, conditional whitening replaces these with\n\n2\n\n2\n\n\u2212 1\nxx (x \u2212 mx) , \u02c6y = W(y \u2212 CyxC\n\n\u2212 1\nxx \u02c6x \u2212 my), W = (Cyy \u2212 CyxC\u22121\n\n2 , (7)\n\u02c6x = C\nwhere Cyx is the covariance of y and x, and mx is the mean of x. In addition to speeding up train-\ning, this variance normalization step helps to make the learning rates less dependent on the training\ndata. When evaluating the conditional log-likelihood, we compensate for the change in variance by\nadding the log-Jacobian log | det W|. Note that this preconditioning introduces a shortcut connec-\ntion from the pixel neighborhood to the predicted pixel which is not shown in Figure 1C.\n\nxx C(cid:62)\n\nyx)\u2212 1\n\n3.1 Ensembles\n\nUria et al. [45] found that forming ensembles of their autoregressive model over different pixel\norderings signi\ufb01cantly improved performance. We here consider a simple trick to produce an en-\n(cid:80)\nsemble without the need for training different models or to change training procedures. If Tk are\nlinear transformations leaving the targeted image distribution invariant (or approximately invariant)\nk p(Tkx)| det Tk|.\nand if p is the distribution of a pretrained model, then we form the ensemble 1\nK\nNote that this is simply a mixture model over images x. We considered rotating as well as \ufb02ipping\nimages along the horizontal and vertical axes (yielding an ensemble over 8 transformations). While\nit could be argued that most of these transformations do not leave the distribution over natural images\ninvariant, we nevertheless observed a noticeable boost in performance.\n\n3.2 Natural images\n\nSeveral recent image models have been evaluated on small image patches sampled from the Berkeley\nsegmentation dataset (BSDS300) [25]. Although our model\u2019s strength lies in its ability to scale to\nlarge images and to capture long-range correlations, we include results on BSDS300 to make a\nconnection to this part of the literature. We followed the protocol of Uria et al. [44]. The RGB\nimages were turned to grayscale, uniform noise was added to account for the integer discretization,\nand the resulting values were divided by 256. The training set of 200 images was split into 180\nimages for training and 20 images for validation, while the test set contained 100 images. We\n\n4\n\n\f63 dim.\nModel\n[nat]\nRNADE [44]\n152.1\nRNADE, 1 hl [45]\n143.2\nRNADE, 6 hl [45]\n155.2\nEoRNADE, 6 layers [45]\n157.0\nGMM, 200 comp. [47, 50] 153.7\n155.3\nSTM, 200 comp. [46]\n156.2\nDeep GMM, 3 layers [47]\n155.1\nMCGSM, 16 comp.\nMCGSM, 32 comp.\n155.8\n156.2\nMCGSM, 64 comp.\n156.4\nMCGSM, 128 comp.\n158.1\nEoMCGSM, 128 comp.\n150.7\nRIDE, 1 layer\nRIDE, 2 layers\n152.1\n154.5\nEoRIDE, 2 layers\n\n64 dim.\n[bit/px]\n3.346\n3.146\n3.416\n3.457\n3.360\n3.418\n3.439\n3.413\n3.430\n3.439\n3.443\n3.481\n3.293\n3.346\n3.400\n\n\u221e dim.\n[bit/px]\n\n-\n-\n-\n-\n-\n-\n-\n\n3.688\n3.706\n3.716\n3.717\n3.748\n3.802\n3.869\n3.899\n\nModel\nGRBM [13]\nICA [1, 48]\nGSM\nISA [7, 16]\nMoGSM, 32 comp. [40]\nMCGSM, 32 comp.\nRIDE, 1 layer, 64 hid.\nRIDE, 1 layer, 128 hid.\nRIDE, 2 layers, 64 hid.\nRIDE, 2 layers, 128 hid.\nEoRIDE, 2 layers, 128 hid.\n\n256 dim.\n[bit/px]\n0.992\n1.072\n1.349\n1.441\n1.526\n1.615\n1.650\n\n-\n-\n-\n-\n\n\u221e dim.\n[bit/px]\n\n-\n-\n-\n-\n-\n\n1.759\n1.816\n1.830\n1.829\n1.839\n1.859\n\nTable 1: Average log-likelihoods and log-likelihood\nrates for image patches (without/with DC comp.) and\nlarge images extracted from BSDS300 [25].\n\nTable 2: Average log-likelihood rates for im-\nage patches and large images extracted from\nvan Hateren\u2019s dataset [48].\n\nextracted 8 by 8 image patches from each set and subtracted the average pixel intensity such that\neach patch\u2019s DC component was zero. Because the resulting image patches live on a 63 dimensional\nsubspace, the bottom-right pixel was discarded. We used 1.6 \u00b7 106 patches for training, 1.8 \u00b7 105\npatches for validation, and 106 test patches for evaluation.\nMCGSMs have not been evaluated on this dataset and so we \ufb01rst tested MCGSMs by training a\nsingle factorized MCGSM for each pixel conditioned on all previous pixels in a \ufb01xed ordering.\nWe \ufb01nd that already an MCGSM (with 128 components and 48 quadratic features) outperforms all\nsingle models including a deep Gaussian mixture model [46] (Table 1). Our ensemble of MCGSMs1\noutperforms an ensemble of RNADEs with 6 hidden layers, which to our knowledge is currently the\nbest result reported on this dataset.\nTraining the recurrent image density estimator (RIDE) on the 63 dimensional dataset is more cum-\nbersome. We tried padding image patches with zeros, which was necessary to be able to compute a\nhidden state at every pixel. The bottom-right pixel was ignored during training and evaluation. This\nsimple approach led to a reduction in performance relative to the MCGSM (Table 1). A possible\nexplanation is that the model cannot distinguish between pixel intensities which are zero and zeros\nin the padded region. Supplying the model with additional binary indicators as inputs (one for each\nneighborhood pixel) did not solve the problem.\nHowever, we found that RIDE outperforms the MCGSM by a large margin when images were\ntreated as instances of a stochastic process (that is, using in\ufb01nitely large images). MCGSMs were\ntrained for up to 3000 iterations of L-BFGS on 106 pixels and corresponding causal neighborhoods\nextracted from the training images. Causal neighborhoods were 9 pixels wide and 5 pixels high.\nRIDE was trained for 8 epochs on image patches of increasing size ranging from 8 by 8 to 22 by\n22 pixels (that is, gradients were approximated as in backpropagation through time [32]). The right\ncolumn in Table 1 shows average log-likelihood rates for both models. Analogously to the entropy\nrate [3], we have for the expected log-likelihood rate:\n\nE(cid:2)log p(x)/N 2(cid:3) = E[log p(xij | x<ij)],\n\n(8)\n\nlim\nN\u2192\u221e\n\nwhere x is an N by N image patch. An average log-likelihood rate can be directly computed for the\nMCGSM, while for RIDE and ensembles we approximated it by splitting the test images into 64 by\n64 patches and evaluating on those.\nTo make the two sets of numbers more comparable, we transformed nats as commonly reported on\nthe 63 dimensional data, (cid:96)1:63, into a bit per pixel log-likelihood rate using the formula ((cid:96)1:63+(cid:96)DC +\nln| det A|)/64/ ln(2). This takes into account a log-likelihood for the missing DC component,\n1Details on how the ensemble of transformations can be applied despite the missing bottom-right pixel are\n\ngiven in Supplementary Section 2.1.\n\n5\n\n\f[bit/px]\nModel\n1.244\nMCGSM, 12 comp. [41]\n1.294\nMCGSM, 32 comp.\n1.489\nDiffusion [35]\n1.402\nRIDE, 64 hid., 1 layer\nRIDE, 64 hid., 1 layer, ext. 1.416\n1.438\nRIDE, 64 hid., 2 layers\n1.454\nRIDE, 64 hid., 3 layers\nRIDE, 128 hid., 3 layers\n1.489\nEoRIDE, 128 hid., 3 layers 1.501\n\n]\nx\np\n/\nt\ni\nb\n[\n\nd\no\no\nh\n\ni\nl\n\ne\nk\n\ni\nl\n-\ng\no\nL\n\n1.5\n\n1.4\n\n1.3\n\n1.2\n\n1.1\n\n1\n\nMCGSM\nRIDE\n\n3\n\n5\n\n7\n\n9 11 13\n\nNeighborhood size\n\nFigure 2: Model performance on dead leaves\nas a function of the causal neighborhood width.\nSimply increasing the neighborhood size of the\nMCGSM is not suf\ufb01cient to improve perfor-\nmance.\n\nTable 3: Average log-likelihood rates on dead\nleaf images. A deep recurrent image model is\non a par with a deep diffusion model [35]. Us-\ning ensembles we are able to further improve the\nlikelihood.\n(cid:96)DC = 0.5020, and the Jacobian of the transformations applied during preprocessing, ln| det A| =\n\u22124.1589 (see Supplementary Section 2.2 for details). The two rates in Table 1 are comparable in the\nsense that their differences express how much better one model would be at losslessly compressing\nBSDS300 test images than another, where patch-based models would compress patches of an image\nindependently. We highlighted the best result achieved with each model in gray. Note that most\nmodels in this list do not scale as well to large images as the MCGSM or RIDE (GMMs in particular)\nand are therefore unlikely to bene\ufb01t as much from increasing the patch size.\nA comparison of the log-likelihood rates reveals that an MCGSM with 16 components applied to\nlarge images already captures more correlations than any model applied to small image patches.\nThe difference is particularly striking given that the factorized MCGSM has approximately 3,000\nparameters while a GMM with 200 components has approximately 400,000 parameters. Using an\nensemble of RIDEs, we are able to further improve this number signi\ufb01cantly (Table 1).\nAnother dataset frequently used to test generative image models is the dataset published by van\nHateren and van der Schaaf [48]. Details of the preprocessing used in this paper are given in Sup-\nplementary Section 3. We reevaluated several models for which the likelihood has been reported\non this dataset [7, 40, 41, 42]. Likelihood rates as well as results on 16 by 16 patches are given in\nTable 2. Because of the larger patch size, RIDE here already outperforms the MCGSM on patches.\n\n3.3 Dead leaves\n\nDead leaf images are generated by superimposing disks of random intensity and size on top of each\nother [22, 26]. This simple procedure leads to images which already share many of the statistical\nproperties and challenges of natural images, such as occlusions and long-range correlations, while\nleaving out others such as non-stationary statistics. They therefore provide an interesting test case\nfor natural image models.\nWe used a set of 1,000 images, where each image is 256 by 256 pixels in size. We compare the\nperformance of RIDE to the MCGSM and a very recently introduced deep multiscale model based\non a diffusion process [35]. The same 100 images as in previous literature [35, 41] were used\nfor evaluation and we used the remaining images for training. We \ufb01nd that the introduction of an\nSLSTM with 64 hidden units greatly improves the performance of the MCGSM. We also tried an\nextended version of the SLSTM which included memory units as additional inputs (right-hand side\nof Equation 6). This yielded a small improvement in performance (5th row in Table 3) while adding\nlayers or using more hidden units led to more drastic improvements. Using 3 layers with 128 hidden\nunits in each layer, we \ufb01nd that our recurrent image model is on a par with the deep diffusion model.\nBy using ensembles, we are able to beat all previously published results for this dataset (Table 3).\nFigure 2 shows that the improved performance of RIDE is not simply due to an effectively larger\ncausal neighborhood but that the nonlinear transformations performed by the SLSTM units matter.\nSimply increasing the neighborhood size of an MCGSM does not yield the same improvement.\nInstead, the performance quickly saturates. We also \ufb01nd that the performance of RIDE slightly\ndeteriorates with larger neighborhoods, which is likely caused by optimization dif\ufb01culties.\n\n6\n\n\fD106\n\nD93\n\nD12\n\nD104\n\nD34\n\nD110\n\nFigure 3: From top to bottom: A 256 by 256 pixel crop of a texture [2], a sample generated by\nan MCGSM trained on the full texture [7], and a sample generated by RIDE. This illustrates that\nour model can capture a variety of different statistical patterns. The addition of the recurrent neural\nnetwork seems particularly helpful where there are strong long-range correlations (D104, D34).\n\n3.4 Texture synthesis and inpainting\n\nTo get an intuition for the kinds of correlations which RIDE can capture or fails to capture, we tried\nto use it to synthesize textures. We used several 640 by 640 pixel textures published by Brodatz [2].\nThe textures were split into sixteen 160 by 160 pixel regions of which 15 were used for training and\none randomly selected region was kept for testing purposes. RIDE was trained for up to 6 epochs\non patches of increasing size ranging from 20 by 20 to 40 by 40 pixels.\nSamples generated by an MCGSM and RIDE are shown in Figure 3. Both models are able to\ncapture a wide range of correlation structures. However, the MCGSM seems to struggle with tex-\ntures having bimodal marginal distributions and periodic patterns (D104, D34, and D110). RIDE\nclearly improves on these textures, although it also struggles to faithfully reproduce periodic struc-\nture. Possible explanations include that LSTMs are not well suited to capture periodicities, or that\nthese failures are not penalized strong enough by the likelihood. For some textures, RIDE produces\nsamples which are nearly indistinguishable from the real textures (D106 and D110).\nOne application of generative image models is inpainting [e.g., 12, 33, 35]. As a proof of concept,\nwe used our model to inpaint a large (here, 71 by 71 pixels) region in textures (Figure 4). Missing\npixels were replaced by sampling from the posterior of RIDE. Unlike the joint distribution, the\nposterior distribution cannot be sampled directly and we had to resort to Markov chain Monte Carlo\nmethods. We found the following Metropolis within Gibbs [43] procedure to be ef\ufb01cient enough.\nThe missing pixels were initialized via ancestral sampling. Since ancestral sampling is cheap, we\ngenerated 5 candidates and used the one with the largest posterior density. Following initialization,\nwe sequentially updated overlapping 5 by 5 pixel regions via Metropolis sampling. Proposals were\ngenerated via ancestral sampling and accepted using the acceptance probability\n\np(xij|x<ij )\n1, p(x(cid:48))\n(9)\nij|x<ij )\np(x(cid:48)\nwhere here xij represents a 5 by 5 pixel patch and x(cid:48)\nij its proposed replacement. Since evaluating the\njoint and conditional densities on the entire image is costly, we approximated p using RIDE applied\nto a 19 by 19 pixel patch surrounding ij. Randomly \ufb02ipping images vertically or horizontally in\nbetween the sampling further helped. Figure 4 shows results after 100 Gibbs sampling sweeps.\n\n\u03b1 = min\n\n(cid:111)\n\n,\n\n(cid:110)\n\np(x)\n\n4 Conclusion\n\nWe have introduced RIDE, a deep but tractable recurrent image model based on spatial LSTMs.\nThe model exempli\ufb01es how recent insights in deep learning can be exploited for generative image\n\n7\n\n\fFigure 4: The center portion of a texture (left and center) was reconstructed by sampling from the\nposterior distribution of RIDE (right).\n\nmodeling and shows superior performance in quantitative comparisons. RIDE is able to capture\nmany different statistical patterns, as demonstrated through its application to textures. This is an\nimportant property considering that on an intermediate level of abstraction natural images can be\nviewed as collections of textures.\nWe have furthermore introduced a factorized version of the MCGSM which allowed us to use more\nexperts and larger causal neighborhoods. This model has few parameters, is easy to train and already\non its own performs very well as an image model. It is therefore an ideal building block and may be\nused to extend other models such as DRAW [11] or video models [31, 37].\nDeep generative image models have come a long way since deep belief networks have \ufb01rst been\napplied to natural images [29]. Unlike convolutional neural networks in object recognition, however,\nno approach has as of yet proven to be a likely solution to the problem of generative image modeling.\nFurther conceptual work will be necessary to come up with a model which can handle both the more\nabstract high-level as well as the low-level statistics of natural images.\n\nAcknowledgments\n\nThe authors would like to thank A\u00a8aron van den Oord for insightful discussions and Wieland Brendel,\nChristian Behrens, and Matthias K\u00a8ummerer for helpful input on this paper. This study was \ufb01nan-\ncially supported by the German Research Foundation (DFG; priority program 1527, BE 3848/2-1).\n\nReferences\n[1] A. J. Bell and T. J. Sejnowski. The \u201cindependent components\u201d of natural scenes are edge \ufb01lters. Vision\n\nResearch, 37(23):3327\u20133338, 1997.\n\n[2] P. Brodatz. Textures: A Photographic Album for Artists and Designers. Dover, New York, 1966. URL\n\nhttp://www.ux.uis.no/\u02dctranden/brodatz.html.\n\n[3] T. Cover and J. Thomas. Elements of Information Theory. Wiley, 2nd edition, 2006.\n[4] E. Denton, S. Chintala, A. Szlam, and R. Fergus. Deep Generative Image Models using a Laplacian\n\nPyramid of Adversarial Networks. In Advances in Neural Information Processing Systems 28, 2015.\n\n[5] J. Domke, A. Karapurkar, and Y. Aloimonos. Who killed the directed model? In CVPR, 2008.\n[6] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. DeCAF: A deep convo-\n\nlutional activation feature for generic visual recognition. In ICML 31, 2014.\n\n[7] H. E. Gerhard, L. Theis, and M. Bethge. Modeling natural image statistics.\n\nComputer Vision\u2014Fundamentals and Applications. Wiley VCH, 2015.\n\nIn Biologically-inspired\n\n[8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Ben-\n\ngio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, 2014.\n\n[9] A. Graves and J. Schmidhuber. Of\ufb02ine handwriting recognition with multidimensional recurrent neural\n\nnetworks. In Advances in Neural Information Processing Systems 22, 2009.\n\n[10] K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra. Deep AutoRegressive Networks. In\n\nProceedings of the 31st International Conference on Machine Learning, 2014.\n\n[11] K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: A recurrent neural network for image\n\ngeneration. In Proceedings of the 32nd International Conference on Machine Learning, 2015.\n\n[12] N. Heess, C. Williams, and G. E. Hinton. Learning generative texture models with extended \ufb01elds-of-\n\nexperts. In BMCV, 2009.\n\n[13] G. Hinton, S. Osindero, and Y. Teh. A fast learning algorithm for deep belief nets. Neural Comp., 2006.\n[14] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8), 1997.\n[15] R. Hosseini, F. Sinz, and M. Bethge. Lower bounds on the redundancy of natural images. Vis. Res., 2010.\n[16] A. Hyv\u00a8arinen and P. O. Hoyer. Emergence of phase and shift invariant features by decomposition of\n\nnatural images into independent feature subspaces. Neural Computation, 12(7):1705\u2014-1720, 2000.\n\n8\n\n\f[17] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe:\n\nConvolutional architecture for fast feature embedding, 2014. arXiv:1408.5093.\n\n[18] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In ICLR, 2014.\n[19] D. P. Kingma, D. J. Rezende, S. Mohamed, and M. Welling. Semi-supervised learning with deep gener-\n\native models. In Advances in Neural Information Processing Systems 27, 2014.\n\n[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classi\ufb01cation with deep convolutional neural\n\nnetworks. In Advances in Neural Information Processing Systems 25, 2012.\n\n[21] H. Larochelle and I. Murray. The neural autoregressive distribution estimator. In Proceedings of the 14th\n\nInternational Conference on Arti\ufb01cial Intelligence and Statistics, 2011.\n\n[22] A. B. Lee, D. Mumford, and J. Huang. Occlusion models for natural images: A statistical study of a\n\nscale-invariant dead leaves model. International Journal of Computer Vision, 2001.\n\n[23] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for scalable unsu-\n\npervised learning of hierarchical representations. In ICML 26, 2009.\n\n[24] Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In ICML 32, 2015.\n[25] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its\n\napplication to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.\n\n[26] G. Matheron. Modele s\u00b4equential de partition al\u00b4eatoire. Technical report, CMM, 1968.\n[27] R. M. Neal. Connectionist learning of belief networks. Arti\ufb01cial Intelligence, 56:71\u2013113, 1992.\n[28] J. Ngiam, Z. Chen, P. W. Koh, and A. Y. Ng. Learning deep energy models. In ICML 28, 2011.\n[29] S. Osindero and G. E. Hinton. Modelling image patches with a directed hierarchy of markov random\n\n\ufb01elds. In Advances In Neural Information Processing Systems 20, 2008.\n\n[30] M. A. Ranzato, J. Susskind, V. Mnih, and G. E. Hinton. On deep generative models with applications to\n\nrecognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.\n\n[31] M. A. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling:\n\na baseline for generative models of natural videos, 2015. arXiv:1412.6604v2.\n\n[32] A. J. Robinson and F. Fallside. The utility driven dynamic error propagation network. Technical report,\n\nCambridge University, 1987.\n\n[33] S. Roth and M. J. Black. Fields of experts. International Journal of Computer Vision, 82(2), 2009.\n[34] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In\n\nInternational Conference on Learning Represenations, 2015.\n\n[35] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using\n\nnonequilibrium thermodynamics. In ICML 32, 2015.\n\n[36] N. Srivastava and R. Salakhutdinov. Multimodal learning with deep Boltzmann machines. JMLR, 2014.\n[37] N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using\n\nLSTMs. In Proceedings of the 32nd International Conference on Machine Learning, 2015.\n\n[38] M. Sundermeyer, R. Schluter, and H. Ney. LSTM neural networks for language modeling. In INTER-\n\nSPEECH, 2010.\n\n[39] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances\n\nin Neural Information Processing Systems 27, 2014.\n\n[40] L. Theis, S. Gerwinn, F. Sinz, and M. Bethge. In all likelihood, deep belief is not enough. JMLR, 2011.\n[41] L. Theis, R. Hosseini, and M. Bethge. Mixtures of conditional Gaussian scale mixtures applied to multi-\n\nscale image representations. PLoS ONE, 7(7), 2012.\n\n[42] L. Theis, J. Sohl-Dickstein, and M. Bethge. Training sparse natural image models with a fast Gibbs\n\nsampler of an extended state space. In Advances in Neural Information Processing Systems 25, 2012.\n\n[43] L. Tierney. Markov chains for exploring posterior distributions. The Annals of Statistics, 1994.\n[44] B. Uria, I. Murray, and H. Larochelle. RNADE: the real-valued neural autoregressive density-estimator.\n\nIn Advances in Neural Information Processing Systems 26, 2013.\n\n[45] B. Uria, I. Murray, and H. Larochelle. A deep and tractable density estimator. In ICML 31, 2014.\n[46] A. van den Oord and B. Schrauwen. The student-t mixture as a natural image patch prior with application\n\nto image compression. Journal of Machine Learning Research, 15(1):2061\u20132086, 2014.\n\n[47] A. van den Oord and B. Schrauwen. Factoring variations in natural images with deep Gaussian mixture\n\nmodels. In Advances in Neural Information Processing Systems 27, 2014.\n\n[48] J. H. van Hateren and A. van der Schaaf. Independent component \ufb01lters of natural images compared with\nsimple cells in primary visual cortex. Proc. of the Royal Society B: Biological Sciences, 265(1394), 1998.\n[49] D. Zoran and Y. Weiss. From learning models of natural image patches to whole image restoration. In\n\nIEEE International Conference on Computer Vision, 2011.\n\n[50] D. Zoran and Y. Weiss. Natural images, Gaussian mixtures and dead leaves. In NIPS 25, 2012.\n\n9\n\n\f", "award": [], "sourceid": 1186, "authors": [{"given_name": "Lucas", "family_name": "Theis", "institution": "U.Tuebingen"}, {"given_name": "Matthias", "family_name": "Bethge", "institution": "CIN, University T\u00fcbingen"}]}