{"title": "Multiple Cause Vector Quantization", "book": "Advances in Neural Information Processing Systems", "page_first": 1041, "page_last": 1048, "abstract": null, "full_text": "Multiple Cause Vector Quantization\n\nDavid A. Ross and Richard S. Zemel\n\nDepartment of Computer Science\n\nUniversity of Toronto\n\nfdross,zemelg@cs.toronto.edu\n\nAbstract\n\nWe propose a model that can learn parts-based representations of high-\ndimensional data. Our key assumption is that the dimensions of the data\ncan be separated into several disjoint subsets, or factors, which take on\nvalues independently of each other. We assume each factor has a small\nnumber of discrete states, and model it using a vector quantizer. The\nselected states of each factor represent the multiple causes of the input.\nGiven a set of training examples, our model learns the association of\ndata dimensions with factors, as well as the states of each VQ. Inference\nand learning are carried out ef\ufb01ciently via variational algorithms. We\npresent applications of this model to problems in image decomposition,\ncollaborative \ufb01ltering, and text classi\ufb01cation.\n\n1 Introduction\n\nMany collections of data exhibit a common underlying structure: they consist of a number\nof parts or factors, each of which has a small number of discrete states. For example, in a\ncollection of facial images, every image contains eyes, a nose, and a mouth (except under\nocclusion), each of which has a range of different appearances. A speci\ufb01c image can be\ndescribed as a composite sketch: a selection of the appearance of each part, depending on\nthe individual depicted.\n\nIn this paper, we describe a stochastic generative model for data of this type. This model\nis well-suited to decomposing images into parts (it can be thought of as a Mr. Potato Head\nmodel), but also applies to domains such as text and collaborative \ufb01ltering in which the\nparts correspond to latent features, each having several alternative instantiations. This rep-\nresentational scheme is powerful due to its combinatorial nature: while a standard clus-\ntering/VQ method containing N states can represent at most N items, if we divide the N\ninto j-state VQs, we can represent j N=j items. MCVQ is also especially appropriate for\nhigh-dimensional data in which many values may be unspeci\ufb01ed for a given input case.\n\n2 Generative Model\n\nIn MCVQ we assume there are K factors, each of which is modeled by a vector quantizer\nwith J states. To generate an observed data example of D dimensions, x 2