{"title": "Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding", "book": "Advances in Neural Information Processing Systems", "page_first": 919, "page_last": 927, "abstract": "This paper presents a new semi-supervised framework with convolutional neural networks (CNNs) for text categorization.  Unlike the previous approaches that rely on word embeddings, our method learns embeddings of small text regions from unlabeled data for integration into a supervised CNN.  The proposed scheme for embedding learning is based on the idea of two-view semi-supervised learning, which is intended to be useful for the task of interest even though the training is done on unlabeled data.  Our models achieve better results than previous approaches on sentiment classification and topic classification tasks.", "full_text": "Semi-supervised Convolutional Neural Networks for\n\nText Categorization via Region Embedding\n\nRie Johnson\n\nRJ Research Consulting\nTarrytown, NY, USA\n\nriejohnson@gmail.com\n\nTong Zhang\u2217\n\nBaidu Inc., Beijing, China\n\nRutgers University, Piscataway, NJ, USA\n\ntzhang@stat.rutgers.edu\n\nAbstract\n\nThis paper presents a new semi-supervised framework with convolutional neural\nnetworks (CNNs) for text categorization. Unlike the previous approaches that rely\non word embeddings, our method learns embeddings of small text regions from\nunlabeled data for integration into a supervised CNN. The proposed scheme for\nembedding learning is based on the idea of two-view semi-supervised learning,\nwhich is intended to be useful for the task of interest even though the training\nis done on unlabeled data. Our models achieve better results than previous ap-\nproaches on sentiment classi\ufb01cation and topic classi\ufb01cation tasks.\n\n1\n\nIntroduction\n\nConvolutional neural networks (CNNs) [15] are neural networks that can make use of the internal\nstructure of data such as the 2D structure of image data through convolution layers, where each\ncomputation unit responds to a small region of input data (e.g., a small square of a large image). On\ntext, CNN has been gaining attention, used in systems for tagging, entity search, sentence modeling,\nand so on [4, 5, 26, 7, 21, 12, 25, 22, 24, 13], to make use of the 1D structure (word order) of text\ndata. Since CNN was originally developed for image data, which is \ufb01xed-sized, low-dimensional and\ndense, without modi\ufb01cation it cannot be applied to text documents, which are variable-sized, high-\ndimensional and sparse if represented by sequences of one-hot vectors. In many of the CNN studies\non text, therefore, words in sentences are \ufb01rst converted to low-dimensional word vectors. The word\nvectors are often obtained by some other method from an additional large corpus, which is typically\ndone in a fashion similar to language modeling though there are many variations [3, 4, 20, 23, 6, 19].\nUse of word vectors obtained this way is a form of semi-supervised learning and leaves us with\nthe following questions. Q1. How effective is CNN on text in a purely supervised setting without\nthe aid of unlabeled data? Q2. Can we use unlabeled data with CNN more effectively than using\ngeneral word vector learning methods? Our recent study [11] addressed Q1 on text categorization\nand showed that CNN without a word vector layer is not only feasible but also bene\ufb01cial when\nnot aided by unlabeled data. Here we address Q2 also on text categorization: building on [11], we\npropose a new semi-supervised framework that learns embeddings of small text regions (instead of\nwords) from unlabeled data, for use in a supervised CNN.\nThe essence of CNN, as described later, is to convert small regions of data (e.g., \u201clove it\u201d in a docu-\nment) to feature vectors for use in the upper layers; in other words, through training, a convolution\nlayer learns an embedding of small regions of data. Here we use the term \u2018embedding\u2019 loosely to\nmean a structure-preserving function, in particular, a function that generates low-dimensional fea-\ntures that preserve the predictive structure. [11] applies CNN directly to high-dimensional one-hot\nvectors, which leads to directly learning an embedding of small text regions (e.g., regions of size 3\n\u2217Tong Zhang would like to acknowledge NSF IIS-1250985, NSF IIS-1407939, and NIH R01AI116744 for\n\nsupporting his research.\n\n1\n\n\flike phrases, or regions of size 20 like sentences), eliminating the extra layer for word vector con-\nversion. This direct learning of region embedding was noted to have the merit of higher accuracy\nwith a simpler system (no need to tune hyper-parameters for word vectors) than supervised word\nvector-based CNN in which word vectors are randomly initialized and trained as part of CNN train-\ning. Moreover, the performance of [11]\u2019s best CNN rivaled or exceeded the previous best results on\nthe benchmark datasets.\nMotivated by this \ufb01nding, we seek effective use of unlabeled data for text categorization through\ndirect learning of embeddings of text regions. Our new semi-supervised framework learns a re-\ngion embedding from unlabeled data and uses it to produce additional input (additional to one-hot\nvectors) to supervised CNN, where a region embedding is trained with labeled data. Speci\ufb01cally,\nfrom unlabeled data, we learn tv-embeddings (\u2018tv\u2019 stands for \u2018two-view\u2019; de\ufb01ned later) of a text\nregion through the task of predicting its surrounding context. According to our theoretical \ufb01nding,\na tv-embedding has desirable properties under ideal conditions on the relations between two views\nand the labels. While in reality the ideal conditions may not be perfectly met, we consider them as\nguidance in designing the tasks for tv-embedding learning.\nWe consider several types of tv-embedding learning task trained on unlabeled data; e.g., one task\nis to predict the presence of the concepts relevant to the intended task (e.g., \u2018desire to recommend\nthe product\u2019) in the context, and we indirectly use labeled data to set up this task. Thus, we seek\nto learn tv-embeddings useful speci\ufb01cally for the task of interest. This is in contrast to the previous\nword vector/embedding learning methods, which typically produce a word embedding for general\npurposes so that all aspects (e.g., either syntactic or semantic) of words are captured. In a sense,\nthe goal of our region embedding learning is to map text regions to high-level concepts relevant\nto the task. This cannot be done by word embedding learning since individual words in isolation\nare too primitive to correspond to high-level concepts. For example, \u201ceasy to use\u201d conveys positive\nsentiment, but \u201cuse\u201d in isolation does not. We show that our models with tv-embeddings outper-\nform the previous best results on sentiment classi\ufb01cation and topic classi\ufb01cation. Moreover, a more\ndirect comparison con\ufb01rms that our region tv-embeddings provide more compact and effective rep-\nresentations of regions for the task of interest than what can be obtained by manipulation of a word\nembedding.\n\n1.1 Preliminary: one-hot CNN for text categorization [11]\n\nA CNN is a feed-forward network equipped with convolution layers interleaved with pooling layers.\nA convolution layer consists of computation units, each of which responds to a small region of\ninput (e.g., a small square of an image), and the small regions collectively cover the entire data. A\ncomputation unit associated with the (cid:96)-th region of input x computes:\n\n\u03c3(W \u00b7 r(cid:96)(x) + b) ,\n\n(1)\nwhere r(cid:96)(x) \u2208 Rq is the input region vector that represents the (cid:96)-th region. Weight matrix W \u2208\nRm\u00d7q and bias vector b \u2208 Rm are shared by all the units in the same layer, and they are learned\nthrough training. In [11], input x is a document represented by one-hot vectors (Figure 1); therefore,\nwe call [11]\u2019s CNN one-hot CNN; r(cid:96)(x) can be either a concatenation of one-hot vectors, a bag-of-\nword vector (bow), or a bag-of-n-gram vector: e.g., for a region \u201clove it\u201d\n\nI\nr(cid:96)(x) =[ 0\nI\nr(cid:96)(x) =[ 0\n\nit\n0\nit\n1\n\nlove\n1\nlove\n1\n\nI\n| 0\n](cid:62)\n\nit\n1\n\nlove\n0\n\n](cid:62)\n\n(concatenation)\n\n(bow)\n\n(2)\n\n(3)\n\nThe bow representation (3) loses word order within the region but is more robust to data sparsity,\nenables a large region size such as 20, and speeds up training by having fewer parameters. This is\nwhat we mainly use for embedding learning from unlabeled data. CNN with (2) is called seq-CNN\nand CNN with (3) bow-CNN. The region size and stride (distance between the region centers) are\nmeta-parameters. Note that we used a tiny three-word vocabulary for the vector examples above to\nsave space, but a vocabulary of typical applications could be much larger. \u03c3 in (1) is a component-\nwise non-linear function (e.g., applying \u03c3(x) = max(x, 0) to each vector component). Thus, each\ncomputation unit generates an m-dimensional vector where m is the number of weight vectors (W\u2019s\nrows) or neurons. In other words, a convolution layer embodies an embedding of text regions, which\nproduces an m-dim vector for each text region. In essence, a region embedding uses co-presence\nand absence of words in a region as input to produce predictive features, e.g., if presence of \u201ceasy\n\n2\n\n\fFigure 1: One-hot CNN example. Re-\ngion size 2, stride 1.\n\nFigure 2: Tv-embedding learning by training\nto predict adjacent regions.\n\nto use\u201d with absence of \u201cnot\u201d is a predictive indicator, it can be turned into a large feature value by\nhaving a negative weight on \u201cnot\u201d (to penalize its presence) and positive weights on the other three\nwords in one row of W. A more formal argument can be found in the supplementary material. The\nm-dim vectors from all the text regions of each document are aggregated by the pooling layer, by\neither component-wise maximum (max-pooling) or average (average-pooling), and used by the top\nlayer (a linear classi\ufb01er) as features for classi\ufb01cation. Here we focused on the convolution layer; for\nother details, [11] should be consulted.\n\n2 Semi-supervised CNN with tv-embeddings for text categorization\n\nIt was shown in [11] that one-hot CNN is effective on text categorization, where the essence is direct\nlearning of an embedding of text regions aided by new options of input region vector representation.\nWe go further along this line and propose a semi-supervised learning framework that learns an em-\nbedding of text regions from unlabeled data and then integrates the learned embedding in supervised\ntraining. The \ufb01rst step is to learn an embedding with the following property.\nDe\ufb01nition 1 (tv-embedding). A function f1 is a tv-embedding of X1 w.r.t. X2 if there exists a\nfunction g1 such that P (X2|X1) = g1(f1(X1), X2) for any (X1, X2) \u2208 X1 \u00d7 X2.\nA tv-embedding (\u2018tv\u2019 stands for two-view) of a view (X1), by de\ufb01nition, preserves everything re-\nquired to predict another view (X2), and it can be trained on unlabeled data. The motivation of tv-\nembedding is our theoretical \ufb01nding (formalized in the Appendix) that, essentially, a tv-embedded\nfeature vector f1(X1) is as useful as X1 for the purpose of classi\ufb01cation under ideal conditions.\nThe conditions essentially state that there exists a set H of hidden concepts such that two views\nand labels of the classi\ufb01cation task are related to each other only through the concepts in H. The\nconcepts in H might be, for example, \u201cpricey\u201d, \u201chandy\u201d, \u201chard to use\u201d, and so on for sentiment\nclassi\ufb01cation of product reviews. While in reality the ideal conditions may not be completely met,\nwe consider them as guidance and design tv-embedding learning accordingly.\nTv-embedding learning is related to two-view feature learning [2] and ASO [1], which learn a linear\nembedding from unlabeled data through tasks such as predicting a word (or predicted labels) from\nthe features associated with its surrounding words. These studies were, however, limited to a linear\nembedding. A related method in [6] learns a word embedding so that left context and right context\nmaximally correlate in terms of canonical correlation analysis. While we share with these studies\nthe general idea of using the relations of two views, we focus on nonlinear learning of region em-\nbeddings useful for the task of interest, and the resulting methods are very different. An important\ndifference of tv-embedding learning from co-training is that it does not involve label guessing, thus\navoiding risk of label contamination. [8] used a Stacked Denoising Auto-encoder to extract features\ninvariant across domains for sentiment classi\ufb01cation from unlabeled data. It is for fully-connected\nneural networks, which underperformed CNNs in [11].\nNow let B be the base CNN model for the task of interest, and assume that B has one convolution\nlayer with region size p. Note, however, that the restriction of having only one convolution layer is\nmerely for simplifying the description. We propose a semi-supervised framework with the following\ntwo steps.\n1. Tv-embedding learning: Train a neural network U to predict the context from each region\nof size p so that U\u2019s convolution layer generates feature vectors for each text region of size\np for use in the classi\ufb01er in the top layer. It is this convolution layer, which embodies the\ntv-embedding, that we transfer to the supervised learning model in the next step. (Note that\nU differs from CNN in that each small region is associated with its own target/output.)\n\n3\n\nI really love it ! Output1 (positive)Convolution layer (size 2)Top layerreally love it ! Pooling layer1 (positive)Input: One-hot vectorsgood goodacting fun plot :) plot :) :)good acting , fun plot :) \u2026Convolution layer f1Top layer g1acting OutputX2fun plot :) Input X1\u2026\f2. Final supervised learning: Integrate the learned tv-embedding (the convolution layer of U)\ninto B, so that the tv-embedded regions (the output of U\u2019s convolution layer) are used as an\nadditional input to B\u2019s convolution layer. Train this \ufb01nal model with labeled data.\n\nThese two steps are described in more detail in the next two sections.\n\n2.1 Learning tv-embeddings from unlabeled data\n\nu(cid:96)(x) = \u03c3(U )(cid:16)\n\nW(U ) \u00b7 r(U )\n\n(cid:96)\n\n(x) + b(U )(cid:17)\n\nWe create a task on unlabeled data to predict the context (adjacent text regions) from each region of\nsize p de\ufb01ned in B\u2019s convolution layer. To see the correspondence to the de\ufb01nition of tv-embeddings,\nit helps to consider a sub-task that assigns a label (e.g., positive/negative) to each text region (e.g., \u201c,\nfun plot\u201d) instead of the ultimate task of categorizing the entire document. This is sensible because\nCNN makes predictions by building up from these small regions. In a document \u201cgood acting, fun\nplot :)\u201d as in Figure 2, the clues for predicting a label of \u201c, fun plot\u201d are \u201c, fun plot\u201d itself (view-\n1: X1) and its context \u201cgood acting\u201d and \u201c:)\u201d (view-2: X2). U is trained to predict X2 from X1,\ni.e., to approximate P (X2|X1) by g1(f1(X1), X2)) as in De\ufb01nition 1, and functions f1 and g1 are\nembodied by the convolution layer and the top layer, respectively.\nGiven a document x, for each text region indexed by (cid:96), U\u2019s convolution layer computes:\n\n,\n\n(x) can be either sequential, bow, or bag-of-n-gram, independent of r(cid:96)(x) in B.\n\n(4)\nwhich is the same as (1) except for the superscript \u201c(U)\u201d to indicate that these entities belong to U.\nThe top layer (a linear model for classi\ufb01cation) uses u(cid:96)(x) as features for prediction. W(U ) and b(U )\n(and the top-layer parameters) are learned through training. The input region vector representation\nr(U )\n(cid:96)\nThe goal here is to learn an embedding of text regions (X1), shared with all the text regions at\nevery location. Context (X2) is used only in tv-embedding learning as prediction target (i.e., not\ntransferred to the \ufb01nal model); thus, the representation of context should be determined to optimize\nthe \ufb01nal outcome without worrying about the cost at prediction time. Our guidance is the conditions\non the relationships between the two views mentioned above; ideally, the two views should be\nrelated to each other only through the relevant concepts. We consider the following two types of\ntarget/context representation.\nUnsupervised target A straightforward vector encoding of context/target X2 is bow vectors of\nthe text regions on the left and right to X1. If we distinguish the left and right, the target vector is\n2|V |-dimensional with vocabulary V , and if not, |V |-dimensional. One potential problem of this\nencoding is that adjacent regions often have syntactic relations (e.g., \u201cthe\u201d is often followed by an\nadjective or a noun), which are typically irrelevant to the task (e.g., to identify positive/negative\nsentiment) and therefore undesirable. A simple remedy we found effective is vocabulary control\nof context to remove function words (or stop-words if available) from (and only from) the target\nvocabulary.\nPartially-supervised target Another context representation that we consider is partially super-\nvised in the sense that it uses labeled data. First, we train a CNN with the labeled data for the\nintended task and apply it to the unlabeled data. Then we discard the predictions and only retain\nthe internal output of the convolution layer, which is an m-dimensional vector for each text region\nwhere m is the number of neurons. We use these m-dimensional vectors to represent the context.\n[11] has shown, by examples, that each dimension of these vectors roughly represents concepts rel-\nevant to the task, e.g., \u2018desire to recommend the product\u2019, \u2018report of a faulty product\u2019, and so on.\nTherefore, an advantage of this representation is that there is no obvious noise between X1 and X2\nsince context X2 is represented only by the concepts relevant to the task. A disadvantage is that it\nis only as good as the supervised CNN that produced it, which is not perfect and in particular, some\nrelevant concepts would be missed if they did not appear in the labeled data.\n\n2.2 Final supervised learning: integration of tv-embeddings into supervised CNN\nWe use the tv-embedding obtained from unlabeled data to produce additional input to B\u2019s convolu-\ntion layer, by replacing \u03c3 (W \u00b7 r(cid:96)(x) + b) (1) with:\n\n\u03c3 (W \u00b7 r(cid:96)(x) + V \u00b7 u(cid:96)(x) + b) ,\n\n(5)\n\n4\n\n\f(cid:32)\n\nk(cid:88)\n\n(cid:33)\n\nwhere u(cid:96)(x) is de\ufb01ned by (4), i.e., u(cid:96)(x) is the output of the tv-embedding applied to the (cid:96)-th\nregion. We train this model with the labeled data of the task; that is, we update the weights W, V,\nbias b, and the top-layer parameters so that the designated loss function is minimized on the labeled\ntraining data. W(U ) and b(U ) can be either \ufb01xed or updated for \ufb01ne-tuning, and in this work we \ufb01x\nthem for simplicity.\nNote that while (5) takes a tv-embedded region as input, (5) itself is also an embedding of text\nregions; let us call it (and also (1)) a supervised embedding, as it is trained with labeled data, to\ndistinguish it from tv-embeddings. That is, we use tv-embeddings to improve the supervised em-\nbedding. Note that (5) can be naturally extended to accommodate multiple tv-embeddings by\n\nW \u00b7 r(cid:96)(x) +\n\n\u03c3\n\nV(i) \u00b7 u(i)\n\n(cid:96) (x) + b\n\n,\n\n(6)\n\nso that, for example, two types of tv-embedding (i.e., k = 2) obtained with the unsupervised target\nand the partially-supervised target can be used at once, which can lead to performance improvement\nas they complement each other, as shown later.\n\ni=1\n\n3 Experiments\n\nOur code and the experimental settings are available at riejohnson.com/cnn download.html.\nData We used the three datasets used in [11]: IMDB, Elec, and RCV1, as summarized in Table\n1. IMDB (movie reviews) [17] comes with an unlabeled set. To facilitate comparison with previous\nstudies, we used a union of this set and the training set as unlabeled data. Elec consists of Amazon\nreviews of electronics products. To use as unlabeled data, we chose 200K reviews from the same\ndata source so that they are disjoint from the training and test sets, and that the reviewed products\nare disjoint from the test set. On the 55-way classi\ufb01cation of the second-level topics on RCV1\n(news), unlabeled data was chosen to be disjoint from the training and test sets. On the multi-label\ncategorization of 103 topics on RCV1, since the of\ufb01cial LYRL04 split for this task divides the entire\ncorpus into a training set and a test set, we used the entire test set as unlabeled data (the transductive\nlearning setting).\n\n#unlabeled\n\n#train\nIMDB 25,000\n25,000\nElec\n15,564\nRCV1\n23,149\nTable 1: Datasets. \u2020The multi-label RCV1 is used only in Table 6.\n\n75K (20M words)\n200K (24M words)\n669K (183M words)\n781K (214M words)\n\n#test\n25,000\n25,000\n49,838\n781,265\n\n55 (single)\n103 (multi)\u2020\n\n#class\n\n2\n2\n\nsentiment\nTopic(s)\n\noutput\n\nPositive/negative\n\nmized weighted square loss(cid:80)\n\nImplementation We used the one-layer CNN models found to be effective in [11] as our base\nmodels B, namely, seq-CNN on IMDB/Elec and bow-CNN on RCV1. Tv-embedding training mini-\ni,j \u03b1i,j(zi[j]\u2212 pi[j])2 where i goes through the regions, z represents\nthe target regions, and p is the model output. The weights \u03b1i,j were set to balance the loss originat-\ning from the presence and absence of words (or concepts in case of the partially-supervised target)\nand to speed up training by eliminating some negative examples, similar to negative sampling of\n[19]. To experiment with the unsupervised target, we set z to be bow vectors of adjacent regions\non the left and right, while only retaining the 30K most frequent words with vocabulary control;\non sentiment classi\ufb01cation, function words were removed, and on topic classi\ufb01cation, numbers and\nstop-words provided by [16] were removed. Note that these words were removed from (and only\nfrom) the target vocabulary. To produce the partially-supervised target, we \ufb01rst trained the super-\nvised CNN models with 1000 neurons and applied the trained convolution layer to unlabeled data\nto generate 1000-dimensional vectors for each region. The rest of implementation follows [11]; i.e.,\nsupervised models minimized square loss with L2 regularization and optional dropout [9]; \u03c3 and\n\u03c3(U ) were the recti\ufb01er; response normalization was performed; optimization was done by SGD.\nModel selection On all the tested methods, tuning of meta-parameters was done by testing the\nmodels on the held-out portion of the training data, and then the models were re-trained with the\nchosen meta-parameters using the entire training data.\n\n5\n\n\f3.1 Performance results\n\nOverview After con\ufb01rming the effectiveness of our new models in comparison with the supervised\nCNN, we report the performances of [13]\u2019s CNN, which relies on word vectors pre-trained with a\nvery large corpus (Table 3). Besides comparing the performance of approaches as a whole, it is\nalso of interest to compare the usefulness of what was learned from unlabeled data; therefore, we\nshow how it performs if we integrate the word vectors into our base model one-hot CNNs (Figure\n3). In these experiments we also test word vectors trained by word2vec [19] on our unlabeled data\n(Figure 4). We then compare our models with two standard semi-supervised methods, transductive\nSVM (TSVM) [10] and co-training (Table 3), and with the previous best results in the literature\n(Tables 4\u20136). In all comparisons, our models outperform the others. In particular, our region tv-\nembeddings are shown to be more compact and effective than region embeddings obtained by simple\nmanipulation of word embeddings, which supports our approach of using region embedding instead\nof word embedding.\n\nnames in Table 3 X1: r(U )\nbow vector\nunsup-tv.\nparsup-tv.\nbow vector\nbag-of-{1,2,3}-gram vector\nunsup3-tv.\n\n(x)\n\n(cid:96)\n\nX2: target of U training\nbow vector\noutput of supervised embedding\nbow vector\n\nTable 2: Tested tv-embeddings.\n\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n\nlinear SVM with 1-3grams [11]\n\nlinear TSVM with 1-3grams\n\n[13]\u2019s CNN\n\nOne-hot CNN (simple) [11]\n\nOne-hot CNN (simple) co-training best\n\nOur CNN\n\nunsup-tv.\n\nparsup-tv.\n\nunsup3-tv.\nall three\n\n100-dim\n200-dim\n100-dim\n200-dim\n100-dim\n200-dim\n100\u00d73\n\nIMDB\n10.14\n9.99\n9.17\n8.39\n(8.06)\n7.12\n6.81\n7.12\n7.13\n7.05\n6.96\n6.51\n\nElec RCV1\n10.68\n9.16\n16.41\n10.77\n10.44\n8.03\n9.17\n7.64\n(8.73)\n(7.63)\n6.96\n8.10\n7.97\n6.69\n8.19\n6.58\n7.99\n6.57\n8.13\n6.66\n6.84\n8.02\n7.71\n6.27\n\nTable 3: Error rates (%). For comparison, all the CNN models were constrained to have 1000 neurons. The\nparentheses around the error rates indicate that co-training meta-parameters were tuned on test data.\n\nOur CNN with tv-embeddings We tested three types of tv-embedding as summarized in Table\n2. The \ufb01rst thing to note is that all of our CNNs (Table 3, row 6\u201312) outperform their supervised\ncounterpart in row 4. This con\ufb01rms the effectiveness of the framework we propose. In Table 3, for\nmeaningful comparison, all the CNNs are constrained to have exactly one convolution layer (except\nfor [13]\u2019s CNN) with 1000 neurons. The best-performing supervised CNNs within these constraints\n(row 4) are: seq-CNN (region size 3) on IMDB and Elec and bow-CNN (region size 20) on RCV11.\nThey also served as our base models B (with region size parameterized on IMDB/Elec). More\ncomplex supervised CNNs from [11] will be reviewed later. On sentiment classi\ufb01cation (IMDB\nand Elec), the region size chosen by model selection for our models was 5, larger than 3 for the\nsupervised CNN. This indicates that unlabeled data enabled effective use of larger regions which are\nmore predictive but might suffer from data sparsity in supervised settings.\n\u2018unsup3-tv.\u2019 (rows 10\u201311) uses a bag-of-n-gram vector to initially represent each region, thus, re-\ntains word order partially within the region. When used individually, unsup3-tv. did not outperform\nthe other tv-embeddings, which use bow instead (rows 6\u20139). But we found that it contributed to\nerror reduction when combined with the others (not shown in the table). This implies that it learned\nfrom unlabeled data predictive information that the other two embeddings missed. The best perfor-\nmances (row 12) were obtained by using all the three types of tv-embeddings at once according to\n(6). By doing so, the error rates were improved by nearly 1.9% (IMDB) and 1.4% (Elec and RCV1)\ncompared with the supervised CNN (row 4), as a result of the three tv-embeddings with different\nstrengths complementing each other.\n\n1 The error rate on RCV1 in row 4 slightly differs from [11] because here we did not use the stopword list.\n\n6\n\n\fIMDB\nElec\nRCV1\n\nconcat\n8.31\n7.37\n8.70\n\navg\n7.83\n7.24\n8.62\n\nFigure 3: GN word vec-\ntors integrated into our base\nmodels. Better than [13]\u2019s\nCNN (Table 3, row 3).\n\nFigure 4: Region tv-embeddings vs. word2vec word embeddings. Trained\non our unlabeled data. x-axis: dimensionality of the additional input to\nsupervised region embedding. \u2018r:\u2019: region, \u2018w:\u2019: word.\n\n[13]\u2019s CNN It was shown in [13] that CNN that uses the Google News word vectors as input is\ncompetitive on a number of sentence classi\ufb01cation tasks. These vectors (300-dimensional) were\ntrained by the authors of word2vec [19] on a very large Google News (GN) corpus (100 billion\nwords; 500\u20135K times larger than our unlabeled data). [13] argued that these vectors can be useful\nfor various tasks, serving as \u2018universal feature extractors\u2019. We tested [13]\u2019s CNN, which is equipped\nwith three convolution layers with different region sizes (3, 4, and 5) and max-pooling, using the\nGN vectors as input. Although [13] used only 100 neurons for each layer, we changed it to 400,\n300, and 300 to match the other models, which use 1000 neurons. Our models clearly outperform\nthese models (Table 3, row 3) with relatively large differences.\nComparison of embeddings Besides comparing the performance of the approaches as a whole,\nit is also of interest to compare the usefulness of what was learned from unlabeled data. For this\npurpose, we experimented with integration of a word embedding into our base models using two\nmethods; one takes the concatenation, and the other takes the average, of word vectors for the words\nin the region. These provide additional input to the supervised embedding of regions in place of\nu(cid:96)(x) in (5). That is, for comparison, we produce a region embedding from a word embedding\nto replace a region tv-embedding. We show the results with two types of word embeddings: the\nGN word embedding above (Figure 3), and word embeddings that we trained with the word2vec\nsoftware on our unlabeled data, i.e., the same data as used for tv-embedding learning and all others\n(Figure 4). Note that Figure 4 plots error rates in relation to the dimensionality of the produced\nadditional input; a smaller dimensionality has an advantage of faster training/prediction.\nOn the results, \ufb01rst, the region tv-embedding is more useful for these tasks than the tested word\nembeddings since the models with a tv-embedding clearly outperform all the models with a word\nembedding. Word vector concatenations of much higher dimensionality than those shown in the\n\ufb01gure still underperformed 100-dim region tv-embedding. Second, since our region tv-embedding\ntakes the form of \u03c3(W \u00b7 r(cid:96)(x) + b) with r(cid:96)(x) being a bow vector, the columns of W correspond\nto words, and therefore, W \u00b7 r(cid:96)(x) is the sum of W\u2019s columns whose corresponding words are\nin the (cid:96)-th region. Based on that, one might wonder why we should not simply use the sum or\naverage of word vectors obtained by an existing tool such as word2vec instead. The suboptimal\nperformances of \u2018w: average\u2019 (Figure 4) tells us that this is a bad idea. We attribute it to the fact that\nregion embeddings learn predictiveness of co-presence and absence of words in a region; a region\nembedding can be more expressive than averaging of word vectors. Thus, an effective and compact\nregion embedding cannot be trivially obtained from a word embedding. In particular, effectiveness\nof the combination of three tv-embeddings (\u2018r: 3 tv-embed.\u2019 in Figure 4) stands out.\nAdditionally, our mechanism of using information from unlabeled data is more effective than [13]\u2019s\nCNN since our CNNs with GN (Figure 3) outperform [13]\u2019s CNNs with GN (Table 3, row 3). This\nis because in our model, one-hot vectors (the original features) compensate for potential information\nloss in the embedding learned from unlabeled data. This, as well as region-vs-word embedding, is a\nmajor difference between our model and [13]\u2019s model.\nStandard semi-supervised methods Many of the standard semi-supervised methods are not ap-\nplicable to CNN as they require bow vectors as input. We tested TSVM with bag-of-{1,2,3}-gram\nvectors using SVMlight. TSVM underperformed the supervised SVM2 on two of the three datasets\n\n2 Note that for feasibility, we only used the 30K most frequent n-grams in the TSVM experiments, thus,\nshowing the SVM results also with 30K vocabulary for comparison, though on some datasets SVM performance\ncan be improved by use of all the n-grams (e.g., 5 million n-grams on IMDB) [11]. This is because the\ncomputational cost of TSVM (single-core) turned out to be high, taking several days even with 30K vocabulary.\n\n7\n\n6.577.588.50150300Error rate (%)additional dimIMDB66.577.580150300Error rate (%)additional dimElec7.58.59.5Error rate (%)7.588.599.50150300additional dimRCV1supervisedw: concatw: averager: unsup-tvr: 3 tv-embed.\fNB-LM 1-3grams [18]\n[11]\u2019s best CNN\nParagraph vectors [14]\nEnsemble of 3 models [18]\nOur best\nTable 4: IMDB: previous error rates (%).\n\n8.13\n7.67\n7.46\n7.43\n6.51\n\n\u2013\n\u2013\n\nUnlab.data\nEns.+unlab.\nUnlab.data\n\nSVM 1-3grams [11]\ndense NN 1-3grams [11]\nNB-LM 1-3grams [11]\n[11]\u2019s best CNN\nOur best\nTable 5: Elec: previous error rates (%).\n\n8.71\n8.48\n8.11\n7.14\n6.27 Unlab.data\n\n\u2013\n\u2013\n\u2013\n\u2013\n\nmodels\nSVM [16]\nbow-CNN [11]\nbow-CNN w/ three tv-embed.\n\n81.6\n84.0\n85.7\n\n60.7\n64.8\n67.1\n\nmicro-F macro-F\n\nextra resource\n\n\u2013\n\u2013\n\nUnlabeled data\n\nTable 6: RCV1 micro- and macro-averaged F on the multi-label task (103 topics) with the LYRL04 split.\n\n(Table 3, rows 1\u20132). Since co-training is a meta-learner, it can be used with CNN. Random split of\nvocabulary and split into the \ufb01rst and last half of each document were tested. To reduce the computa-\ntional burden, we report the best (and unrealistic) co-training performances obtained by optimizing\nthe meta-parameters including when to stop on the test data. Even with this unfair advantage to co-\ntraining, co-training (Table 3, row 5) clearly underperformed our models. The results demonstrate\nthe dif\ufb01culty of effectively using unlabeled data on these tasks, given that the size of the labeled data\nis relatively large.\nComparison with the previous best results We compare our models with the previous best re-\nsults on IMDB (Table 4). Our best model with three tv-embeddings outperforms the previous best\nresults by nearly 0.9%. All of our models with a single tv-embed. (Table 3, row 6\u201311) also perform\nbetter than the previous results. Since Elec is a relatively new dataset, we are not aware of any previ-\nous semi-supervised results. Our performance is better than [11]\u2019s best supervised CNN, which has\na complex network architecture of three convolution-pooling pairs in parallel (Table 5). To compare\nwith the benchmark results in [16], we tested our model on the multi-label task with the LYRL04\nsplit [16] on RCV1, in which more than one out of 103 categories can be assigned to each document.\nOur model outperforms the best SVM of [16] and the best supervised CNN of [11] (Table 6).\n\n4 Conclusion\n\nThis paper proposed a new semi-supervised CNN framework for text categorization that learns em-\nbeddings of text regions with unlabeled data and then labeled data. As discussed in Section 1.1,\na region embedding is trained to learn the predictiveness of co-presence and absence of words in\na region. In contrast, a word embedding is trained to only represent individual words in isolation.\nThus, a region embedding can be more expressive than simple averaging of word vectors in spite\nof their seeming similarity. Our comparison of embeddings con\ufb01rmed its advantage; our region tv-\nembeddings, which are trained speci\ufb01cally for the task of interest, are more effective than the tested\nword embeddings. Using our new models, we were able to achieve higher performances than the\nprevious studies on sentiment classi\ufb01cation and topic classi\ufb01cation.\n\nAppendix A Theory of tv-embedding\nSuppose that we observe two views (X1, X2) \u2208 X1 \u00d7 X2 of the input, and a target label Y \u2208 Y of\ninterest, where X1 and X2 are \ufb01nite discrete sets.\nAssumption 1. Assume that there exists a set of hidden states H such that X1, X2, and Y are\nconditionally independent given h in H, and that the rank of matrix [P (X1, X2)] is |H|.\nTheorem 1. Consider a tv-embedding f1 of X1 w.r.t. X2. Under Assumption 1, there exists\na function q1 such that P (Y |X1) = q1(f1(X1), Y ). Further consider a tv-embedding f2 of\nX2 w.r.t. X1. Then, under Assumption 1, there exists a function q such that P (Y |X1, X2) =\nq(f1(X1), f2(X2), Y ).\nThe proof can be found in the supplementary material.\n\n8\n\n\fReferences\n[1] Rie K. Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and\n\nunlabeled data. Journal of Machine Learning Research, 6:1817\u20131853, 2005.\n\n[2] Rie K. Ando and Tong Zhang. Two-view feature generation model for semi-supervised learning.\n\nProceedings of ICML, 2007.\n\nIn\n\n[3] Yoshua Bengio, R\u00b4ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language\n\nmodel. Journal of Marchine Learning Research, 3:1137\u20131155, 2003.\n\n[4] Ronan Collobert and Jason Weston. A uni\ufb01ed architecture for natural language processing: Deep neural\n\nnetworks with multitask learning. In Proceedings of ICML, 2008.\n\n[5] Ronan Collobert, Jason Weston, L\u00b4eon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.\nNatural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493\u2013\n2537, 2011.\n\n[6] Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. Multi-view learning of word embeddings via CCA.\n\nIn Proceedings of NIPS, 2011.\n\n[7] Jianfeng Gao, Patric Pantel, Michael Gamon, Xiaodong He, and Li dent. Modeling interestingness with\n\ndeep neural networks. In Proceedings of EMNLP, 2014.\n\n[8] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classi-\n\n\ufb01cation: A deep learning approach. In Proceedings of ICML, 2011.\n\n[9] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov.\n\nImproving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012.\n[10] Thorsten Joachims. Transductive inference for text classi\ufb01cation using support vector machines.\n\nIn\n\nProceedings of ICML, 1999.\n\n[11] Rie Johnson and Tong Zhang. Effective use of word order for text categorization with convolutional\n\nneural networks. In Proceedings of NAACL HLT, 2015.\n\n[12] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modeling\n\nsentences. In Proceedings of ACL, pages 655\u2013665, 2014.\n\n[13] Yoon Kim. Convolutional neural networks for sentence classi\ufb01cation. In Proceedings of EMNLP, pages\n\n1746\u20131751, 2014.\n\n[14] Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Proceedings of\n\nICML, 2014.\n\n[15] Yann LeCun, Le\u00b4on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to\n\ndocument recognition. In Proceedings of the IEEE.\n\n[16] David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. RCV1: A new benchmark collection for text\n\ncategorization research. Journal of Marchine Learning Research, 5:361\u2013397, 2004.\n\n[17] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.\n\nLearning word vectors for sentiment analysis. In Proceedings of ACL, 2011.\n\n[18] Gr\u00b4egoire Mesnil, Tomas Mikolov, Marc\u2019Aurelio Ranzato, and Yoshua Bengio. Ensemble of generative\nand discriminative techniques for sentiment analysis of movie reviews. arXiv:1412.5335v5 (4 Feb 2015\nversion), 2014.\n\n[19] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations\n\nof words and phrases and their compositionality. In Proceedings of NIPS, 2013.\n\n[20] Andriy Mnih and Geoffrey E. Hinton. A scalable hierarchical distributed language model. In NIPS, 2008.\n[21] Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gr\u00b4egoire Mensnil. A latent semantic model with\n\nconvolutional-pooling structure for information retrieval. In Proceedings of CIKM, 2014.\n\n[22] Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. Learning sentiment-speci\ufb01c word\n\nembedding for twitter sentiment classi\ufb01cation. In Proceedings of ACL, pages 1555\u20131565, 2014.\n\n[23] Joseph Turian, Lev Rainov, and Yoshua Bengio. Word representations: A simple and general method for\n\nsemi-supervised learning. In Proceedings of ACL, pages 384\u2013394, 2010.\n\n[24] Jason Weston, Sumit Chopra, and Keith Adams. #tagspace: Semantic embeddings from hashtags. In\n\nProceedings of EMNLP, pages 1822\u20131827, 2014.\n\n[25] Liheng Xu, Kang Liu, Siwei Lai, and Jun Zhao. Product feature mining: Semantic clues versus syntactic\n\nconstituents. In Proceedings of ACL, pages 336\u2013346, 2014.\n\n[26] Puyang Xu and Ruhi Sarikaya. Convolutional neural network based triangular CRF for joint intent detec-\n\ntion and slot \ufb01lling. In ASRU, 2013.\n\n9\n\n\f", "award": [], "sourceid": 593, "authors": [{"given_name": "Rie", "family_name": "Johnson", "institution": "RJ Research Consuulting"}, {"given_name": "Tong", "family_name": "Zhang", "institution": "Rutgers"}]}