{"title": "Decoding with Value Networks for Neural Machine Translation", "book": "Advances in Neural Information Processing Systems", "page_first": 178, "page_last": 187, "abstract": "Neural Machine Translation (NMT) has become a popular technology in recent years, and beam search is its de facto decoding method due to the shrunk search space and reduced computational complexity. However, since it only searches for local optima at each time step through one-step forward looking, it usually cannot output the best target sentence. Inspired by the success and methodology of AlphaGo, in this paper we propose using a prediction network to improve beam search, which takes the source sentence $x$, the currently available decoding output $y_1,\\cdots, y_{t-1}$ and a candidate word $w$ at step $t$ as inputs and predicts the long-term value (e.g., BLEU score) of the partial target sentence if it is completed by the NMT model. Following the practice in reinforcement learning, we call this prediction network \\emph{value network}. Specifically, we propose a recurrent structure for the value network, and train its parameters from bilingual data. During the test time, when  choosing a word $w$ for decoding, we consider both its conditional probability given by the NMT model and its long-term value predicted by the value network. Experiments show that such an approach can significantly improve the translation accuracy on several translation tasks.", "full_text": "Decoding with Value Networks for Neural Machine\n\nTranslation\n\nDi He1\n\nHanqing Lu2\n\nYingce Xia3\n\ndi_he@pku.edu.cn\n\nhanqinglu@cmu.edu\n\nxiayingc@mail.ustc.edu.cn\n\nTao Qin4\n\nLiwei Wang1,5\n\nTie-Yan Liu4\n\ntaoqin@microsoft.com\n\nwanglw@cis.pku.edu.cn\n\ntie-yan.liu@microsoft.com\n\n1Key Laboratory of Machine Perception, MOE, School of EECS, Peking University\n\n2Carnegie Mellon University\n\n3University of Science and Technology of China\n\n5Center for Data Science, Peking University, Beijing Institute of Big Data Research\n\n4Microsoft Research\n\nAbstract\n\nNeural Machine Translation (NMT) has become a popular technology in recent\nyears, and beam search is its de facto decoding method due to the shrunk search\nspace and reduced computational complexity. However, since it only searches\nfor local optima at each time step through one-step forward looking, it usually\ncannot output the best target sentence. Inspired by the success and methodology of\nAlphaGo, in this paper we propose using a prediction network to improve beam\nsearch, which takes the source sentence x, the currently available decoding output\ny1,\u00b7\u00b7\u00b7 , yt\u22121 and a candidate word w at step t as inputs and predicts the long-term\nvalue (e.g., BLEU score) of the partial target sentence if it is completed by the NMT\nmodel. Following the practice in reinforcement learning, we call this prediction\nnetwork value network. Speci\ufb01cally, we propose a recurrent structure for the value\nnetwork, and train its parameters from bilingual data. During the test time, when\nchoosing a word w for decoding, we consider both its conditional probability\ngiven by the NMT model and its long-term value predicted by the value network.\nExperiments show that such an approach can signi\ufb01cantly improve the translation\naccuracy on several translation tasks.\n\n1\n\nIntroduction\n\nNeural Machine Translation (NMT), which is based on deep neural networks and provides an end-\nto-end solution to machine translation, has attracted much attention from the research community\n[2, 6, 12, 20] and gradually been adopted by industry in past several years [18, 22]. NMT uses\nan RNN-based encoder-decoder framework to model the entire translation process. In training, it\nmaximizes the likelihood of a target sentence given a source sentence. In testing, given a source\nsentence x, it tries to \ufb01nd a sentence y\u2217 in the target language that maximizes the conditional\nprobability P (y|x). Since the number of possible target sentences is exponentially large, \ufb01nding the\noptimal y\u2217 is NP-hard. Thus beam search is commonly employed to \ufb01nd a reasonably good y.\nBeam search is a heuristic search algorithm that maintains the top-scoring partial sequences expanded\nin a left-to-right fashion. In particular, it keeps a pool of candidates each of which is a partial sequence.\nAt each time step, the algorithm expands each candidate by appending a new word, and then keeps\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fthe top-ranked new candidates scored by the NMT model. The algorithm terminates if it meets the\nmaximum decoding depth or all sentences are completely generated, i.e., all sentences are ended with\nthe end-of-sentence (EOS) symbol.\nWhile NMT with beam search has been proved to be successful, it has several obvious issues,\nincluding exposure bias [9], loss-evaluation mismatch [9] and label bias [16], which have been studied.\nHowever, we observe that there is still an important issue associated with beam search of NMT, the\nmyopic bias, which unfortunately is largely ignored, to the best of our knowledge. Beam search tends\nto focus more on short-term reward. At iteration t, for a candidate y1,\u00b7\u00b7\u00b7 , yt\u22121 (refers to y<t) and\ntwo words w and w(cid:48), we denote y<t + w if we append w to y<t. If P (y<t + w|x) > P (y<t + w(cid:48)|x),\nnew candidate y<t + w is more likely to be kept, even if w(cid:48) is the ground truth translation at step t or\ncan offer a better score in future decodings. Such search errors coming from short sighted actions\nsometimes provide a bad translation even if the translation model is good.\nTo address the myopic bias, for each word w and each candidate y<t, we propose to design a\nprediction model to estimate the long-term reward if we append w to y<t and follow the current\nNMT model until the decoding \ufb01nishes. Then we can leverage the predicted score from this model\nduring each decoding step to help \ufb01nd a better w that can contribute to the long-term translation\nperformance. This prediction model, which predicts long-term reward we will receive in the future, is\nexactly the concept of value function in Reinforcement Learning (RL).\nIn this work, we develop a neural network-based prediction model, which is called value network\nfor NMT. The value network takes the source sentence and any partial target sequence as input, and\noutputs a predicted value to estimate the expected total reward (e.g. BLEU) generated from this\npartial sequence by the NMT model. In any decoding step, we select the best candidates not only\nbased on the conditional probability of the partial sequence outputted by the NMT model, but also\nbased on the estimated long-term reward outputted by the value network.\nThe main contributions of this work are summarized as follows. First, we develop a decoding scheme\nthat considers long-term reward while generating words one by one for machine translation, which is\nnew in NMT literature. At each step, the new decoding scheme not only considers the probability of\nthe word sequence conditioned on the source sentence, but also relies on the predicted future reward.\nWe believe that considering the two aspects can lead to better \ufb01nal translation.\nSecond, we design a novel structure for the value network. On the top of the encoder-decoder layer\nof NMT, we develop another two modules for the value network, a semantic matching module and a\ncontext-coverage module. The semantic matching module aims at estimating the similarity between\nthe source and target sentences, which can contribute to the quality of the translation. It is often\nobserved that the more context used in the attention mechanism, the better translation we will generate\n[14, 15]. Thus we build a context-coverage module to measure the coverage of context used in the\nencoder-decoder layer. With the outputs of the two modules, the value prediction is done via fully\nconnected layers.\nWe conduct a set of experiments on several translation tasks. All the results demonstrate the\neffectiveness and robustness of the new decoding mechanism compared to several baseline algorithms.\nThe remaining parts of the paper are organized as follows. In Section 2, we brie\ufb02y review the\nliterature of neural machine translation. After that, we describe the myopic bias problem of NMT in\nSection 3 and introduce our method for value network learning in Section 4. Experimental results are\nprovided and analyzed in Section 5. We discuss future directions in the last section.\n\n2 Neural Machine Translation\n\nNeural machine translation systems are typically implemented with a Recurrent Neural Network\n(RNN)-based encoder-decoder framework. Such a framework directly models the probability P (y|x)\nof a target sentence y = {y1, y2, ..., yTy} conditioned on the source sentence x = {x1, x2, ..., xTx},\nwhere Tx and Ty are the length of sentence x and y.\nThe encoder of NMT reads the source sentence x word by word and generates a hidden representation\nfor each word xi:\n\nhi = f (hi\u22121, xi),\n\n2\n\n(1)\n\n\fFigure 1: Architecture of Value Network\n\nin which function f is the recurrent unit such as Long Short-Term Memory (LSTM) unit [12] or Gated\nRecurrent Unit (GRU) [4]. Afterwards, the decoder of NMT computes the conditional probability of\neach target word yt conditioned on its proceeding words y<t as well as the source sentence:\n\nP (yt|y<t, x) \u221d exp(yt; rt, ct),\nrt = g(rt\u22121, yt\u22121, ct),\nct = q(rt\u22121, h1,\u00b7\u00b7\u00b7 , hTx ),\n\n(2)\n(3)\n(4)\nwhere rt is the decoder RNN hidden representation at step t, similarly computed by an LSTM or GRU,\nand ct denotes the weighted contextual information summarizing the source sentence x using some\nattention mechanism [4]. Denote all the parameters to be learned in the encoder-decoder framework\nas \u0398. For ease of reference, we also use \u03c0\u0398 to represent the translation model with parameter \u0398.\nDenote D as the training dataset that contains source-target sentence pairs. The training process\naims at seeking the optimal parameters \u0398\u2217 to correctly encode source sentence and decode it into the\ntarget sentence. While there are different objectives to achieve this [2, 10, 1, 5, 19, 17], maximum\nlikelihood estimation is the most popular one [2]:\n\n(cid:89)\n(cid:89)\n\n\u0398\u2217 = argmax\n\n\u0398\n\n= argmax\n\n\u0398\n\nP (y|x; \u0398)\n\n(x,y)\u2208D\n\nTy(cid:89)\n\n(x,y)\u2208D\n\nt=1\n\nP (yt|y<t, x; \u0398).\n\n(5)\n\n3 The Myopic Bias\nSince during training, one aims to \ufb01nd the conditional probability P (y|x), ideally in testing, the\ntranslation of a source sentence x should be the target sentence y with the maximum conditional\nprobability P (y|x). However, as there are exponentially many candidates in the target language, one\ncannot compute the probability for every candidate and \ufb01nd the maximum one. Thus, beam search is\nwidely used to \ufb01nd a reasonable good target sentence [4, 14, 12].\nNote that the training objective of NMT is usually de\ufb01ned on the full target sentence y instead of\npartial sentences. One issue with beam search is that a locally good word might not lead to a good\ncomplete sentence. From the example mentioned in the introduction, we can see that such search\nerrors from short sighted actions can provide a bad translation even if we hold a perfect translation\nmodel. We call such errors the myopic bias. To reduce myopic bias, we hope to predict the long-term\nvalue of each action and use the value in decoding, which is the exact motivation of our work.\nThere exist several works weakly related to this issue. [3] develops the scheduled sampling approach,\nwhich takes the generated outputs from the model as well as the golden truth sentence in training,\nto help the model learn from its own errors. Although it can (to some extent) handle the negative\n\n3\n\nh1h2h3htxc1c2c3r3r2r1\u2026x1x2x3xtxy1y2y3\u2026\u2026SM ModuleCC ModuleEncoderDecoderSemantic Matching (SM) Modulec1c2c3r3r2r1\u2026\u2026AttentionContext-Coverage (CC) Modulec1c2c3htxh2h1\u2026\u2026Mean poolingMean pooling\ud835\udf07\ud835\udc46\ud835\udc40Mean poolingMean pooling\ud835\udf07\ud835\udc36\ud835\udc36\fimpact of choosing an incorrect word at middle steps, it still follows beam search during testing,\nwhich cannot avoid the myopic bias. Another related work is [16]. It learns a predictor to predict\nthe ranking score of a certain word at step t, and use this score to replace the conditional probability\noutputted by the NMT model for beam search during testing. Unfortunately, this work still looks\nonly one step forward and cannot address the problem.\n\n4 Value Network for NMT\n\nAs discussed in the previous section, it is not reasonable to fully rely on the conditional probability\nin beam search. This motivates us to estimate the expected performance of any sequence during\ndecoding, which is exactly the concept of value function in reinforcement learning.\n\n4.1 Value Network Structure\n\ny(cid:48)\u2208Y:y(cid:48)\n\n<t=y<t\n\nas the ground truth translation, and then v(x, y<t) =(cid:80)\n\nIn conventional reinforcement learning, a value function describes how much cumulated reward could\nbe collected from state s by following certain policy \u03c0. In machine translation, we can consider any\ninput sentence x paired with partial output sentence y<t as the state, and consider the translation\nmodel \u03c0\u0398 as policy which can generate a word (action) given any state. Given policy \u03c0\u0398, the value\nfunction characterizes what the expected translation performance (e.g. BLEU score) is if we use \u03c0\u0398\nto translate x with the \ufb01rst t \u2212 1 words being y<t. Denote v(x, y<t) as the value function and y\u2217(x)\nBLEU(y\u2217(x), y(cid:48))P (y(cid:48)|x; \u0398),\nwhere Y is the space of complete sentences.\nThe \ufb01rst important problem is how to design the input and the parametric form of the value function.\nAs the translation model is built up on an encoder-decoder framework, we also build up our value\nnetwork on the top of this architecture. To fully exploit the information in the encoder-decoder\nframework, we develop a value network with two new modules, the semantic matching module and\nthe context-coverage module.\nSemantic Matching (SM) Module: In the semantic matching module, at time step t, we use mean\npooling over the decoder RNN hidden states \u00afrt = 1\nl=1 rl as a summarization of the partial target\nt\nsentence, and use mean pooling over context states \u00afct = 1\nl=1 cl as a summarization of the context\nt\nin source language. We concatenate \u00afrt and \u00afct, and use a feed-forward network \u00b5SM = fSM ([\u00afrt, \u00afct])\nto evaluate semantic information between the source sentence and the target sentence.\nContext-Coverage (CC) Module: It is often observed that the more context covered in the attention\nmodel, the better translation we will generate [14, 15]. Thus we build a context-coverage module to\nmeasure the coverage of information used in the encoder-decoder framework. We argue that using\nmean pooling over the context layer and the encoding states should give some effective knowledge.\nSimilarly, denote \u00afh = 1\nl=1 hl, we use another feed-forward network \u00b5CC = fCC([\u00afct, \u00afh]) to\nTx\nprocess such information.\nIn the end, we concatenate both \u00b5SM and \u00b5CC and then use another fully connected layer with\nsigmoid activation function to output a scalar as the value prediction. The whole architecture is shown\nin Figure 1.\n\n(cid:80)Tx\n\n(cid:80)t\n\n(cid:80)t\n\n4.2 Training Data Generation\n\nBased on the designed value network structure, we aim at \ufb01nding a model that can correctly predict the\nperformance after the decoding ends. Popular value function learning algorithms include Monte-Carlo\nmethods and Temple-Difference methods, and both of them have been adopted in many challenging\ntasks [13, 7, 11]. In this paper, we adopt the Monte-Carlo method to learn the value function. Given\na well-learnt NMT model \u03c0\u0398, the training of the value network for \u03c0\u0398 is shown in Algorithm 1. For\nrandomly picked source sentence x in the training corpus, we generate a partial target sentence yp\nusing \u03c0\u0398 with random early stop, i.e., we randomly terminate the decoding process before its end.\nThen for the pair (x, yp), we use \u03c0\u0398 to \ufb01nish the translation starting from yp and obtain a set S(yp)\nof K complete target sentences, e.g., using beam search. In the end, we compute the BLEU score of\n\n4\n\n\feach complete target sentence and calculate the averaged BLEU score of (x, yp):\n\navg_bleu(x, yp) =\n\n1\nK\n\nBLEU(y\u2217(x), y).\n\n(6)\n\n(cid:88)\n\ny\u2208S(yp)\n\navg_bleu(x, yp) can be considered as an estimation of the long-term reward of state (x, yp) used in\nvalue network training.\n\n4.3 Learning\n\nAlgorithm 1 Value network training\n1: Input: Bilingual corpus, a trained neural machine translation model \u03c0\u0398, hyperparameter K.\n2: repeat\n3:\n4:\n5:\n6:\n\nt = t + 1.\nRandomly pick a source sentence x from the training dataset.\nGenerate two partial translations yp,1, yp,2 for x using \u03c0\u0398 with random early stop.\nGenerate K complete translations for each partial translation using \u03c0\u0398 and beam search.\n\nDenote this set of complete target sentences as S(yp,1) and S(yp,2).\n\nCompute the BLEU score for each sentence in S(yp,1) and S(yp,2).\nCalculate the average BLEU score for each partial translation according to Eqn.(6)\nGradient Decent on stochastic loss de\ufb01ned in Eqn.(7).\n\n7:\n8:\n9:\n10: until converge\n11: Output: Value network with parameter \u03c9\n\nIn conventional Monte-Carlo method for value function estimation, people usually use a regression\nmodel to approximate the value function, i.e., learn a mapping from (x, yp) \u2192 avg_bleu(x, yp) by\nminimizing the mean square error (MSE). In this paper, we take an alternative objective function\nwhich is shown to be more effective in experiments. We hope the value network we learn is accurate\nand useful in differentiating good and bad examples. Thus we use pairwise ranking loss instead of\nMSE loss.\nTo be concrete, we sample two partial sentences yp,1 and yp,2 for each x. We hope the predicted\nscore of (x, yp,1) can be larger than that of (x, yp,2) by certain margin if avg_bleu(x, yp,1) >\navg_bleu(x, yp,2). Denote \u03c9 as the parameter of the value function described in Section 4.1. We\ndesign the loss function as follows:\n\n(cid:88)\n\nL(\u03c9) =\n\nev\u03c9(x,yp,2)\u2212v\u03c9(x,yp,1),\n\n(7)\n\nwhere avg_bleu(x, yp,1) > avg_bleu(x, yp,2).\n\n(x,yp,1,yp,2)\n\nvalue network model v(x, y), beam search size K, maximum search depth L, weight \u03b1.\n\nAlgorithm 2 Beam search with value network in NMT\n1: Input: Testing example x, neural machine translation model P (y|x) with target vocabulary V ,\n2: Set S = \u2205, U = \u2205 as candidate sets.\n3: repeat\n4:\n5:\n6:\n\nt log P (y|x) + (1 \u2212 \u03b1) \u00d7 log v(x, y)|y \u2208 Uexpand}\n\nt = t + 1.\nUexpand \u2190 {yi + {w}|yi \u2208 U, w \u2208 V }.\nU \u2190 {top (K \u2212 |S|) candidates that maximize\n\u03b1 \u00d7 1\nUcomplete \u2190 {y|y \u2208 U, yt = EOS}\n7:\nU \u2190 U \\ Ucomplete\n8:\nS \u2190 S \u222a Ucomplete\n9:\n10: until |S| = K or t = L\n11: Output: y = argmaxy\u2208S\u222aU \u03b1 \u00d7 1|y| log P (y|x) + (1 \u2212 \u03b1) \u00d7 log v(x, y)\n\n5\n\n\f4.4\n\nInference\n\nSince the value network estimates the long-term reward of a state, it will be helpful to enhance the\ndecoding process of NMT. For example, in a certain decoding step, the NMT model prefers word\nw1 over w2 according to the conditional probability, but it does not know that picking w2 will be a\nbetter choice for future decoding. As the value network provides suf\ufb01cient information on the future\nreward, if the value network outputs show that picking w2 is better than picking w1, we can take both\nNMT probability and future reward into consideration to choose a better action.\nIn this paper, we simply linearly combine the outputs of the NMT model and the value network,\nwhich is motivated by the success of AlphaGo [11]. We \ufb01rst compute the normalized log probability\nof each candidate, and then linearly combine it with the logarithmic value of the reward. In detail,\ngiven a translation model P (y|x), a value network v(x, y) and a hyperparameter \u03b1 \u2208 (0, 1), the score\nof partial sequence y for x is computed by\n\n\u03b1 \u00d7 1\n\n|y| log P (y|x) + (1 \u2212 \u03b1) \u00d7 log v(x, y),\n\n(8)\n\nwhere |y| is the length of y. The details of the decoding process are presented in Algorithm 2, and we\ncall our neural network-based decoding algorithm NMT-VNN for short.\n\n5 Experiments\n\n5.1 Settings\n\nWe compare our proposed NMT-VNN with two baselines. The \ufb01rst one is classic NMT with beam\nsearch [2] (NMT-BS). The second one [16] trains a predictor that can evaluate the quality of any\npartial sequence, e.g., partial BLEU score 1, and then it uses the predictor to select words instead of\nthe probability. The main difference between [16] and ours is that they predict the local improvement\nof BLEU for any single word, while ours aims at predicting the \ufb01nal BLEU score and use the predicted\nscore to select words. We refer their work as beam search optimization (we call it NMT-BSO). For\nNMT-BS, we directly used the open source code [2]. NMT-BSO was implemented by ourselves\nbased on the open source code [2].\nWe tested our proposed algorithms and the baselines on three pairs of languages: English\u2192French\n(En\u2192Fr), English\u2192German (En\u2192De), and Chinese\u2192English (Zh\u2192En). In detail, we used the same\nbilingual corpora from WMT\u2019 14 as used in [2] , which contains 12M, 4.5M and 10M training data\nfor each task. Following common practices, for En\u2192Fr and En\u2192De, we concatenated newstest2012\nand newstest2013 as the validation set, and used newstest2014 as the testing set. For Zh\u2192En, we\nused NIST 2006 and NIST 2008 datasets for testing, and use NIST 2004 dataset for validation. For\nall datasets in Chinese, we used a public tool for word segmentation. In all experiments, validation\nsets were only used for early-stopping and hyperparameter tuning.\nFor NMT-VNN and NMT-BS, we need to train an NMT model \ufb01rst. We followed [2] to set\nexperimental parameters to train the NMT model. For each language, we constructed the vocabulary\nwith the most common 30K words in the parallel corpora, and out-of-vocabulary words were replaced\nwith a special token \u201cUNK\". Each word was embedded into a vector space of 620 dimensions, and\nthe dimension of the recurrent unit was 1000. We removed sentences with more than 50 words from\nthe training set. Batch size was set as 80 with 20 batches pre-fetched and sorted by sentence lengths.\nThe NMT model was trained with asynchronized SGD on four K40m GPUs for about seven days.\nFor NMT-BSO, we implemented the algorithm and the model was trained in the same environment.\nFor the value network used in NMT-VNN, we set the same parameters for the encoder-decoder\nlayers as the NMT model. Additionally, in the SM module and CC module, we set function \u00b5SM\nand \u00b5CC as single-layer feed forward networks with 1000 output nodes. In Algorithm 1, we set\nthe hyperparameter K = 20 to estimate the value of any partial sequence. We adapted mini-batch\ntraining with batch size to be 80, and the value network model was trained with AdaDelta [21] on\none K40m GPU for about three days.\n\n1If the ground truth is y\u2217, the partial bleu on the partial sequence y<t at step t is de\ufb01ned as the BLEU score\n\non y<t and y\u2217\n<t.\n\n6\n\n\fDuring testing, the hyperparameter \u03b1 for NMT-VNN was set by cross validation. For En\u2192Fr, En\u2192De\nand Zh\u2192En tasks, we found setting \u03b1 to be 0.85, 0.9 and 0.8 respectively are the best choices. We\nused the BLEU score [8] as the evaluation metric, which is computed by the multi-bleu.perl script2.\nWe set the beam search size to be 12 for all the algorithms following the common practice [12].\n\n(a) En\u2192Fr\n\n(b) En\u2192De\n\n(c) Zh\u2192En NIST 2006\n\n(d) Zh\u2192En NIST 2008\n\nFigure 2: Translation results on the test sets of three tasks\n\n5.2 Overall Results\n\nWe report the experimental results in this subsection. From Table 1 we can see that our NMT-VNN\nalgorithm outperforms the baseline algorithms on all tasks.\nFor English\u2192French task and English\u2192German task, NMT-VNN outperforms the baseline NMT-BS\nby about 1.03/1.3 points. As the only difference between the two algorithms is that our NMT-VNN\nadditionally uses the outputs of value network to enhance decoding, we can conclude that such\nadditional knowledge provides useful information to help the NMT model. Our method outperforms\nNMT-BSO by about 0.31/0.33 points. Since NMT-BSO only uses a local BLEU predictor to estimate\nthe partial BLEU score while ours predicts the future performance, our proposed value network which\nconsiders long-term bene\ufb01t is more powerful.\nThe performance of our NMT-VNN is much better\nthan NMT-BS and NMT-BSO for\nChinese\u2192English tasks. NMT-VNN outperforms the baseline NMT-BS by about 1.4/1.82 points on\nNIST 2006 and NIST 2008, and outperforms NMT-BSO by about 1.01/0.72 points. We plot BLEU\nscores with respect to the length of source sentences in Figure 2 for all the tasks. From the \ufb01gures, we\ncan see that our NMT-VNN algorithm outperforms the baseline algorithms in almost all the ranges of\nlength.\nFurthermore, we also test our value network on a deep NMT model in which the encoder and decoder\nare both stacked 4-layer LSTMs. The result also shows that we can get 0.33 points improvement on\nEnglish\u2192French task. These results demonstrate the effectiveness and robustness of our NMT-VNN\nalgorithm.\n\n2https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl. For \ufb01nal eval-\nuation we use corpus-level BLEU, while for the value network training we use sentence-level BLEU as in\n[1].\n\n7\n\n\fTable 1: Overall Performance\n\nEn\u2192Fr En\u2192De Zh\u2192En NIST06 Zh\u2192En NIST08 En\u2192Fr Deep\nNMT-BS\n30.51\nNMT-BSO 31.23\nNMT-VNN 31.54\n\n36.2\n36.59\n37.6\n\n29.4\n30.5\n31.22\n\n15.67\n16.64\n16.97\n\n37.86\n\n\u2013\n\n38.19\n\n5.3 Analysis on Value Network\n\nWe further look into the learnt value network and conduct some analysis to better understand it.\nFirst, as we use an additional component during decoding, it will affect the ef\ufb01ciency of the trans-\nlation process. As the designed value network architecture is similar to the basic NMT model, the\ncomputational complexity is similar to the NMT model and the two processes can be run in parallel.\nSecond, it has been observed that the accuracy of NMT is sometimes very sensitive to the size of\nbeam search on certain tasks. As the beam size grows, the accuracy will drop drastically. [14] argues\nthis is because the training of NMT favors short but inadequate translation candidates. We also\nobserve this phenomenon on English\u2192German translation. However, we show that by using value\nnetwork, such shortage can be largely avoided. We tested the accuracy of our algorithm with different\nbeam sizes, as shown in Figure 3.(a). It can be seen that NMT-VNN is much more stable than the\noriginal NMT without value network: its accuracy only differs a little for different beam sizes while\nNMT-BS drops more than 0.5 point when the beam size is large.\n\n(a)\n\n(b)\n\nFigure 3: (a). BLEU scores of En\u2192De task w.r.t different beam size. (b). BLEU scores of En\u2192De\ntask w.r.t different hyperparameter \u03b1.\n\nThird, we tested the performances of NMT-VNN using different hyperparameter \u03b1 during decoding\nfor English\u2192German task. As can be seen from the \ufb01gure, the performance is stable for the \u03b1 ranging\nfrom 0.7 to 0.95, and slightly drops for a smaller \u03b1. This shows that our proposed algorithm is robust\nto the hyperparameter.\n\n6 Conclusions and Future Work\n\nIn this work we developed a new decoding scheme that incorporates value networks for neural\nmachine translation. By introducing the value network, the new decoding scheme considers not\nonly the local conditional probability of a candidate word, but also its long-term reward for future\ndecoding. Experiments on three translation tasks verify the effectiveness of the new scheme. We plan\nto explore the following directions in the future. First, it is interesting to investigate how to design\nbetter structures for the value network. Second, the idea of using value networks is quite general, and\nwe will extend it to other sequence-to-sequence learning tasks, such as image captioning and dialog\nsystems.\n\n8\n\n\fAcknowledgments\n\nThis work was partially supported by National Basic Research Program of China (973 Program)\n(grant no. 2015CB352502), NSFC (61573026). We would like to thank the anonymous reviewers for\ntheir valuable comments on our paper.\n\nReferences\n[1] D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio. An\n\nactor-critic algorithm for sequence prediction. ICLR, 2017.\n\n[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align\n\nand translate. ICLR, 2015.\n\n[3] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction\nwith recurrent neural networks. In Advances in Neural Information Processing Systems, pages\n1171\u20131179, 2015.\n\n[4] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and\nY. Bengio. Learning phrase representations using rnn encoder\u2013decoder for statistical machine\ntranslation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan-\nguage Processing (EMNLP), pages 1724\u20131734, Doha, Qatar, October 2014. Association for\nComputational Linguistics.\n\n[5] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T. Liu, and W.-Y. Ma. Dual learning for machine\n\ntranslation. In Advances in Neural Information Processing Systems, pages 820\u2013828, 2016.\n\n[6] S. Jean, K. Cho, R. Memisevic, and Y. Bengio. On using very large target vocabulary for\nneural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for\nComputational Linguistics and the 7th International Joint Conference on Natural Language\nProcessing (Volume 1: Long Papers), pages 1\u201310, Beijing, China, July 2015. Association for\nComputational Linguistics.\n\n[7] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves,\nM. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein-\nforcement learning. Nature, 518(7540):529\u2013533, 2015.\n\n[8] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of\nmachine translation. In Proceedings of the 40th annual meeting on association for computational\nlinguistics, pages 311\u2013318. Association for Computational Linguistics, 2002.\n\n[9] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural\n\nnetworks. ICLR, 2016.\n\n[10] S. Shen, Y. Cheng, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. Minimum risk training for neural\n\nmachine translation. In ACL, 2016.\n\n[11] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser,\nI. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural\nnetworks and tree search. Nature, 529(7587):484\u2013489, 2016.\n\n[12] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In\n\nAdvances in neural information processing systems, pages 3104\u20133112, 2014.\n\n[13] G. Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play.\n\nNeural computation, 6(2):215\u2013219, 1994.\n\n[14] Z. Tu, Y. Liu, L. Shang, X. Liu, and H. Li. Neural machine translation with reconstruction. In\n\nAAAI, pages 3097\u20133103, 2017.\n\n[15] Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li. Coverage-based neural machine translation. CoRR,\n\nabs/1601.04811, 2016.\n\n9\n\n\f[16] S. Wiseman and A. M. Rush. Sequence-to-sequence learning as beam-search optimization. In\n\nEMNLP, 2016.\n\n[17] L. Wu, Y. Xia, L. Zhao, F. Tian, T. Qin, J. Lai, and T.-Y. Liu. Adversarial neural machine\n\ntranslation. arXiv preprint arXiv:1704.06933, 2017.\n\n[18] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao,\nK. Macherey, et al. Google\u2019s neural machine translation system: Bridging the gap between\nhuman and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\n[19] Y. Xia, T. Qin, W. Chen, J. Bian, N. Yu, and T.-Y. Liu. Dual supervised learning. In ICML,\n\n2017.\n\n[20] Y. Xia, F. Tian, L. Wu, J. Lin, T. Qin, N. Yu, and T. Liu. Deliberation networks: Sequence\nIn 31st Annual Conference on Neural Information\n\ngeneration beyond one-pass decoding.\nProcessing Systems (NIPS), 2017.\n\n[21] M. D. Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701,\n\n2012.\n\n[22] J. Zhou, Y. Cao, X. Wang, P. Li, and W. Xu. Deep recurrent models with fast-forward\n\nconnections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\n10\n\n\f", "award": [], "sourceid": 145, "authors": [{"given_name": "Di", "family_name": "He", "institution": "Peking University"}, {"given_name": "Hanqing", "family_name": "Lu", "institution": "Zhejiang University"}, {"given_name": "Yingce", "family_name": "Xia", "institution": "University of Science and Technology of China"}, {"given_name": "Tao", "family_name": "Qin", "institution": "Microsoft Research"}, {"given_name": "Liwei", "family_name": "Wang", "institution": "Peking University"}, {"given_name": "Tie-Yan", "family_name": "Liu", "institution": "Microsoft Research"}]}