{"title": "Attention-Based Models for Speech Recognition", "book": "Advances in Neural Information Processing Systems", "page_first": 577, "page_last": 585, "abstract": "Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis and image caption generation. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation reaches a competitive 18.6\\% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18\\% PER in single utterances and 20\\% in 10-times longer (repeated) utterances.  Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6\\% level.", "full_text": "Attention-Based Models for Speech Recognition\n\nJan Chorowski\n\nUniversity of Wroc\u0142aw, Poland\n\njan.chorowski@ii.uni.wroc.pl\n\nDzmitry Bahdanau\n\nJacobs University Bremen, Germany\n\nDmitriy Serdyuk\n\nUniversit\u00b4e de Montr\u00b4eal\n\nKyunghyun Cho\n\nUniversit\u00b4e de Montr\u00b4eal\n\nYoshua Bengio\n\nUniversit\u00b4e de Montr\u00b4eal\nCIFAR Senior Fellow\n\nAbstract\n\nRecurrent sequence generators conditioned on input data through an attention\nmechanism have recently shown very good performance on a range of tasks in-\ncluding machine translation, handwriting synthesis [1, 2] and image caption gen-\neration [3]. We extend the attention-mechanism with features needed for speech\nrecognition. We show that while an adaptation of the model used for machine\ntranslation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the\nTIMIT phoneme recognition task, it can only be applied to utterances which are\nroughly as long as the ones it was trained on. We offer a qualitative explanation of\nthis failure and propose a novel and generic method of adding location-awareness\nto the attention mechanism to alleviate this issue. The new method yields a model\nthat is robust to long inputs and achieves 18% PER in single utterances and 20%\nin 10-times longer (repeated) utterances. Finally, we propose a change to the at-\ntention mechanism that prevents it from concentrating too much on single frames,\nwhich further reduces PER to 17.6% level.\n\n1\n\nIntroduction\n\nRecently, attention-based recurrent networks have been successfully applied to a wide variety of\ntasks, such as handwriting synthesis [1], machine translation [2], image caption generation [3] and\nvisual object classi\ufb01cation [4].1 Such models iteratively process their input by selecting relevant\ncontent at every step. This basic idea signi\ufb01cantly extends the applicability range of end-to-end\ntraining methods, for instance, making it possible to construct networks with external memory [6, 7].\nWe introduce extensions to attention-based recurrent networks that make them applicable to speech\nrecognition. Learning to recognize speech can be viewed as learning to generate a sequence (tran-\nscription) given another sequence (speech). From this perspective it is similar to machine translation\nand handwriting synthesis tasks, for which attention-based methods have been found suitable [2, 1].\nHowever, compared to machine translation, speech recognition principally differs by requesting\nmuch longer input sequences (thousands of frames instead of dozens of words), which introduces a\nchallenge of distinguishing similar speech fragments2 in a single utterance. It is also different from\nhandwriting synthesis, since the input sequence is much noisier and does not have as clear structure.\nFor these reasons speech recognition is an interesting testbed for developing new attention-based\narchitectures capable of processing long and noisy inputs.\nApplication of attention-based models to speech recognition is also an important step toward build-\ning fully end-to-end trainable speech recognition systems, which is an active area of research. The\n\n1An early version of this work was presented at the NIPS 2014 Deep Learning Workshop [5].\n2Explained in more detail in Sec. 2.1.\n\n1\n\n\fdominant approach is still based on hybrid systems consisting of a deep neural acoustic model, a tri-\nphone HMM model and an n-gram language model [8, 9]. This requires dictionaries of hand-crafted\npronunciation and phoneme lexicons, and a multi-stage training procedure to make the components\nwork together. Excellent results by an HMM-less recognizer have recently been reported, with the\nsystem consisting of a CTC-trained neural network and a language model [10]. Still, the language\nmodel was added only at the last stage in that work, thus leaving open a question of how much an\nacoustic model can bene\ufb01t from being aware of a language model during training.\nIn this paper, we evaluate attention-based models on the widely-used TIMIT phoneme recognition\ntask. For each generated phoneme, an attention mechanism selects or weighs the signals produced\nby a trained feature extraction mechanism at potentially all of the time steps in the input sequence\n(speech frames). The weighted feature vector then helps to condition the generation of the next\nelement of the output sequence. Since the utterances in this dataset are rather short (mostly under\n5 seconds), we measure the ability of the considered models in recognizing much longer utterances\nwhich were created by arti\ufb01cially concatenating the existing utterances.\nWe start with a model proposed in [2] for the machine translation task as the baseline. This model\nseems entirely vulnerable to the issue of similar speech fragments but despite our expectations it\nwas competitive on the original test set, reaching 18.7% phoneme error rate (PER). However, its\nperformance degraded quickly with longer, concatenated utterances. We provide evidence that this\nmodel adapted to track the absolute location in the input sequence of the content it is recognizing, a\nstrategy feasible for short utterances from the original test set but inherently unscalable.\nIn order to circumvent this undesired behavior, in this paper, we propose to modify the attention\nmechanism such that it explicitly takes into account both (a) the location of the focus from the\nprevious step, as in [6] and (b) the features of the input sequence, as in [2]. This is achieved by\nadding as inputs to the attention mechanism auxiliary convolutional features which are extracted by\nconvolving the attention weights from the previous step with trainable \ufb01lters. We show that a model\nwith such convolutional features performs signi\ufb01cantly better on the considered task (18.0% PER).\nMore importantly, the model with convolutional features robustly recognized utterances many times\nlonger than the ones from the training set, always staying below 20% PER.\nTherefore, the contribution of this work is three-fold. For one, we present a novel purely neural\nspeech recognition architecture based on an attention mechanism, whose performance is compara-\nble to that of the conventional approaches on the TIMIT dataset. Moreover, we propose a generic\nmethod of adding location awareness to the attention mechanism. Finally, we introduce a modi\ufb01-\ncation of the attention mechanism to avoid concentrating the attention on a single frame, and thus\navoid obtaining less \u201ceffective training examples\u201d, bringing the PER down to 17.6%.\n\n2 Attention-Based Model for Speech Recognition\n\n2.1 General Framework\n\nAn attention-based recurrent sequence generator (ARSG) is a recurrent neural network that stochas-\ntically generates an output sequence (y1, . . . , yT ) from an input x. In practice, x is often processed\nby an encoder which outputs a sequential input representation h = (h1, . . . , hL) more suitable for\nthe attention mechanism to work with.\nIn the context of this work, the output y is a sequence of phonemes, and the input x = (x1, . . . , xL(cid:48))\nis a sequence of feature vectors. Each feature vector is extracted from a small overlapping window\nof audio frames. The encoder is implemented as a deep bidirectional recurrent network (BiRNN),\nto form a sequential representation h of length L = L(cid:48).\nAt the i-th step an ARSG generates an output yi by focusing on the relevant elements of h:\n\n\u03b1i = Attend(si\u22121, \u03b1i\u22121, h)\n\ngi =\n\n\u03b1i,jhj\nyi \u223c Generate(si\u22121, gi),\n\nL(cid:88)\n\nj=1\n\n2\n\n(1)\n\n(2)\n\n(3)\n\n\fFigure 1:\nTwo steps of the pro-\nposed attention-based recurrent se-\nquence generator (ARSG) with a hy-\nbrid attention mechanism (computing\n\u03b1), based on both content (h) and lo-\ncation (previous \u03b1) information. The\ndotted lines correspond to Eq. (1),\nthick solid lines to Eq. (2) and dashed\nlines to Eqs. (3)\u2013(4).\n\nwhere si\u22121 is the (i \u2212 1)-th state of the recurrent neural network to which we refer as the gener-\nator, \u03b1i \u2208 RL is a vector of the attention weights, also often called the alignment [2]. Using the\nterminology from [4], we call gi a glimpse. The step is completed by computing a new generator\nstate:\n\nsi = Recurrency(si\u22121, gi, yi)\n\n(4)\n\nLong short-term memory units (LSTM, [11]) and gated recurrent units (GRU, [12]) are typically\nused as a recurrent activation, to which we refer as a recurrency. The process is graphically illus-\ntrated in Fig. 1.\nInspired by [6] we distinguish between location-based, content-based and hybrid attention mecha-\nnisms. Attend in Eq. (1) describes the most generic, hybrid attention. If the term \u03b1i\u22121 is dropped\nfrom Attend arguments, i.e., \u03b1i = Attend(si\u22121, h), we call it content-based (see, e.g., [2] or [3]).\nIn this case, Attend is often implemented by scoring each element in h separately and normalizing\nthe scores:\n\nei,j = Score(si\u22121, hj),\n\n(cid:44) L(cid:88)\n\n\u03b1i,j = exp(ei,j)\n\nexp(ei,j) .\n\n(5)\n\n(6)\n\nj=1\n\nThe main limitation of such scheme is that identical or very similar elements of h are scored equally\nregardless of their position in the sequence. This is the issue of \u201csimilar speech fragments\u201d raised\nabove. Often this issue is partially alleviated by an encoder such as e.g. a BiRNN [2] or a deep\nconvolutional network [3] that encode contextual information into every element of h . However,\ncapacity of h elements is always limited, and thus disambiguation by context is only possible to a\nlimited extent.\nAlternatively, a location-based attention mechanism computes the alignment from the generator\nstate and the previous alignment only such that \u03b1i = Attend(si\u22121, \u03b1i\u22121). For instance, Graves\n[1] used the location-based attention mechanism using a Gaussian mixture model in his handwriting\nsynthesis model. In the case of speech recognition, this type of location-based attention mechanism\nwould have to predict the distance between consequent phonemes using si\u22121 only, which we expect\nto be hard due to large variance of this quantity.\nFor these limitations associated with both content-based and location-based mechanisms, we argue\nthat a hybrid attention mechanism is a natural candidate for speech recognition. Informally, we\nwould like an attention model that uses the previous alignment \u03b1i\u22121 to select a short list of elements\nfrom h, from which the content-based attention, in Eqs. (5)\u2013(6), will select the relevant ones without\nconfusion.\n\n2.2 Proposed Model: ARSG with Convolutional Features\n\nWe start from the ARSG-based model with the content-based attention mechanism proposed in [2].\nThis model can be described by Eqs. (5)\u2013(6), where\n\nei,j = w(cid:62) tanh(W si\u22121 + V hj + b).\n\n(7)\n\nw and b are vectors, W and V are matrices.\n\n3\n\nyigihj-1hjhj+1gi+1yi+1si+1\u00d7\u03b1i-1,j-1\u00d7\u03b1i-1,j\u00d7\u03b1i-1,j+1si-1si\u00d7\u03b1i,j-1\u00d7\u03b1i,j\u00d7\u03b1i,j+1\u03b1i\fWe extend this content-based attention mechanism of the original model to be location-aware by\nmaking it take into account the alignment produced at the previous step. First, we extract k vectors\nfi,j \u2208 Rk for every position j of the previous alignment \u03b1i\u22121 by convolving it with a matrix\nF \u2208 Rk\u00d7r:\n\nfi = F \u2217 \u03b1i\u22121.\n\nThese additional vectors fi,j are then used by the scoring mechanism ei,j:\nei,j = w(cid:62) tanh(W si\u22121 + V hj + U fi,j + b)\n\n2.3 Score Normalization: Sharpening and Smoothing\n\n(8)\n\n(9)\n\nThere are three potential issues with the normalization in Eq. (6).\nFirst, when the input sequence h is long, the glimpse gi is likely to contain noisy information from\nmany irrelevant feature vectors hj, as the normalized scores \u03b1i,j are all positive and sum to 1. This\nmakes it dif\ufb01cult for the proposed ARSG to focus clearly on a few relevant frames at each time\ni. Second, the attention mechanism is required to consider all the L frames each time it decodes a\nsingle output yi while decoding the output of length T , leading to a computational complexity of\nO(LT ). This may easily become prohibitively expensive, when input utterances are long (and issue\nthat is less serious for machine translation, because in that case the input sequence is made of words,\nnot of 20ms acoustic frames).\nThe other side of the coin is that the use of softmax normalization in Eq. (6) prefers to mostly focus\non only a single feature vector hj. This prevents the model from aggregating multiple top-scored\nframes to form a glimpse gi.\n\n(cid:46)(cid:80)L\n\nSharpening There is a straightforward way to address the \ufb01rst issue of a noisy glimpse by \u201csharp-\nening\u201d the scores \u03b1i,j. One way to sharpen the weights is to introduce an inverse temperature \u03b2 > 1\nto the softmax function such that ai,j = exp(\u03b2ei,j)\nj=1 exp(\u03b2ei,j), or to keep only the top-k\nframes according to the scores and re-normalize them. These sharpening methods, however, still\nrequires us to compute the score of every frame each time (O(LT )), and they worsen the second\nissue, of overly narrow focus.\nWe also propose and investigate a windowing technique. At each time i, the attention mechanism\nconsiders only a subsequence \u02dch = (hpi\u2212w, . . . , hpi+w\u22121) of the whole sequence h, where w (cid:28) L is\nthe prede\ufb01ned window width and pi is the median of the alignment \u03b1i\u22121. The scores for hj /\u2208 \u02dch are\nnot computed, resulting in a lower complexity of O(L + T ). This windowing technique is similar\nto taking the top-k frames, and similarly, has the effect of sharpening.\nThe proposed sharpening based on windowing can be used both during training and evaluation.\nLater, in the experiments, we only consider the case where it is used during evaluation.\n\nSmoothing We observed that the proposed sharpening methods indeed helped with long utter-\nances. However, all of them, and especially selecting the frame with the highest score, negatively\naffected the model\u2019s performance on the standard development set which mostly consists of short\nutterances. This observations let us hypothesize that it is helpful for the model to aggregate selec-\ntions from multiple top-scored frames. In a sense this brings more diversity, i.e., more effective\ntraining examples, to the output part of the model, as more input locations are considered. To facil-\nitate this effect, we replace the unbounded exponential function of the softmax function in Eq. (6)\nwith the bounded logistic sigmoid \u03c3 such that ai,j = \u03c3(ei,j)\nj=1 \u03c3(ei,j). This has the effect of\nsmoothing the focus found by the attention mechanism.\n\n(cid:46)(cid:80)L\n\n3 Related Work\n\nSpeech recognizers based on the connectionist temporal classi\ufb01cation (CTC, [13]) and its extension,\nRNN Transducer [14], are the closest to the ARSG model considered in this paper. They follow\nearlier work on end-to-end trainable deep learning over sequences with gradient signals \ufb02owing\n\n4\n\n\fFigure 2: Decoding performance w.r.t. the beam size. For rigorous comparison, if decoding failed\nto generate (cid:104)eos(cid:105), we considered it wrongly recognized without retrying with a larger beams size.\nThe models, especially with smooth focus, perform well even with a beam width as small as 1.\n\nthrough the alignment process [15]. They have been shown to perform well on the phoneme recog-\nnition task [16]. Furthermore, the CTC was recently found to be able to directly transcribe text from\nspeech without any intermediate phonetic representation [17].\nThe considered ARSG is different from both the CTC and RNN Transducer in two ways. First,\nwhereas the attention mechanism deterministically aligns the input and the output sequences, the\nCTC and RNN Transducer treat the alignment as a latent random variable over which MAP (max-\nimum a posteriori) inference is performed. This deterministic nature of the ARSG\u2019s alignment\nmechanism allows beam search procedure to be simpler. Furthermore, we empirically observe that a\nmuch smaller beam width can be used with the deterministic mechanism, which allows faster decod-\ning (see Sec. 4 and Fig. 2). Second, the alignment mechanism of both the CTC and RNN Transducer\nis constrained to be \u201cmonotonic\u201d to keep marginalization of the alignment tractable. On the other\nhand, the proposed attention mechanism can result in non-monotonic alignment, which makes it\nsuitable for a larger variety of tasks other than speech recognition.\nA hybrid attention model using a convolution operation was also proposed in [6] for neural Turing\nmachines (NTM). At each time step, the NTM computes content-based attention weights which are\nthen convolved with a predicted shifting distribution. Unlike the NTM\u2019s approach, the hybrid mech-\nanism proposed here lets learning \ufb01gure out how the content-based and location-based addressing\nbe combined by a deep, parametric function (see Eq. (9).)\nSukhbaatar et al. [18] describes a similar hybrid attention mechanism, where location embeddings\nare used as input to the attention model. This approach has an important disadvantage that the model\ncannot work with an input sequence longer than those seen during training. Our approach, on the\nother hand, works well on sequences many times longer than those seen during training (see Sec. 5.)\n\n4 Experimental Setup\n\nWe closely followed the procedure in [16]. All experiments were performed on the TIMIT corpus\n[19]. We used the train-dev-test split from the Kaldi [20] TIMIT s5 recipe. We trained on the\nstandard 462 speaker set with all SA utterances removed and used the 50 speaker dev set for early\nstopping. We tested on the 24 speaker core test set. All networks were trained on 40 mel-scale \ufb01lter-\nbank features together with the energy in each frame, and \ufb01rst and second temporal differences,\nyielding in total 123 features per frame. Each feature was rescaled to have zero mean and unit\nvariance over the training set. Networks were trained on the full 61-phone set extended with an\nextra \u201cend-of-sequence\u201d token that was appended to each target sequence. Similarly, we appended\nan all-zero frame at the end of each input sequence to indicate the end of the utterance. Decoding\nwas performed using the 61+1 phoneme set, while scoring was done on the 39 phoneme set.\n\nTraining Procedure One property of ARSG models is that different subsets of parameters are\nreused different number of times; L times for those of the encoder, LT for the attention weights\nand T times for all the other parameters of the ARSG. This makes the scales of derivatives w.r.t.\nparameters vary signi\ufb01cantly. We used an adaptive learning rate algorithm, AdaDelta [21] which\nhas two hyperparameters \u0001 and \u03c1. All the weight matrices were initialized from a normal Gaussian\ndistribution with its standard deviation set to 0.01. Recurrent weights were orthogonalized.\nAs TIMIT is a relatively small dataset, proper regularization is crucial. We used the adaptive weight\nnoise as a main regularizer [22]. We \ufb01rst trained our models with a column norm constraint [23] with\n\n5\n\nlllllllllllllllllllllBaselineConv FeatsSmooth Focus16171819125102050100125102050100125102050100Beam widthPhoneme Error Rate [%]DatasetldevtestDependency of error rate on beam search width.\fFigure 3: Alignments produced by the baseline model. The vertical bars indicate ground truth\nphone location from TIMIT. Each row of the upper image indicates frames selected by the attention\nmechanism to emit a phone symbol. The network has clearly learned to produce a left-to-right\nalignment with a tendency to look slightly ahead, and does not confuse between the repeated \u201ckcl-\nk\u201d phrase. Best viewed in color.\n\nthe maximum norm 1 until the lowest development negative log-likelihood is achieved.3 During this\ntime, \u0001 and \u03c1 are set to 10\u22128 and 0.95, respectively. At this point, we began using the adaptive\nweight noise, with the model complexity cost LC divided by 10, while disabling the column norm\nconstraints. Once the new lowest development log-likelihood was reached, we \ufb01ne-tuned the model\nwith a smaller \u0001 = 10\u221210, until we did not observe the improvement in the development phoneme\nerror rate (PER) for 100K weight updates. Batch size 1 was used throughout the training.\n\nEvaluated Models We evaluated the ARSGs with different attention mechanisms. The encoder\nwas a 3-layer BiRNN with 256 GRU units in each direction, and the activations of the 512 top-layer\nunits were used as the representation h. The generator had a single recurrent layer of 256 GRU units.\nGenerate in Eq. (3) had a hidden layer of 64 maxout units. The initial states of both the encoder\nand generator were treated as additional parameters.\nOur baseline model is the one with a purely content-based attention mechanism (See Eqs. (5)\u2013(7).)\nThe scoring network in Eq. (7) had 512 hidden units. The other two models use the convolutional\nfeatures in Eq. (8) with k = 10 and r = 201. One of them uses the smoothing from Sec. 2.3.\n\nDecoding Procedure A left-to-right beam search over phoneme sequences was used during de-\ncoding [24]. Beam search was stopped when the \u201cend-of-sequence\u201d token (cid:104)eos(cid:105) was emitted. We\nstarted with a beam width of 10, increasing it up to 40 when the network failed to produce (cid:104)eos(cid:105) with\nthe narrower beam. As shown in Fig. 2, decoding with a wider beam gives little-to-none bene\ufb01t.\n\n5 Results\n\nTable 1: Phoneme error rates (PER). The bold-faced PER corresponds to the best error rate with an\nattention-based recurrent sequence generator (ARSG) incorporating convolutional attention features\nand a smooth focus.\n\nModel\n\nBaseline Model\nBaseline + Conv. Features\nBaseline + Conv. Features + Smooth Focus\nRNN Transducer [16]\nHMM over Time and Frequency Convolutional Net [25]\n\nDev\nTest\n15.9% 18.7%\n16.1% 18.0%\n15.8% 17.6%\nN/A\n17.7%\n13.9% 16.7%\n\nAll the models achieved competitive PERs (see Table 1). With the convolutional features, we see\n3.7% relative improvement over the baseline and further 5.9% with the smoothing.\nTo our surprise (see Sec. 2.1.), the baseline model learned to align properly. An alignment pro-\nduced by the baseline model on a sequence with repeated phonemes (utterance FDHC0 SX209) is\npresented in Fig. 3 which demonstrates that the baseline model is not confused by short-range repe-\ntitions. We can also see from the \ufb01gure that it prefers to select frames that are near the beginning or\n\n3 Applying the weight noise from the beginning of training caused severe under\ufb01tting.\n\n6\n\nh#maykclkelkclkahlerdcldhixbclbehdcldruxmwaolwixthkclkreyaansh#FDHC0_SX209: Michael colored the bedroom wall with crayons.\fFigure 4: Results of force-aligning the concatenated utterances. Each dot represents a single utter-\nance created by either concatenating multiple copies of the same utterance, or of different, randomly\nchosen utterances. We clearly see that the highest robustness is achieved when the hybrid attention\nmechanism is combined with the proposed sharpening technique (see the bottom-right plot.)\n\neven slightly before the phoneme location provided as a part of the dataset. The alignments produced\nby the other models were very similar visually.\n\n5.1 Forced Alignment of Long Utterances\n\nThe good performance of the baseline model led us to the question of how it distinguishes between\nrepetitions of similar phoneme sequences and how reliably it decodes longer sequences with more\nrepetitions. We created two datasets of long utterances; one by repeating each test utterance, and the\nother by concatenating randomly chosen utterances. In both cases, the waveforms were cross-faded\nwith a 0.05s silence inserted as the \u201cpau\u201d phone. We concatenated up to 15 utterances.\nFirst, we checked the forced alignment with these longer utterances by forcing the generator to emit\nthe correct phonemes. Each alignment was considered correct if 90% of the alignment weight lies\ninside the ground-truth phoneme window extended by 20 frames on each side. Under this de\ufb01nition,\nall phones but the (cid:104)eos(cid:105) shown in Fig. 3 are properly aligned.\nThe \ufb01rst column of Fig. 4 shows the number of correctly aligned frames w.r.t. the utterance length\n(in frames) for some of the considered models. One can see that the baseline model was able to\ndecode sequences up to about 120 phones when a single utterance was repeated, and up to about 150\nphones when different utterances were concatenated. Even when it failed, it correctly aligned about\n50 phones. On the other hand, the model with the hybrid attention mechanism with convolutional\nfeatures was able to align sequences up to 200 phones long. However, once it began to fail, the\nmodel was not able to align almost all phones. The model with the smoothing behaved similarly to\nthe one with convolutional features only.\nWe examined failed alignments to understand these two different modes of failure. Some of the\nexamples are shown in the Supplementary Materials.\nWe found that the baseline model properly aligns about 40 \ufb01rst phones, then makes a jump to the\nend of the recording and cycles over the last 10 phones. This behavior suggests that it learned to\ntrack its approximate location in the source sequence. However, the tracking capability is limited to\nthe lengths observed during training. Once the tracker saturates, it jumps to the end of the recording.\nIn contrast, when the location-aware network failed it just stopped aligning \u2013 no particular frames\nwere selected for each phone. We attribute this behavior to the issue of noisy glimpse discussed in\nSec. 2.3. With a long utterance there are many irrelevant frames negatively affecting the weight as-\nsigned to the correct frames. In line with this conjecture, the location-aware network works slightly\nbetter on the repetition of the same utterance, where all frames are somehow relevant, than on the\nconcatenation of different utterances, where each misaligned frame is irrelevant.\nTo gain more insight we applied the alignment sharpening schemes described in Sec. 2.3. In the\nremaining columns of Fig. 4, we see that the sharpening methods help the location-aware network\nto \ufb01nd proper alignments, while they show little effect on the baseline network. The windowing\n\n7\n\n\fFigure 5:\nPhoneme error rates obtained on decoding long sequences. Each network was de-\ncoded with alignment sharpening techniques that produced proper forced alignments. The proposed\nARSG\u2019s are clearly more robust to the length of the utterances than the baseline one is.\n\ntechnique helps both the baseline and location-aware networks, with the location-aware network\nproperly aligning nearly all sequences.\nDuring visual inspection, we noticed that in the middle of very long utterances the baseline model\nwas confused by repetitions of similar content within the window, and that such confusions did not\nhappen in the beginning. This supports our conjecture above.\n\n5.2 Decoding Long Utterances\n\nWe evaluated the models on long sequences. Each model was decoded using the alignment sharp-\nening techniques that helped to obtain proper forced alignments. The results are presented in Fig. 5.\nThe baseline model fails to decode long utterances, even when a narrow window is used to constrain\nthe alignments it produces. The two other location-aware networks are able to decode utterances\nformed by concatenating up to 11 test utterances. Better results were obtained with a wider window,\npresumably because it resembles more the training conditions when at each step the attention mech-\nanism was seeing the whole input sequence. With the wide window, both of the networks scored\nabout 20% PER on the long utterances, indicating that the proposed location-aware attention mecha-\nnism can scale to sequences much longer than those in the training set with only minor modi\ufb01cations\nrequired at the decoding stage.\n\n6 Conclusions\n\nWe proposed and evaluated a novel end-to-end trainable speech recognition architecture based on a\nhybrid attention mechanism which combines both content and location information in order to select\nthe next position in the input sequence for decoding. One desirable property of the proposed model\nis that it can recognize utterances much longer than the ones it was trained on. In the future, we\nexpect this model to be used to directly recognize text from speech [10, 17], in which case it may\nbecome important to incorporate a monolingual language model to the ARSG architecture [26].\nThis work has contributed two novel ideas for attention mechanisms: a better normalization ap-\nproach yielding smoother alignments and a generic principle for extracting and using features from\nthe previous alignments. Both of these can potentially be applied beyond speech recognition. For\ninstance, the proposed attention can be used without modi\ufb01cation in neural Turing machines, or by\nusing 2\u2013D convolution instead of 1\u2013D, for improving image caption generation [3].\n\nAcknowledgments\n\nAll experiments were conducted using Theano [27, 28], PyLearn2 [29], and Blocks [30] libraries.\nThe authors would like to acknowledge the support of the following agencies for research funding\nand computing support: National Science Center (Poland) grant Sonata 8 2014/15/D/ST6/04402,\nNSERC, Calcul Qu\u00b4ebec, Compute Canada, the Canada Research Chairs and CIFAR. D. Bahdanau\nalso thanks Planet Intelligent Systems GmbH and Yandex.\n\n8\n\nBaselineConv FeatsSmooth Focuslllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll18202224369369369Number of repetitionsPhoneme error rate [%]Decoding algorithmllKeep 20Keep 50Win\u201375Win\u2013150DatasetMixed Utt.Same Utt.Phoneme error rates on long utterances\fReferences\n[1] A. Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850, August 2013.\n[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate.\n\nIn Proc. of the 3rd ICLR, 2015. arXiv:1409.0473.\n\n[3] K. Xu, J. Ba, R. Kiros, et al. Show, attend and tell: Neural image caption generation with visual attention.\n\nIn Proc. of the 32nd ICML, 2015. arXiv:1502.03044.\n\n[4] V. Mnih, N. Heess, A. Graves, et al. Recurrent models of visual attention. In Proc. of the 27th NIPS,\n\n2014. arXiv:1406.6247.\n\n[5] J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio. End-to-end continuous speech recognition using\n\nattention-based recurrent NN: First results. arXiv:1412.1602 [cs, stat], December 2014.\n[6] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv:1410.5401, 2014.\n[7] J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv:1410.3916, 2014.\n[8] M. Gales and S. Young. The application of hidden markov models in speech recognition. Found. Trends\n\nSignal Process., 1(3):195\u2013304, January 2007.\n\n[9] G. Hinton, L. Deng, D. Yu, et al. Deep neural networks for acoustic modeling in speech recognition: The\nshared views of four research groups. IEEE Signal Processing Magazine, 29(6):82\u201397, November 2012.\n[10] A. Hannun, C. Case, J. Casper, et al. Deepspeech: Scaling up end-to-end speech recognition.\n\narXiv:1412.5567, 2014.\n\n[11] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural. Comput., 9(8):1735\u20131780, 1997.\n[12] K. Cho, B. van Merrienboer, C. Gulcehre, et al. Learning phrase representations using RNN encoder-\n\ndecoder for statistical machine translation. In EMNLP, October 2014. to appear.\n\n[13] A. Graves, S. Fern\u00b4andez, F. Gomez, and J. Schmidhuber. Connectionist temporal classi\ufb01cation: Labelling\n\nunsegmented sequence data with recurrent neural networks. In Proc. of the 23rd ICML-06, 2006.\n\n[14] A. Graves. Sequence transduction with recurrent neural networks. In Proc. of the 29th ICML, 2012.\n[15] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient based learning applied to document recognition.\n\nProc. IEEE, 1998.\n\n[16] A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In\n\nICASSP 2013, pages 6645\u20136649. IEEE, 2013.\n\n[17] A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In Proc.\n\nof the 31st ICML, 2014.\n\n[18] S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. Weakly supervised memory networks.\n\narXiv:1503.08895, 2015.\n\n[19] J. S. Garofolo, L. F. Lamel, W. M. Fisher, et al. DARPA TIMIT acoustic phonetic continuous speech\n\ncorpus, 1993.\n\n[20] D. Povey, A. Ghoshal, G. Boulianne, et al. The kaldi speech recognition toolkit. In Proc. ASRU, 2011.\n[21] M. D. Zeiler. ADADELTA: An adaptive learning rate method. arXiv:1212.5701, 2012.\n[22] A. Graves. Practical variational inference for neural networks. In Proc of the 24th NIPS, 2011.\n[23] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov.\n\nImproving neural\n\nnetworks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012.\n\n[24] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Proc. of\n\nthe 27th NIPS, 2014. arXiv:1409.3215.\n\n[25] L. T\u00b4oth. Combining time-and frequency-domain convolution in convolutional neural network-based\n\nphone recognition. In Proc. ICASSP, 2014.\n\n[26] C. Gulcehre, O. Firat, K. Xu, et al. On using monolingual corpora in neural machine translation.\n\narXiv:1503.03535, 2015.\n\n[27] J. Bergstra, O. Breuleux, F. Bastien, et al. Theano: a CPU and GPU math expression compiler. In Proc.\n\nSciPy, 2010.\n\n[28] F. Bastien, P. Lamblin, R. Pascanu, et al. Theano: new features and speed improvements. Deep Learning\n\nand Unsupervised Feature Learning NIPS 2012 Workshop, 2012.\n\n[29] I. J. Goodfellow, D. Warde-Farley, P. Lamblin, et al. Pylearn2: a machine learning research library. arXiv\n\npreprint arXiv:1308.4214, 2013.\n\n[30] B. van Merri\u00a8enboer, D. Bahdanau, V. Dumoulin, et al. Blocks and fuel: Frameworks for deep learning.\n\narXiv:1506.00619 [cs, stat], June 2015.\n\n9\n\n\f", "award": [], "sourceid": 410, "authors": [{"given_name": "Jan", "family_name": "Chorowski", "institution": "University of Wroclaw"}, {"given_name": "Dzmitry", "family_name": "Bahdanau", "institution": "Jacobs University, Germany"}, {"given_name": "Dmitriy", "family_name": "Serdyuk", "institution": "Universit\u00e9 de Montr\u00e9al"}, {"given_name": "Kyunghyun", "family_name": "Cho", "institution": "NYU"}, {"given_name": "Yoshua", "family_name": "Bengio", "institution": "U. Montreal"}]}