{"title": "Connectionist Temporal Classification with Maximum Entropy Regularization", "book": "Advances in Neural Information Processing Systems", "page_first": 831, "page_last": 841, "abstract": "Connectionist Temporal Classification (CTC) is an objective function for end-to-end sequence learning, which adopts dynamic programming algorithms to directly learn the mapping between sequences. CTC has shown promising results in many sequence learning applications including speech recognition and scene text recognition. However, CTC tends to produce highly peaky and overconfident distributions, which is a symptom of overfitting. To remedy this, we propose a regularization method based on maximum conditional entropy which penalizes peaky distributions and encourages exploration. We also introduce an entropy-based pruning method to dramatically reduce the number of CTC feasible paths by ruling out unreasonable alignments. Experiments on scene text recognition show that our proposed methods consistently improve over the CTC baseline without the need to adjust training settings. Code has been made publicly available at: https://github.com/liuhu-bigeye/enctc.crnn.", "full_text": "Connectionist Temporal Classi\ufb01cation with\n\nMaximum Entropy Regularization\n\nHu Liu Sheng Jin Changshui Zhang\n\nInstitute for Arti\ufb01cial Intelligence, Tsinghua University (THUAI)\n\nBeijing National Research Center for Information Science and Technology (BNRist)\n\nState Key Lab of Intelligent Technologies and Systems\n\nDepartment of Automation, Tsinghua University, Beijing, P.R.China\n\n{liuhu15, js17}@mails.tsinghua.edu.cn\n\nzcs@mail.tsinghua.edu.cn\n\nAbstract\n\nConnectionist Temporal Classi\ufb01cation (CTC) is an objective function for end-to-\nend sequence learning, which adopts dynamic programming algorithms to directly\nlearn the mapping between sequences. CTC has shown promising results in\nmany sequence learning applications including speech recognition and scene text\nrecognition. However, CTC tends to produce highly peaky and overcon\ufb01dent\ndistributions, which is a symptom of over\ufb01tting. To remedy this, we propose a\nregularization method based on maximum conditional entropy which penalizes\npeaky distributions and encourages exploration. We also introduce an entropy-\nbased pruning method to dramatically reduce the number of CTC feasible paths by\nruling out unreasonable alignments. Experiments on scene text recognition show\nthat our proposed methods consistently improve over the CTC baseline without\nthe need to adjust training settings. Code has been made publicly available at:\nhttps://github.com/liuhu-bigeye/enctc.crnn.\n\n1\n\nIntroduction\n\nPast few years have witnessed signi\ufb01cant progress in sequence learning tasks. Currently recurrent\nneural network (RNN) with Connectionist Temporal Classi\ufb01cation (CTC) [5] has become a popular\nframework and is widely used in areas such as speech recognition [6, 8, 1, 2, 22], sign language\nrecognition [4], video segmentation [10, 18] and scene text recognition [29, 19, 7]. CTC views the\noutputs of RNN as a probability distribution over all possible alignments and directly learns the\nmapping from input sequences to target sequences. It is proven to be effective in weakly supervised\nsequence modeling with only temporal order supervision but no alignment information provided.\nCTC can be regarded as a kind of Multiple Instance Learning (MIL) [21]. From the perspective of\nMIL, the label sequence is a bag containing all feasible paths. CTC learns by maximum likelihood\nestimation (MLE) over the summation of all feasible path probabilities. However, as the number of\nfeasible paths grows exponentially with the input sequence length, it is hard for CTC to \ufb01nd the most\nsuitable one. More seriously, once CTC \ufb01nds a dominant feasible path during the training process,\nthe error signal will concentrate on the vicinity of this path, and the prediction of this feasible path\nwill continuously strengthen until this path completely dominates the prediction output. As blanks are\nincluded in most of the feasible paths, dominant paths are often overwhelmed by blanks, interspersed\nby sharp spikes (narrow regions along the time axis) of non-blank labels, which is known as the CTC\npeaky distribution problem [5, 22].\nThis problem of CTC will lead to the following consequences.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fexploration and is prone to fall into worse local minima.\n\n\u2022 Harm the training process. The positive feedback-like error signal means CTC lacks\n\u2022 Output overcon\ufb01dent paths. CTC tends to concentrate all its output distribution over\none speci\ufb01c path. On the one hand, it is not suitable to handle the situation where the\nsegmentation boundary is ambiguous, e.g. adjacent syllables in speech recognition and\naction switching in continuous sign language recognition. On the other hand, the low-entropy\noutput distribution is a symptom of over\ufb01tting [31], leading to low prediction accuracy.\n\u2022 Output paths with peaky distribution. The peaky distribution is not desirable for sequence\nsegmentation tasks when the model needs to densely predict labels for each time-step.\nEven if only the temporal order of labels is required, learning the correct segmentation will\nimprove model generalization and interpretation abilities.\n\nMotivated by the maximum entropy principle [13], we propose a maximum conditional entropy based\nregularization for CTC (EnCTC). We consider the conditional distribution of feasible paths given the\ninput sequence and label sequence. EnCTC prevents the entropy of feasible paths from decreasing\ntoo fast, thus alleviating the impact of CTC\u2019s positive feedback-like error signal and encouraging\nexploration during training. It also prevents the probability from being dominated by a single path\nand solves the peaky distribution problem. It mitigates over\ufb01tting and is more suitable for depicting\nambiguous segmentation boundaries.\nWe further consider another solution to the problem of \ufb01nding the reasonable feasible path\u2014limit\nthe size of the feasible set. We observe that in many sequence learning tasks, the intervals between\nthe adjacent elements in the input sequence are almost the same, e.g. characters in text recognition,\nsyllables in speech recognition and gestures in continuous sign language recognition. We summarize\nthis phenomenon as the equal spacing prior. Therefore, we propose an algorithm to limit the size\nof the CTC feasible set by eliminating these unreasonable paths that seriously violate the equal\nspacing prior (EsCTC). Moreover, the equal spacing prior can give theoretical explanations from the\nperspective of maximum entropy, which indicates that equal spacing is the best prior without any\nadditional subjective assumptions.\nThe main contributions of this paper can be summarized as: (1) We propose a maximum conditional\nentropy regularization for CTC (EnCTC), which encourages exploration for CTC training and\nprevents peaky output distributions. (2) We derive from equal spacing prior a pruning algorithm\n(EsCTC) to effectively limit the size of CTC feasible set and give theoretical explanations from the\nperspective of maximum entropy. (3) We provide polynomial-time dynamic programming algorithms\nfor calculating EnCTC, EsCTC and their combination (EnEsCTC). (4) We validate the proposed\nmethods on scene text recognition tasks and show that these methods are able to improve the baseline\nmodel without changing training settings.\n\n2 Related Work\n\nCTC [5] is a popular framework for end-to-end sequence learning tasks, such as speech recogni-\ntion [6, 8, 1, 2, 22], scene text recognition [29, 19, 7], sign language recognition [4] and video\nsegmentation [10, 18]. However, CTC tends to output highly peaky distribution [5, 22], which is a\nsign for model overcon\ufb01dence. To remedy this, some intuitive smoothing methods are proposed. [32]\nintroduces path sampling to speed up CTC training, which has a side-effect of reducing the posterior\nspikiness. [18, 22] smooth the estimation of path label priors by discounting the majority number\nof \"background\" labels and increasing the counts of rare actions, then normalizing the posterior\ndistribution using the priors when decoding. However, their smoothing methods are performed during\ndecoding and do not affect the training process.\nModel overcon\ufb01dence is a universal problem in the \ufb01elds of machine learning and several regulariza-\ntion approaches have been proposed to handle it. Weight decay [16] regularize by limiting the range\nof parameters; Dropout [9] and DropConnect [33] by adding noise to the model structure; Stochastic\nPooling [37] by adding noises to the pooling operations. Their methods regularize model parameters,\nbut do not directly regularize the output distributions.\nThe maximum entropy based regularization [23] has long been investigated in the literature to\nregularize model behavior. The maximum entropy estimation [13] involves no additional assumptions\nwhen estimating the distribution. It assigns the positive weight to every possible situation and produces\n\n2\n\n\fthe maximum entropy predictions under certain constraints. In reinforcement learning, [36, 25], the\nmaximum entropy regularization is proposed to encourage exploration and prevents early convergence.\nIn supervised learning, [27] proposes to penalize the entropy of high-con\ufb01dence output softmax\ndistributions by adding negative entropy to the objective function.\nThis work is mostly related to those CTC based methods that use regularization to reduce the over-\ncon\ufb01dent prediction. In [3], label smoothing [31] and increasing temperature of the SoftMax function\nare employed to improve beam search. [15] simply adds con\ufb01dence penalty regularization term [27]\nto regularize the output distribution. To some extent, their approaches improve generalization and\nreduce the peaky distribution. However label smoothing [31] and con\ufb01dence penalty [27] regularize\nthe model prediction at each time-step, which corresponds to regularize both feasible and invalid\npaths. We instead regularize the entropy among the feasible paths of CTC to handle peaky distribution\nproblem. Alignment constraints on CTC paths have been previously explored in [28, 32]. We further\npropose an entropy based pruning method to reduce the searching space of possible alignments and\nfacilitate convergence.\n\n3 Method\n\n3.1 Problem De\ufb01nition\n\nThroughout the paper, the sequence learning problem is de\ufb01ned as follows:\n\n\u2022 The dataset consists of pairs of input sequences X and corresponding target sequences l.\n\u2022 Each element of the target sequence is de\ufb01ned in a \ufb01xed-length label alphabet L.\n\u2022 Each input sequence X should be longer than its corresponding target sequence l.\n\u2022 The alignment between X and l is unknown but in a sequential manner.\n\n3.2 Connectionist Temporal Classi\ufb01cation (CTC)\n\nCTC [5] is a popular method for sequence learning. It enables the end-to-end model training with no\npre-de\ufb01ned alignment information required. In the framework of CTC, given an input sequence X1:T\nof length T , the model predicts a sequence y1:T of length T , where yt denotes the probability vector\nof observing labels over the \ufb01xed-length label alphabet L(cid:48). L(cid:48) = L \u222a \u2205 contains all the pre-de\ufb01ned\nlabels including a \u2019blank\u2019 label \u2205 at time-step t. We call the concatenation of observed labels at all\ntime-steps as a path \u03c0.\nIn order to access the relationship between path \u03c0 and target sequence l, CTC de\ufb01nes a many-to-one\nmapping operation B. B \ufb01rstly removes the repeated labels then removes all blanks from the given\npath. Given a label sequence l, we de\ufb01ne feasible paths as all those \u03c0 that can be mapped onto l\nthrough B. The conditional probability of a given target sequence is de\ufb01ned as the sum of probabilities\nof all feasible paths.\n\np(l|X1:T ) =\n\np(\u03c0|X1:T ),\n\n(cid:88)\n\n\u03c0\u2208B\u22121(l)\n\nwhere the probability of \u03c0 is de\ufb01ned as\n\np(\u03c0|X1:T ) =\n\nT(cid:89)\n\nt=1\n\n,\u2200\u03c0 \u2208 L(cid:48)T .\n\nyt\n\u03c0t\n\n(1)\n\n(2)\n\n(3)\n\nCTC guides the end-to-end model training by directly optimizing the loss function Lctc =\n\u2212 log p(l|X1:T ). It uses dynamic programming to ef\ufb01ciently sum up all the feasible paths.\nHowever, due to the massive amount of feasible path in Equation 1, directly optimizing CTC loss\nmay not result in a good alignment. More speci\ufb01cally, the error signal of CTC loss with respect to yt\nk\nis computed as:\n\n= \u2212\n\n\u2202Lctc\n\u2202yt\nk\n\n1\n\np(l|X)yt\n\nk\n\n(cid:88)\n\np(\u03c0|X).\n\n{\u03c0|\u03c0\u2208B\u22121(l),\u03c0t=k}\n\n3\n\n\fWe can see that the error signal is proportional to the fraction of all feasible paths that go through\nsymbol k at time t. That means once a feasible path is dominant, the error signal of yt\n\u03c0t will\ndominate yt at all time-steps t, causing all the probabilities to focus on a single path while ignoring\nits alternatives. This positive feedback-like error signal makes CTC lack exploration during training\nand tend to over\ufb01t.\n\n3.3 Maximum Conditional Entropy Regularization for CTC (EnCTC)\n\nIn order to facilitate training and be more likely to \ufb01nd the feasible paths, we propose a regularization\nmethod based on maximum conditional entropy (EnCTC). EnCTC constitutes an entropy-based\nregularization term that prevents the entropy of the feasible paths from decreasing too fast, leading to\nbetter generalization and exploration.\n\nLenctc = Lctc \u2212 \u03b2H(p(\u03c0|l, X)),\n\nwhere \u03b2 controls the strength of the maximum conditional entropy regularization.\nThe entropy of the feasible paths given input sequence X and target sequence l is de\ufb01ned as:\n\nH(p(\u03c0|l, X)) = \u2212 (cid:88)\n\np(\u03c0|X, l) log p(\u03c0|X, l)\n\n\u03c0\u2208B\u22121(l)\n\n= \u2212 1\n\np(l|X)\n\n(cid:88)\n\n\u03c0\u2208B\u22121(l)\n\np(\u03c0|X) log p(\u03c0|X) + log p(l|X).\n\n(4)\n\n(5)\n\n=\n\nQ(l)\np(l|X)yt\n\nk\n\n(\n\nFast convergence to one feasible path means that the entropy term of Equation 5 reduces rapidly.\nMore speci\ufb01cally, the error signal of entropy regularization term \u2212H(p(\u03c0|l, X)) with respect to yt\nis computed as:\n\u2202\u2212H(p(\u03c0|l, X))\n\n(cid:80){\u03c0|\u03c0\u2208B\u22121(l),\u03c0t=k} p(\u03c0|X) log p(\u03c0|X)\n\n(cid:80){\u03c0|\u03c0\u2208B\u22121(l),\u03c0t=k} p(\u03c0|X)\n\nk\n\n\u2212\n\np(l|X)\n\n),\n\nQ(l)\n\n\u2202yt\nk\n\nwhere Q(l) =(cid:80)\n\n(6)\n\u03c0\u2208B\u22121(l) p(\u03c0|X) log p(\u03c0|X). We can see from Equation 6 that this error signal is\nproportional to the fraction of p(\u03c0|X) log p(\u03c0|X) minus the fraction of p(\u03c0|X) for all feasible paths\nthat go through symbol k at time t. Notice that these two fractions are zero when p(\u03c0|X) = 0, but\np(\u03c0|X) log p(\u03c0|X) decreases more rapidly around zero and reaches its minimum when p(\u03c0|X) =\ni.e. paths near the dominant path,\n1/e. Therefore, paths with probability between 0 and 1/e,\ncontribute the most to the error signal. This error signal will in turn increase the probability of the\nnearby paths and improve the exploration during training.\n\n3.4 Equal Spacing CTC (EsCTC)\n\nAs discussed previously, CTC is prone to output degenerated paths due to large searching space. In\nthis section, we present a pruning method to dramatically reduce the searching space of possible\nalignments.\nWe observe that in many sequence learning tasks, such as scene text recognition, the spacing of two\nconsecutive elements (or the width of elements) is nearly the same. We assume that this property\nwill hold for all reasonable alignments. We thus propose to measure the spacing equality and set a\nthreshold to rule out those unreasonable alignments. This protects our model from distractions and\nhelps convergence.\nHere we provide some theoretical demonstration of the equal spacing prior based on maximum\nentropy over segmentation.\nGiven an input sequence X1:T of length T and a label sequence l, de\ufb01ne the segmentation sequence\nz1:|l| splitting X into a series of short segments. Each segment corresponds to a duration zs and\ns=1 zs \u2264 T (with an optional all-blank suf\ufb01x). Sequence Z consists of the starting\ni<s zi. Feasible paths that map onto l through B can\nbe split into feasible paths satisfying different segmentation sequence z, with \u03c0Zs+1\u22121 = ls and\n\nlabel ls,(cid:80)|l|\npositions of each segment, where Zs = (cid:80)\n\n4\n\n\fz (l). Therefore, we have \u02d9(cid:83)\n\nB(\u03c0Zs:Zs+1 ) = ls for each segment. We denote feasible paths that satisfy the segmentation sequence\nz as B\u22121\ndisjoint subsets.\nThe entropy of feasible paths satisfying a speci\ufb01c segmentation sequence is de\ufb01ned as\n\nz (l) = B\u22121(l), where \u02d9(cid:83) is the union operation on pairwise\n\nzB\u22121\n\nH(p(\u03c0|z, l, X)) = \u2212 (cid:88)\n\np(\u03c0|z, l, X) log p(\u03c0|z, l, X).\n\n(7)\n\n\u03c0\u2208B\u22121\n\nz (l)\n\nTheorem 3.1. Among all segmentation sequences, the equal spacing one has the maximum entropy.\n(8)\n\nH(p(\u03c0|z, l, X)) = zes.\n\nargmax\n\nmax\n\nz\n\np\n\nProof. As to the de\ufb01nition of B\u22121\nz , the feasible path of each segment must start with several blanks,\nfollowed by the element corresponding to that segment more than once. Therefore, we can \ufb01nd the\nmaximum of entropy for every segment\n\nH(p(\u03c0Zs:Zs+1\u22121|z, l, X)) = log zs.\n\nmax\n\np\n\n(9)\n\nAccording to Equation 2 and the de\ufb01nition of segmentation, the entropy of feasible paths satisfying a\nspeci\ufb01c segmentation sequence can be disassembled into the summation of the entropy of segments.\n\nH(p(\u03c0|z, l, X)) =\n\nmax\n\np\n\n|l|(cid:88)\n\ns=1\n\n= log\n\nH(p(\u03c0Zs:Zs+1\u22121|z, l, X))\n\n(10)\n\nmax\n\np\n\n|l|(cid:89)\n\ns=1\n\nzs.\n\nWith the constraint(cid:80)|l|\n\ntion maximized the entropy term.\n\ns=1 zs \u2264 T , it can be derived from Jensen inequality that equidistant segmenta-\n\nIn the case of strict equal spacing, the length of each element in the segmentation sequence is equal\nand sums to T . As the real-world sequence learning tasks do not strictly satisfy the equal-spacing\nconstraints, we introduce a slight relaxation by requiring each segment being no longer than \u03c4\n(1 \u2264 \u03c4 \u2264 |l|) times the average segmentation length, including the remaining suf\ufb01x.\n\nC\u03c4,T = {z|T \u2212 \u03c4\n\nT\n\n|l| \u2264\n\nzs \u2264 T, zs \u2264 \u03c4\n\nT\n\n|l|}.\n\n(11)\n\n|l|(cid:88)\n\ns=1\n\nThus the Equal Spacing CTC deformation (EsCTC) guides the end-to-end model training by optimiz-\ning the loss function:\n\nLesctc = \u2212 log p\u03c4 (l|X)\n\n= \u2212 log\n\n(cid:88)\n\n(cid:88)\n\nz\u2208C\u03c4,T\n\n\u03c0\u2208B\u22121\n\nz (l)\n\np(\u03c0|X).\n\n(12)\n\nWe consider the EsCTC as complementary to EnCTC that explicitly rules out all unreasonable\nalignments. Therefore, we also combined the Equal Spacing CTC deformation with maximum\nconditional entropy regularization de\ufb01ned in section 3.3.\n\nLenesctc = Lesctc \u2212 \u03b2H(p\u03c4 (\u03c0|l, X)).\n\n(13)\n\n3.5 Algorithm and Complexity Analysis\n\nEnCTC, EsCTC and EnEsCTC can be ef\ufb01ciently calculated by dynamic programming. Due to space\nlimit, the details of the dynamic programming are presented in the supplementary material.\n\n5\n\n\fParallelizing all independent computing units, the time complexity of CTC and EnCTC forward-\nbackward dynamic programming is O(T ). The time complexity of EsCTC and EnEsCTC is O( T 2\u03c4|l| ),\nwhich is between O( T 2\n|l| ) and O(T 2) depending on \u03c4. The space complexity of CTC and EnCTC is\nO(T|l|) since forward and backward variable are kept for gradient computing. The space complexity\nof EsCTC and EnEsCTC is O(T 2|l|).\n\n4 Experiments\n\nWe evaluate our proposed method on several standard benchmarks for scene text recognition tasks.\nWe use CRNN [29] as our baseline model. CRNN trains a fully-convolutional network (FCN)\nwith 2 bidirectional LSTM layers on the top. The model is optimized by CTC loss. For all the\nexperiments, we use the same training settings and parameters as CRNN [29], except for the proposed\nregularization term.\n\n4.1 Datasets and Evaluation Metrics\n\nWhen compared with the state-of-the-art approaches, we follow the standard experimental settings [29,\n19, 17, 30] to train on the synthetic dataset (Synth90K) [11] once and test on four challenging real-\nworld benchmarks for scene text recognition without \ufb01ne-tuning. In such experimental settings,\nthe model is purely trained with synthetic text data, but tested on real-world scene text datasets,\nwhich requires generalization across datasets. The four real-world benchmarks include ICDAR-2003\n(IC03) [20], ICDAR-2013 (IC13) [14], IIIT5k-word (IIIT5k) [24] and Street View Text (SVT) [34]\ndatasets. Synth90K [11] consists of 8M training images and 1M testing images generated by a\nsynthetic data engine. ICDAR-2003 (IC03) [20] test set consists of 251 full scene images and 860\ncropped image patches containing words. We follow the standard evaluation protocol as [34, 11, 35]\nto only consider words with alphanumeric characters and at least three characters. ICDAR-2013\n(IC13) [14] extends IC03 and contains 1015 groundtruth cropped word images from real scenes.\nIIIT5k-word (IIIT5k) [24] is a real-world scene text recognition dataset with 3,000 cropped word\nimages downloaded from Google Image Search. Street View Text (SVT) [34] test dataset contains\n249 Google Street View images, from which 647 word images are cropped for testing. We further\nevaluate the model generalization performance on a small dataset Synth5K. Synth5K is a small-scale\ndataset with 5K training data and 5K testing data randomly sampled from Synth90K.\nFor all the experiments, sequence accuracy is used as the evaluation metric, i.e. the percentage of\ntesting images correctly recognized. We only evaluate the model in lexicon-free mode, that the\npredictions are made only based on the input image without any prede\ufb01ned lexicon.\n\n4.2\n\nImplementation Details\n\nWe use RMSProp to train our model and set the batch size to 100. The learning rate is \ufb01xed at\n1 \u00d7 10\u22123 during training. The training stops at 150 epochs. Models and loss functions are all\nimplemented using Pytorch [26]. Computations of the forward and backward variables are carried\nout in log-space to avoid over\ufb02ow.\nFor EnCTC, we varied the strength term \u03b2 in the range [0.1, 0.2, 0.5] and found 0.2 work best. For\nEsCTC, we varied the equal-spacing term \u03c4 in the range [1.1, 1.3, 1.5, 1.7, 2.0, 2.5] and found 1.5\nwork best. We simply apply the same parameters to the EnEsCTC algorithm. For all the experiments,\nwe set \u03b2 as 0.2 and \u03c4 as 1.5 without further tuning.\n\n4.3 Qualitative Analysis\n\nIn this section, we provide some qualitative analysis. All the experiments are performed on Synth5K.\nError Signal in Training\nFigure 1 visualizes the evolution of the model prediction and error signal. In the early stage of training\n(on the top), the prediction is near uniform distribution. Since the entropy term has zero gradient\nfor uniform distribution, the error signal of EnCTC coincides exactly with CTC. We also see that\nEsCTC and EnEsCTC produce more concentrated error signal, that is, the algorithms consider that\n\n6\n\n\fthe element at the head of the label sequence cannot appear at the rear of the input sequence, reducing\nthe size of feasible path set. In the interim training period (shown on the bottom), we observe that the\nerror signal of EnCTC gets smoother than that of CTC, which helps to enhance exploration during\ntraining and avoid the peaky distribution. EsCTC tends to pull the blue and green labels closer to\nget more equally spaced, producing more reasonable alignment. Error signal of EnEsCTC has the\nproperties of both algorithms.\n\nFigure 1: The evolution of prediction and error signal. Colors indicate different labels and the\nhorizontal axis correspond to the position in the input image. (a) the prediction of CTC model\n(b)(c)(d) the gradient of EnCTC, EsCTC and EntEsCTC versus CTC respectively, in which the solid\nlines correspond to CTC and the dashed lines to the three proposed algorithms. Top: Initial training\nstage. Bottom: Interim training period. The CTC predicts a correct label sequence in this case.\n\nAlignment Evaluation\nWe observe from Figure 2 that CTC outputs a highly peaky distribution and shows inferior alignment\nto the three proposed algorithms. It is attributed to that CTC lacks exploration during training and\nconsiders equally for the massive amount of feasible paths. EnCTC strengthened the exploration\nduring CTC training process, producing a better and more reasonable alignment where each label\noccupies a certain range along the horizontal axis. The probabilities overlap at the junction of two\nconsecutive characters, effectively modeling the boundary ambiguity. EsCTC eliminates unreasonable\nCTC feasible paths based on the equal spacing prior, which results in a more accurate alignment. The\nalignment result of EnEsCTC shows it combines the advantage of both EnCTC and EsCTC.\nPruning Analysis for EsCTC\n\nFigure 2: Illustration of the prediction results\nof an example image. The vertical dotted line\nindicates the ground truth segmentation results.\n\nFigure 3: The in\ufb02uence of \u03c4 on path pruning for\ndifferent data scales.\n\n7\n\n(a)(b)(c)(d)parityCTCEnCTCEsCTCEnEsCTC00.20.40.60.812652104208\ud835\udf0f=1\ud835\udf0f=1.2\ud835\udf0f=1.5\ud835\udf0f=2\ud835\udf0f=| \ud835\udc59| (CTC)XLength(T)Ratio\fFigure 3 illustrates the in\ufb02uence of \u03c4 (see Equation 11) on path pruning for different data scales. The\nx-axis is the length of the input sequence, while the y-axis is the ratio of feasible path numbers of\nEsCTC to that of CTC. The length of the label |l| is 12 in the experiment. We see that \u03c4 (1 \u2264 \u03c4 \u2264 |l|)\ncontrols the searching space reduction rate. Note that when \u03c4 = 1, EsCTC requires strict equality of\nspacing; when \u03c4 = |l|, EsCTC degenerates to original CTC. As \u03c4 approaches 1, the ratio of remaining\npaths quickly drops to zero, indicating the path pruning effect of EsCTC. We also observe that as the\nlength of input sequence (T ) increases, the path pruning effect gets more signi\ufb01cant. Please refer to\nthe supplementary material for more details.\n\n4.4 Evaluation of Model Generalization Performance\n\nWe further evaluate the model generalization performance on Synth5K. Note that over 97% of the\nwords in the training set only appear once and 94% of words in the test set does not appear in the\ntraining set. Therefore, the model needs strong generalization ability. We also compare with two\nregularization methods, namely label smoothing (LS) [31] and con\ufb01dence penalty (CP) [27].\nAs shown in Table 1, our proposed regularization methods show superior performance over LS and CP.\nLS and CP directly regularize the model prediction at each time-step and improve the generalization\nof the CTC baseline. However, their methods indiscriminately regularizes both feasible and unfeasible\npaths. EnCTC instead regularizes the entropy of feasible paths, which is more reasonable for CTC.\nWhen combined with EsCTC, our \ufb01nal EnEsCTC achieves the best generalization performance.\n\n4.5 Comparisons with the State-of-the-art Methods\n\nWe compare our proposed methods with several state-of-the-art approaches. STAR-Net [19] and\nCRNN [29] are CTC-based methods, while R2AM [17] and RARE [30] are attention-based methods.\nSTAR-Net [19] employs a much deeper network than [29] to extract features. STAR-Net [19] and\nRARE [30] also apply STN [12] to remove the distortions. Note that we use the same training\nsettings and parameters as our baseline CRNN [29], except only for the proposed regularization term.\nExperimental settings and implementation details are summarized in Section 4.1 and Section 4.2.\nTable 2 presents the results of the scene text recognition task. We \ufb01nd that our method outperforms\nthe baseline CRNN without changing the training settings. It also achieves comparable performance\nto the state-of-the-art methods with a much simpler architecture. In particular, our proposed methods\nachieve the best performance on IC03 and IC13 datasets.\n\nTable 1: Evaluation of model generalization.\n\nTable 2: Comparisons with the state-of-the-art methods.\n\nMethod\nCTC\nCTC + LS [31]\nCTC + CP [27]\nEnCTC\nEsCTC\nEnEsCTC\n\nSynth5K\n\n38.1\n42.9\n44.4\n45.5\n46.3\n47.2\n\nMethod\nCRNN [29]\nSTAR-Net [19]\nR2AM [17]\nRARE [30]\nEnCTC\nEsCTC\nEnEsCTC\n\nIC03\n89.4\n89.9\n88.7\n90.1\n90.8\n92.6\n92.0\n\nIC13\n86.7\n89.1\n90.0\n88.6\n90.0\n87.4\n90.6\n\nIIIT5K SVT\n80.8\n78.2\n83.6\n83.3\n78.4\n80.7\n81.9\n81.9\n81.5\n82.6\n81.5\n81.7\n82.0\n80.6\n\n5 Conclusions\n\nIn this paper, we have presented a novel maximum entropy based regularization for CTC (EnCTC),\nwhich maintains reasonable possibilities among all the feasible paths, to enhance generalization\nand exploration. Moreover, we derive from equal spacing prior a pruning algorithm (EsCTC) to\neffectively reduce the space of the feasible set and give theoretical explanations from the perspective\nof maximum entropy. The experiments on scene text recognition benchmarks demonstrate that our\nproposed methods achieve superior performance than the baseline and show better generalization\nability. Our proposed method achieves comparable performance to the state-of-the-art methods with\na much simpler architecture. We believe that the proposed regularization is general, which can be\nused to consistently improve the performance of the original CTC model.\n\n8\n\n\fAcknowledgments\n\nThis work is supported by NSFC (Grant No. 61876095, No. 61751308 and No. 61473167) and\nBeijing Natural Science Foundation (Grant No. L172037).\n\nReferences\n[1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg,\nCarl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech\n2: End-to-end speech recognition in english and mandarin. In International Conference on\nMachine Learning (ICML), pages 173\u2013182, 2016.\n\n[2] Eric Battenberg, Jitong Chen, Rewon Child, Adam Coates, Yashesh Gaur, Yi Li, Hairong Liu,\nSanjeev Satheesh, David Seetapun, Anuroop Sriram, et al. Exploring neural transducers for\nend-to-end speech recognition. Automatic Speech Recognition and Understanding Workshop\n(ASRU), pages 206\u2013213, 2017.\n\n[3] Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in\n\nsequence to sequence models. arXiv preprint arXiv:1612.02695, 2016.\n\n[4] Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for\ncontinuous sign language recognition by staged optimization. In IEEE Conference on Computer\nVision and Pattern Recognition (CVPR), pages 1610\u20131618, 2017.\n\n[5] Alex Graves and Faustino Gomez. Connectionist temporal classi\ufb01cation:labelling unsegmented\nsequence data with recurrent neural networks. In International Conference on Machine Learning\n(ICML), pages 369\u2013376, 2006.\n\n[6] Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural\nnetworks. In International Conference on Machine Learning (ICML), pages 1764\u20131772, 2014.\n\n[7] Alex Graves, Marcus Liwicki, Santiago Fern\u00e1ndez, Roman Bertolami, Horst Bunke, and\nJ\u00fcrgen Schmidhuber. A novel connectionist system for unconstrained handwriting recognition.\nTransactions on Pattern Analysis and Machine Intelligence (TPAMI), 31(5):855\u2013868, 2009.\n\n[8] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan\nPrenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up\nend-to-end speech recognition. arXiv preprint arXiv:1412.5567, 2014.\n\n[9] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut-\ndinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv\npreprint arXiv:1207.0580, 2012.\n\n[10] De An Huang, Fei Fei Li, and Juan Carlos Niebles. Connectionist temporal modeling for\nweakly supervised action labeling. In European Conference on Computer Vision (ECCV), pages\n137\u2013153, 2016.\n\n[11] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and\narti\ufb01cial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227,\n2014.\n\n[12] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In\n\nAdvances in Neural Information Processing Systems (NIPS), pages 2017\u20132025, 2015.\n\n[13] Edwin T Jaynes. Information theory and statistical mechanics. Physical Review, 106(4):620,\n\n1957.\n\n[14] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Big-\norda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and\nLluis Pere De Las Heras. Icdar 2013 robust reading competition. In International Conference\non Document Analysis and Recognition (ICDAR), pages 1484\u20131493, 2013.\n\n[15] Suyoun Kim, Michael L Seltzer, Jinyu Li, and Rui Zhao. Improved training for online end-to-end\n\nspeech recognition systems. arXiv preprint arXiv:1711.02212, 2017.\n\n9\n\n\f[16] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In\n\nAdvances in Neural Information Processing Systems (NIPS), pages 950\u2013957, 1992.\n\n[17] Chen-Yu Lee and Simon Osindero. Recursive recurrent nets with attention modeling for ocr\nin the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n2231\u20132239, 2016.\n\n[18] Mengxi Lin, Nakamasa Inoue, and Koichi Shinoda. Ctc network with statistical language\nmodeling for action sequence recognition in videos. In Thematic Workshops of ACM Multimedia,\npages 393\u2013401, 2017.\n\n[19] Wei Liu, Chaofeng Chen, Kwan Yeek. Wong, Zhizhong Su, and Junyu Han. Star-net: A spatial\nattention residue network for scene text recognition. In British Machine Vision Conference\n(BMVC), pages 43.1\u201343.13, 2016.\n\n[20] Simon M Lucas, Alex Panaretos, Luis Sosa, Anthony Tang, Shirley Wong, Robert Young,\nKazuki Ashida, Hiroki Nagai, Masayuki Okamoto, Hiroaki Yamamoto, et al. Icdar 2003 robust\nreading competitions: entries, results, and future directions. International Journal of Document\nAnalysis and Recognition (IJDAR), 7(2-3):105\u2013122, 2005.\n\n[21] Oded Maron and Tom\u00e1s Lozano-P\u00e9rez. A framework for multiple-instance learning.\n\nAdvances in Neural Information Processing Systems (NIPS), pages 570\u2013576, 1998.\n\nIn\n\n[22] Yajie Miao, Mohammad Gowayyed, and Florian Metze. Eesen: End-to-end speech recognition\nusing deep rnn models and wfst-based decoding. In IEEE Workshop on Automatic Speech\nRecognition and Understanding (ASRU), pages 167\u2013174, 2015.\n\n[23] David Miller, Ajit V Rao, Kenneth Rose, and Allen Gersho. A global optimization technique\nfor statistical classi\ufb01er design. IEEE Transactions on Signal Processing, 44(12):3108\u20133122,\n1996.\n\n[24] Anand Mishra, Karteek Alahari, and C.V. Jawahar. Scene text recognition using higher order\n\nlanguage priors. In British Machine Vision Conference (BMVC), 2012.\n\n[25] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap,\nTim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce-\nment learning. In International Conference on Machine Learning (ICML), pages 1928\u20131937,\n2016.\n\n[26] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,\nZeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in\npytorch. 2017.\n\n[27] Gabriel Pereyra, George Tucker, Jan Chorowski, \u0141ukasz Kaiser, and Geoffrey Hinton. Reg-\narXiv preprint\n\nularizing neural networks by penalizing con\ufb01dent output distributions.\narXiv:1701.06548, 2017.\n\n[28] Ha\u00b8sim Sak, F\u00e9lix de Chaumont Quitry, Tara Sainath, Kanishka Rao, et al. Acoustic mod-\nelling with cd-ctc-smbr lstm rnns. In IEEE Workshop on Automatic Speech Recognition and\nUnderstanding (ASRU), pages 604\u2013609, 2015.\n\n[29] B. Shi, X. Bai, and C. Yao. An end-to-end trainable neural network for image-based sequence\nrecognition and its application to scene text recognition. Transactions on Pattern Analysis\nMachine Intelligence (TPAMI), 39(11):2298\u20132304, 2016.\n\n[30] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Robust scene text\nrecognition with automatic recti\ufb01cation. In IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR), pages 4168\u20134176, 2016.\n\n[31] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink-\ning the inception architecture for computer vision. In IEEE Conference on Computer Vision\nand Pattern Recognition (CVPR), pages 2818\u20132826, 2016.\n\n10\n\n\f[32] Ehsan Variani, Tom Bagby, Kamel Lahouel, Erik McDermott, and Michiel Bacchiani. Sampled\nconnectionist temporal classi\ufb01cation. In International Conference on Acoustics, Speech and\nSignal Processing (ICASSP), pages 4959\u20134963, 2018.\n\n[33] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural\nnetworks using dropconnect. In International Conference on Machine Learning (ICML), pages\n1058\u20131066, 2013.\n\n[34] Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In Interna-\n\ntional Conference on Computer Vision (ICCV), pages 1457\u20131464, 2011.\n\n[35] Tao Wang, David J Wu, Adam Coates, and Andrew Y Ng. End-to-end text recognition with\nconvolutional neural networks. In International Conference on Pattern Recognition (ICPR),\npages 3304\u20133308, 2012.\n\n[36] Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement\n\nlearning algorithms. Connection Science, 3(3):241\u2013268, 1991.\n\n[37] Matthew D Zeiler and Rob Fergus. Stochastic pooling for regularization of deep convolutional\n\nneural networks. arXiv preprint arXiv:1301.3557, 2013.\n\n11\n\n\f", "award": [], "sourceid": 454, "authors": [{"given_name": "Hu", "family_name": "Liu", "institution": "Tsinghua University"}, {"given_name": "Sheng", "family_name": "Jin", "institution": "Tsinghua University"}, {"given_name": "Changshui", "family_name": "Zhang", "institution": "Tsinghua University"}]}