{"title": "Phonetic Classification and Recognition Using the Multi-Layer Perceptron", "book": "Advances in Neural Information Processing Systems", "page_first": 248, "page_last": 254, "abstract": null, "full_text": "Phonetic Classification and Recognition \n\nUsing the Multi-Layer Perceptron \n\nHong C. Leung, James R. Glass, \n\nMichael S. Phillips, and Victor W. Zue \n\nSpoken Language Systems Group \nLaboratory for Computer Science \n\nMassachusetts Institute of Technology \n\nCambridge, Massachusetts 02139, U.S.A. \n\nAbstract \n\nIn this paper, we will describe several extensions to our earlier work, utiliz(cid:173)\ning a segment-based approach. We will formulate our segmental framework \nand report our study on the use of multi-layer perceptrons for detection \nand classification of phonemes. We will also examine the outputs of the \nnetwork, and compare the network performance with other classifiers. Our \ninvestigation is performed within a set of experiments that attempts to \nrecognize 38 vowels and consonants in American English independent of \nspeaker. When evaluated on the TIMIT database, our system achieves an \naccuracy of 56%. \n\n1 \n\nIntroduction \n\nThus far, the neural networks research community has placed heavy emphasis on \nthe problem of pattern classification. In many applications, including speech recog(cid:173)\nnition, one must also address the issue of detection. Thus, for example, one must \ndetect the presence of phonetic segments as well as classify them. Recently, the \ncommunity has moved more towards recognition of continuous speech. A network \nis typically used to label every frame of speech in a frame-based recognition sys(cid:173)\ntem [Franzini 90, Morgan 90, Tebelskis 90]. \n\nOur goal is to study and exploit the capability of ANN for speech recognition, \nbased on the premise that ANN may offer a flexible framework for us to utilize our \n\n248 \n\n\fPhonetic Classification and Recognition Using the Multi-Layer Perceptron \n\n249 \n\nimproved, albeit incomplete, speech knowledge. As an intermediate milestone, this \npaper extends our earlier work on phonetic classification to context-independent \nphonetic recognition. Thus we need to locate as well as identify the phonetic units. \nOur system differs from the majority of approaches in that a; segmental framework is \nadopted. The network is used in conjunction with acoustic segmentation procedures \nto provide a phonetic string for the entire utterance. \n\n2 Segmental Formulation \n\nIn our segmental framework, a phonetic unit is mapped to a segment explicitly \ndelineated by a begin and end time in the speech signal. This is motivated by the \nbelief that a segmental framework offers us more flexibility in applying our speech \nknowledge than is afforded by a frame-based approach. As a result, a segment-based \napproach could ultimately lead to superior modelling of the temporal variations in \nthe realization of underlying phonological units. \n\nLet & denote the best sequence of phonetic units in an utterance. To simplify the \nproblem, we assume that P(SI) = p(sllQ'j), where SI stand for the ith time segment \nthat has one and only one phoneme in it, and Q' j stands for the best phoneme label \nin SI. Thus the probability of the best sequence, p( &), is: \n\n1 <j< N \n\n(1) \n\nwhere sis any possible sequence of time segments consisting of {S1' S2, ... }, P(SI) is \nthe probability of a valid time segment, and N is the number of possible phonetic \nunits. In order to perform recognition, the two probabilities in Equation 1 must be \nestimated. The first term, p(Q'j), is a set of phoneme probabilities and thus can be \nviewed as a classification problem. The second term, P(SI), is a set of probabilities \nof valid time regions and thus can be estimated as a segmentation problem. \n\n2.1 Segmentation \n\nIn order to estimate the segment probabilities, P(SI), in Equation 1, we have formu(cid:173)\nlated segmentation into a boundary classification problem. Let bl and br be the left \nand right boundary of a time segment, SI, respectively, as shown in Figure 1a. Let \n{b1 , b2 , \u2022\u2022 , bK } be the set of boundaries that might exist within SI. These boundaries \ncan be proposed by a boundary detector, or they can simply occur at every frame of \nspeech. We define p( SI) to be the joint probability that the left and right boundaries \nexist and all other boundaries within SI do not exist. To reduce the complexity of \nthe problem, assume b j is statistically independent of bk for \\;f j # k. Thus, \n\np(Si) = p(bl , b1 , b2 , \u2022\u2022 , bK , br ) \n\n= p(bdp(b1 )p(b2 ) \u2022\u2022. p(bK )p(br ), \n\n(2) \n\n\f250 \n\nLeung, Glass, Phillips, and Zue \n\nt.2 \n1-\n\nt.1 \n1-\n\nb\u00b7 I \n\nt. \nI \n(b) \n\n(a) \n\nFigure 1: Schematic diagrams for estimation of (a) segment probability, P(Si), and \n(b) boundary probability, p(bk ). The boundaries can be proposed by a boundary \ndetector, or they can simply occur at every frame. See text. \n\nwhere p(b,) and p(br2 stand for the probability that the left and right boundary \nexist, respectively, P(bk) stands for the probability that the kth boundary does not \nexist. As a result, the probability of a segment, p(sd can be obtained by computing \nthe probabilities of the boundaries, P(bk), subsumed by the segment. As we will \ndiscuss in a later section, by using the time-aligned transcription, we can train the \nboundary probabilities in a supervised manner. \n\n2.2 Phonetic Classification \n\nOnce the probability of a segment, P(SI), is obtained, we still need to classify it, \ni.e. compute the probabilities of the phonetic units in the segment, p(aj). Again, \nthe time-aligned transcription can be used to train the probabilities in a supervised \nmanner. We have discussed this in earlier papers [Leung 89, Leung 90]. In a later \nsection, we will discuss some of our recent experimental results. \n\n3 Experiments \n\n3.1 Tasks and Corpora \n\nThe experiments described in this paper deal with classification and recognition of \n38 phonetic labels representing 14 vowels, 3 semivowels, 3 nasals, 8 fricatives, 2 \naffricates, 6 stops, 1 flap and 1 silence. Within the context of classification, the \nnetworks are given a segment of the speech signal, and are asked to determine \nits phonetic identity. Within the context of recognition, the networks are given \nan utterance, and are asked to determine the identity and locations of the pho(cid:173)\nnetic units in the utterance. All experiments were based on the sentences in the \nTIMIT database [Lamel 86]. As summarized in Table 1, Corpus I contains 1,750 sx \nsentences spoken by 350 male and female speakers, resulting in a total of 64,000 \nphonetic tokens. Corpus II contains 4,400 sx and si sentences spoken by 550 male \nand female speakers, resulting in a total of 165,000 phonetic tokens . \n\n3.2 Phonetic Classification \n\nAs previously discussed, estimation of the probability, p(aj) in Equation 1 can be \nviewed as a classification problem. Many statistical classifiers can be used. We have \n\n\fPhonetic Classification and Recognition Using the Multi-Layer Perceptron \n\n251 \n\nCorpus \n\nSet \n\nSpeakers Sentences Tokens Type \n\nI \n\nII \n\ntrainmg \ntesting \ntraIning \ntesting \n\n300 \n50 \n500 \n50 \n\n1500 \n250 \n4000 \n400 \n\n55,000 \n9,000 \n150,000 \n15,000 \n\nsx \nsx \n\nsxfsi \nsx/si \n\nTable 1: Corpora I and II extracted from the TIMIT database. Corpus I contains \nonly sx sentences, whereas Corpus II contains both sx and si sentences. The speakers \nin the testing sets for both Corpus I and Corpus II are the same. \n\nchosen to use the MLP, due to its discriminatory capability, as well as its flexibility \nin that it does not make assumptions about specific statistical distributions or \ndistance metrics. In addition, earlier work shows that the outputs of MLP can \napproximate posteriori probabilities [Bourlard 88]. To train the network, we adopt \nprocedures such as center initialization, input normalization, adaptive gain, and \nmodular training [Leung 90]. The input representation was identical to that in the \nSUMMIT system, and consisted of 82 acoustic attributes [Zue 89]. These segmental \nattributes were generated automatically by a search procedure that uses the training \ndata to determine the settings of the free parameters of a set of generic property \ndetectors using an optimization procedure [Phillips 88]. \n\n3.3 Boundary Classification \n\nIn our segmental framework formulated in Equation 1, the main difference between \nclassification and recognition is the incorporation of a probability for each segment, \nP(Si). As described previously in Equation 2, we have simplified the problem of \nestimating P(SI) to one of determining the probability that a boundary exists, p(bt). \nTo estimate P(bk), a MLP with two output units is used, one for the valid bound(cid:173)\naries and the other for the extraneous boundaries. By referencing the time-aligned \nphonetic transcription, the desired outputs of the network can be determined. In \nour current implementation p(bt) is determined using four abutting segments, as \nshown in Figure lb. These segments are proposed by the boundary detector in \nthe SUMMIT system. Let tl stand for the time at which bl is located, and SI stand \nfor the segment between tl and tl+1, where tl+1 > tl. The boundary probability, \np( bl ), is then determined by using the average mean-rate response [Seneff 88] in \nSI-2, SI-1, 81, and SI+1 as inputs to the MLP. Thus the network has altogether 160 \ninput units. \n\n3.4 Results \n\n3.4.1 Phonetic Classification \n\nIn the phonetic classification experiments, the system classified a token extracted \nfrom a phonetic transcription that had been aligned with the speech waveform. \nSince there was no detection involved in these experiments only substitution errors \nwere possible. \n\n\f252 \n\nLeung, Glass, Phillips, and Zue \n\nClassifier Correct Parameters \nI \n2,200 \nSUMMIT \nI Gaussian \n128,000 \nMLP \nI \n15,000 \nII MLP \n30,000 \n\n70% \n70% \n74% \n76% \n\nTable 2: Phonetic classification results using the SUMMIT classifier, Gaussian clas(cid:173)\nsifier, and MLP. Also shown are the number of parameters in the classifiers. \n\nIn the first set of experiments, we compared results based on Corpus I, using dif(cid:173)\nferent classifiers. As Table 2 shows, the baseline speaker-independent classification \nperformance of SUMMIT on the testing data was 70%. When Gaussian classifiers \nwith full covariance matrices were used, we found that the performance is also about \n70%. Finally, when the MLP is used, a performance of 74% is achieved. \n\nAlthough the sx sentences were designed to be phonetically balanced, the 1,750 \nsentences in Corpus I are not distinct. In the second set of experiments, we evaluated \nthe MLP classifier on Corpus II, which include both the sx and si sentences. 1 As \nshown in Table 2, the classifier achieves 76%. \n\nParameters: The networks used as described in Table 2 have only 1 hidden layer. \nThe number of hidden units in the network can be 128 or 256, resulting in 15,000 \nor 30,000 connections. For comparison, Table 2 also shows the number of parame(cid:173)\nters for the SUMMIT and Gaussian classifiers. While the SUMMIT classifier requires \nonly about 2,200 parameters, the Gaussian classifiers require as much as 128,000 \nparameters, an order of magnitude more than the MLP. These numbers also give us \nsome idea about the computational requirements for different classifiers, since the \nrequired number of multiplications is about the same as the number of parameters. \n\nNetwork Outputs: We have chosen the network to estimate the phoneme proba(cid:173)\nbilities. When the network is trained, the target values are either lor O. However, if \nthe network is over-trained, its output values may approach either 1 or 0, resulting \nin poor estimates of the posterior probabilities. Figure 2 shows two distributions \nfor the output values of the network for 3600 tokens from the test set. Figure 2a \ncorresponds to the ratio of the highest output value to the sum of the network out(cid:173)\nput values, whereas Figure 2b corresponds to the second highest output value. We \ncan see that both distributions are quite broad, suggesting that the network often \nmakes \"soft\" decisions about the phoneme labels. We feel that this is important \nsince in speech recognition, we often need to combine scores or probabilities from \ndifferent parts of the system. \n\n3.4.2 Boundary Classification \n\nWe have evaluated the boundary classifier using the training and testing data in \nCorpus I. By using 32 hidden units, the network can classify 87% of the boundaries \nin the test set correctly. \n\n1 All the si sentences in TIMIT are distinct. \n\n\fPhonetic Classification and Recognition Using the Multi-Layer Perceptron \n\n253 \n\n:g400 \nQ. \nE \nc2l \n'0 .x 200 \n\nE \n:3 \nZ \n\no+-~~~~~~~~~~ \n0.0 \n\n1.0 \n\nNetwork Output Value \n\nFigure 2: Histograms for the output values of the network extracted from 3600 \nsamples: (a) the highest output values, and (b) the second highest output values. \n\nCorpus Classifer \nBaseline \n\nI \nI \nI \nII \n\nMLP \nMLP \nMLP \n\nSegment \n\nAccuracy \n\nBinary Hierarchy \nBinary Hierarchy \nStochastic Pruning \nStochastic Pruning \n\n47% \n50% \n54% \n56% \n\nTable 3: Phonetic recognition results using binary hierarchy (dendrogram), and \nboundary pruning. No duration, bigram, or trigram statistics have been used. \nErrors include substitutions, deletions, and insertions. \n\n3.4.3 Phonetic Recognition \n\nOne of the disadvantages of our segmental framework is that the amount of com(cid:173)\nputation involved can be very significant, since a segment can begin and end at any \nframe of an utterance. We have explored various pruning strategies. In this paper, \nwe will report our results using stochastic pruning and binary hierarchy [Leung gOa]. \nWe have found that such pruning strategies can reduce the amount of computation \nby about 3 orders of magnitude. \n\nThe results of the phonetic recognition experiments are shown in Table 3. No \nduration, bigram, or trigram statistics have been used. The baseline performance \nof the current SUMMIT system on Corpus I is 47%, including substitution, deletion, \nand insertion errors. When the MLP was used in place of the classifier in the current \nSUMMIT system using also the binary hierarchical representation, the performance \nimproved to 50%. When the MLP was used with stochastic pruning technique, the \nperformance improved to 54%. Finally, by using the network trained and tested on \nCorpus II, the performance improved to 56%. \n\n\f254 \n\nLeung, Glass, Phillips, and Zue \n\n4 Discussion \n\nIn summary, we have discussed a segmental approach for phonetic recognition. \nWe have also examined the outputs of the network, and compared performance \nresults and computational requirements with different classifiers. We have shown \nthat decisions made by the network are quite \"soft\", and that the network yields \nresults favorable to other more traditional classifiers. Future work includes the use \nof context-dependent models for phonetic and boundary classification, utilization \nof other phonological units, and extension to recognition of continuous speech. \n\nReferences \n\n[Bourlard 88] Bourlard, H., and C.J. Wellekens, \"Links between Markov Models \nand Multilayer Perceptrons,\" Advances in Neural Information Processing Systems \n1, Morgan Kaufmann, 1988. \n\n[Franzini 90] Franzini, M.A., K.F. Lee, and A. Waibel, \"Connectionist Viterbi Train(cid:173)\ning: A New Hybrid Method for Continuous Speech Recognition,\" Proc. ICASSP-90, \nAlbuquerque, NM, USA, 1990. \n[LameI86] Lamel, L.F., R.H. Kassel, and S. Seneff, \"Speech Database Development: \nDesign and Analysis of the Acoustic Phonetic Corpus,\" Proc. DARPA Speech Recog(cid:173)\nnition Workshop, 1986. \n[Leung 89] Leung, H.C., The Use of Artificial Neural Networks of Phonetic Recog(cid:173)\nnition, Ph.D. Thesis, Mass. Inst. of Tech., 1989. \n[Leung 90] Leung, H.C., and V.W. Zue, \"Phonetic Classification Using Multi-Layer \nPerceptrons,\" Proc. ICASSP-90, Albuquerque, 1990. \n[Leung 90a] Leung, H., Glass, J., Phillips, M., and Zue, V., \"Detection and Clas(cid:173)\nsification of Phonemes Using Context-independent Error Back-Propagation,\" Proc. \nInternational Conference on Spoken Language Processing, Kobe, Japan, 1990. \n[Morgan 90] Morgan, N., and H. Bourlard, \"Continuous Speech Recognition Using \nMultilayer Perceptrons with Hidden Markov Models,\" Proc. \nquerque, NM, USA, 1990. \n[Phillips 88] Phillips, M.S., \"Automatic Discovery of Acoustic Measurements for \nAcoustic Classification,\" J. Acoust. Soc. Amer., Vol. 84, 1988. \n\nICASSP-90, Albu(cid:173)\n\n[Seneff 88] Seneff, S. \"A Joint Synchrony/Mean-Rate Model of Auditory Speech \nProcessing,\" Proc. J. of Phonetics, 1988. \n[Tebelskis] Tebelskis, J., and A. Waibel, \"Large Vocabulary Recognition Using \nLinked Predictive Neural Networks,\" Proc. ICASSP-90, Albuquerque, NM, USA, \n1990. \n[Zue 89] Zue, V., J. Glass, M. Phillips, and S. Seneff, \"The MIT SUMMIT Speech \nRecognition System: A Progress Report,\" Proceedings of DARPA Speech and Nat(cid:173)\nural Language Workshop, February, 1989. \n\n\f", "award": [], "sourceid": 341, "authors": [{"given_name": "Hong", "family_name": "Leung", "institution": null}, {"given_name": "James", "family_name": "Glass", "institution": null}, {"given_name": "Michael", "family_name": "Phillips", "institution": null}, {"given_name": "Victor", "family_name": "Zue", "institution": null}]}