{"title": "A Large-Scale Neural Network Which Recognizes Handwritten Kanji Characters", "book": "Advances in Neural Information Processing Systems", "page_first": 415, "page_last": 422, "abstract": null, "full_text": "A Large-Scale Neural Network \n\n415 \n\nA  LARGE-SCALE  NEURAL  NETWORK \nWHICH  RECOGNIZES  HANDWRITTEN \n\nKANJI  CHARACTERS \n\nYoshihiro Mori \n\nKazuki Joe \n\nA TR Auditory and Visual Perception Research Laboratories \n\nSanpeidani Inuidani Seika-cho Soraku-gun Kyoto 619-02 Japan \n\nABSTRACT \n\nWe propose a  new  way  to construct a  large-scale neural  network  for \n3.000 handwritten  Kanji  characters  recognition.  This  neural  network \nconsists  of 3  parts:  a  collection  of small-scale  networks  which  are \ntrained individually on a small number of Kanji characters; a network \nwhich  integrates  the  output  from  the  small-scale  networks,  and  a \nprocess to facilitate  the integration of these neworks. The recognition \nrate of the  total  system  is  comparable  with  those  of the  small-scale \nnetworks. Our results indicate that the proposed method is effective for \nconstructing  a  large-scale  network  without  loss  of  recognition \nperformance. \n\n1  INTRODUCTION \n\nNeural networks have been applied to recognition tasks in many fields. with good results \n[Denker, 1988][Mori,1988][Weideman, 1989].  They  have  performed  better  than \nconventional  methods.  However  these  networks  currently  operate  with  only  a  few \ncategories, about 20 to 30. The Japanese writing system at present is composed of about \n3,000 characters. For a  network to  recognize this  many  characters,  it must be given a \nlarge number of categories while maintaining its level of performance. \n\nTo train small-scale neural networks is not a difficult task. Therefore. exploring methods \nfor integrating these small-scale neural networks is important to construct a large-scale \nnetwork.  If such  methods  could  integrate  small-scale  networks  without \nloss  of the \nperformance, the scale of neural networks would be extended dramatically. In this paper, \nwe propose such  a  method  for  constructing a  large-scale  network  whose  object is  to \nrecognize  3,000  handwritten  Kanji  characters,  and  report  the  result  of a  part of this \nnetwork.  This  method  is  not limited to  systems for  character recognition,  and can  be \napplied to any system which recognizes many categories. \n\n2  STRATEGIES  FOR  A  LARGE-SCALE  NETWORK \n\nKnowing  the current recognition and generalization capacity of a  neural  network.  we \nrealized  that constructing  a  large-scale  monolithic  network  would  not be  efficient or \n\n\f416  Mori and Joe \n\neffective.  Instead,  from  the  start  we  decided  on  a  building  blocks  approach \n[Mori,1988] [Waibel,1988]. There are two strategies to mix many small-scale networks. \n\n2.1  Selective  Neural  Network  (SNN) \n\nIn  this strategy, a  large-scale neural  network is made from  many  small-scale networks \nwhich are  trained individually on a small number of categories, and a  network  (SNN) \nwhich selects the appropriate small-scale network (Fig.  I). The advantage of this strategy \nis that the information passed to a selected small-scale networks is always appropriate for \nthat  network.  Therefore,  training  these  small-scale networks  is  very easy.  But on  the \nother hand,  increasing the number of categories  will substantially increase the training \ntime  of the  SNN,  and  may  make  it harder for  the  SNN  to  retain  high  perfonnance. \nFurthennore, the error rate of the SNN will limit the perfonnance of the whole system. \n\n2.2  Integrative  Neural  Network  (INN) \n\nIn this strategy, a  large-scale neural  network is made from  many  small-scale networks \nwhich are trained individually on a  small  number of categories. and a  network (INN) \nwhich  integrates  the  output from  these  small-scale networks(Fig. 2).  The advantage of \nthis strategy is that every small-scale network gets information and contributes to finding \nthe right answer. Therefore,  it is possible to use  the knowledge distributed among each \nsmall-scale  network.  But in  some  respects.  various  devices  are  needed  to  make  the \nintegration easier. \n\nThe common advantage with both strategies just mentioned is that the size of each neural \nnetwork is relatively small, and it does not take a long time to train these networks. Each \nsmall-scale networks is considered an independent part of the whole system. Therefore, \nretraining these networks (to improve the performance of the whole system) will not take \ntoo long. \n\n~ __ .... O,utput \n\nSub \nNet \n1 \n\n':1U:U/W:::::::/:::/E::::::::. \n:::: Suspending \n/ Network \n(:::::{::::::::::::::::.:::::::: ~ \n\n\u2022  \u2022 \n\nNeural Network \n(Selection Type) \n\nFig.  1 SNN Strategy \n\n\fA Large-Scale Neural Network \n\n417 \n\nOutput \n\nNeural Network \n(Integration Type) \n\n\u2022  \u2022 \n\nFig. 2  INN Strategy \n\n3  STRUCTURE  OF  LARGE-SCALE  NETWORK \n\nThe whole system  is constructed using  three kinds of neural  networks.  The ftrst  one, \ncalled a  SubNet, is an ordinary three layered feed forward  type neural network trained \nusing the Back Propagation learning algorithm. The second kind of network is called a \nSuperNet. This neural network makes its decision by integrating the outputs from all the \nSubNets. This network is also a 3-layered feed-forward net, but is larger than the Subnets. \nThe last network, which we call an OtherFilter, is devised to improve the integration of \nthe  S uperNet.  This  OtherFilter  network  was  designed  using  the  L VQ  algorithm \n[Khonen,1988].  There  are  also  some  changes  made  in  the  BP  learning  algorithm \nespecially for pattern recognition [Joe,1989]. \n\nWe decided that, based on the time it takes for learning, there should be 9 categories in \neach  small-scale network.  The 3,000 characters  are separated into  these  small groups \nthrough the K-means clustering method, which allows similar characters to be grouped \ntogether. The separation occurs in two stages. First, 11  groups of 270 characters each are \nformed,  then each group is separated into 30 smaller units.  In this way, 330 groups of 9 \ncharacters each are obtained. We choose the INN strategy to use distributed knowledge to \nfull advantage. The 9-character units are SubNets, which are integrated in 2 stages. First \n30 SubNets are integrated by a higher level network SuperNet. Altogether, 11  SuperNets \nare needed to recognize all 3,000 characters. SuperNets are in turn integrated by a higher \nlevel  network,  the HyperNet.  More  precisely,  the role and  structure of these  kinds  of \nnetworks are as follows: \n\n3.1  SubNet \n\nA  feature  vector extracted from  handwritten patterns is  used as the input (described in \nSection  4.1).  The  number  of units  in  the  output  layer  is  the  same as  the  number  of \ncategories to be recognized by the SubNet. In short, the role of a SubNet is to output the \nsimilarity between the input pattern and the categories allotted to the SubNet. (Fig. 3) \n\n3.2  SuperNet \n\nThe outputs from each SubNet fIltered by the OtherFilter network are used as the input to \n\n\f418 \n\nMori and Joe \n\nthe  SuperNet.  The  number  of units  in  an  output layer  is  the  same  as  the  number of \nSubNets belonging to a SuperNet. In shortt the role of SuperNet is to select the SubNet \nwhich covers the category corresponding to the input patterns. (Fig. 5) \n\nOutput \n\nHorizontal \n\nVertical \n\n+45\u00b0diagonal \n\nOriginal Pattern \n\n3.3  OtherFIIter \n\nFig.  3 S ubNet \n\n45(9x5)  reference  vectors  are  assigned  to  each  SubNet.  LVQ  is  used  to  adapt  these \nreference vectorst so that each  input vector has a reference of the correct SubNet as its \nclosest  reference  vector.  The \nOtherFilter method is to frrst measure \nthe distance between all the reference \nvectors  and  one  input  vector.  The \nmean distance and normal deviation of \ndistance are calculated. The distance \nbetween a S ubNet and an input vector \nis defmed to be the smallest distance \nof that SubNet's reference vectors to \nthe input vector . \n\n\u2022 \n\n\u2022 \n\n\u2022 \n\nd \n\n\u2022  References \nX Input Vector \n\nf(xo}=l 1(1+ e (xn-M+2d)/Cd) (1) \n: The Distance of Nth SubNet \nXn \nM \n: The Mean of Xn \n: The Variance of Xn \nd \nC \n: Constant \n\nFig4. Shape of OtherFilter \n\n\fA Large-Scale Neural Network \n\n419 \n\nThis distance modified by equation (1) is multiplied by the outputs of the SubNet. and fed \ninto  the SuperNet.  The outputs  of SubNets  whose  distance  is  greater  than  the mean \ndistance are suppressed. and the outputs of SubNets whose distance is  smaller than  the \nmean distance are amplified. In this way. the outputs of SubNets are modified to improve \nthe integration of the higher level SuperNet. (Fig. 5) \n\n1 \nHyperNet \nSuperNet \n11 \n330 \nSubNet \nOtherFilter  12 \n\nOther-Filter \n\nFigS. Outline of  the Whole System \n\n4  RECOGNITION  EXPERIMENT \n\n4.1  TRAINING  PATTERN \n\nThe training samples for this network were chosen from a database of about 3000 Kanji \ncharacters  [Saito  1985].  For each  character.  there  are  200 handwritten  samples  from \ndifferent writers.  100 are used as training samples. and the remaining 100 are used to test \nrecognition accuracy of the trained network. All samples in the database consist of 64 by \n63 dots. \n\n\f420  Mori and Joe \n\n~ JlQ  ~~  ~ ~ \n~~lJ  ~ ,~ \n~ .J-~ \n~ ~~  ~~  ~ \n~ .orfffi \n~i2  ~~ \no/N{ \n\n~~  J..~ \nO~  ~ \nDI~J \n\n-\n\nV'#f) \n\nFig.  6  Examples  of  training  pattern \n\n4.2  LDCD  FEATURE \n\nIf we were to use this pattern as the input to our neural net, the number of units required \nin the input layer would be too large for the computational abilities of current computers. \nTherefore, a feature vector extracted from the handwritten patterns is used as the input. In \nthe \"LDCD feature\"  [Hagita 1983], there are 256 dimensions computing a line segment \nlength along four directions: horizontal, vertical, and two diagonals in the 8 by 8 squares \ninto which the handwritten samples are divided. \n\no \n\nt\" \n:61 \n\nhorizontal \ncomponent \nFig 7.  LDCD Feature \n\n4.3  RECOGNITION  RESULTS \n\nIn  the  work  reported  here,  one  SuperNet,  30  SubNets  and  one  OtherFilter  were \nconstructed for recognition experiments.  SubNets  were trained  until  the recognition of \ntraining samples reaches at least 99%. With these SubNets, the mean recognition rate of \ntest patterns was 92%. This recognition rate is higher than that of conventional methods. \nA SuperNet which integrates the output modified by OtherFilter from  30 trained SubNets \n\n\fA Large-Scale Neural Network \n\n421 \n\nwas  then constructed. The number of units in  the input layer of the SuperNet was 270. \nThis  SuperNet was  trained  until  the performance of training samples becomes at least \n93%. With this  SuperNet, the recognition rate of test patterns was 74%, though  that of \nOtherFilter was 72%.  The recognition rate of a  system  without the OtherFilter of test \npatterns was 55%. \n\n5  CONCLUSION \n\nWe have here proposed a  new way of constructing a large-scale neural network for the \nrecognition  of  3,000  handwritten  Kanji  characters.  With  this  method,  a  system \nrecognizing 270 Kanji characters was constructed. This system will become a part of a \nsystem recognizing  3,000 Kanji characters. Only a modest training time was necessary \nowing to the modular nature of the system. Moreover, this modularity means that only a \nmodest re-training  time is necessary  for retraining an erroneous  neural  network in the \nwhole system. The overall system performance can be improved by retraining just that \nneural  network,  and  there  is  no  need  to  retrain  the  whole  system.  However,  the \nperformance of the OtherFilter is not satisfactory. We intend to improve the OtherFilter, \nand build a large-scale network for the recognition of 3,000 handwritten Kanji characters \nby the method reported here. \n\nAcknowledgments \n\nWe are grateful to Dr. Yodogawa for his support and encouragement. Special thanks  to \nDr. Sei Miyake for the ideas he provided in our many discussions. The authors would like \nto acknowledge, with thanks, the help of Erik McDermott for his valuable assistance in \nwriting this paper in English. \n\nReferences \n\nl.S.Denker,  W.R.Gardner,  H.P.  Graf,  D.Henderson,  R.E.  Howard, \n\n[Denier, 1988] \nW.Hubbard, L.DJackel. H.S.Baird, I.Guyon  :  \"Neural Network Recognizer for  Hand(cid:173)\nWritten  ZIP Code  Digits\", NEURAL  INFORMATION PROCESSING  SYSTEMS  1. \npp.323-331, Morgan Kaufmann. 1988 \n[Mori,1988] \nY.Mori. K.Yokosawa :  \"Neural Networks  that Learn to  Discriminate \nSimilar Kanji  Characters\".  NEURAL  INFORMATION PROCESSING  SYSTEMS  1, \npp.332-339, Morgan Kaufmann. 1988 \n[Weideman.1989]W.E.Weideman.  M.T.Manry.  H.C.Yau  ;  tI A  COMPARISON OF A \nNEAREST NEIGHBOR CLASSIFIER AND A NEURAL NETWORK FOR NUMERIC \nHANDPRINT CHARACTER RECOGNITION\". UCNN89(Washington), VoLl, pp.117-\n120, June 1989 \n\n\f422  Mori and Joe \n\nKJoo,  Y.Mori,  S.Miyake  :  \"Simulation  of a  Large-Scale  Neural \n\n[Waibel, 1988]  Alex  Waibel,  \"Consonant Recognition  by  Modular Construction  of \nLarge  Phonemic  Time-Delay  Neural  Networks\",  NEURAL  INFORMATION \nPROCESSING SYSTEMS  1, pp.215-223, Morgan Kaufmann,  1988 \n[Joo,1989] \nNetworks on a Parallel Computer\", 4th Hypercube Concurrent Computers,1989 \n[Khonen,1988]  T.Kohonen,  G.Barna,  R.Chrisley  :  \"Statistical  Pattern  Recognition \nwith Neural Networks\", IEEE, Proc.of ICNN, YoU, pp.61-68, July  1988 \nT.Saito,  H.Yamada,  K.Yamamoto  :  \"On  the  Data  Base  ETL9  of \n[Saito,1985] \nHandprinted Characters in 1IS Chinese Characters and Its Analysis\", J68-D, 4, 757-764, \n1985 \n[Hagita,1983] \nN.Hagita,  S.Naito,  I.Masuda  :  \"Recognition of Handprinted Chinese \nCharacters  by  Global  and Local  Direction  Contributivity  Density-Feature\",  J66-D,  6, \n722-729,1983 \n\n\f", "award": [], "sourceid": 211, "authors": [{"given_name": "Yoshihiro", "family_name": "Mori", "institution": null}, {"given_name": "Kazuki", "family_name": "Joe", "institution": null}]}