{"title": "Information Geometrical Framework for Analyzing Belief Propagation Decoder", "book": "Advances in Neural Information Processing Systems", "page_first": 407, "page_last": 414, "abstract": null, "full_text": "Information Geometrical Framework for\nAnalyzing Belief Propagation Decoder\n\nShiro Ikeda\n\nKyushu Inst. of Tech., & PRESTO, JST\n\nWakamatsu, Kitakyushu, Fukuoka, 808-0196 Japan\n\nshiro@brain.kyutech.ac.jp\n\nToshiyuki Tanaka\n\nTokyo Metropolitan Univ.\n\nHachioji, Tokyo, 192-0397 Japan\n\nShun-ichi Amari\n\nRIKEN BSI\n\nWako, Saitama, 351-0198 Japan\n\ntanaka@eei.metro-u.ac.jp\n\namari@brain.riken.go.jp\n\nAbstract\n\nThe mystery of belief propagation (BP) decoder, especially of the turbo\ndecoding, is studied from information geometrical viewpoint. The loopy\nbelief network (BN) of turbo codes makes it dif\ufb01cult to obtain the true\n\u201cbelief\u201d by BP, and the characteristics of the algorithm and its equilib-\nrium are not clearly understood. Our study gives an intuitive understand-\ning of the mechanism, and a new framework for the analysis. Based on\nthe framework, we reveal basic properties of the turbo decoding.\n\n1 Introduction\n\nSince the proposal of turbo codes[2], they have been attracting a lot of interests because\nof their high performance of error correction. Although the thorough experimental results\nstrongly support the potential of this iterative decoding method, the mathematical back-\nground is not suf\ufb01ciently understood. McEliece et al.[5] have shown its relation to the\nPearl\u2019s BP, but the BN for the turbo decoding is loopy, and the BP solution gives only an\napproximation.\n\nThe problem of the turbo decoding is a speci\ufb01c example of a general problem of marginaliz-\ning an exponential family distribution. The distribution includes higher order correlations,\nand its direct marginalization is intractable. But the partial model with a part of the corre-\nlations, can be marginalized with BP algorithm exactly, since it does not have any loop. By\ncollecting and exchanging the BP results of the partial models, the true \u201cbelief\u201d is approxi-\nmated. This structure is common among various iterative methods, such as Gallager codes,\nBeth\u00b4e approximation in statistical physics[4], and BP for loopy BN.\n\nIt gives a new\nWe investigate the problem from information geometrical viewpoint[1].\nframework for analyzing these iterative methods, and shows an intuitive understanding of\nthem. Also it reveals a lot of basic properties, such as characteristics of the equilibrium, the\ncondition of stability, the cost function related to the decoder, and the decoding error. In this\n\n\fpaper, we focus on the turbo decoding, because its structure is simple, but the framework\nis general, and the main results can be generalized.\n\n2 Information Geometrical Framework\n\n2.1 Marginalization, MPM Decoding, and Belief\n\n(1)\nis the higher order correlations\n\n\u0003\u0017\u0018\u0012\n2 . The problem of turbo codes and similar iterative methods are to marginalize this\n\u0012 . The\n\n\u0003\u0017\u0018\u0012)'*\u000b\r\u000b\f\u000b+',\"\u001e-.\u0003\u0017\u0018\u0012/\u0012$\t\n\u0003\u0017\u0018\u0012\u0018\u0001\u001a\u0019\u001c\u001b\u001e\u001d \u001f!\u0003\u0017\"$#%\u0003&\u0018\u0012!'(\"\n\u0003&\u0018\u0012\u00156\f798\n\u0001;:\ndenote the operator of marginalization as, 4\u001c5\n\u0014HG\n\nLet us consider a distribution of\u0002\u0001\u0004\u0003\u0006\u0005\b\u0007\n\t\f\u000b\r\u000b\f\u000b\u000e\t\u000f\u0005\u0011\u0010\u0013\u0012\u0015\u0014 which is de\ufb01ned as follows\n\u0005\u00111/2 , and each\"\f3 \u0003\u0017\u0018\u0012\nwhere,\"\nis the linear function of0\nof0\ndistribution. Let4\nmarginalization is equivalent to take the expectation of  as\n?@6\f798\n\u0001;A\fBC\n\t\f\u000b\f\u000b\r\u000b\u000e\t/F\nIn the case of MPM (maximization of the posterior marginals) decoding, \u0005\nis the decoding result. In the belief network, \u0005\u00111\nand the sign of eachFN1\n\t\r\u000b\f\u000b\f\u000b\u000e\t\\[]\t^T\n\u0003&SR/T\u0011\u0012\u0018\u0001*\u001b\f\u001d%\u001f\u0013\u0003U\"\n\u0003\u0017\u0018\u0012)',\"\n\u0003&\u0018\u0012>'VT.\u000b\r\n\u0016\u0011Q\n\u0003&SR\u0015T\u0011\u0012\nEach\u0016\nincludes only one of the 0\nused to adjust linear part of\nfor each\u0016aQ , and \ufb01nally approximate4\u001c5\n\n\t$'\n0LK\u0013M\n2 andFN1\n\u0003XT\u0011\u0012/\u0012\u000e\tZY\u0013\u0001\nin eq.(1), and additional parameter T\n\nis\nthe belief. In these iterative methods, the marginalization of eq.(1) is not tractable, but the\nmarginalization of the following distribution is tractable.\n\n(2)\nis\n. The iterative methods are exchanging information through\n\n1=<>\u0007\n1JI\n0PO\nI`_\n\n\u0003&\u0018\u0012\u001e\tD?\u0002\u0001E\u0003&F\n\nK\u0002W\n\u0003&\u0018\u0012$2\n\u0003&\u0018\u0012 .\n\n\u0003\u0006\u0005\n\n2.2 The Case of Turbo Decoding\n\nb\bcNd9e%f\u0013g\u000ehNi\\f\njNk\\d\nw\u001ex\nm\rn\u0015oXprq\u0015sXtvu\nw{z\nm\rn\u0015oXprq\u0015sXtNy\n\nl+}\n\nw\\z\n\n~\u0080\u007fL\u0081\u0083\u0082X\u0084\u000e\u0085\n\nl+}\n\nw{z\n\nFigure 1: Turbo codes\n\nl+}\n\nw$x\n\nb\bcNd9e%f\u0013\u0087\u0088k\\i\\f+jNk\\d\n\nsXoXprq\u0015sXtLu\n\nsXoXprq\u0015sXt\ny\n\nl+}\n\n\u008f)\u0012\u0015\u0014\n\nis the information bits, from which the turbo encoder generates\n\nis then transmitted over a\nnoisy channel, which we assume BSC (binary symmetric channel) with \ufb02ipping probability\n\nIn the case of turbo codes,\ntwo sets of parity bits, \u008c\n0LK\u0013M\ntaken over a subset of 0LM\n\u0094\u001c\u0095\nThe ultimate goal of the turbo decoding is the MPM decoding of based on\u0016\n\u0097N\u00a2\n\n\u0001\u0091\u0003&\u008e\n\u0007,\u0001\u008d\u0003\u0006\u008ev\u0007\\\u0007\n\t\f\u000b\f\u000b\r\u000b\u000e\t/\u008ev\u0007/\u008f\u000e\u0012\u0015\u0014\n, and \u008c\u000e\u0090\n(Fig.1). Each parity bit is expressed as the form :\n\t{'\n\t\r\u000b\f\u000b\r\u000b\u000e\t\u000f\u0093V2 . The codeword \u0003&S\t\n\u0012 , \u0098\n\u0007\u009a\t\u0011\u0098\nM+\u0096\n\u0097 . The receiver observes \u0003\n\u0098\nS\t\u0099\u0098\n\u008ev\u00079\u0092N\t\n\u0005\u00111\u000f\t\n\u008c\u000e\u0090\n0vK\u0013M\n\u0007\u0088'\u00a0\u009d@\u0098\n\u001c\u000b\r\u009f'\u00a0\u009d@\u0098\n\u0018\u0012\u0018\u0001*\u001b\f\u001d%\u001f\u0013\u0003\u0017\u009d\u009e\u0098\n\u0007\u0083\u000b\n\u008c\u000e\u0090SK\n\u008c\u000e\u0090\n6\r798\n\u0003\u0017\u00b0\f\u00b1\u00b2'(\u00b0N\u00b3\u0011\u00b1\u0011\u0012\n\u009d>\u0012\u001e\t\u00ac\u00a3\u00ad\u0003&\u009d>\u0012\n\u0001\u00af\u00ae\nM\u009cK\u00a7\u00a6{\u00a8\n\u00a9\u00ab\u00aa\n\n\u0007+\t\r\u000b\f\u000b\f\u000b\u000e\t\u000f\u008e\n\u0005\u00111 , where the product is\n2 .\n\u0012 .\n\u0003\u0017\u0093\u00a1'\n\nSince the channel is memoryless, the following relation holds\n\n\u0003\u0017\u009c\u009b\nS\t\n\u0012r\u00a3\u00a4\u0003\u0006\u009d>\u0012\u000f\u0012\n\n\u0003\n\u0098\nS\t\u0099\u0098\n\u009d\u00a0\u00a5\n\n, \u008e%\u0007U\u0092N\t\u000f\u008e\n\n\u0007\u009a\t\u0011\u0098\n\u008c\u000e\u0090\n\n\u008c\u000e\u0090\n\n\t$'\n\n\u0007+\t\n\n\u0016\n\u0007\n#\n\u0005\n1\n\u0016\n\u0010\n\u0016\n1\n\u0016\n\u0007\n\u0010\n\u0012\nM\n2\nI\n\t\nM\n#\nQ\nQ\nM\n\u0010\nG\nQ\n\"\n3\nT\n\u0016\nl\n|\n|\n|\n|\n|\n|\n|\n|\nw\nx\n}\n|\n\u0086\n\u0086\nw\nx\n}\n\u0086\n|\n\u0089\n\u008a\nz\n\u0089\n\u008a\nx\n\u0086\n\u0086\n\u0086\n\u0086\nw\nz\n|\n|\n\u008b\n\u008b\n|\n\u008b\n\u0090\n\u0090\n\u0090\n\u0092\nI\nM\n2\n1\n\u008c\n\u0007\n\t\n\u008c\n\u0090\n\u0012\n\u008c\n\u0098\n\u0098\n\u008e\n\u0090\n\u0092\nI\nM\n\u0098\n\u0098\n\u008c\n\u0098\n\u0016\n\u008c\n\u009b\n\u008c\n\u008c\n\u000b\nO\n\t\n\u0094\n\u0001\nM\n\u0097\n\u0003\n\u00a9\nG\n\f\u0018\u0012\n\n\u0012\u0018\u0001\n\n'\u00a0\u009d\n\n, the posterior distribution is given as follows\n\nBy assuming the uniform prior on\nS\t\n\u0007+\t\n\u0018\u0012\n\u008c\u000e\u0090\n\u0003&\u009c\u009b\n\u0083\t\n\u0019\u001c\u001b\f\u001d%\u001f\u009e\u0003&\u009d\n\u001c\u000b\r\u009f'\u00a0\u009d\n\u0003+\u0098\nS\t\u0099\u0098\n\u0007+\t\u0011\u0098\n\u008c\u000e\u0090\n\u0003&\u0018\u0012!'(\"\r\u0007+\u0003\u0017\u0018\u0012!'(\"\n\u0003U\"\n\u0019\u00a7\u001b\f\u001d \u001f\n\u0003\u0017\u0018\u0012/\u0012\n\u0003\u0006Y\u0004\u0001\nis the normalizing factor, and \"$#v\u0003&\u0018\u0012(\u0001\u008d\u009d\nHere \u0019\n\u0003\u0017\u0018\u0012(\u0001\u0091\u009d\n, \"\nS\u000b\nEquation(3) is equivalent to eq.(1), where [^\u0001\n\u0097 . When \u0093\n\u0003\u0017\u009c\u009br\u0098\nS\t\u0011\u0098\n\u0007\u009a\t\u0099\u0098\n\u008c\u000e\u0090\nis intractable since it needs summation over \u0097\n\u0003\u0017SR\u0015T\u0011\u0012\u00ad\u0003\u0006Y\u001c\u0001\nlize two decoders which solve the MPM decoding of\u0016aQ\n\u0018\u0012 and the prior of which has the form of\n\u0083\t\ndistribution is derived from\u0016\n\u001b\f\u001d \u001f!\u0003XT\n\u0003\u0017SR\u0015T\u0011\u0012\u0018\u0001\n\u00a3\u00a4\u0003XT\u0011\u0012\u000f\u0012\n\u0003&SR/T\u0011\u0012\n\u0083\t\nis a factorizable distribution. The marginalization of\u0016\nits BN is loop free. The parameterT\nbetween the two decoders. The MPM decoding is approximated by updating T\n\nin \u201cturbo\u201d like way.\n\n\u0018\u0012\n\n\u000b\f\n\nis feasible since\nserves as the window of exchanging the information\niteratively\n\nis large, marginalization of\nterms. Turbo codes uti-\nin eq.(2). The\n\n\u0012 .\n\n(3)\n\n2.3 Information Geometrical View of MPM Decoding\n\nas\n\n,\n\n(4)\n\nwhich is de\ufb01ned as\n\nis decomposable, or factorizable. From the informa-\n\nLet us consider the family of all the probability distributions over \n. We denote it by\u0002\n\u0003&\u0018\u0012S\u00a5\n\u0003\u0017\u0018\u0012\u0018\u0001\n\t\u000f\n\t$'\n\u0003&\u0018\u0012\u0005\u0004\nM\u0007\u0006\n0LK\u0013M\nWe consider an\u00b0 \u2013\ufb02at submanifold\b\n\u0003\u0017SR\n\t\u000e\u0012 de\ufb01ned\n. This is the submanifold of\u0016\nin\u0002\n\u0010\u0014\u0013\n\u0001\f\u000b\n\u0003&SR\r\t\u000e\u0012\u0080\u0001\u001a\u001b\u001e\u001d \u001f\u009e\u0003\u0017\"\n\u0003\u0017\u0018\u0012!'\u000e\t`\u000b\f\n\u0003\u000f\t\u000e\u0012\u000f\u0012)\u009b\n\u0001E\u0003\u0011\u0010\n\t\f\u000b\f\u000b\r\u000b\u000e\t\u0012\u0010\nK\u0002W\n# can be rewritten as follows\n\u0003&\u0018\u0012\u0088\u0001*\u009d\u009e\u0098\nSince\"\n\u0002\u000b\f\n, every distribution of\b\n\u000b\r\n\u0003&SR\r\t\u000e\u0012\u0080\u0001\u001a\u001b\u001e\u001d \u001f\u0013\u0003U\"\n\u0003\u0017\u0018\u0012!'\u000e\t\n\u000b\r\n\u001b\u001e\u001d \u001f\u009e\u0003\u000f\u0003\u0006\u009d\u009e\u0098\n]'\u0015\t\u000e\u0012\n\u0003\u000f\t\u000e\u0012\u000f\u0012\u0018\u0001\n\u0003\u0016\t\u000e\u0012/\u0012\nK\u0002W\nK\u0002W\nIt shows that every distribution of\b\ntion geometry[1], we have the following theorem of\u0017 \u2013projection.\n\u0003&\u0018\u0012\nbe an\u00b0 \u2013\ufb02at submanifold in\u0002\n. The point in\b\nTheorem 1. Let\b\n, and let\u0018\n\u0003&\u0018\u0012\nto\b\nminimizes the KL-divergence from\u0018\nB%$'&\n\u0003&\u0018\u0012\u0088\u0001\n\u0003&\u0018\u0012-,U\t\n\u0003\u0017\u0018\u0012$R\n\u00a8\u0005\u001c\n\u001d\u001f\u001e\u0014 \n4\u001a\u0019\u009f5\u001b\u0018\n!#\"\n\u0019)(+*\n\u0003&\u0018\u0012\nand is called the\u0017 \u2013projection of\u0018\n. The\u0017 \u2013projection is unique.\nto\b\n# [7]. Since\nIt is easy to show that the marginalization corresponds to the\u0017 \u2013projection to\b\n\u0017 \u2013projection to\b\n\u0003&\u0018\u0012 denote the parameters in\b\nof the\u0017 \u2013projected distribution,\nLet./\u0019\n&#576\n\u0003\u0017\u0018\u0012\u0018\u0001\n\u00a8\u001f\u001c1\u001d2\u001e3 \n(8*\n\nMPM decoding and marginalization is equivalent, MPM decoding is also equivalent to the\n\n2.4 Information Geometry of Turbo Decoding\n\nThe turbo decoding process is written as follows,\n\n\u0003&SR\r\t\u000e\u0012',\n\n.0\u0019\u009f5#\u0018\n\n, is denoted by,\n\n\u0003\u0017\u0018\u0012$R\n\n5\u0005\u0018\n\n# .\n\nthat\n\n\u0016\n\u0098\n\u0098\n\u008c\n\u0007\n\t\n\u0098\n\u008c\n\u0090\n\u0016\n\u0003\n\u0098\n\u0098\n\u008c\n\u0098\n\u009b\n\nB\n\u0016\n\u008c\n\u009b\n\u0001\n\u0098\n\u0098\n\u008c\n\u0007\n\u000b\n\u008c\n\u0007\n\u0098\n\u008c\n\u0090\n\u000b\n\u008c\n\u0090\n\u0012\n\u0001\n#\n\u0090\nG\n\u0098\n\nQ\n\u0098\n\u008c\nQ\n\u000b\n\u008c\nQ\nM\n\t\n\u0097\n\u0016\n\u008c\n\u0012\n\u0010\nM\n\t\n\u0097\n\u0012\n\u0003\n\u0098\n\u0098\n\u008c\nQ\n\u009b\n\u0001\nK\nG\n\u0001\n\u0003\n\u0098\n\u0098\n\u008c\nQ\n\u009b\n\u0002\n\u0001\n\u0003\n\u0016\n\u0004\n\u0004\n\u0016\nO\nI\nM\n2\n\u0010\n\t\nA\nB\n\u0016\nG\n#\n#\n\b\n#\n\u0016\n#\n#\n#\n\t\n\u0007\n\u0010\n\u0012\n\u0014\nI\n_\nG\n#\n\u0016\n#\n#\n#\n#\nG\n#\nI\n\u0002\n\u00a9\n\u0018\n\u0016\n\u00a9\n4\n\u0018\n\u0016\nG\n\f\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\n\u0006\u0005\n\nby\n\nby\n\nfor \u0002\n\n\u0012 , and calculateT\n\nThe turbo decoding approximates the estimated parameter\n\n1. LetT\u0001\nO , and \u0002\nM .\n# as\t\n\u0012 onto\b\n\u0003&SR/T\u0001\n\u0003&SR/T\u0001\n2. Project\u0016\n./\u0019\u0004\u0003\u009a5\n\u0003&SR\u0015T\n.0\u0019\u0007\u0003P5\n# as\t\n\u0012 onto\b\n\u0003&SR/T\n\u0003&SR\u0015T\n\u0012 , and calculateT\n3. Project\u0016\b\u0007\n\u0016\b\u0007\n\u0003\u009a5\n\u0007\n\u0003&SR/T\n./\u0019\u0004\u0003\u009a5\n\u0012 , go to step 2.\n\u0003&SR/T\n\u0012\u0007\b\n\u0007\n\u0003&SR\u0015T\n4. If.0\u0019\u0007\u0003P5\n.0\u0019\u0007\u0003P5\n\t\n\t ,\n# , as\t\n\t\u009c\u0001\n'VT\u000b\t\n\u0012 onto\b\nT\u000b\t\n\u0007\u009a\t\nS\t\n\u0003\u0017\u009c\u009b\n\u008c\u000e\u0090\n\u0090 , where the estimated distribution is\n'\u00a0T\n\u0012\u0018\u0001\u001a\u001b\u001e\u001d \u001f\u009e\u0003&\"\n\u0003&\u0018\u0012!'\u00a0T\n'VT\n\u0003&SR\r\t\n\u0012/\u0012\n\u000bP\n\u000b\r\nKVW\nAn intuitive understanding of the turbo decoding is as follows. In step 2, \u0003\u0006T\u000b\t\n\u000b\u0006\u0018\u0012\n\u0012 , andT\u000b\t\n\u0003\u0017SR\u0015T\u000b\t\n\u0003&\u0018\u0012 . The distribution becomes\u0016\nreplaced with\"\n# . In step 3, \u0003XT\u000b\t\nin eq.(5) is replaced with\"\r\u0007\n\u0003&\u0018\u0012 , andT\u000b\t\n\u000b\r\u0018\u0012\nis estimated by\u0017 \u2013\nit onto\b\n\u0007+\u0003\u0017SR\u0015T\u000b\t\n\u0012 .\nprojection of\u0016\n\u0010\u0014\u0013\n\u0003\u0017SR\u0015T\u0011\u0012\u0018\u0001\u001a\u001b\u001e\u001d \u001f\u0013\u0003\u0017\"$#v\u0003&\u0018\u0012!'(\"\n\u000bP\n\t\f\u000b\r\u000b\f\u000b\u000e\t\r\f\n\u0001\u0004\u0003\u0006\f\n\u0003XT\u0011\u0012\u000f\u0012\f\u009b\nI`_\nY\u009e\u0001\nis also an \u00b0 \u2013\ufb02at submanifold.\nQ .\n\b\u00a0\u0090 and\nis the coordinate system of\b\n# hold because \"\n\u0003\u0017\u0018\u0012\n\u0003&\u0018\u0012\nand \"\r\u0007N\u0003&\u0018\u0012\u000f\b\n\u0001J\"\nincludes cross terms of \n\nThe information geometrical view of the turbo decoding is schematically shown in Fig.2.\n\n(5)\nin eq.(5) is\nis estimated by projecting\n\nWe now de\ufb01ne the submanifold corresponding to each decoder,\n\n3 The Properties of Belief Propagation Decoder\n\n\u0003\u0017\u0018\u0012!'\u00a0T\n\nthe projection of\n\nin general.\n\nKVW\n\n\u0007\u000e\b\n\n\u0003XT\n\n3.1 Equilibrium\n\n\u0012\u0088\u0001\n\n(6)\n(7)\n\n\u0003\u0017SR\u0015T\n\nWhen the the turbo decoding converges, equilibrium solution de\ufb01nes three important dis-\n\n\u0003\u0017SR\u0015T\u000b\t\n\u0012 ,\u0016\n#L\u0003\u0017SR\n\t\n\t\f\u0012 . They satisfy the following two conditions:\n\u0003\u0017SR\u0015T\u000b\t\n\u0012 , and\u0016\ntributions,\u0016\b\u0007\n\u0007+\u0003&SR/T\n\u0012H\u0001\n\u0003&SR\r\t\n4\u001c5\n4\u001c5\n'VT\n\u0003\u000f\t\u000e\u0012 as\nLet us de\ufb01ne a manifold\b\n\u0003\u0017\u0018\u0012\n\t\u009fA\n\u0003&\u0018\u0012\u0015\u00a0\u0001\n\u0003&\u0018\u0012\u0005\u0004\n\u0003\u0016\t\u000e\u0012\u0080\u0001\n\u0003&SR\r\t\u000e\u0012\u0015\n\u0003&\u0018\u0012\n\u0003\u000f\t\u000e\u0012 , the expectation of \nFrom its de\ufb01nition, for any \u0016\nis the same, and its\u0017 \u2013\n# coincides with\u0016\n\u0003\u0017SR\n\t\u000e\u0012 . This is an\u0017 \u2013\ufb02at submanifold[1], and we call\nprojection to\b\n\u0003\u0017SR\u0015T\u000b\t\n\u0003\u0017SR\u0015T\u000b\t\n#L\u0003\u0017SR\n\t\n\t\r\u0012$\t\n\u0003\u000f\t\u000e\u0012 an equimarginal submanifold. Since eq.(6) holds,\u0016\n\u0012$\t\n\u0016\b\u0007\n\u0003\u000f\t\n\t\f\u0012\nLet us de\ufb01ne an \u00b0 \u2013\ufb02at version of the submanifold as\n\u0003\u0016\t\n\t\f\u0012 , which connects\u0016\n\u0003\u0017SR\n\t\n\t\u001e\u0012 ,\n\u0003&SR/T\u000b\t\n\u0012 , and\u0016\n\u0007\n\u0003&SR/T\u000b\t\n\u0003\u0017SR\n\u0007+\u0003&SR\n\u0003&SR\n\u0012\u0018\u0001\n\u0003\u000f\t\n\u0003\u000f\t\u000e\u0012 .\nSince eq.(7) holds, \u0016\n\n\u0003\u0017\u0018\u0012\u0018\u0001\n\u0003&\u009c\u009b\n\u0083\t\nM .\n\nIt can be proved by taking\n\nin log-linear manner\n\nis included in the\n\nK\u0013M , \u0002\n\nis satis\ufb01ed.\n\n\u008c\b\u0090\n\n\u0007\u009a\t\n\n\u0012\u0011\n\n\u0006\u0013\n\n\u0007\n\u0001\nO\n\u0001\n\u0001\n\u0090\n\u0007\n\u0001\n\u0016\n\u0090\n\u0007\n\u0007\n\u0090\nT\n\u0007\n\u0090\n\u0001\n\u0016\n\u0090\n\n\u0007\n\u0012\nK\nT\n\n\u0007\nG\n\u0007\n\u0090\n\u0001\n.\n\u0019\n\u0007\n\u0090\n\u0007\n\u0007\nT\n\u0007\n\u0007\n\u0001\n\u0016\n\u0007\n\u0090\n\u0012\nK\nT\n\u0007\n\u0090\nG\n\u0016\n\u0007\n\u0090\n\u0001\n\u0016\n\u0090\n\u0007\n\u0007\n\u0016\n\u0098\n\u0098\n\u008c\n\u0098\n\u0007\n\u0016\n#\n\t\n#\n\t\n\u0007\n\t\n\u0090\n#\n\t\n\u0007\n\t\n\u0090\nG\n\u0090\n\u0090\n\u0090\n\u0007\n\u0090\n\u0007\n\u0007\n\u0090\n\b\nQ\n\u0001\n\u000b\n\u0016\nQ\nQ\nQ\nT\n\u0007\n\u0010\n\u0012\n\u0014\nM\n\t\n\u0097\nG\nT\n\b\nQ\n\b\n\u0001\n\b\nQ\n\b\n\u0001\n\b\nQ\n\u0090\n\u0007\n\u0090\n\u0090\nM\nG\n\u0016\n\t\n\u0090\n\u0016\n\u0090\n\t\n\u0007\n\u0016\n#\n\t\n\u0012\nG\n\u0097\nG\n\t\n\t\n\u0001\nT\n\t\n\u0007\n\t\n\u0090\nG\n\b\n\u0003\n\u0016\n\u0004\n\u0004\n\u0016\nI\n\u0002\nB\n\u0016\nA\nB\n\u0016\n#\n\u0006\nG\nI\n\b\n#\n\b\n\u0090\n\u0016\n\u0090\n\u0007\n\u0012\nI\n\b\n\u0010\n#\n\u0016\n\u0090\n\u0090\n\u0007\n\u0012\n\u0010\n\t\n\u0003\n\u0016\n\u0019\n\u0016\n#\n\t\n\t\n\u0012\n\n\u0003\n\u0016\nT\n\t\n\u0090\n\u0012\n\u0016\n\u0090\nT\n\t\n\u0007\n\u0012\n\u0004\n\u0004\n\u0004\n\u0090\nA\nQ\n<\n#\n\u0002\nQ\n\u0001\nM\n\u0006\nG\n\u0098\n\u0098\n\u008c\n\u0098\n\u0012\n\u0010\n\u0002\n#\n\u0001\n\u0007\n\u0001\n\u0002\n\u0090\n\u0001\n\fpqxzw\n\nFigure 2: Turbo decoding\n\nincludes these three distributions and also the pos-\n\n#%$'&)(\n-/.\n\n\u001f1!\n\n\t\u0014\u0013\n\t\u0014\u0013\n\nHowever,\n\ndecoding error.\n\n3.2 Condition of Stability\n\nUXWZY=s\n\nBNQ\n\nBPO\nBNM\n\n\u001b\u001d\u001c\n\u001e \u001f\"!\n\u001b\u001d\u001c\n-/.\n\u001f1!\n\u001b\u001d\u001c\n-/.\n\u001f\"!\n\nTheorem 2. When the turbo decoding procedure converges, the convergent probability\n\n\u001e+*\n#\u0014,\n\u001f\"!\n\u0002\u0004\u0003\u0006\u0005\b\u0007\n\t\f\u000b\u000e\r\u0010\u000f\u0010\u0011\u000e\u0012\n2'3'4\u000e57698\f:\n;=<\n>7?\n\u0002\u0004\u0003\u0006\u0005\b\u0007\n\t\f\u000b\u000e\r\u0010\u000f\u0010\u0011\u000e\u0012\n\u001e+*\n\nR\u001dU\u0014Wvu\bw\nWrx+w\nKLCFEHGJI\nR\u0004pqU\u0014WrY9s\nBDCFEHGJI\nRTSVUXWZY\\[^]`_\na+b=cVdfehg\u001deji\bk'l\ncJm=n\nk'o\n\u0003\u000f\t\n\t\r\u0012\n\u0003\u000f\t\n\t\f\u0012 and\nFigure 3:\b\n\u0012 , and \u0016\n\u0007+\u0003\u0017SR\u0015T\u000b\t\n\u0003&SR/T\u000b\t\n\u0012 belong to equimarginal submanifold\n\u0003&SR\r\t\n\t\u001e\u0012 , \u0016\ndistributions \u0016\n\u0003\u000f\t\n\t\r\u0012\n\u0003\u000f\t\n\t\f\u0012 , while its\u00b0 \u2013\ufb02at version\n\u0083\t\n\u0003&\u009c\u009b\n\u0012 (Fig.3).\nterior distribution\u0016\n\u0003&SR\r\t\n\t\u001e\u0012\n\u0012 ,\u0016\n\u0007+\t\n\u0003\u0016\t\n\t\u001e\u0012\n\u0003\u0017\u009c\u009b\n\u0083\t\n\u0012 .\nS\t\n\u0003\u0017\u009c\u009b\n\u0007+\t\nincludes\u0016\nis the true marginalization of\u0016\n\u008c\u000e\u0090\n\u008c\u000e\u0090\nIf\b\n\u0003\u0016\t\n\t\u001e\u0012 does not necessarily include \u0016\n\u0012 . This fact means that\n\u0012 and\u0016\n\u0007\u009a\t\u0099\u0098\n\u0003\u0017\u009c\u009br\u0098\nS\t\u0011\u0098\n\u0003&SR\r\t\n\t\f\u0012 are not necessarily equimarginal, which is the origin of the\n\u008c\u000e\u0090\nThe expectation parameters are de\ufb01ned as follows withW\n\u0003XT\u0011\u0012%6\r798\n#L\u0003\u0016\t\u000e\u0012$\tZ?\n\u0001L{\n#v\u0003&SR\n\t\u000e\u0012\n\u0001EA\n?\u000e#L\u0003\u0016\t\u000e\u0012%6\f798\n\u0001\u0004A\n\u0012\u0018\u0001\u001a?\n\u0012\u0018\u0001\u001a?>\u0007+\u0003\u0006T\n\u0003\u0016\t\ntoT\u000b\t\n\u0007 and apply one turbo decoding step. The\nWe give a suf\ufb01ciently small perturbation ~\n# gives,\n\u0003&SR\u0015T\n\u0017 \u2013projection from\u0016\nto\b\n\u0003XT\n'\u0080\u007f\u0014\t\u000e\u0012\u0018\u0001*?\n\u0003XT\n#v\u0003\u0016\t\n\u0001\u0082\u0081\n\u007f\u0014\t\n#v\u0003&SR\n\t\u000e\u0012 , and \u0081\n#L\u0003\u0016\t\u000e\u0012\nis the Fisher information matrix of\u0016\nHere, \u0081\n\u0003\u0016\t\u000e\u0012\n\u0012 . Note that\u0081\n\u0003\u0006Y\u009e\u0001\n\u0003\u000f\t\u000e\u0012{\t\u0084\u0081\n|\\|\n\u0003XT\u0011\u0012\u0088\u0001\u0082{\n\u0003\u000f\t\u000e\u0012\u0018\u0001L{\n\u0003\u000f\t\u000e\u0012\u0018\u0001L{\n\u0003XT\u0011\u0012\u0018\u0001L{\n4\u001b4\"\u0083\n\u0010\u008a\u0089\n\u0003XT\n\u0012$\u00b3\n'\u0086\u0085\u0014\u0081\n\u0003\u0016\t\nK\u0088\u0087\nis an identity matrix of size\u0093\n\nis that of\u0016\n\u0003XT\u0011\u0012\u001e\tZY\u0013\u0001\n\nis a diagonal matrix. The Fisher information matrix is de\ufb01ned\n\nin eq.(4) andW\n\u0003&SR\u0015T\u0011\u0012\u0018\u0001L{}|\n\nin eq.(2)\n\n\u0003XT\u0011\u0012^Y\u009e\u0001\n\nEquation (6) is rewritten as follows with these parameters,\n\ntheorem which coincides with the result of Richardson[6].\n\n\u0003\u0006T\u0011\u0012\n\n\u0003&SR\u0015T\u0011\u0012 ,\n\nas follows\n\nin step 2 will be,\n\n\u0003\u0017\u009c\u009b\n\n\u0083\t\n\n\u0003\u0006T\n\nHere,\u0087\n\n. Following the same line for step 3, we derive the\n\n\u0003\u0016\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u0001\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0016\n\u0016\n\u0016\n\u0016\n\u0017\n\u0018\n\u0018\n\u0019\n\u001a\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0016\n\u0016\n\u0016\n\u0016\n\u0017\n-\n-\n\u0016\n\u0016\n\u0016\n\u0016\n\u0017\n\u0019\n0\n\u001e\n\u001f\n!\n&\n\u001f\n!\n#\n&\n#\n\u0018\n\u0018\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n\u0015\n-\n\u0018\n\u0018\n\u0019\n@\n\u001e\n&\n$\n#\n(\n\u001f\n!\n&\n,\nA\n]\nt\n_\nR\nt\n]\np\n_\ny\ny\nt\n_\n\u0010\n#\n\u0090\n\u0090\n\u0007\n\b\n\u0010\n\u0098\n\u0098\n\u008c\n\u0007\n\t\n\u0098\n\u008c\n\u0090\n\u0098\n\u0098\n\u008c\n\u0098\n\u0098\n\u0098\n\u008c\n\u0098\n\b\n\u0098\n\u0098\n\u008c\n\u0007\n\t\n\u0098\n\u008c\n\u0090\n\u0016\n\u008c\n#\n#\nQ\nB\n\n\u0016\n4\nW\nQ\nB\n\n\u0016\nQ\nW\nQ\nM\n\t\n\u0097\nG\n?\n#\n\t\n\t\n\u0090\n\u0090\n\t\n\u0007\n\u0012\nG\n\u0090\n\t\n'\n~\n\u0012\n?\n#\n\t\n\u0090\n\t\n\u0007\n'\n~\n\u0012\n\t\n\u0012\n\u00b3\n\u0007\n\u0081\n\u0090\n\t\n\u0007\n\u0012\n~\nG\nQ\nQ\nM\n\t\n\u0097\n#\n\u0081\n#\nW\n#\n4\n?\n#\nQ\n\u0083\nW\nQ\n|\n?\nQ\nM\n\t\n\u0097\nG\nT\n\u0090\nT\n\u0090\n\u0001\nT\n\t\n\u0090\n#\n\t\n\u0007\n\u0081\n\u0090\n\t\n\u0007\n\u0012\n~\nG\n\u0010\n\fde\ufb01ned as\n\nis\n\nis de\ufb01ned as,\n\n.\n\n\u0012 . For the\n\n\u0012{\u00b3\n\n\u0003\u0006T\n\n\u0089Z\u0085\n\nis small,\n\n\u0006\u0005\n\u0006\u0005\n\n\u0003\u0006T\n\n\u0010\u008a\u0089\n\nK\u0088\u0087\n\n3.3 Cost Function and Characteristics of Equilibrium\n\nTheorem 3. Let\nWhen \u009b\n\n1{\u009b\n\nWe give the cost function which plays an important role in turbo decoding.\n\nThis shows how the algorithm works, but it does not give the characteristics of the equilib-\n\n1 be the eigenvalues of the matrix\u0001\n#%\u0003\u000f\t\n#%\u0003\u000f\t\n\u0012$\u00b3\nM holds for all\u0002 , the equilibrium point is stable.\n\u0012\u0018\u0001\n\t/T\n#L\u0003\u0016\t\u000e\u0012\n\u0003XT\n\u0012!'\n\u0003\u0006T\n\u0003XT\n\u0012/\u0012\n\u0001*T \u0007\u0018'\u00a0T\nHere,\t\n\u0090 . This function is identical to the \u201cfree energy\u201d de\ufb01ned in [4].\nG\rG\fG\n\t/T\u000b\t\nTheorem 4. The equilibrium stateT\u000b\t\nis the critical point of\u0003\n\u0003\u000f\t\u000e\u0012\n\u0003XT\u00ab\u0007\f\u0012 ,{\n\u0003\u000f\t\u000e\u0012\n\u0001*?\n?>\u0007\u009a\u0003XT\n\u0001*?\nProof. Direct calculation gives{\n\u0012 holds, and the proof is completed.\n\u0003\u0006T\u000b\t\n\u0012\u0018\u0001\u001a?\n\u0003\u0006T\u000b\t\nequilibrium,?\u000e#L\u0003\u0016\t\n\t\f\u0012\u0018\u0001\u001a?\nT\u0001\nWhen\u0003XT\u0001\u0006\u0005\n#%\u0003\u000f\t\u000e\u0012\nT\u0001\n{}|\nT\u0001\n#%\u0003\u000f\t\u000e\u0012\n\u0090\u0006\u0005\n\u0005\b\u0007\nrium point. The Hessian of\u0003\n\u0081\u00b2\u0007\nT \u0007\u0018'\u00a0T\nAnd by transforming the variables as,\t\n\u0090 and\u000b\n\u0090 , we have\n\u0003\u000f\t\u000e\u0012\n\u0081\u00b2\u0007H'\n\u0003X\u0081\u00b2\u0007\n4\r\f\n4\u001b4\n\u0081\u00b2\u0007\u0018'\u0080\u0081\n\u0003X\u0081\u00b2\u0007\n\f\u000e\f\nis positive de\ufb01nite but {\nMost probably,{\nis always negative, and\u0003\n\f\u0010\f\n\u0003&SR\r\t\n\t\u0012\u0011!\u0012 as\nFor the following discussion, we de\ufb01ne a distribution\u0016\n\t\u0013\u0017\n\u000b\u0016\u0015\u0011\u0003\u0017\u0018\u0012\n\t\u0013\u0011!\u0012\u0018\u0001\n\u0003\u0017SR\n\t\n\t\u0013\u0011!\u0012/\u0012\b\t\u0006\u0011]\u0001\u0004\u0003\u0018\u0017\n\u0003\u000f\t\n\u001b\u001e\u001d \u001f\u009e\u0003\u0017\"\n\u0003\u0017\u0018\u0012!'\u000e\t`\u000b\f]'\u0014\u0011\nK\u0002W\n\u0003&\u0018\u0012$\t\\\"\n\u0001(\u0003&\"\n\t\u0012\u0011!\u0012H\u0001\n\u0003\u000f\t\n\u000b\u0019\u0015\u0011\u0003\u0017\u0018\u0012/\u0012\b\t\u001a\u0015a\u0003&\u0018\u0012%6\r798\n\u001b\f\u001d \u001f\u009e\u0003&\"$#v\u0003&\u0018\u0012!'\u0015\t\n'\u0014\u0011\n\u000b\r\n\u0003&\u0018\u0012/\u0012\n,\u0011\u0002\u0001\u001f\u001e ), and\u0016\n\u0012 (\t\u001c\u0001\u001d\u001b\n\u0003&\u009c\u009b\n#v\u0003&SR\n\t\u000e\u0012 (\u0011\u0002\u0001\u001c\u001b ),\u0016\nS\t\nThis distribution includes\u0016\n\u0012/\u0014\n\u0012/\u0014\n\u0012/\u0014\n(\t,\u0001;T\n\u0001\u001f \nQ ), where\u001e\u00a7\u0001\n, \u0011\u0007@\u0001\n, and \n,\u0011\nparameter?\u0083\u0003\u0016\t\n\t\u0012\u0011!\u0012\n\t\u0012\u0011!\u0012\n\u0003\u0017SR\n\t\n\t\u0013\u0011!\u0012\u0018\u0001\n\u0003\u000f\t\n\u0001L{\n\t\u0012\u0011!\u0012\n?\u0083\u0003\u0016\t\n\u0012 has the same expecta-\n\u0003\u0017SR\n\t\n\u0012 , where every distribution\u0016\n\u0003\u000f\t\n\u0003\u000f\t\n\t\u0012\u0011!\u0012\nLet us consider\b\ntion parameter, that is,?S\u0003\u000f\t\n\t\u0013\u0011!\u0012H\u0001\n?S\u0003\u000f\t\n\t\f\u0012 holds. Here, we de\ufb01ne,?S\u0003\u000f\t\n\t\r\u0012H\u0001\n?S\u0003\u000f\t\n\t+\t!\u001ba\u0012 . From\n\u0012\"\u0017\nFL1\u000f\u0003\u0016\t\n\u0012\"\u0017\nFN1\\\u0003\u000f\t\n\u00129\u007f\n{\n\u0092PFN1/\u0003\u0016\t\n\u0012!'\nFL1\u000f\u0003\u0016\t\n\t\u0012\u0011!\u0012\u0018\u0001\nFL1\u000f\u0003\u0016\t\nQ\u0013#\n\u0012\"\u0017\n\u0003\u000f\t\n'\u0002A\n\u0003\u0013&1\u007f\u0014\t*&+)\f\u0012\n\u0003!&'\u0011(&\u0019)P\u0012!'\n\nthe Taylor expansion, we have,\n\n3.4 Perturbation Analysis\n\n. The expectation\n\n\u0003&SR/T\u0011\u0012\n\nsaddle at equilibrium.\n\n4\u001b4\n\nT \u0007\n\nis generally\n\n(8)\n\n\u00129\u007f\n\n\u0003\u0016\t\n\n\u0001\n\u0001\n\u0085\n\u0081\n\t\n\u0007\n\u0081\n\u0007\n\t\n\u0090\n\u0012\nK\n\u0087\n\u0010\n\u0081\n\t\n\u0007\n\u0081\n\u0090\n\t\n\u0007\n\u0012\nG\n\n\u0095\n\u0003\n\u0007\n\u0090\nW\nK\n\u0003\nW\n\u0007\n\u0090\nW\n\u0090\n\u0007\nG\n\u0007\n\t\n-\n|\n\u0011\n\u0003\n#\nK\n?\n\u0090\n|\n\u0013\n\u0003\n#\nK\n\u0090\n\u0007\n\u0090\n\u0090\n\u0007\n\u0007\nQ\nK\nQ\n\u0012\n\u0004\nT\n\u0007\n\u0007\nT\n\u0007\nK\n\u0004\n\u0007\n\u0090\nK\n\u0004\n\t\n\u0081\n\u00b3\n\u0007\n\u0081\n\u00b3\n\u0007\n\t\n\u0005\n\u0004\n\u0011\n\u0003\n{\n|\n\u0013\n\u0003\n\u0005\nG\n\n\u0001\n\u0004\n{\n|\n\u0011\n|\n\u0011\n\u0003\n{\n|\n\u0011\n|\n\u0013\n\u0003\n{\n|\n\u0013\n|\n\u0011\n\u0003\n{\n|\n\u0013\n|\n\u0013\n\u0003\n\u0005\n\u0001\n\u0004\n\u0081\n#\nK\n\u0081\n#\n\u0081\n#\n\u0081\n#\nK\n\u0081\n\u0090\n\u0005\nG\n\u0001\n\u0001\nK\nT\n\u0004\n{\n\u0003\n{\n\u0003\n{\n\f\n4\n\u0003\n{\n\u0003\n\u0005\n\u0001\nM\n\u000f\n\u0004\n\u000f\n\u0081\n#\nK\n\u0003\n\u0081\n\u0090\n\u0012\nK\n\u0081\n\u0090\n\u0012\nK\n\u0081\n\u0090\n\u0012\nK\n\u0003\n\u0090\n\u0012\n\u0005\nG\n\u0003\n\u0003\n\u0016\n#\n\u0007\n\u0090\n\u0012\n\u0014\n\t\nW\n\u00ae\n\u00a9\nA\nB\n\u0007\n\u0090\n\u0014\nG\n\u0098\n\u0098\n\u008c\n\u0007\n\t\n\u0098\n\u008c\n\u0090\nQ\n\u0003\nM\n\t\nM\n\u0003\nM\n\t\nO\n\u0090\n\u0001\n\u0003\nO\n\t\nM\n4\nW\nA\nB\n\n\u0016\nG\n\t\nI\n\b\n\t\n\t\nA\n\u0092\n\t\n\u0010\n\u0092\n'\nA\nQ\n{\nQ\n\t\nQ\n'\nM\n\u0097\nA\n$\n{\nQ\n{\n$\n\t\nQ\n\u0017\n$\n\u0092\n#\nQ\n{\nQ\n{\n\u0092\nF\n1\n\t\nQ\n\u007f\n\u0010\n\u0092\n'\nM\n\u0097\nA\n3\n#\n%\n{\n3\n{\n%\nF\n1\n\t\n\u0010\n3\n\u007f\n\u0010\n%\n'\n\t\n\t\nG\n\ffollowing result,\n\n\t . After adding some\nis the Fisher\n\n\t\u0001L\t\u0003\u0002\u000e\t\u0005\u0004r2 are for\t\n, and \u007f\u0014\ta6\r798\n\u0001+\t\nY+\t\u0003\u0006N2 are for\u0011\nThe indexes 0\r\u0002\n, 0\nde\ufb01nitions, that is,F\n\u0003\u0016\t\n\t\u0012\u0011!\u0012\u009c\u0001EF\n\u0003\u0016\t\n\t\u001e\u0012J\u0001\b\u0007\n\u0003\u0016\t\n\t\f\u0012 , where 0\n\u0003\u000f\t\n\t\u001e\u0012 , and {\n1 with func-\n\u0003&SR\r\t\n\t\u009a\t!\u001ba\u0012 which is a diagonal matrix, we substitute \u007f\ninformation matrix of\u0016\nQ . And we have,\nQ up to its 2nd order, and neglect the higher orders of\u0017\ntion of\u0017\n3\u001e3\n\u0092r\u0092\n\u0003\u000f\t\n\u0012\"\u0017\n\u0092\r\u000b\n3\f\u000b\nQ\u0013#\n\u0003\u000f\t\n\t\f\u0012 .\n\u0001L{\u000f\u000e\u0011\u0010\rF\n1=1 , and \t\nM\u009a\u0096\nwhere,\u0007\n\u0007+\u0003\u0017SR\n\t\u000e\u0012\n\u0012 holds,\t\n\u0003\u0016\t\n\u0003&SR\r\t\n\t\u0013 \u0011\u0007$\u0012\u001c\u0001\n a\u0007 , and since \u0016\n\u0090 and \u007f\u0014\t\nLet\u0011\nT\u000b\t\n\u0007 . Also when we put\u0011V\u0001\u001c \n\t\n\t\u0013\u0001\n\u0090 , \u007f\u0014\t\u00a7\u0001\nT\u000b\t\nT\u000b\t\n\u0090 holds. From eq.(9), we have the\n\u0092r\u0092\n3\u001e3\n1=1\n{%3\u0012\u000b\n{N\u0092\u0013\u000b\bFN1\\\u0003\u000f\t\n\u0003\u0016\t\n\t\u001e\u0012 , where \u0014\n\u0003&SR\u0015\u0014\n\t!\t\r\u001e%\u0012\n, and we consider\u0016\nNext, let\u0011\nS\t\n\u0003&\u009c\u009b\n\u0003\u0017SR!\u001b!\t\u0019\u001e \u0012\nsatis\ufb01es this equation. Since\u0016\n\u0003\u000f\t\n\t\f\u0012 , \u0014\nis generally not equal to\u001b\n\u0092r\u0092\n3\u001e3\n{N\u0092\u0013\u000b\bFN1\u000f\u0003\u0016\t\n{ 3\f\u000b\n'VT\nFrom the condition\t\n\u0090 and eq.(10), we have the following approximation,\n3\u001e3\n\u0092r\u0092\n{ 3\nFL1\u000f\u0003\u0016\t\n{\n\u0092\nand?S\u0003\u000f\t\n\t\r\u0012 on\b\n, and we evaluate the difference between ?\n# . The result is\n\u0019\u0019\u0018\n\u0019\u0019\u0018\n, which is?\n\u0001\u001a?\u0083\u0003\nTheorem 5. The true expectation of\n\u001b)\t\r\u001e%\u0012 , is approximated as,\n\u0019\u001a\u0018\n\u0092r\u0092\n3\u001e3\n?S\u0003\u000f\t\n\u0092\u0013\u000b\n3\f\u000b\n\u0019\u0019\u0018\nWhere?\u0083\u0003\u0016\t\n\t\f\u0012\n\u0003\u0016\t\n\t\f\u0012 (Fig.3). The result can be\nEquation (11) is related to the\u0017 \u2013embedded\u2013curvature of\nextended to general case where[\u001c\u001b\u001e\u001d\n\nThis result gives the approximation accuracy of the BP decoding. Let the true belief be\n\nis the parameter which\nis not necessarily included in\n\n\u0012!'\n\n?\u0083\u0003\u0016\t\n\nQ\u0017\u0016\n\nsummarized in the following theorem.\n\n(9)\n\n(10)\n\n(11)\n\nis the solution of the turbo decoding.\n\n[3, 8].\n\n4 Discussion\n\n. From eq.(9),\n\nQ\u0017\u0016\n\nWe have shown a new framework for understanding and analyzing the belief propagation\ndecoder.\n\nSince the BN of turbo codes is loopy, we don\u2019t have enough theoretical results for BP\nalgorithm, while a lot of experiments show that it works surprisingly well in such cases.\nThe mystery of the BP decoders is summarized in 2 points, the approximation accuracy\nand the convergence property.\n\nOur results elucidate the mathematical background of the BP decoding algorithm. The\ninformation geometrical structure of the equilibrium is summarized in Theorem 2. It shows\n\nK\n\t\n1\n1\n\u0092\nF\n1\n1\n\u0092\n\u0007\n1\n\u0092\n2\n\u0010\n\u007f\n\u0010\n1\n\u0007\nK\n\u0007\n1\n1\nA\nQ\n\t\n1\nQ\n\u0017\nQ\nK\n\u0007\n1\n1\n\u0097\nA\n$\n\n{\nQ\nK\nA\n3\n\u0007\n\t\n3\nQ\n{\n\n{\n$\nK\nA\n\u0092\n\u0007\n\t\n\u0092\n$\n{\nF\n1\n\t\nQ\n\u0017\n$\n\t\n1\n1\n\u0001\n\u0007\n1\nQ\n1\n\u0001\n\u0016\nI\n\b\n\t\n\u0001\nT\n\t\n\u0001\n\u0090\nK\nK\nK\nK\n\f\n1\n#\n\t\nQ\n\u0007\nK\n\u0007\n\t\n1\nQ\nK\n\u0007\n1\n1\n\u0097\n\n{\nQ\nK\nA\n3\n\u0007\n\t\n3\nQ\n\n{\nQ\nK\nA\n\u0092\n\u0007\n\t\n\u0092\nQ\n\t\n\u0012\nG\n\u0001\n\u001e\nI\n\b\n\t\n\u0001\n\u0016\n\u0098\n\u0098\n\u008c\n\u0007\n\t\n\u0098\n\u008c\n\u0090\n\u0012\n\b\n\t\n\u0014\n\u0010\n1\nK\n\u0010\n1\n#\n\t\n\u0007\nK\n\u0007\n1\n1\nA\nQ\n\t\n1\nQ\nK\n\u0007\n1\n1\n\u0097\nA\nQ\n\n{\nQ\nK\nA\n3\n\u0007\n\t\n3\nQ\n\n{\nQ\nK\nA\n\u0092\n\u0007\n\t\n\u0092\nQ\n\t\n\u0012\nG\n\t\n\u0001\nT\n1\n\u0007\n1\n\u0014\n\u0010\n1\n\u0007\nK\n\u0007\n1\n1\n\u0097\nA\n<\n$\n\n{\nQ\nK\nA\n3\n\u0007\n\t\n3\nQ\n\u000b\n\n{\n$\nK\nA\n\u0092\n\u0007\n\t\n\u0092\n$\n\u000b\n\t\n\u0012\nG\n?\n\u0019\n\u0019\n\u0019\n?\n\u0019\n\u0007\n\t\nM\n\u0097\nA\n<\n$\n\n{\nQ\nK\nA\n3\n\u0007\n\t\n3\nQ\n{\n\n{\n$\nK\nA\n\u0092\n\u0007\n\t\n\u0092\n$\n{\n\t\n\u0012\nG\n\u0010\n\fthe\u00b0 \u2013\ufb02at submanifold\n\nthe relation between\nand the principal component of the error is the curvature of\nstrongly depends on the codeword, we can control it by the encoder design. This shows a\nroom for improvement of the \u201cnear optimum error correcting code\u201d[2].\n\n\u0003\u000f\t\n\t\r\u0012 plays an important role. Furthermore, Theorem 5 shows that\n\u0003\u000f\t\n\t\r\u0012 and the\u0017 \u2013\ufb02at submanifold\b\n\u0003\u000f\t\n\t\f\u0012 causes the decoding error,\n\u0003\u0016\t\n\t\f\u0012 . Since the curvature\n\nFor the convergent property, we have shown the energy function, which is known as Beth\u00b4e\nfree energy[4, 9]. Unfortunately, the \ufb01xed point of the turbo decoding algorithm is gener-\nally a saddle of the function, which makes further analysis dif\ufb01cult. We have only shown a\nlocal stability condition, and the global property is one of our future works.\n\nThis paper gives a \ufb01rst step to the information geometrical understanding of the belief\npropagation decoder. The main results are for the turbo decoding, but the mechanism is\ncommon with wider class, and the framework is valid for them. We believe further study\nin this direction will lead us to better understanding and improvements of these methods.\n\nAcknowledgments\n\nWe thank Chiranjib Bhattacharyya who gave us the opportunity to face this problem. We\nare also grateful to Yoshiyuki Kabashima and Motohiko Isaka for useful discussions.\n\nReferences\n\n[1] S. Amari and H. Nagaoka. (2000) Methods of Information Geometry, volume 191 of\n\nTranslations of Mathematical Monographs. American Mathematical Society.\n\n[2] C. Berrou and A. Glavieux. (1996) Near optimum error correcting coding and decod-\n\ning: Turbo-codes. IEEE Transactions on Communications, 44(10):1261\u20131271.\n\n[3] S. Ikeda, T. Tanaka, and S. Amari. (2001) Information geometry of turbo codes and\nlow-density parity-check codes. submitted to IEEE transaction on Information Theory.\n[4] Y. Kabashima and D. Saad. (2001) The TAP approach to intensive and extensive con-\nnectivity systems. In M. Opper and D. Saad, editors, Advanced Mean Field Methods \u2013\nTheory and Practice, chapter 6, pages 65\u201384. The MIT Press.\n\n[5] R. J. McEliece, D. J. C. MacKay, and J.-F. Cheng. (1998) Turbo decoding as an in-\nstance of Pearl\u2019s \u201cbelief propagation\u201d algorithm. IEEE Journal on Selected Areas in\nCommunications, 16(2):140\u2013152.\n\n[6] T. J. Richardson. (2000) The geometry of turbo-decoding dynamics.\n\ntions on Information Theory, 46(1):9\u201323.\n\nIEEE Transac-\n\n[7] T. Tanaka. (2001) Information geometry of mean-\ufb01eld approximation. In M. Opper and\nD. Saad, editors, Advanced Mean Field Methods \u2013 Theory and Practice, chapter 17,\npages 259\u2013273. The MIT Press.\n\n[8] T. Tanaka, S. Ikeda, and S. Amari. (2002)\n\nInformation-geometrical signi\ufb01cance of\nsparsity in Gallager codes. in T. G. Dietterich et al. (eds.), Advances in Neural Infor-\nmation Processing Systems, vol. 14 (this volumn), The MIT Press.\n\n[9] J. S. Yedidia, W. T. Freeman, and Y. Weiss. (2001) Bethe free energy, Kikuchi approx-\nimations, and belief propagation algorithms. Technical Report TR2001\u201316, Mitsubishi\nElectric Research Laboratories.\n\n\u0010\n\u0010\n\u0010\n\f", "award": [], "sourceid": 2010, "authors": [{"given_name": "Shiro", "family_name": "Ikeda", "institution": null}, {"given_name": "Toshiyuki", "family_name": "Tanaka", "institution": null}, {"given_name": "Shun-ichi", "family_name": "Amari", "institution": null}]}