{"title": "Error Propagation for Approximate Policy and Value Iteration", "book": "Advances in Neural Information Processing Systems", "page_first": 568, "page_last": 576, "abstract": "We address the question of how the approximation error/Bellman residual at each iteration of the Approximate Policy/Value Iteration algorithms influences the quality of the resulted policy. We quantify the performance loss as the Lp norm of the approximation error/Bellman residual at each iteration. Moreover, we show that the performance loss depends on the expectation of the squared Radon-Nikodym derivative of a certain distribution rather than its supremum -- as opposed to what has been suggested by the previous results.  Also our results indicate that the contribution of the approximation/Bellman error to the performance loss is more prominent in the later iterations of API/AVI, and the effect of an error term in the earlier iterations decays exponentially fast.", "full_text": "Error Propagation for Approximate Policy and\n\nValue Iteration\n\nAmir massoud Farahmand\n\nDepartment of Computing Science\n\nUniversity of Alberta\n\nEdmonton, Canada, T6G 2E8\n\namirf@ualberta.ca\n\nSequel Project, INRIA Lille\n\nR\u00b4emi Munos\n\nLille, France\n\nremi.munos@inria.fr\n\nCsaba Szepesv\u00b4ari \u2217\n\nDepartment of Computing Science\n\nUniversity of Alberta\n\nEdmonton, Canada, T6G 2E8\nszepesva@ualberta.ca\n\nAbstract\n\nWe address the question of how the approximation error/Bellman residual at each\niteration of the Approximate Policy/Value Iteration algorithms in\ufb02uences the qual-\nity of the resulted policy. We quantify the performance loss as the Lp norm of the\napproximation error/Bellman residual at each iteration. Moreover, we show that\nthe performance loss depends on the expectation of the squared Radon-Nikodym\nderivative of a certain distribution rather than its supremum \u2013 as opposed to what\nhas been suggested by the previous results. Also our results indicate that the\ncontribution of the approximation/Bellman error to the performance loss is more\nprominent in the later iterations of API/AVI, and the effect of an error term in the\nearlier iterations decays exponentially fast.\n\n1\n\nIntroduction\n\nThe exact solution for the reinforcement learning (RL) and planning problems with large state space\nis dif\ufb01cult or impossible to obtain, so one usually has to aim for approximate solutions. Approximate\nPolicy Iteration (API) and Approximate Value Iteration (AVI) are two classes of iterative algorithms\nto solve RL/Planning problems with large state spaces. They try to approximately \ufb01nd the \ufb01xed-\npoint solution of the Bellman optimality operator.\nAVI starts from an initial value function V0 (or Q0), and iteratively applies an approximation of\nT \u2217, the Bellman optimality operator, (or T \u03c0 for the policy evaluation problem) to the previous\nestimate, i.e., Vk+1 \u2248 T \u2217Vk. In general, Vk+1 is not equal to T \u2217Vk because (1) we do not have\ndirect access to the Bellman operator but only some samples from it, and (2) the function space\nin which V belongs is not representative enough. Thus there would be an approximation error\n\u03b5k = T \u2217Vk \u2212 Vk+1 between the result of the exact VI and AVI.\nSome examples of AVI-based approaches are tree-based Fitted Q-Iteration of Ernst et al. [1], multi-\nlayer perceptron-based Fitted Q-Iteration of Riedmiller [2], and regularized Fitted Q-Iteration of\nFarahmand et al. [3]. See the work of Munos and Szepesv\u00b4ari [4] for more information about AVI.\n\n\u2217Csaba Szepesv\u00b4ari is on leave from MTA SZTAKI. We would like to acknowledge the insightful comments\nby the reviewers. This work was partly supported by AICML, AITF, NSERC, and PASCAL2 under no216886.\n\n1\n\n\fAPI is another iterative algorithm to \ufb01nd an approximate solution to the \ufb01xed point of the Bellman\noptimality operator. It starts from a policy \u03c00, and then approximately evaluates that policy \u03c00, i.e.,\nit \ufb01nds a Q0 that satis\ufb01es T \u03c00 Q0 \u2248 Q0. Afterwards, it performs a policy improvement step, which\nis to calculate the greedy policy with respect to (w.r.t.) the most recent action-value function, to get\na new policy \u03c01, i.e., \u03c01(\u00b7) = arg maxa\u2208A Q0(\u00b7, a). The policy iteration algorithm continues by\napproximately evaluating the newly obtained policy \u03c01 to get Q1 and repeating the whole process\nagain, generating a sequence of policies and their corresponding approximate action-value functions\nQ0 \u2192 \u03c01 \u2192 Q1 \u2192 \u03c02 \u2192 \u00b7\u00b7\u00b7 . Same as AVI, we may encounter a difference between the ap-\nproximate solution Qk (T \u03c0k Qk \u2248 Qk) and the true value of the policy Q\u03c0k, which is the solution\nof the \ufb01xed-point equation T \u03c0k Q\u03c0k = Q\u03c0k. Two convenient ways to describe this error is either\nby the Bellman residual of Qk (\u03b5k = Qk \u2212 T \u03c0k Qk) or the policy evaluation approximation error\n(\u03b5k = Qk \u2212 Q\u03c0k).\nAPI is a popular approach in RL literature. One well-known algorithm is LSPI of Lagoudakis and\nParr [5] that combines Least-Squares Temporal Difference (LSTD) algorithm (Bradtke and Barto\n[6]) with a policy improvement step. Another API method is to use the Bellman Residual Mini-\nmization (BRM) and its variants for policy evaluation and iteratively apply the policy improvement\nstep (Antos et al. [7], Maillard et al. [8]). Both LSPI and BRM have many extensions: Farah-\nmand et al. [9] introduced a nonparametric extension of LSPI and BRM and formulated them as\nan optimization problem in a reproducing kernel Hilbert space and analyzed its statistical behavior.\nKolter and Ng [10] formulated an l1 regularization extension of LSTD. See Xu et al. [11] and Jung\nand Polani [12] for other examples of kernel-based extension of LSTD/LSPI, and Taylor and Parr\n[13] for a uni\ufb01ed framework. Also see the proto-value function-based approach of Mahadevan and\nMaggioni [14] and iLSTD of Geramifard et al. [15].\nA crucial question in the applicability of API/AVI, which is the main topic of this work, is to un-\nderstand how either the approximation error or the Bellman residual at each iteration of API or AVI\naffects the quality of the resulted policy. Suppose we run API/AVI for K iterations to obtain a policy\n\u03c0K. Does the knowledge that all \u03b5ks are small (maybe because we have had a lot of samples and\nused powerful function approximators) imply that V \u03c0K is close to the optimal value function V \u2217\ntoo? If so, how does the errors occurred at a certain iteration k propagate through iterations of\nAPI/AVI and affect the \ufb01nal performance loss?\nThere have already been some results that partially address this question. As an example, Propo-\nsition 6.2 of Bertsekas and Tsitsiklis [16] shows that for API applied to a \ufb01nite MDP, we have\nlim supk\u2192\u221e (cid:107)V \u2217 \u2212 V \u03c0k(cid:107)\u221e \u2264 2\u03b3\n(1\u2212\u03b3)2 lim supk\u2192\u221e (cid:107)V \u03c0k \u2212 Vk(cid:107)\u221e where \u03b3 is the discount facto.\nSimilarly for AVI, if the approximation errors are uniformly bounded ((cid:107)T \u2217Vk \u2212 Vk+1(cid:107)\u221e \u2264 \u03b5), we\nhave lim supk\u2192\u221e (cid:107)V \u2217 \u2212 V \u03c0k(cid:107)\u221e \u2264 2\u03b3\nNevertheless, most of these results are pessimistic in several ways. One reason is that they are\nexpressed as the supremum norm of the approximation errors (cid:107)V \u03c0k \u2212 Vk(cid:107)\u221e or the Bellman error\n(cid:107)Qk \u2212 T \u03c0k Qk(cid:107)\u221e. Compared to Lp norms, the supremum norm is conservative. It is quite possible\nthat the result of a learning algorithm has a small Lp norm but a very large L\u221e norm. Therefore, it\nis desirable to have a result expressed in Lp norm of the approximation/Bellman residual \u03b5k.\nIn the past couple of years, there have been attempts to extend L\u221e norm results to Lp ones [18, 17,\n7]. As a typical example, we quote the following from Antos et al. [7]:\nProposition 1 (Error Propagation for API \u2013 [7]). Let p \u2265 1 be a real and K be a positive integer.\nThen, for any sequence of functions {Q(k)} \u2282 B(X \u00d7 A; Qmax)(0 \u2264 k < K), the space of Qmax-\nbounded measurable functions, and their corresponding Bellman residuals \u03b5k = Qk \u2212 T \u03c0Qk, the\nfollowing inequalities hold:\n\n(1\u2212\u03b3)2 \u03b5 (Munos [17]).\n\nwhere Rmax is an upper bound on the magnitude of the expected reward function and\n\n(cid:16)\n\nC 1/p\n\n\u03c1,\u03bd max\n0\u2264k<K\n\n2\u03b3\n\n(1 \u2212 \u03b3)2\n\n(cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)p,\u03c1 \u2264\n\nC\u03c1,\u03bd = (1 \u2212 \u03b3)2(cid:88)\n\nm\u03b3m\u22121\n\nsup\n\n\u03c01,...,\u03c0m\n\nm\u22651\n\n(cid:17)\n\n,\n\nK\n\np \u22121 Rmax\n\n(cid:107)\u03b5k(cid:107)p,\u03bd + \u03b3\n\n(cid:13)(cid:13)(cid:13)(cid:13) d (\u03c1P \u03c01 \u00b7\u00b7\u00b7 P \u03c0m)\n\nd\u03bd\n\n(cid:13)(cid:13)(cid:13)(cid:13)\u221e\n\n.\n\nThis result indeed uses Lp norm of the Bellman residuals and is an improvement over results\nlike Bertsekas and Tsitsiklis [16, Proposition 6.2], but still is pessimistic in some ways and does\n\n2\n\n\fd\u03bd\n\n||\u221e is intrinsic to the dif\ufb01culty of the problem or can be relaxed.\n\nnot answer several important questions. For instance, this result implies that the uniform-over-all-\niterations upper bound max0\u2264k<K (cid:107)\u03b5k(cid:107)p,\u03bd is the quantity that determines the performance loss. One\nmay wonder if this condition is really necessary, and ask whether it is better to put more emphasis\non earlier/later iterations? Or another question is whether the appearance of terms in the form of\n|| d(\u03c1P \u03c01\u00b7\u00b7\u00b7P \u03c0m )\nThe goal of this work is to answer these questions and to provide tighter upper bounds on the\nperformance loss of API/AVI algorithms. These bounds help one understand what factors contribute\nto the dif\ufb01culty of a learning problem. We base our analysis on the work of Munos [17], Antos et al.\n[7], Munos [18] and provide upper bounds on the performance loss in the form of (cid:107)V \u2217 \u2212 V \u03c0k(cid:107)1,\u03c1\n(the expected loss weighted according to the evaluation probability distribution \u03c1 \u2013 this is de\ufb01ned\nin Section 2) for API (Section 3) and AVI (Section 4). This performance loss depends on a certain\nfunction of \u03bd-weighted L2 norms of \u03b5ks, in which \u03bd is the data sampling distribution, and C\u03c1,\u03bd(K)\nthat depends on the MDP, two probability distributions \u03c1 and \u03bd, and the number of iterations K.\nIn addition to relating the performance loss to Lp norm of the Bellman residual/approximation er-\nror, this work has three main contributions that to our knowledge have not been considered before:\n(1) We show that the performance loss depends on the expectation of the squared Radon-Nikodym\nderivative of a certain distribution, to be speci\ufb01ed in Section 3, rather than its supremum. The dif-\nference between this expectation and the supremum can be considerable. For instance, for a \ufb01nite\nstate space with N states, the ratio can be of order O(N 1/2). (2) The contribution of the Bell-\nman/approximation error to the performance loss is more prominent in later iterations of API/AVI.\nand the effect of an error term in early iterations decays exponentially fast. (3) There are certain\nstructures in the de\ufb01nition of concentrability coef\ufb01cients that have not been explored before. We\nthoroughly discuss these qualitative/structural improvements in Section 5.\n\n2 Background\n\nIn this section, we provide a very brief summary of some of the concepts and de\ufb01nitions from\nthe theory of Markov Decision Processes (MDP) and reinforcement learning (RL) and a few other\nnotations. For further information about MDPs and RL the reader is referred to [19, 16, 20, 21].\nA \ufb01nite-action discounted MDP is a 5-tuple (X ,A, P,R, \u03b3), where X is a measurable state space, A\nis a \ufb01nite set of actions, P is the probability transition kernel, R is the reward kernel, and 0 \u2264 \u03b3 < 1\nis the discount factor. The transition kernel P is a mapping with domain X \u00d7 A evaluated at\n(x, a) \u2208 X \u00d7 A that gives a distribution over X , which we shall denote by P (\u00b7|x, a). Likewise,\nR is a mapping with domain X \u00d7 A that gives a distribution of immediate reward over R, which\nis denoted by R(\u00b7|x, a). We denote r(x, a) = E [R(\u00b7|x, a)], and assume that its absolute value is\nbounded by Rmax.\nA mapping \u03c0 : X \u2192 A is called a deterministic Markov stationary policy, or just a policy in\nshort. Following a policy \u03c0 in an MDP means that at each time step At = \u03c0(Xt). Upon taking\naction At at Xt, we receive reward Rt \u223c R(\u00b7|x, a), and the Markov chain evolves according to\nXt+1 \u223c P (\u00b7|Xt, At). We denote the probability transition kernel of following a policy \u03c0 by P \u03c0,\ni.e., P \u03c0(dy|x) = P (dy|x, \u03c0(x)).\n\nThe value function V \u03c0 for a policy \u03c0 is de\ufb01ned as V \u03c0(x) (cid:44) E(cid:104)(cid:80)\u221e\naction-value function is de\ufb01ned as Q\u03c0(x, a) (cid:44) E(cid:104)(cid:80)\u221e\n\n(cid:12)(cid:12)(cid:12) X0 = x\n(cid:12)(cid:12)(cid:12) X0 = x, A0 = a\n(cid:105)\n\nt=0 \u03b3tRt\n\n. For a discounted\nMDP, we de\ufb01ne the optimal value and action-value functions by V \u2217(x) = sup\u03c0 V \u03c0(x) (\u2200x \u2208 X )\nand Q\u2217(x, a) = sup\u03c0 Q\u03c0(x, a) (\u2200x \u2208 X ,\u2200a \u2208 A). We say that a policy \u03c0\u2217 is optimal\nif it achieves the best values in every state, i.e., if V \u03c0\u2217 = V \u2217. We say that a policy \u03c0 is\ngreedy w.r.t. an action-value function Q and write \u03c0 = \u02c6\u03c0(\u00b7; Q), if \u03c0(x) \u2208 arg maxa\u2208A Q(x, a)\nholds for all x \u2208 X . Similarly, the policy \u03c0 is greedy w.r.t. V , if for all x \u2208 X , \u03c0(x) \u2208\nargmaxa\u2208A\nis chosen in an arbitrary deterministic manner). Greedy policies are important because a greedy pol-\nicy w.r.t. Q\u2217 (or V \u2217) is an optimal policy. Hence, knowing Q\u2217 is suf\ufb01cient for behaving optimally\n(cf. Proposition 4.3 of [19]).\n\n(cid:82) P (dx(cid:48)|x, a)[r(x, a) + \u03b3V (x(cid:48))] (If there exist multiple maximizers, some maximizer\n\n(cid:105)\n\nt=0 \u03b3tRt\n\nand the\n\n3\n\n\ferator is de\ufb01ned as (T \u2217V )(x) (cid:44) maxa\n\nWe de\ufb01ne the Bellman operator for a policy \u03c0 as (T \u03c0V )(x) (cid:44) r(x, \u03c0(x)) + \u03b3(cid:82) V \u03c0(x(cid:48))P (dx(cid:48)|x, a)\nand (T \u03c0Q)(x, a) (cid:44) r(x, a) + \u03b3(cid:82) Q(x(cid:48), \u03c0(x(cid:48)))P (dx(cid:48)|x, a). Similarly, the Bellman optimality op-\n(cid:110)\n(cid:111)\nr(x, a) + \u03b3(cid:82) V (x(cid:48))P (dx(cid:48)|x, a)\nr(x, a) + \u03b3(cid:82) maxa(cid:48) Q(x(cid:48), a(cid:48))P (dx(cid:48)|x, a).\n\u03c1P \u03c0(dx(cid:48)) =(cid:82) P (dx(cid:48)|x, \u03c0(x))d\u03c1(x). In words, \u03c1(P \u03c0)m \u2208 M(X ) is an m-step-ahead probability\ndistribution of states if the starting state distribution is \u03c1 and we follow P \u03c0 for m steps. In what\nfollows we shall use (cid:107)V (cid:107)p,\u03bd to denote the Lp(\u03bd)-norm of a measurable function V : X \u2192 R:\n(cid:80)\n(cid:107)V (cid:107)p\n(cid:44)\n1|A|\n\nFor a measurable space X , with a \u03c3-algebra \u03c3X , we de\ufb01ne M(X ) as the set of all probability\nmeasures over \u03c3X . For a probability measure \u03c1 \u2208 M(X ) and the transition kernel P \u03c0, we de\ufb01ne\n\nX |V (x)|pd\u03bd(x). For a function Q : X \u00d7 A (cid:55)\u2192 R, we de\ufb01ne (cid:107)Q(cid:107)p\n\n(cid:44) \u03bd|V |p (cid:44) (cid:82)\n(cid:82)\n\nX |Q(x, a)|pd\u03bd(x).\n\nand (T \u2217Q)(x, a) (cid:44)\n\na\u2208A\n\np,\u03bd\n\np,\u03bd\n\n3 Approximate Policy Iteration\nConsider the API procedure and the sequence Q0 \u2192 \u03c01 \u2192 Q1 \u2192 \u03c02 \u2192 \u00b7\u00b7\u00b7 \u2192 QK\u22121 \u2192 \u03c0K,\nwhere \u03c0k is the greedy policy w.r.t. Qk\u22121 and Qk is the approximate action-value function for policy\n\u03c0k. For the sequence {Qk}K\u22121\nk=0 , denote the Bellman Residual (BR) and policy Approximation Error\n(AE) at each iteration by\n\nk = Qk \u2212 T \u03c0k Qk,\n\u03b5BR\nk = Qk \u2212 Q\u03c0k .\n\u03b5AE\n\n(1)\n(2)\n\nk }K\u22121\n\nk }K\u22121\n\nk=0 or the policy evaluation approximation error sequence {\u03b5AE\n\nThe goal of this section is to study the effect of \u03bd-weighted L2p norm of the Bellman residual\nsequence {\u03b5BR\nk=0 on the per-\nformance loss (cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)p,\u03c1 of the outcome policy \u03c0K.\nThe choice of \u03c1 and \u03bd is arbitrary, however, a natural choice for \u03bd is the sampling distribution of the\ndata, which is used by the policy evaluation module. On the other hand, the probability distribution\n\u03c1 re\ufb02ects the importance of various regions of the state space and is selected by the practitioner. One\ncommon choice, though not necessarily the best, is the stationary distribution of the optimal policy.\nBecause of the dynamical nature of MDP, the performance loss (cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)p,\u03c1 depends on the\ndifference between the sampling distribution \u03bd and the future-state distribution in the form of\n\u03c1P \u03c01 P \u03c02 \u00b7\u00b7\u00b7 . The precise form of this dependence will be formalized in Theorems 3 and 4. Before\nstating the results, we require to de\ufb01ne the following concentrability coef\ufb01cients.\nDe\ufb01nition 2 (Expected Concentrability of the Future-State Distribution). Given \u03c1, \u03bd \u2208 M(X ),\n\u03bd (cid:28) \u03bb1 (\u03bb is the Lebesgue measure), m \u2265 0, and an arbitrary sequence of stationary policies\n{\u03c0m}m\u22651, let \u03c1P \u03c01 P \u03c02 . . . P \u03c0m \u2208 M(X ) denote the future-state distribution obtained when the\n\ufb01rst state is distributed according to \u03c1 and then we follow the sequence of policies {\u03c0k}m\nDe\ufb01ne the following concentrability coef\ufb01cients that is used in API analysis:\n\nk=1.\n\ncPI1,\u03c1,\u03bd(m1, m2; \u03c0) (cid:44)\n\ncPI2,\u03c1,\u03bd(m1, m2; \u03c01, \u03c02) (cid:44)\n\ncPI3,\u03c1,\u03bd (cid:44)\n\n\uf8eb\uf8edEX\u223c\u03bd\n\uf8eb\uf8edEX\u223c\u03bd\n\uf8eb\uf8edEX\u223c\u03bd\n\nd\u03bd\n\n2\n\n,\n\n(X)\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)2\uf8f9\uf8fb\uf8f6\uf8f8 1\n\uf8ee\uf8f0(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d(cid:0)\u03c1(P \u03c0\u2217)m1(P \u03c0)m2(cid:1)\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)2\uf8f9\uf8fb\uf8f6\uf8f8 1\n\uf8ee\uf8f0(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d(cid:0)\u03c1(P \u03c0\u2217)m1(P \u03c01)m2P \u03c02(cid:1)\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)2\uf8f9\uf8fb\uf8f6\uf8f8 1\n\uf8ee\uf8f0(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d(cid:0)\u03c1P \u03c0\u2217(cid:1)\n\n(X)\n\n(X)\n\nd\u03bd\n\nd\u03bd\n\n2\n\n,\n\n2\n\n,\n\n1For two measures \u03bd1 and \u03bd2 on the same measurable space, we say that \u03bd1 is absolutely continuous with\n\nrespect to \u03bd2 (or \u03bd2 dominates \u03bd1) and denote \u03bd1 (cid:28) \u03bd2 iff \u03bd2(A) = 0 \u21d2 \u03bd1(A) = 0.\n\n4\n\n\fthe\n\nunderstanding\n\nwith\nthat\nthe\n\u03c1(P \u03c0\u2217)m1(P \u03c01)m2P \u03c02 or \u03c1P \u03c0\u2217\ncPI1,\u03c1,\u03bd(m1, m2; \u03c0) = \u221e (similar for others).\nAlso de\ufb01ne the following concentrability coef\ufb01cient that is used in AVI analysis:\n\nif\nis not absolutely continuous w.r.t.\n\ndistribution\n\nfuture-state\n\n)\n\n\u03c1(P \u03c0\u2217)m1(P \u03c0)m2\n\n(or\nthen we take\n\n\u03bd,\n\ncVI,\u03c1,\u03bd(m1, m2; \u03c0) (cid:44)\n\n\uf8eb\uf8edEX\u223c\u03bd\n\n\uf8ee\uf8f0(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d(cid:0)\u03c1(P \u03c0)m1(P \u03c0\u2217)m2(cid:1)\n\nd\u03bd\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)2\uf8f9\uf8fb\uf8f6\uf8f8 1\n\n2\n\n,\n\n(X)\n\nwith the understanding that if the future-state distribution \u03c1(P \u03c0\u2217)m1(P \u03c0)m2 is not absolutely con-\ntinuous w.r.t. \u03bd, then we take cVI,\u03c1,\u03bd(m1, m2; \u03c0) = \u221e.\nIn order to compactly present our results, we de\ufb01ne the following notation:\n\n\u03b1k =\n\n(1 \u2212 \u03b3)\u03b3K\u2212k\u22121\n\n1 \u2212 \u03b3K+1\n\n0 \u2264 k < K.\n\n1\u2212\u03b3 . Then for any sequence {Qk}K\u22121\n\nTheorem 3 (Error Propagation for API). Let p \u2265 1 be a real number, K be a positive integer,\nand Qmax \u2264 Rmax\nk=0 \u2282 B(X \u00d7 A, Qmax) (space of Qmax-bounded\nmeasurable functions de\ufb01ned on X \u00d7 A) and the corresponding sequence {\u03b5k}K\u22121\nk=0 de\ufb01ned in (1)\nor (2) , we have\n(cid:21)\n\n2p (\u03b50, . . . , \u03b5K\u22121; r) + \u03b3\n\nK\n\np \u22121Rmax\n\n.\n\n(cid:20)\nwhere E(\u03b50, . . . , \u03b5K\u22121; r) =(cid:80)K\u22121\n\n(cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)p,\u03c1 \u2264\n\n(1 \u2212 \u03b3)2\n\n2\u03b3\n\ninf\nr\u2208[0,1]\n\nk=0 \u03b12r\n\n(a) If \u03b5k = \u03b5BR for all 0 \u2264 k < K, we have\n\n1\n2p\n\n2p,\u03bd.\n\nPI(BR/AE),\u03c1,\u03bd(K; r)E 1\nC\nk (cid:107)\u03b5k(cid:107)2p\nK\u22121(cid:88)\n\n(cid:32)(cid:88)\n\n\u03b12(1\u2212r)\n\n\u03b3m(cid:16)\n\nk\n\nk=0\n\nm\u22650\n\nCPI(BR),\u03c1,\u03bd(K; r) = (\n\n1 \u2212 \u03b3\n2\n\n)2\n\nsup\n\u03c0(cid:48)\n0,...,\u03c0(cid:48)\n\nK\n\ncPI1,\u03c1,\u03bd(K \u2212 k \u2212 1, m + 1; \u03c0(cid:48)\n\nk+1)+\n\n(cid:17)(cid:33)2\n\n.\n\ncPI1,\u03c1,\u03bd(K \u2212 k, m; \u03c0(cid:48)\nk)\n\n(b) If \u03b5k = \u03b5AE for all 0 \u2264 k < K, we have\n\nCPI(AE),\u03c1,\u03bd(K; r, s) = (\n\n1 \u2212 \u03b3\n2\n\n)2\n\nsup\n\u03c0(cid:48)\n0,...,\u03c0(cid:48)\n\nK\n\nK\u22121(cid:88)\n\nk=0\n\n\u03b12(1\u2212r)\n\nk\n\n(cid:32)(cid:88)\n(cid:88)\n\nm\u22650\n\nm\u22651\n\n\u03b3mcPI1,\u03c1,\u03bd(K \u2212 k \u2212 1, m + 1; \u03c0(cid:48)\n\nk+1)+\n\n\u03b3mcPI2,\u03c1,\u03bd(K \u2212 k \u2212 1, m; \u03c0(cid:48)\n\nk+1, \u03c0(cid:48)\n\nk) + cPI3,\u03c1,\u03bd\n\n(cid:33)2\n\n.\n\n4 Approximate Value Iteration\nConsider the AVI procedure and the sequence V0 \u2192 V1 \u2192 \u00b7\u00b7\u00b7 \u2192 VK\u22121, in which Vk+1 is the\nresult of approximately applying the Bellman optimality operator on the previous estimate Vk, i.e.,\nVk+1 \u2248 T \u2217Vk. Denote the approximation error caused at each iteration by\n\n(3)\nThe goal of this section is to analyze AVI procedure and to relate the approximation error sequence\n{\u03b5k}K\u22121\nk=0 to the performance loss (cid:107)V \u2217 \u2212 V \u03c0K(cid:107)p,\u03c1 of the obtained policy \u03c0K, which is the greedy\npolicy w.r.t. VK\u22121.\n\n\u03b5k = T \u2217Vk \u2212 Vk+1.\n\n5\n\n\f1\u2212\u03b3 . Then for any sequence {Vk}K\u22121\n\nTheorem 4 (Error Propagation for AVI). Let p \u2265 1 be a real number, K be a positive integer, and\nVmax \u2264 Rmax\nk=0 \u2282 B(X , Vmax), and the corresponding sequence\n(cid:21)\n{\u03b5k}K\u22121\n\n(cid:20)\n\nk=0 de\ufb01ned in (3), we have\n(cid:107)V \u2217 \u2212 V \u03c0K(cid:107)p,\u03c1 \u2264\n\n2\u03b3\n\n(1 \u2212 \u03b3)2\n\n1\n2p\n\nVI,\u03c1,\u03bd(K; r)E 1\nC\n\ninf\nr\u2208[0,1]\n\n2p (\u03b50, . . . , \u03b5K\u22121; r) +\n\nK\n\n\u03b3\n\np Rmax\n\n,\n\n2\n1 \u2212 \u03b3\n\nwhere\n\nCVI,\u03c1,\u03bd(K; r) = (\n\nK\u22121(cid:88)\nand E(\u03b50, . . . , \u03b5K\u22121; r) =(cid:80)K\u22121\n\n1 \u2212 \u03b3\n2\n\n)2 sup\n\u03c0(cid:48)\n\nk=0\n\nk=0 \u03b12r\n\nk (cid:107)\u03b5k(cid:107)2p\n\n2p,\u03bd.\n\n\uf8eb\uf8ed(cid:88)\n\nm\u22650\n\n\u03b12(1\u2212r)\n\nk\n\n\u03b3m (cVI,\u03c1,\u03bd(m, K \u2212 k; \u03c0(cid:48)) + cVI,\u03c1,\u03bd(m + 1, K \u2212 k \u2212 1; \u03c0(cid:48)))\n\n\uf8f6\uf8f82\n\n,\n\n5 Discussion\n\nIn this section, we discuss signi\ufb01cant improvements of Theorems 3 and 4 over previous results such\nas [16, 18, 17, 7].\n\n2\u03b3\n\n5.1 Lp norm instead of L\u221e norm\nAs opposed to most error upper bounds, Theorems 3 and 4 relate (cid:107)V \u2217 \u2212 V \u03c0K(cid:107)p,\u03c1 to the Lp norm\nof the approximation or Bellman errors (cid:107)\u03b5k(cid:107)2p,\u03bd of iterations in API/AVI. This should be con-\ntrasted with the traditional, and more conservative, results such as lim supk\u2192\u221e (cid:107)V \u2217 \u2212 V \u03c0k(cid:107)\u221e \u2264\n(1\u2212\u03b3)2 lim supk\u2192\u221e (cid:107)V \u03c0k \u2212 Vk(cid:107)\u221e for API (Proposition 6.2 of Bertsekas and Tsitsiklis [16]). The\nuse of Lp norm not only is a huge improvement over conservative supremum norm, but also allows\nus to bene\ufb01t from the vast literature on supervised learning techniques, which usually provides error\nupper bounds in the form of Lp norms, in the context of RL/Planning problems. This is especially\ninteresting for the case of p = 1 as the performance loss (cid:107)V \u2217 \u2212 V \u03c0K(cid:107)1,\u03c1 is the difference between\nthe expected return of the optimal policy and the resulted policy \u03c0K when the initial state distribu-\ntion is \u03c1. Convenient enough, the errors appearing in the upper bound are in the form of (cid:107)\u03b5k(cid:107)2,\u03bd\nwhich is very common in the supervised learning literature. This type of improvement, however,\nhas been done in the past couple of years [18, 17, 7] - see Proposition 1 in Section 1.\n\n5.2 Expected versus supremum concentrability of the future-state distribution\n\nThe concentrability coef\ufb01cients (De\ufb01nition 2) re\ufb02ect the effect of future-state distribution on the per-\nformance loss (cid:107)V \u2217 \u2212 V \u03c0K(cid:107)p,\u03c1. Previously it was thought that the key contributing factor to the per-\nformance loss is the supremum of the Radon-Nikodym derivative of these two distributions. This is\nevident in the de\ufb01nition of C\u03c1,\u03bd in Proposition 1 where we have terms in the form of || d(\u03c1(P \u03c0)m)\n||\u221e\ninstead of\n\n(cid:104)| d(\u03c1(P \u03c0)m)\n\n(X)|2(cid:105)(cid:17) 1\n\n2 that we have in De\ufb01nition 2.\n\n(cid:16)EX\u223c\u03bd\n\nd\u03bd\n\nd\u03bd\n\nNevertheless, it turns out that the key contributing factor that determines the performance loss is\nthe expectation of the squared Radon-Nikodym derivative instead of its supremum. Intuitively this\nimplies that even if for some subset of X (cid:48) \u2282 X the ratio d(\u03c1(P \u03c0)m)\nis large but the probability \u03bd(X (cid:48))\nis very small, performance loss due to it is still small. This phenomenon has not been suggested by\nprevious results.\nAs an illustration of this difference, consider a Chain Walk with 1000 states with a single policy that\n201 for x \u2208 [400, 600] and zero everywhere\ndrifts toward state 1 of the chain. We start with \u03c1(x) = 1\nelse. Then we evaluate both || d(\u03c1(P \u03c0)m)\n) 1\n2 for m = 1, 2, . . . when \u03bd\nis the uniform distribution. The result is shown in Figure 1a. One sees that the ratio is constant in the\nbeginning, but increases when the distribution \u03c1(P \u03c0)m concentrates around state 1, until it reaches\nsteady-state. The growth and the \ufb01nal value of the expectation-based concentrability coef\ufb01cient is\nmuch smaller than that of supremum-based.\n\n(cid:104)| d(\u03c1(P \u03c0)m)\n\n||\u221e and (EX\u223c\u03bd\n\n|2(cid:105)\n\nd\u03bd\n\nd\u03bd\n\nd\u03bd\n\n6\n\n\f(a)\n\n(cid:104)| d(\u03c1(P \u03c0)m)\n\n|2(cid:105) 1\n\n(cid:13)(cid:13)(cid:13) d(\u03c1(P \u03c0)m)\n\nd\u03bd\n\n(b)\n\n(cid:13)(cid:13)(cid:13)\u221e\n\n(a) Comparison of EX\u223c\u03bd\n\n(b) Comparison of\nFigure 1:\n(cid:107)Q\u2217 \u2212 Qk(cid:107)1 for uniform and exponential data sampling schedule. The total number of samples\nis the same. [The Y -scale of both plots is logarithmic.]\n\n2 and\n\nd\u03bd\n\n|2(cid:105)\n\nd\u03bd\n\n2 \u2192 \u221a\n) 1\n\n||\u221e \u2192 N, while (EX\u223c\u03bd\n\nIt is easy to show that if the Chain Walk has N states and the policy has the same concentrating\nbehavior and \u03bd is uniform, then || d(\u03c1(P \u03c0)m)\nN when\n\u221a\nm \u2192 \u221e. The ratio, therefore, would be of order \u0398(\nN). This clearly shows the improvement of\nthis new analysis in a simple problem. One may anticipate that this sharper behavior happens in\nmany other problems too.\nMore generally, consider C\u221e = || d\u00b5\n) 1\n2 . For a \ufb01nite state space\nwith N states and \u03bd is the uniform distribution, C\u221e \u2264 N but CL2 \u2264 \u221a\nN. Neglecting all\nother differences between our results and the previous ones, we get a performance upper bound\nin the form of (cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)1,\u03c1 \u2264 c1(\u03b3)O(N 1/4) supk (cid:107)\u03b5k(cid:107)2,\u03bd, while Proposition 1 implies that\n(cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)1,\u03c1 \u2264 c2(\u03b3)O(N 1/2) supk ||\u0001k||2,\u03bd. This difference between O(N 1/4) and O(N 1/2)\nshows a signi\ufb01cant improvement.\n\nd\u03bd ||\u221e and CL2 = (EX\u223c\u03bd\n\nd\u03bd\n\n(cid:104)| d(\u03c1(P \u03c0)m)\n(cid:104)| d\u00b5\nd\u03bd |2(cid:105)\n\n5.3 Error decaying property\n\nk=0 \u03b12r\n\nk (cid:107)\u03b5k(cid:107)2p\n\nk=0 is in the form of E(\u03b50, . . . , \u03b5K\u22121; r) =(cid:80)K\u22121\n\nthe dependence of performance loss (cid:107)V \u2217 \u2212 V \u03c0K(cid:107)p,\u03c1 (or\nTheorems 3 and 4 show that\n(cid:107)Q\u2217 \u2212 Q\u03c0K(cid:107)p,\u03c1) on {\u03b5k}K\u22121\n2p,\u03bd. This has\na very special structure in that the approximation errors at later iterations have more contribution to\nthe \ufb01nal performance loss. This behavior is obscure in previous results such as [17, 7] that the depen-\ndence of the \ufb01nal performance loss is expressed as E(\u03b50, . . . , \u03b5K\u22121; r) = maxk=0,...,K\u22121 (cid:107)\u03b5k(cid:107)p,\u03bd\n(see Proposition 1).\nThis property has practical and algorithmic implications too. It says that it is better to put more\neffort on having a lower Bellman or approximation error at later iterations of API/AVI. This, for\ninstance, can be done by gradually increasing the number of samples throughout iterations, or to use\nmore powerful, and possibly computationally more expensive, function approximators for the later\niterations of API/AVI.\nTo illustrate this property, we compare two different sampling schedules on a simple MDP. The\nMDP is a 100-state, 2-action chain similar to Chain Walk problem in the work of Lagoudakis and\nParr [5]. We use AVI with a lookup-table function representation. In the \ufb01rst sampling schedule,\nevery 20 iterations we generate a \ufb01xed number of fresh samples by following a uniformly random\nwalk on the chain (this means that we throw away old samples). This is the \ufb01xed strategy. In the\nexponential strategy, we again generate new samples every 20 iterations but the number of samples\nat the kth iteration is ck\u03b3. The constant c is tuned such that the total number of both sampling\nstrategy is almost the same (we give a slight margin of about 0.1% of samples in favor of the \ufb01xed\nstrategy). What we compare is (cid:107)Q\u2217 \u2212 Qk(cid:107)1,\u03bd when \u03bd is the uniform distribution. The result can be\nseen in Figure 1b. The improvement of the exponential sampling schedule is evident. Of course, one\n\n7\n\n150010001500100101102103Step (m)Concentrability Coefficients  Infinity norm\u2212based concentrabilityExpectation\u2212base concentrability10204060801001201401601802000.20.30.40.50.60.811.52345IterationL1 error  UniformExponential\fmay think of more sophisticated sampling schedules but this simple illustration should be suf\ufb01cient\nto attract the attention of practitioners to this phenomenon.\n\n5.4 Restricted search over policy space\n\nk in cPI1,\u03c1,\u03bd(K \u2212 k, m; \u03c0(cid:48)\n\nOne interesting feature of our results is that it puts more structure and restriction on the way policies\nmay be selected. Comparing CPI,\u03c1,\u03bd(K; r) (Theorem 3) and CVI,\u03c1,\u03bd(K; r) (Theorem 4) with C\u03c1,\u03bd\n(Proposition 1) we see that:\n(1) Each concentrability coef\ufb01cient in the de\ufb01nition of CPI,\u03c1,\u03bd(K; r) depends only on a single or\ntwo policies (e.g., \u03c0(cid:48)\nk)). The same is true for CVI,\u03c1,\u03bd(K; r). In contrast, the\nmth term in C\u03c1,\u03bd has \u03c01, . . . , \u03c0m as degrees of freedom, and this number is growing as m \u2192 \u221e.\n(2) The operator sup in CPI,\u03c1,\u03bd and CVI,\u03c1,\u03bd appears outside the summation. Because of that, we\nonly have K + 1 degrees of freedom \u03c0(cid:48)\nK to choose from in API and remarkably only a\nsingle degree of freedom in AVI. On the other other hand, sup appears inside the summation in the\nde\ufb01nition of C\u03c1,\u03bd. One may construct an MDP that this difference in the ordering of sup leads to an\narbitrarily large ratio of two different ways of de\ufb01ning the concentrability coef\ufb01cients.\n(3) In API, the de\ufb01nitions of concentrability coef\ufb01cients cPI1,\u03c1,\u03bd, cPI2,\u03c1,\u03bd, and cPI3,\u03c1,\u03bd (De\ufb01ni-\ntion 2) imply that if \u03c1 = \u03c1\u2217, the stationary distribution induced by an optimal policy \u03c0\u2217, then\ncPI1,\u03c1,\u03bd(m1, m2; \u03c0) = cPI1,\u03c1,\u03bd(\u00b7, m2; \u03c0) = (EX\u223c\u03bd\n) 1\n2 (similar for the other two\ncoef\ufb01cients). This special structure is hidden in the de\ufb01nition of C\u03c1,\u03bd in Proposition 1, and instead\nwe have an extra m1 degrees of \ufb02exibility.\nRemark 1. For general MDPs, the computation of concentrability coef\ufb01cients in De\ufb01nition 2 is\ndif\ufb01cult, as it is for similar coef\ufb01cients de\ufb01ned in [18, 17, 7].\n\n(cid:20)(cid:12)(cid:12)(cid:12) d(\u03c1\u2217(P \u03c0)m2 )\n\n0, . . . , \u03c0(cid:48)\n\n(cid:12)(cid:12)(cid:12)2(cid:21)\n\nd\u03bd\n\n6 Conclusion\n\nTo analyze an API/AVI algorithm and to study its statistical properties such as consistency or con-\nvergence rate, we require to (1) analyze the statistical properties of the algorithm running at each\niteration, and (2) study the way the policy approximation/Bellman errors propagate and in\ufb02uence\nthe quality of the resulted policy.\nThe analysis in the \ufb01rst step heavily uses tools from the Statistical Learning Theory (SLT) literature,\ne.g., Gy\u00a8or\ufb01 et al. [22]. In some cases, such as AVI, the problem can be cast as a standard regression\nwith the twist that extra care should be taken to the temporal dependency of data in RL scenario.\nThe situation is a bit more complicated for API methods that directly aim for the \ufb01xed-point solution\n(such as LSTD and its variants), but still the same kind of tools from SLT can be used too \u2013 see Antos\net al. [7], Maillard et al. [8].\nThe analysis for the second step is what this work has been about. In our Theorems 3 and 4, we\nhave provided upper bounds that relate the errors at each iteration of API/AVI to the performance\nloss of the whole procedure. These bounds are qualitatively tighter than the previous results such as\nthose reported by [18, 17, 7], and provide a better understanding of what factors contribute to the\ndif\ufb01culty of the problem. In Section 5, we discussed the signi\ufb01cance of these new results and the\nway they improve previous ones.\nFinally, we should note that there are still some unaddressed issues. Perhaps the most important one\nis to study the behavior of concentrability coef\ufb01cients cPI1,\u03c1,\u03bd(m1, m2; \u03c0), cPI2,\u03c1,\u03bd(m1, m2; \u03c01, \u03c02),\nand cVI,\u03c1,\u03bd(m1, m2; \u03c0) as a function of m1, m2, and of course the transition kernel P of MDP. A\nbetter understanding of this question alongside a good understanding of the way each term \u03b5k in\nE(\u03b50, . . . , \u03b5K\u22121; r) behaves, help us gain more insight about the error convergence behavior of the\nRL/Planning algorithms.\n\nReferences\n[1] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement\n\nlearning. Journal of Machine Learning Research, 6:503\u2013556, 2005.\n\n8\n\n\f[2] Martin Riedmiller. Neural \ufb01tted Q iteration \u2013 \ufb01rst experiences with a data ef\ufb01cient neural\nreinforcement learning method. In 16th European Conference on Machine Learning, pages\n317\u2013328, 2005.\n\n[3] Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesv\u00b4ari, and Shie Mannor.\nRegularized \ufb01tted Q-iteration for planning in continuous-space markovian decision problems.\nIn Proceedings of American Control Conference (ACC), pages 725\u2013730, June 2009.\n\n[4] R\u00b4emi Munos and Csaba Szepesv\u00b4ari. Finite-time bounds for \ufb01tted value iteration. Journal of\n\nMachine Learning Research, 9:815\u2013857, 2008.\n\n[5] Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine\n\nLearning Research, 4:1107\u20131149, 2003.\n\n[6] Steven J. Bradtke and Andrew G. Barto. Linear least-squares algorithms for temporal differ-\n\nence learning. Machine Learning, 22:33\u201357, 1996.\n\n[7] Andr\u00b4as Antos, Csaba Szepesv\u00b4ari, and R\u00b4emi Munos. Learning near-optimal policies with\nBellman-residual minimization based \ufb01tted policy iteration and a single sample path. Machine\nLearning, 71:89\u2013129, 2008.\n\n[8] Odalric Maillard, R\u00b4emi Munos, Alessandro Lazaric, and Mohammad Ghavamzadeh. Finite-\nsample analysis of bellman residual minimization. In Proceedings of the Second Asian Con-\nference on Machine Learning (ACML), 2010.\n\n[9] Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesv\u00b4ari, and Shie Mannor.\nRegularized policy iteration. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors,\nAdvances in Neural Information Processing Systems 21, pages 441\u2013448. MIT Press, 2009.\n\n[10] J. Zico Kolter and Andrew Y. Ng. Regularization and feature selection in least-squares tempo-\nral difference learning. In ICML \u201909: Proceedings of the 26th Annual International Conference\non Machine Learning, pages 521\u2013528, New York, NY, USA, 2009. ACM.\n\n[11] Xin Xu, Dewen Hu, and Xicheng Lu. Kernel-based least squares policy iteration for reinforce-\n\nment learning. IEEE Trans. on Neural Networks, 18:973\u2013992, 2007.\n\n[12] Tobias Jung and Daniel Polani. Least squares SVM for least squares TD learning. In In Proc.\n\n17th European Conference on Arti\ufb01cial Intelligence, pages 499\u2013503, 2006.\n\n[13] Gavin Taylor and Ronald Parr. Kernelized value function approximation for reinforcement\nlearning. In ICML \u201909: Proceedings of the 26th Annual International Conference on Machine\nLearning, pages 1017\u20131024, New York, NY, USA, 2009. ACM.\n\n[14] Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A Laplacian framework\nfor learning representation and control in markov decision processes. Journal of Machine\nLearning Research, 8:2169\u20132231, 2007.\n\n[15] Alborz Geramifard, Michael Bowling, Michael Zinkevich, and Richard S. Sutton. iLSTD: El-\nigibility traces and convergence analysis. In B. Sch\u00a8olkopf, J. Platt, and T. Hoffman, editors,\nAdvances in Neural Information Processing Systems 19, pages 441\u2013448. MIT Press, Cam-\nbridge, MA, 2007.\n\n[16] Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming (Optimization and\n\nNeural Computation Series, 3). Athena Scienti\ufb01c, 1996.\n\n[17] R\u00b4emi Munos. Performance bounds in lp norm for approximate value iteration. SIAM Journal\n\non Control and Optimization, 2007.\n\n[18] R\u00b4emi Munos. Error bounds for approximate policy iteration. In ICML 2003: Proceedings of\n\nthe 20th Annual International Conference on Machine Learning, 2003.\n\n[19] Dimitri P. Bertsekas and Steven E. Shreve. Stochastic Optimal Control: The Discrete-Time\n\nCase. Academic Press, 1978.\n\n[20] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction (Adaptive\n\nComputation and Machine Learning). The MIT Press, 1998.\n\n[21] Csaba Szepesv\u00b4ari. Algorithms for Reinforcement Learning. Morgan Claypool Publishers,\n\n2010.\n\n[22] L\u00b4aszl\u00b4o Gy\u00a8or\ufb01, Michael Kohler, Adam Krzy\u02d9zak, and Harro Walk. A Distribution-Free Theory\n\nof Nonparametric Regression. Springer Verlag, New York, 2002.\n\n9\n\n\f", "award": [], "sourceid": 1020, "authors": [{"given_name": "Amir-massoud", "family_name": "Farahmand", "institution": null}, {"given_name": "Csaba", "family_name": "Szepesv\u00e1ri", "institution": null}, {"given_name": "R\u00e9mi", "family_name": "Munos", "institution": null}]}