{"title": "Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress", "book": "Advances in Neural Information Processing Systems", "page_first": 206, "page_last": 214, "abstract": "Formal exploration approaches in model-based reinforcement learning estimate the accuracy of the currently learned model without consideration of the empirical prediction error. For example, PAC-MDP approaches such as Rmax base their model certainty on the amount of collected data, while Bayesian approaches assume a prior over the transition dynamics. We propose extensions to such approaches which drive exploration solely based on empirical estimates of the learner's accuracy and learning progress. We provide a ``sanity check'' theoretical analysis, discussing the behavior of our extensions in the standard stationary finite state-action case. We then provide experimental studies demonstrating the robustness of these exploration measures in cases of non-stationary environments or where original approaches are misled by wrong domain assumptions.", "full_text": "Exploration in Model-based Reinforcement Learning\n\nby Empirically Estimating Learning Progress\n\nManuel Lopes\n\nINRIA\n\nBordeaux, France\n\nTobias Lang\nFU Berlin\nGermany\n\nMarc Toussaint\n\nFU Berlin\nGermany\n\nAbstract\n\nPierre-Yves Oudeyer\n\nINRIA\n\nBordeaux, France\n\nFormal exploration approaches in model-based reinforcement learning estimate\nthe accuracy of the currently learned model without consideration of the empir-\nical prediction error. For example, PAC-MDP approaches such as R-MAX base\ntheir model certainty on the amount of collected data, while Bayesian approaches\nassume a prior over the transition dynamics. We propose extensions to such\napproaches which drive exploration solely based on empirical estimates of the\nlearner\u2019s accuracy and learning progress. We provide a \u201csanity check\u201d theoreti-\ncal analysis, discussing the behavior of our extensions in the standard stationary\n\ufb01nite state-action case. We then provide experimental studies demonstrating the\nrobustness of these exploration measures in cases of non-stationary environments\nor where original approaches are misled by wrong domain assumptions.\n\n1\n\nIntroduction\n\nReinforcement learning (RL) agents need to solve the exploitation-exploration tradeoff. They have\nto exploit their current model of the environment. At the same time they need to explore the en-\nvironment suf\ufb01ciently to learn more about its reward-relevant structure. Established model-based\napproaches like E3 [8] and R-MAX [4] take into account how often a state-action pair has been\nvisited. With an ef\ufb01cient model learner, the estimated transition model can be guaranteed to be\napproximately correct after a suf\ufb01cient, and ef\ufb01cient, number of visitations in a stationary domain.\nAn alternative approach to exploration is Bayesian RL [11]. Bayesian RL exploits prior knowledge\nabout the transition dynamics to reason explicitly about the uncertainty of the estimated model. In-\nterestingly, these existing approaches estimate the accuracy of the currently learned model based\nonly on visitation counts. They do not consider the actual empirical prediction performance or\nlearning rate of the learner w.r.t. the data seen so far.\nWhat happens if the fundamental presumption of R-MAX and Bayesian RL fails that each single\nseen data-point will increase the agent\u2019s certainty about its model? This is the case if the transi-\ntion dynamics change over time (so there is no correct stationary prior), or if we want to be able\nto ignore non-learnable, currently \u201ctoo dif\ufb01cult\u201d parts of the state space, either to the level of noise\nor limitations of a learning algorithm. For example, a household robot cannot learn how to repair\na cupboard until it has achieved a basic understanding for the handling of tools. Such scenarios\nrequire the development of new methods and even new measures of success. Previous work into this\ndirection emphasizes the concept of intrinsic motivation [10, 13, 12] and has shown empirical suc-\ncess in developmental robotics [1]. An interesting aspect about this work is its reliance on empirical\nmeasures of learning progress to drive exploration in reinforcement learning [17, 6]. However, to\nour knowledge this has not been made rigorous or related to fundamental methods like R-MAX or\nBayesian RL.\nIn this paper, we aim to draw these relations and make the following contributions: (i) We propose\nto drive exploration in model-based RL by the estimated progress in model learning. We estimate\nthis progress in terms of the loss over the training data used for model learning. (ii) We introduce\n\n1\n\n\ftwo algorithms based on modi\ufb01cations of R-MAX and the recent Bayesian exploration bonus (BEB)\napproach [9]. In contrast to the existing approaches, our algorithms do not have to assume correct\nprior knowledge or that the visitation counts translate directly to model certainty. Hence, they can\nalso cope with changing dynamics.\n(iii) While our extensions are targeted at scenarios that go\nbeyond the standard domain of stationary unstructured, \ufb01nite state and action spaces, we provide\na kind of theoretical sanity check of our extensions exactly under these standard assumptions: We\ndiscuss exploration guarantees under standard assumptions analogous to those of R-MAX and BEB.\nIn the next section, we review background work on exploration in Markov decision processes. Then\nwe present our approaches for exploration based on an empirical estimate of the future model learn-\ning progress. Thereafter, we discuss guarantees on their exploration ef\ufb01ciency. Finally, we present\nan empirical evaluation before we conclude.\n\n2 Background on Exploration\n\nfor a \ufb01xed horizon H, R = (cid:80)H\n\nt=1 R(st), or for an in\ufb01nite horizon, R = (cid:80)\n\nWe model the interaction of an agent with its environment as a Markov decision process (MDP).\nAn MDP is a discrete-time stochastic control process where at each time-step the process is in one\nof a \ufb01xed set S of states and the agent can choose an action from a set A. The transition model T\nspeci\ufb01es the conditional transition distribution T (s(cid:48) | a, s) over successor states s(cid:48) when executing\nan action a in a given state s. In unstructured \ufb01nite state and action spaces T can be de\ufb01ned by\nseparate multinomial distributions T (s, a) over successor states for each state-action pair s, a. The\nagent receives rewards in states according to a function R : S \u2192 R. A policy \u03c0 : S \u2192 A\nspeci\ufb01es for each state the action to take. The goal of planning in an MDP is to \ufb01nd the optimal\npolicy \u03c0\u2217 which maximizes the expected future rewards E[R]. The future rewards R can be de\ufb01ned\nt \u03b3tR(st), using a\ndiscount factor 0 < \u03b3 < 1. The value of state s when acting according to policy \u03c0 is de\ufb01ned as\nthe expected future rewards when starting from s, V \u03c0(s) = E[R | s1 = s, \u03c0]. The optimal policy\n\u03c0\u2217 can be found by classical algorithms such as value iteration or policy iteration. In reinforcement\nlearning, the agent does not know the transition model T . In a model-based approach, the agent\nestimates \u02c6T from its interaction trace with the environment \u2206 = (cid:104)s1, r1, a1, . . . , sT , rT(cid:105). Based on\n\u02c6T it computes (approximately) optimal policies. A simple approach to the exploitation-exploration\ntradeoff is \u0001-greedy:\nthe agent performs a random action for exploration with probability \u0001 and\nexploits otherwise by executing a greedy policy with respect to \u02c6T . If \u0001 decreases over time towards\n0, \u0001-greedy exploration converges to \u03c0\u2217. However, this may take an inef\ufb01ciently large number of\nnon-optimal actions which is exponential in |S| and |A|.\nIn the PAC-MDP (probably approximately correct) framework, the ef\ufb01ciency of an exploration\nalgorithm A is measured by its sample complexity [7]. This is the number of time-steps when fol-\n(st) \u2212 \u0001.\nlowing A where its policy \u03c0A\nGiven \u03b4 with 0 < \u03b4 < 1, A is said to be PAC-MDP ef\ufb01cient if with probability 1 \u2212 \u03b4 its sample com-\nplexity scales polynomially in quantities describing T as well as in \u03b4 and \u0001 (and \u03b3). The model-based\nRL algorithms E3 [8] and R-MAX [4] are PAC-MDP ef\ufb01cient exploration methods for unstructured\n\ufb01nite state and action spaces: their sample complexity scales polynomially in |S| and |A|. E3\nand R-MAX share the central concept of known states and actions which have been observed suf-\n\ufb01ciently often. (Counts are also used in the theoretical analysis of alternative PAC-MDP ef\ufb01cient\nalgorithms like MBIE-EB [15].) If the visitation count c(s, a) of a state-action pair s, a is larger\nthan some threshold m, the estimate \u02c6T (s, a) is guaranteed to be with high probability \u0001-close to\nthe true model. Following a policy in these known states achieves approximately the same rewards\nin the learned model \u02c6T and the true model T . To drive exploration, R-MAX is \u201coptimistic in the\nface of uncertainty\u201d and assumes maximum reward Rmax in unknown states. This gives the reward\nfunction\n\nt at time t is not \u0001-optimal, that is, where V \u03c0A\n\nt (st) < V \u03c0\u2217\n\n(cid:26) R(s, a) c(s, a)\u2265 m (s, a known)\n\nRmax\n\nc(s, a) < m (s, a unknown) .\n\nRR-MAX(s, a) =\n\nTypically, the threshold m is very large as E3 and R-MAX need to account for all possible model\ninstantiations within the model class of T [14]. For instance, for conservative choices \u0001 = 0.1,\n\u03b3 = 0.9, \u03b4 = 0.1, S = 10, A = 5, we get m > 106, which is unfeasible in practice. PAC-MDP\napproaches like R-MAX ignore the current empirical progress in learning T : the threshold m is \ufb01xed\n\n2\n\n\fa-priori and remains the same for all s, a independently of the agent\u2019s experiences or its estimated\nrelevance of s, a for large rewards.\nBayesian reinforcement learning [11] is an alternative approach to exploration. Here, the agent\ntakes its uncertainty about the learned model \u02c6T explicitly into account. This allows to incorporate\nprior knowledge. More formally, the agent maintains a posterior belief b over all possible transition\nmodels T given its previous experiences \u2206 and a prior. The value function for a deterministic policy\n\u03c0(b, s) is de\ufb01ned in terms of the state s and the belief state b and ful\ufb01lls\n\nV \u03c0(b, s) = R(s, \u03c0(b, s)) +\n\nT (b(cid:48), s(cid:48) | b, s, \u03c0(b, s)) V \u03c0(b(cid:48), s(cid:48)\n\n) .\n\n(cid:90)\n\nb(cid:48),s(cid:48)\n\n(1)\nThe optimal Bayesian policy \u03c0\u2217 = argmax\u03c0 V \u03c0(b, s) solves the exploitation-exploration tradeoff\nimplicitly: \u03c0\u2217 considers how actions affect not only the state of the world, but also the agent\u2019s inter-\nnal belief about the world. In a Bayesian RL context for a \ufb01nite horizon H, the sample complexity\nof an algorithm A can be de\ufb01ned as the number of time-steps when following A where its policy\nt at time t is not \u0001-Bayesian-optimal, that is, where V \u03c0A\nH (bt, st) \u2212 \u0001. Due to\n\u03c0A\nthe complexity of the belief state, however, Bayesian RL is typically intractable in terms of both\nplanning and updating the belief after an action. A recent approximate solution to Bayesian RL\nfor unstructured \ufb01nite representations is the Bayesian exploration bonus (BEB) [9] which resembles\nclosely MBIE-EB [15]. In this setting, the belief b over T can be expressed by means of a separate\n(cid:80)\nDirichlet distribution \u03b1(s, a, s(cid:48)) for each s, a with a mean estimator \u02c6Tb(s(cid:48) | s, a) = \u03b1(s,a,s(cid:48))\ns(cid:48) \u03b1(s,a,s(cid:48)).\nBEB avoids reasoning in the belief space: it solves an MDP built from the mean estimate \u02c6Tb using\nan additional exploration bonus \u03b2/(1 + c(s, a)) to reward state-action pairs inversely according to\ntheir visitation counts c(s, a). In the undiscounted, \ufb01xed horizon H formulation BEB has a Bayesian\nsample complexity which with probability 1\u2212 \u03b4 is polynomial in |S| and |A| when setting \u03b2 = 2H 2\n\nand updating the belief for s, a is stopped once(cid:80)\n\ns(cid:48) \u03b1(s, a, s(cid:48)) > 4H 3/\u0001 [9].\n\nH (bt, st) < V \u03c0\u2217\n\nt\n\nBayesian RL approaches are vulnerable to incorrect priors. Fard and Pineau [5] have combined\nBayesian and PAC approaches to derive bounds on the approximation error in the value function of\npolicies regardless of the correctness of the prior. However, their bounds do not apply to changing\ntransition dynamics and it remains unanswered how to incorporate them in ef\ufb01cient exploration\nalgorithms.\nIn a wider context of RL and developmental robotics, many strategies for ef\ufb01cient exploration have\nbeen subsumed by the concept of intrinsic motivation [10, 13] which is also termed fun or curiosity\n[12]. Many of these approaches take empirical learning progress into account. This includes meth-\nods that estimate from the agent\u2019s experience the amount of potential learning progress in different\nregions of the state space. Thereby, exploration focuses on those areas where learning progress\ncan indeed be made: areas which are neither already well-understood nor currently too dif\ufb01cult to\nlearn. The resulting algorithms enable an agent to develop progressively more complex skills. For\ninstance, this has been demonstrated for learning in robot control [1] and options learning in hierar-\nchical reinforcement learning [16] and has also been considered lately in machine learning under the\nname of curriculum learning [3]. Under the RL formalism only very recently have the concept of in-\ntrinsic motivation been mixed with standard exploration-exploitation strategies [6]. So far, however,\nguarantees about the sample complexity of intrinsic motivation based exploration approaches have\nbeen missing. In this paper, we take up the ideas of intrinsic motivation to extend the theoretically\nfounded exploration approaches described above.\n\n3 Exploration by Empirically Estimated Model Accuracy and Learning\n\nProgress\n\nPAC-MDP approaches like R-MAX and Bayesian RL approaches like BEB have been developed\nin the context of stationary \ufb01nite state-action domains. In those problems, we know that after a\n\ufb01xed number of visits to a state its estimated transition model becomes approximately correct and\nwe can perform exact belief updates to guarantee Bayesian optimality. In the following, we present\nextensions which rely instead on previous exploration and learning experience to estimate in which\nparts of the state and action space further exploration is promising and where not. This is helpful\nin situations where the basic assumptions of R-MAX and BEB about model improvement might be\nviolated: for example, when we have an incorrect prior (large misspeci\ufb01ed priors may impair the\n\n3\n\n\f(cid:26) R(s, a)\n\nRmax\n\nperformance of BEB[9]), when we cannot come up with an a-priori threshold on the data number\nfor learning accurate models (e.g. in domains where a polynomial KWIK learner is not available), or\nwhen the transition dynamics change over time. We can also see our approach as a method to adjust\nthe standard R-MAX threshold online, allowing to adapt to different levels of noise on different parts\nof the state space.\n\n3.1 Exploration Driven by Learning Progress\nLet \u03b6 : S \u00d7 A (cid:55)\u2192 R denote a measure for the expected learning and exploration progress when\nvisiting a state-action pair s, a. We discuss concrete de\ufb01nitions of \u03b6 later. Clearly, \u03b6 is a non-\nstationary function which changes with new experiences. Hence, an exploration strategy based on \u03b6\nneeds to re-estimate \u03b6 with each new experience. We use \u03b6 to de\ufb01ne two exploration algorithms.\nOur \ufb01rst approach \u03b6-R-MAX is based on R-MAX [4]. \u03b6-R-MAX acts greedily with respect to the\noptimal policy for the reward function\n\nR\u03b6-R-MAX(s, a) =\n\n\u03b6(s, a) < m\nelse\n\n.\n\n(2)\n\nInstead of rewarding arbitrary states with low visitation counts (considered unknown) directly as\nin R-MAX, \u03b6-R-MAX gets large reward for exploring such state-action pairs where the expected\nlearning progress is large.\nOur second approach \u03b6-EB is based on Bayesian Exploration Bonus (BEB) [9]. \u03b6-EB acts greedily\nwith respect to the optimal policy for the reward function\nR\u03b6-EB(s, a) = R(s, a) +\n\n(3)\n\n\u03b2\n1\u221a\n\n\u03b6(s,a)\n\n1 +\n\nfor some constant \u03b2.\nInstead of setting the exploration bonus directly proportional to visitation\ncounts as in BEB , \u03b6-EB gets a bonus for exploring state-actions pairs where the expected learning\nprogress is large. The idea of using expected learning progress to drive exploration is that we can\nestimate \u03b6 empirically from the interaction data \u2206 = (cid:104)s1, a1, r1, s2, . . . , sT , rT(cid:105).\n\n3.2 Empirically Estimated Model Accuracy and Learning Progress\n\nWe start by considering an empirical estimate of the current model accuracy. In a classical learning\ncontext, model accuracy is ideally tested on a held-out test data set. However, to exploit the full\navailable data for model selection and algorithm comparison, cross validation methods have become\na standard method. The work [2] discusses the importance of estimating the variance of a cross-\nvalidation estimator of the model accuracy, for instance to include this uncertainty of model accuracy\nin a model selection criterion. We base our following treatment of \u03b6 on this previous work.\nLet \u02c6T denote the estimated transition model based on data \u2206, which approximates the true model\nT . In general, we assume that learning \u02c6T (s, a) implies minimizing the loss L( \u02c6T (s, a); Ds,a) where\nDs,a = {s(cid:48)\ni) from s, a in \u2206. \u02c6T may generalize\nover states and actions, in which case the data for learning \u02c6T (s, a) and evaluating the loss may\ninclude experience sets Ds(cid:48),a(cid:48) with s(cid:48) (cid:54)= s, a(cid:48) (cid:54)= a (this is for example important for relational or\ncontinuous domains). A typical loss is the neg-log data-likelihood,\n\ni}ns,a\ni=1 are the successor states in the transitions (s, a, s(cid:48)\n\nL( \u02c6T ; Ds,a) = \u2212 1\n\n|Ds,a| log\n\nGiven such a loss, the predictive error is de\ufb01ned as\n\n(cid:89)\n\ns(cid:48)\u2208Ds,a\n\n\u02c6T (s(cid:48) | s, a) .\n\n(4)\n\n(5)\nAn empirical estimator of the PE based on the available data Ds,a is the leave-one-out cross-\nvalidation estimator\n\nP E(s, a) = Es(cid:48)\u223cT (s,a) L( \u02c6T ;{s(cid:48)}) .\n\nCV (Ds,a, s, a) =\n\n1\n\n|Ds,a|\n\n(cid:96)loo\ns(cid:48)\n\n,\n\n(cid:96)loo\ns(cid:48)\n\n:= L( \u02c6T \u2212s(cid:48)\n\n;{s(cid:48)})\n\n(6)\n\n(cid:88)\n\ns(cid:48)\u2208D\n\n4\n\n\fis the model learned from data D\u2212s(cid:48)\n\nwhere \u02c6T \u2212s(cid:48)\nPutting an absolute threshold directly on the loss to decide whether a state is known or unknown\n\nis hard. Note that the predictive error P E(s, a) = KL(cid:0)T (s, a)\n\n(cid:12)(cid:12)(cid:12)(cid:12) \u02c6T (s, a)(cid:1) + H(cid:0)T (s, a)(cid:1) has the\n\ns,a = Ds,a \\ {s(cid:48)}.\n\nentropy of the true distribution as a lower bound, which is unknown. Therefore, we propose to\ndrive exploration based on the learning progress instead of the current learner accuracy. Using the\nchange in loss we may gain robustness by becoming independent of the loss\u2019 absolute value and can\npotentially detect time-varying conditions.\ni \u2208\nWe de\ufb01ne \u03b6 in terms of the change in the (empirically estimated) loss as follows. Let D\u2212k\nDs,a | i < ns,a \u2212 k} denote the experiences in Ds,a except the last k. \u02c6T \u2212k is the transition model\ns,a, in contrast to \u02c6T which is learned from all data Ds,a. We\nlearned from the reduced data-set D\u2212k\nde\ufb01ne\n\ns,a = {s(cid:48)\n\n\u02c6\u03b6(s, a) := CV (D\u2212k\n\ns,a, s, a) \u2212 CV (Ds,a, s, a) \u2248 L( \u02c6T \u2212k; Ds,a) \u2212 L( \u02c6T ; Ds,a) .\n\n(7)\nThis estimates to which extent the last k experiences help to learn a better model as evaluated over\nthe complete data. Thus, if \u02c6\u03b6(s, a) is small, then the last k visitations in the data-set Ds,a did not\nhave a signi\ufb01cant effect on improving \u02c6T and in turn s, a does not require further exploration.\nThe estimator \u02c6\u03b6(s, a) de\ufb01ned above is only a mean estimator of the learning progress. The \u03b6(s, a) we\nuse in concrete exploration algorithms \u03b6-EB and \u03b6-R-MAX includes an additional variance margin,\n\n\u03b6(s, a) := \u02c6\u03b6(s, a) + \u03b1(cid:112)\u03bd(s, a) ,\n\nwhere \u03bd(s, a) is an estimate of the CV variance (discussed in more detail below),\n\n\u03bd(s, a) =\n\n1\n\n|Ds,a|\n\ns(cid:48) \u2212 CV (Ds,a, s, a)]2 .\n[(cid:96)loo\n\n(cid:88)\n\ns(cid:48)\u2208Ds,a\n\n(8)\n\n(9)\n\nThe variance margin increases robustness and is motivated by the following analysis.\n\n3.3 Guarantees on the Exploration Ef\ufb01ciency\n\nAs discussed in the introduction, we propose our extensions of R-MAX and BEB to target scenarios\nwhich go beyond the standard setting of stationary unstructured, \ufb01nite state and action spaces. In\nthis subsection, however, we go back and consider the behavior of our extensions exactly under\nthese classical assumptions\u2014this is meant as a sanity check to ensure that our extensions inherit\nthe standard exploration ef\ufb01ciency properties under standard assumptions. We will directly relate a\nthreshold on the empirical \u03b6(s, a) to a threshold on model accuracy.\n\nWe start by providing two properties of the mean\n\nunder random data. First we \ufb01nd\n\nthat the expected \u02c6\u03b6(s, a) converges with 1/n2:\nLemma 1. For a Dirichlet learner in a stationary environment, we have\n\n(cid:69)\n(cid:68)\u02c6\u03b6(s, a)\n(cid:18) 1\n\nDs,a\n\n(cid:19)\n\n.\n\n= O\n\nDs,a\n\nn2\n\ns,a\n\n(cid:68)\u02c6\u03b6(s, a)\n(cid:69)\n\n(cid:68)\u02c6\u03b6(s, a)\n(cid:69)\n\nA proof is given in the supplementary material. Similarly, a threshold on the mean\nimplies a model accuracy threshold:\nLemma 2. Given an approximated model \u02c6T of a true model T , for any \u0001 there exists an \u0001(cid:48) such\nthat:\n\nDs,a\n\n| < \u0001(cid:48) \u21d2 | \u02c6T (s, a) \u2212 T (s, a)| < \u0001 .\n\n(10)\n\n|(cid:68)\u02c6\u03b6(s, a)\n\n(cid:69)\n\nDs,a\n\nSketch of proof. For the case of multinomial variables, we know that the maximum likelihood\nestimator is consistent and unbiased and is equal to the normalized visit counts. In this situation we\nn(\u02c6p \u2212 p) (cid:32) N (0, \u03a3). Our measure of progress \u03b6 is the difference between\nknow that as n \u2192 \u221e,\n\n\u221a\n\n5\n\n\f(cid:68)\u02c6\u03b6(s, a)\n(cid:69)\n\ntwo maximum likelihood estimators and so, as n approaches in\ufb01nity we have the same limiting result\non the model quality, with a higher variance due to the subtraction between two different random\nvariables and the correlation between them. (cid:3)\nunder random\nWith the previous results we know that if we had access to the mean\ndata we would be able to assess model error by looking at its value. Unfortunately, the agent only\nhas access to an empirically estimate. To ensure that we can de\ufb01ne robust criteria for considering a\nstate as known, we have to consider the variability of the estimator \u02c6\u03b6(s, a) under random data Ds,a.\nAs discussed in [2], the estimator is unbiased, that is (cid:104)CV (Ds,a, s, a)(cid:105)D = P E(s, a) and, in the\nlimit |Ds,a| \u2192 \u221e, the ( \u02c6T \u2212T ) becomes Gaussian. Its variance can be described by \ufb01rst considering\nthe covariance matrix C of the vector ((cid:96)loo\ns(cid:48) )s(cid:48)\u2208Ds,a under random Ds,a. The diagonal entries are the\nvariances VarD{(cid:96)loo\ns(cid:48) } of each single (cid:96)loo\ns(cid:48) under random data, which are independent of s(cid:48) (assuming\ni.i.d. data) and therefore equally estimated as \u03bd(s, a) given in Eq. (9). The off-diagonal entries of C\ncapture the correlations between different (cid:96)loo\ns(cid:48)(cid:48) and are constant (due to i.i.d. data; see [2] for\ndetails). By assuming these correlations to vanish we over-estimate the CV variance and therefore\nhave, from Eq. (6),\n\ns(cid:48) and (cid:96)loo\n\nDs,a\n\nVarDs,a{CV (Ds,a, s, a)} \u2264 \u03bd(s, a)\n\n(11)\nHaving an overestimation of the variance of the loss we will consider what is the variance of the\nestimation of \u02c6\u03b6(s, a). Both terms L( \u02c6T ; Ds,a) and L( \u02c6T \u2212k; Ds,a) are estimated using LOO-CV,\nallowing us to bound the \u02c6\u03b6\u2019s variance under random data, from Eq. (11), as:\n\nVarDs,a{\u02c6\u03b6(s, a)} \u2264 2\u03bd(Ds,a, s, a) .\n\nNow that we have a con\ufb01dence measure on the estimator we can show that a threshold on the\n:\n\n(cid:68)\u02c6\u03b6(s, a)\nempirical estimator \u03b6(s, a) = \u02c6\u03b6(s, a) + \u03b1(cid:112)\u03bd(s, a) implies a threshold on the mean\n| < \u0001(cid:3) with probability 1 \u2212 \u03b4 .\n\n(cid:2)|\u02c6\u03b6(s, a)| + \u03b1(cid:112)\u03bd(s, a) < \u0001 \u21d2 |(cid:68)\u02c6\u03b6(s, a)\n(cid:69)\n\nLemma 3. For any given \u03b4 with 0 < \u03b4 < 1 and \u0001 > 0 there exists an \u03b1 such that\n\n(cid:69)\n\n(13)\n\nDs,a\n\n(12)\n\n\u221a\nProof. For a Gaussian variable x with mean \u00b5 and variance \u03bd we know that x < \u00b5 + \u03b1\nprobability given by the error function \u03b4 = 1/2 + erf(\u03b1/\nthe above.\n\n\u03bd with\n2)/2. By inverting this we get \u03b1 to ful\ufb01ll\n\nDs,a\n\n\u221a\n\nFinally, we show that our exploration method using the empirical \u03b6(s, a) to drive exploration is\nPAC-MDP ef\ufb01cient.\nLemma 4. There is a threshold m such that \u03b6-R-MAX using a Dirichlet learner in the standard\nsetting of stationary unstructured, \ufb01nite state and action spaces is PAC-MDP ef\ufb01cient.\n\nProof. From Lemma 3 we know that a threshold on our empirical measure implies a threshold on\nthe mean measure. From Lemma 2 we know that a small\ncorresponds, with high\nprobability, to a low model error. Under these conditions a state is only marked as known if the\nempirical measure is below a certain threshold; this ensures with high probability that the error in\nthe model for the state-action is low. From the standard conditions of R-MAX, \u03b6-R-MAX is PAC-\nMDP ef\ufb01cient.\n\nDs,a\n\n(cid:68)\u02c6\u03b6(s, a)\n\n(cid:69)\n\n4 Evaluation\n\nWe compare the empirical performance of our exploration algorithms \u03b6-R-MAX and \u03b6-EB with\nthe performance of R-MAX, BEB and simple model-based \u0001-greedy exploration with optimistic\ninitialization in unstructured \ufb01nite state and action spaces. We investigate different scenarios where\nthe assumptions of R-MAX and BEB are ful\ufb01lled or violated. We de\ufb01ne these scenarios by varying\nthe level of stochasticity in state transitions. BEB assumes to have a correct a prior about this\nstochasticity. R-MAX assumes to know correct thresholds m for the number of visits to states to\n\n6\n\n\fensure accurate transition models. We simulate satis\ufb01ed or violated assumptions of R-MAX by\nsetting individual thresholds for states: in general, states with higher noise require more samples to\nachieve the same level of model accuracy as states with low noise. Setting individual thresholds is\nequivalent to setting individual initials counts (instead of 0s) for states.\nWe investigate three questions: (a) How close do our algorithms \u03b6-R-MAX and \u03b6-EB get to the\nperformance of the original algorithms when the assumptions of the latter are correct? (b) How\nmuch do \u03b6-R-MAX and \u03b6-EB gain if the assumptions of the original algorithms are violated? (c)\nAnd are our approaches more robust to unexpected changes in the transition dynamics?\nOur evaluation environment (shown in Fig. 1(a)) is a discrete MDP with 25 states and \ufb01ve actions:\nup, down, left, right and stop. There is a single goal state with a reward of 1 (marked with \u201cG\u201d) and\nseveral states with negative rewards \u22120.1 (darker states). The lighter states are noisy states: their\nactions have less predictable effects. The transition probabilities of the noisy states are sampled\nfrom a Dirichlet distribution with parameters \u03b1 = 0.1, while the probabilities for all other states are\nsampled from a Dirichlet with \u03b1 = 1.0, in both cases eventually permuted to ensure that the highest\nprobability corresponds to the expected next state according to the name de\ufb01nition of the actions.\nThe shortest path from start to goal is not optimal due to the uncertainty in the transitions. Instead,\nthe optimal path avoids the lighter states and takes the route below. To \ufb01nd this optimal path and\navoid local minima, an exploration algorithm needs to explore suf\ufb01ciently and estimate the state\nvalues accurately. We evaluate the performance of the algorithms in terms of the reward collected\nin the true model T using the optimal policy \u03c0\u2217\n\u02c6T for their learned model \u02c6T . In our results, we report\nthe policy value error de\ufb01ned as VT (sI ; \u03c0\u2217\nT ) in the value of the start state sI with\nrespect to the optimal policy \u03c0\u2217. In our experiments, the agent is reset to sI every 30 steps. All\n\ufb01gures presented in the following show the means and standard deviations over 20 runs. For the \u03b6\nestimation we use k = 10.\n\n\u02c6T ) \u2212 VT (sI ; \u03c0\u2217\n\nExperiment 1: Correct Assumptions\nIn our \ufb01rst experiment, the assumptions of BEB and R-\nMAX are ful\ufb01lled: BEB is given the correct prior; R-MAX uses appropriate thresholds for states\n(depending on the state stochasticity). \u03b6-R-MAX, \u03b6-EB and \u0001-greedy are not given any knowledge.\nThe results presented in Fig. 1(b) show that our exploration methods \u03b6-R-MAX and \u03b6-EB achieve\nsimilar performance to the original methods, even without having a correct prior or state-dependent\nthresholds. Both \u03b6-R-MAX and \u03b6-EB converge to the correct \ufb01nal policy, requiring only moder-\nately more steps. In contrast, \u0001-greedy does not \ufb01nd the optimal policy in reasonable time. Clearly,\nthe original algorithms could also be executed based on likewise correct, but less informative as-\nsumptions: BEB with an uninformative prior, R-MAX with more conservative (larger) uniform\nthreeholds. Then, R-MAX would need longer learning time; BEB might not converge, see [9].\n\nExperiment 2: Violated Assumptions Here, the assumptions of R-MAX and BEB are violated\n(wrong thresholds/priors). This may well happen in practice where correct priors cannot always be\nspeci\ufb01ed or the counts cannot be translated directly to model certainty. In each run, we initialize\nR-MAX and BEB with a random uniform prior (in the interval of the minimum and maximum values\nof the true prior, translated to counts for R-MAX). The results in Fig. 1(c) show that as expected\nR-MAX and BEB do not converge any longer to the optimal policy: they explore states too long\nwhose dynamics are already well estimated, while neglecting states which require more samples for\nan accurate model estimation. In contrast, \u03b6-R-MAX and \u03b6-EB do not rely on these assumptions and\nagain converge to the optimal policy.\n\nExperiment 3: Change in Dynamics\nIn our \ufb01nal experiment, the transition dynamics for a ran-\ndomly chosen state along the optimal path get permuted after 900 steps. As Fig. 1(d) shows, R-MAX\nand BEB with correct assumptions for the original problem (before time-step 900) cannot compen-\nsate for this as they base their estimate of the model certainty only on the visitation counts, but do\nnot look at the data itself. In contrast, \u03b6-R-MAX and \u03b6-EB detect such an unexpected event and can\nrefocus their exploration efforts.\n\n5 Conclusions and Future Work\n\nWe have proposed to drive exploration in model-based reinforcement learning using the estimated\nfuture progress in model learning. When estimating this learning progress empirically, exploration\n\n7\n\n\f(a) Evaluation Environment\n\n(b) Experiment 1\u2014Correct Assumptions\n\n(c) Experiment 2\u2014Violated Assumptions\n\n(d) Experiment 3\u2014Change in Dynamics\n\nFigure 1: Experiments: (a) The agent starts at state \u201cI\u201d and needs to get to goal \u201cG\u201d. Grey (darker)\nstates incur a negative reward. Green (lighter) states have very noisy transition dynamics. (b) Like\nR-MAX and BEB with correct assumptions, our algorithms \u03b6-R-MAX and \u03b6-EB based on an em-\npirical estimation of the learning progress converge to the optimal policy without relying on these\nassumptions, but take a small extra amount of time. (c) When their assumptions are violated, R-\nMAX and BEB fail to converge, while \u03b6-R-MAX and \u03b6-EB don\u2019t rely on these assumptions and\nagain \ufb01nd the optimal policy. (d) In contrast to existing methods, \u03b6-R-MAX and \u03b6-EB can cope with\nthe change in transition dynamics after 900 steps and refocus their exploration.\n\nalgorithms can be de\ufb01ned which do not rely on correct prior knowledge and can cope with changing\ntransition dynamics. As a theoretical \u201csanity check\u201d we have discussed ef\ufb01ciency guarantees of our\napproaches similar to the ones of the established algorithms. Our novel problem settings provide\ninteresting opportunities for the development of RL algorithms and theoretical analyses for relevant\nreal-world scenarios, in particular for structured, continuous and non-stationary domains. It is also\nworth to investigate in more depth the relation of our approach to the general concept of intrinsic\nmotivation as proposed in developmental robotics. In our view, a combination of methods which\ntrades off both strong prior assumptions together with empirical estimates of the learning progress\nseems to be the most promising direction for future work on exploration in the real world.\n\nAcknowledgments\n\nWork supported by the Flowers Team (INRIA/ENSTA-Paristech), Conseil R\u00b4egional d\u2019Aquitaine\nand the ERC grant EXPLORERS 24007. Tobias Lang and Marc Toussaint were supported by the\nGerman Research Foundation under grants TO 409/1-3 and TO 409/7-1.\n\n8\n\nGI\fReferences\n[1] A. Baranes and P.Y. Oudeyer. Active learning of inverse models with intrinsically motivated\n\ngoal exploration in robots. Robotics and Autonomous Systems, 2012.\n\n[2] Yoshua Bengio and Yves Grandvalet. No unbiased estimator of the variance of k-fold cross-\n\nvalidation. Journal of Machine Learning Research (JMLR), 5:1089\u20131105, 2004.\n\n[3] Yoshua Bengio, J\u00b4er\u02c6ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.\n\nIn Int. Conf. on Machine Learning (ICML), pages 41\u201348, 2009.\n\n[4] Ronen I. Brafman and Moshe Tennenholtz. R-max - a general polynomial time algorithm for\nnear-optimal reinforcement learning. Journal of Machine Learning Research (JMLR), 3:213\u2013\n231, 2002.\n\n[5] Mahdi Milani Fard and Joelle Pineau. Pac-bayesian model selection for reinforcement learn-\n\ning. In Conf. on Neural Information Processing Systems (NIPS). 2010.\n\n[6] Todd Hester and Peter Stone. Intrinsically motivated model learning for a developing curious\n\nagent. In AAMAS Workshop on Adaptive Learning Agents (ALA), 2012.\n\n[7] Sham Kakade. On the Sample Complexity of Reinforcement Learning. PhD thesis, Gatsby\n\nComputational Neuroscience Unit, University College London, 2003.\n\n[8] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time.\n\nMachine Learning Journal, 49(2-3):209\u2013232, 2002.\n\n[9] J. Zico Kolter and Andrew Ng. Near-Bayesian exploration in polynomial time. In Int. Conf.\n\non Machine Learning (ICML), pages 513\u2013520, 2009.\n\n[10] P.Y. Oudeyer, F. Kaplan, and V.V. Hafner. Intrinsic motivation systems for autonomous mental\n\ndevelopment. IEEE Transactions on Evolutionary Computation, 11(2):265\u2013286, 2007.\n\n[11] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete\n\nBayesian reinforcement learning. In Int. Conf. on Machine Learning (ICML), 2006.\n\n[12] J\u00a8urgen Schmidhuber. Curious model-building control systems. In Proc. of Int. Joint Conf. on\n\nNeural Networks, volume 2, pages 1458\u20131463, 1991.\n\n[13] Satinder Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically motivated reinforce-\nment learning. In Conf. on Neural Information Processing Systems (NIPS), pages 1281\u20131288.\n2005.\n\n[14] Alexander L. Strehl, Lihong Li, and Michael Littman. Reinforcement learning in \ufb01nite MDPs:\n\nPAC analysis. Journal of Machine Learning Research (JMLR), 2009.\n\n[15] Alexander L. Strehl and Michael L. Littman. An analysis of model-based interval estimation\n\nfor markov decision processes. J. Comput. Syst. Sci., 74(8):1309\u20131331, 2008.\n\n[16] Christopher M. Vigorito and Andrew G. Barto. Intrinsically motivated hierarchical skill learn-\nIEEE Transactions on Autonomous Mental Development\n\ning in structured environments.\n(TAMD), 2(2), 2010.\n\n[17] Marco Wiering and J\u00a8urgen Schmidhuber. Ef\ufb01cient model-based exploration. In International\n\nConference on Simulation of Adaptive Behavior: From Animals to Animats 6, 1998.\n\n9\n\n\f", "award": [], "sourceid": 122, "authors": [{"given_name": "Manuel", "family_name": "Lopes", "institution": null}, {"given_name": "Tobias", "family_name": "Lang", "institution": null}, {"given_name": "Marc", "family_name": "Toussaint", "institution": null}, {"given_name": "Pierre-yves", "family_name": "Oudeyer", "institution": null}]}