{"title": "Learning Partially Observable Models Using Temporally Abstract Decision Trees", "book": "Advances in Neural Information Processing Systems", "page_first": 818, "page_last": 826, "abstract": "This paper introduces timeline trees, which are partial models of partially observable environments. Timeline trees are given some specific predictions to make and learn a decision tree over history. The main idea of timeline trees is to use temporally abstract features to identify and split on features of key events, spread arbitrarily far apart in the past (whereas previous decision-tree-based methods have been limited to a finite suffix of history). Experiments demonstrate that timeline trees can learn to make high quality predictions in complex, partially observable environments with high-dimensional observations (e.g. an arcade game).", "full_text": "Learning Partially Observable Models Using\n\nTemporally Abstract Decision Trees\n\nDepartment of Mathematics and Computer Science\n\nErik Talvitie\n\nFranklin & Marshall College\n\nLancaster, PA 17604\n\nerik.talvitie@fandm.edu\n\nAbstract\n\nThis paper introduces timeline trees, which are partial models of partially observ-\nable environments. Timeline trees are given some speci\ufb01c predictions to make and\nlearn a decision tree over history. The main idea of timeline trees is to use tem-\nporally abstract features to identify and split on features of key events, spread ar-\nbitrarily far apart in the past (whereas previous decision-tree-based methods have\nbeen limited to a \ufb01nite suf\ufb01x of history). Experiments demonstrate that timeline\ntrees can learn to make high quality predictions in complex, partially observable\nenvironments with high-dimensional observations (e.g. an arcade game).\n\n1\n\nIntroduction\n\nLearning a model of a high-dimensional environment can pose a signi\ufb01cant challenge, but the ability\nto make predictions about future events is key to good decision making. One common approach is\nto avoid learning a complete, monolithic model of the environment, and to instead focus on learning\npartial models that are only capable of making a restricted set of predictions (for instance, predic-\ntions about some particular aspect of the environment, or predictions about future rewards). Partial\nmodels can often be simpler to learn than a complete model. In some cases they can be combined\nto form complete, structured models, which can then be used for planning purposes (e.g. factored\nMDPs [1], collections of partial models [2]). In other cases, partial models can be directly useful for\ncontrol (e.g. U-Tree [3], prediction pro\ufb01le models [4]).\n\nThis paper introduces timeline trees which are partial models for partially observable environments.\nTimeline trees are focused on capturing a particular kind of partial observability; they assume that\ntheir predictions can be made by recalling a (\ufb01nite) sequence of events in the past that may have\noccurred far apart from each other in time. While not all partially observable phenomena take this\nform, a good deal of everyday partial observability has this \ufb02avor. For instance, you may know that\nyour keys are in the next room because you remember putting them there. Most of the experiences\nsince that event are probably irrelevant to making predictions about the location of your keys.\n\nThe main idea of timeline trees is to build a decision tree over history. As with similar approaches,\nthe decision tree can split on features of observations in recent history. However, a timeline tree may\nalso establish new timestamps in the past and is able split on features of observations surrounding\nthose events as well. For instance, there could be a timestamp representing the last time the agent\nsaw its keys, and then features of the neighboring observations could identify the keys\u2019 location. In\nthis way, timeline trees can make use of information arbitrarily spread out in history.\n\n1\n\n\f2 Partial Models\n\nThis paper will focus on discrete dynamical systems. Speci\ufb01cally, time procedes in discrete steps\nt = 1, 2, 3, . . .. At every step t, the agent selects an action at from a \ufb01nite set A and the envi-\nronment (stochastically) emits an observation ot, taken from a \ufb01nite set O. The history at time\nt is the sequence of actions and observations from the beginning of time, up through time t:\ndef= a1o1a2o2 . . . atot. In the general partially observable case, the observation emitted at each\nht\nstep may depend upon the entire history (and the agent\u2019s action). So, an agent wishing to predict the\nnext observation must model the conditional probability distribution Pr(Ot+1 | Ht, At+1).\n\nIf one is able to predict the next observation at any history and for any action (that is, if one has\naccess to this conditional distribution), one can compute the probability of any future sequence of\nobservations given any future sequence of actions and the history [5]. Such a model is called a\ncomplete model because in any situtation, it is capable of making any prediction about the future.\nExamples in the partially observable setting include POMDPs [6, 7] and PSRs [5, 8]. A partial\nmodel is any model that does not represent this full conditional distribution.\n\nThis paper will focus on partial models that make conditional predictions about abstract features of\nthe next observation, though many of the ideas can be straightforwardly adapted to work with pre-\ndictions of other forms. Formally, let \u03c9 and \u03ba be many-to-one mappings over the set of observations\nO. The task of the partial model at time t will be to predict the value of \u03c9(ot+1), conditioned on the\nvalue of \u03ba(ot+1). So it represents the distribution Pr(\u03c9(Ot+1) | Ht, At+1, \u03ba(Ot+1)). For example,\nin the experiment in Section 5.3, observations are images and multiple partial models are learned,\neach predicting the color of a single pixel, conditioned on the colors of pixels above and to the left.\n\n2.1 Related Work: Partial Models in Partially Observable Environments\n\nMcCallum\u2019s U-Tree [3] learns a decision tree over history, where the leaves of the tree map to the\nexpected discounted sum of rewards at the associated history (though the method could be adapted\nto make other predictions, as in [9]). McCallum used binary features of the form \u201cFeature X takes\nvalue Y at time-step t \u2212 k,\u201d where t is the current time-step. Thus the decision tree learns an\nabstraction both over observations (which could be high-dimensional in their own right) and over\nthe sequence of observations (by using features from multiple time-steps). However, because it can\nonly consider a \ufb01nite number of such features, U-Tree has a \ufb01nite memory horizon; events that occur\nbefore some arbitrary cutoff in the past cannot be taken into account when making predictions.\n\nTimeline trees are an extension of UTree that allow it to use features of observations arbitrarily far\nback in the past, though they are not the \ufb01rst attempt to address this issue. Looping predictive suf\ufb01x\ntrees (LPSTs) [10] are prediction suf\ufb01x trees [11] that allow nodes to loop back to their ancestors.\nLocal agent state representations (LASR) [12] map histories to a real number, and then learn a\ndirect mapping from that number to the target predictions. McCallum [13] and Mahmud [14] both\ndeveloped incremental hill-climbing algorithms to learn \ufb01nite state machines (FSMs), where each\nstate is associated with predictions about future rewards, and the transitions depend on both the\naction taken and the observation received. Prediction pro\ufb01le models [4] are similar FSMs, but rather\nthan hill-climbing, they are learned by pre-processing the data and then applying standard complete\nmodel learning methods (they were demonstrated using POMDPs and LPSTs).\n\nAll of these approaches can, in principle, represent arbitrarily long-range dependencies in time.\nHowever, unlike U-Tree, they all treat observations as atomic, which limits their applicability to\ntruly high-dimensional systems. Furthermore, despite their theoretical representational capacity,\ntheir learning algorithms have dif\ufb01culty discovering long-range dependencies in practice.\n\nThe learning algorithm for LPSTs \ufb01rst learns a full suf\ufb01x tree, and then adds loops as appropri-\nate. Thus, to capture very long-range dependencies, one must \ufb01rst build a very deep suf\ufb01x tree.\nMcCallum reported that his FSM-learning method was often unable to detect long-range temporal\ndependencies (since this typically involves multiple elaborations of the FSM, none of which would\nindividually seem valuable to the hill-climbing algorithm). Mahmud\u2019s similar approach would likely\nsuffer a similar limitation. The learning algorithms for LASR and prediction pro\ufb01le models both rely\non estimating predictions at particular histories. Because estimates will only be accurate for histo-\nries that appear many times, these algorithms can only be effectively applied to data consisting of\nmany short trajectories, which limits their ability to discover long-range dependencies in practice.\n\n2\n\n\fIt should be noted that prediction pro\ufb01le models have been combined with an additional pre-\nprocessing step that learns an abstraction before the prediction pro\ufb01le model is learned [15]. Because\nof this, and because their formulation most closely \ufb01ts the setting of this paper, the experiments in\nSection 5 will directly compare against the performance of prediction pro\ufb01le models.\n\n3 Timeline Trees\n\nThe goal of timeline trees is to combine the strengths of U-Tree with the ability to attend to events\narbitrarily far apart in history (rather than limited to a \ufb01nite suf\ufb01x). Unlike several of the above\napproaches, timeline trees are not arbitrarily recurrent (they do not contain loops except in a limited,\nimplicit sense), which does restrict their representational capacity. However, in exchange they retain\nthe straightfoward decision tree training of U-Tree, which allows them to simultaneously learn an\nabstraction over both the history sequence and high-dimensional observations and which further-\nmore allows them to discover long-range temporal depencies in practice (and not just in principle).\n\n3.1 Timestamps\n\nThe decision tree built by U-Tree splits on features of observations at some temporal offset from\nthe current timestep. The key idea of timeline trees is to allow multiple timestamps in history and\nto allow splits on features of observations at temporal offsets relative to any of these timestamps.\nTimeline trees take a set F of binary features, where each feature f (ht, k) takes the history at time t,\nht, and a timestep 0 < k \u2264 t + 11 and returns 1 or 0. For example, if the observations are images, f\ncould return 1 if a black pixel existed anywhere at step k \u2212 1 but did not exist at step k. It is assumed\nthat f (ht, k) makes use of no information after step k (though it may access timestep k or before).\n\nFor a \ufb01xed vector \u03c4 of timestamps, the model is a standard decision tree, and only a small extension\nof U-Tree (which \ufb01xed the number of timestamps to 1: the current timestep). Each internal node in\nthe tree is associated with a feature f , a timestamp index i, and a temporal offset \u03b4 (which may be\nnegative) and has two children representing histories where the value of f (ht, \u03c4 [i] + \u03b4) is 0 or 1, re-\nspectively. The leaves of the tree are associated with estimates of Pr(\u03c9(Ot+1) | ht, at+1, \u03ba(Ot+1)).\nTo use the timeline tree to make a prediction, one simply follows a path from the root to a leaf in the\ntree, choosing the appropriate child at each node according to the feature value f (ht, \u03c4 [i] + \u03b4).\n\nTimeline trees\u2019 real strength lies in their ability to add new timestamps. They do this via a special\ntype of feature. For every feature f \u2208 F , there is an additional timestamp feature \u03bef . The feature\n\u03bef (ht, j, k) = 1 if there is some timestep m such that j < m < k where f (ht, m) = 1. More\nimportantly, the greatest such m (that is, the time of the most recent occurence of f ), call it mf , is\nadded as a timestamp to all nodes in the subtree where \u03bef = 1.\n\nWhen making a prediction for ht, one maintains a growing vector \u03c4 of timestamps (in order of least\nto most recent). Beginning at the root there is only one timestamp: \u03c4root = ht + 1i, where t is the\ncurrent timestep. As one travels from the root to a leaf, one may encounter a node associated with\ntimestamp feature \u03bef . Such a node is also associated with an index i into the current timestamp\nvector. If \u03bef (ht, \u03c4 [i \u2212 1], \u03c4 [i]) = 1, the path moves to the corresponding child and adds mf to \u03c4\n(let \u03c4 [0] def= \u22121). Nodes further down in the tree may refer to this new timestamp. As such, the tree\nis able to establish timestamps based on the occurence of key events (the presence of some feature).\n\nTimestamp features are a form of temporal abstraction; they refer to an event in the past, but abstract\naway how long ago it was. They are limited, however. There are systems that would require an in\ufb01-\nnite timeline tree that approaches in Section 2.1 can capture easily (see Section 5.2). Nevertheless,\nthey do capture a natural and intuitive form of partial observability, as can be seen by example.\n\n3.2 An Example\n\nAs a simple illustrative example, consider an agent that must keep track of its key. The agent\u2019s key\ncan be in room A or B, or in the agent\u2019s pocket (where it starts). The agent has three actions: move\n\n1For simplicity\u2019s sake, the notation f (ht, k) hides the fact that the features may also depend on at+1 and\n\u03ba(ot+1). For this discussion, assume that k may equal t + 1 if the feature makes use of only these aspects of\ntime t + 1. If k = t + 1 and the feature refers to other information, assume f (ht, k) = 0.\n\n3\n\n\fFigure 1: The Key World example.\n\n(which switches its location), stay (which does nothing), and pocket. The last action transfers the\nkey between the agent\u2019s pocket and the current room (in either direction) unless the key is in neither\n(in which case it does nothing). The agent can observe its location and whether the key is in the\nroom. A diagram is shown in the left of Figure 1 (missing arrows are self-loops).\n\nOn the right of Figure 1 an example timeline tree is shown that can predict whether the agent will\nsee the key in the next timestep. At the root, there is only one timestamp: t+1, where t is the current\nstep. The root checks if the agent can currently see the key. If so, the agent will only see the key\nin the next step if it stays. Otherwise, the agent must remember where the key was last seen. The\nsquare-shaped node is meant to indicate a timestamp feature, which checks if the agent has ever seen\nthe key before the only timestamp. If not, the key is in the agent\u2019s pocket. If so, a new timestamp m\nis added that marks the last time the key was seen. If the agent put the key in its pocket after m, it\nmust take the key out to see it. Otherwise, it must be in the other room.\n\n4 Learning Timeline Trees\n\nTimeline trees can be learned using standard decision tree induction algorithms (e.g. ID3 [16] and\nC4.5 [17]). The leaves of the tree contain the estimated predictions (counts of the occurrences of the\nvarious values of \u03c9(o) associated with histories mapping to that leaf). The tree starts as just the root\n(not associated with a feature). Each phase of training expands a single leaf by associating it with a\nfeature and adding the appropriate children under it. At each phase every candidate expansion (every\nleaf and every feature) is tried and the one that results in the highest information gain between the\npredictions of the original tree and the expanded tree is greedily selected.\n\nThe main difference in timeline trees is that different features may be available in different leaf\nnodes (because different timestamps will be available). Speci\ufb01cally for each leaf n, all features of\nthe form f (\u00b7, \u03c4n[i] + k) are considered for all timestamp indices i \u2208 {1, . . . , |\u03c4n|} and all integer\noffsets k in some \ufb01nite range. Similarly, all timestamp features of the form \u03bef (\u00b7, \u03c4n[i \u2212 1], \u03c4n[i]) are\nconsidered for all timestamp indices i. In the experiments below, candidate expansions also include\nall combinations of timestamp features and regular features (essentially two expansions at once).\nThese compound features take the form of \ufb01rst splitting on a timestamp feature, and then splitting\nthe resulting \u201c1 child\u201d with a regular feature. This allows the tree to notice that a timestamp is useful\nfor the subsequent splits it allows, even if it is not inherently informative itself. For instance, in\nthe Key World, knowing whether the agent has ever seen the key may not be very informative, but\nknowing that the pocket action was taken immediately after seeing the key is very informative.\n\nNote that compound features will tend to result in higher information gain than simple features. As a\nresult, there will be a bias toward selecting compound features, which is not necessarily desireable.\nTo combat this, the information gain of compound features was penalized by a factor of \u03b2.\nIn\nthe experiments below, \u03b2 = 0.5. Also note that because the information gain measurement used to\nchoose expansions is estimated from a \ufb01nite number of samples, expanding the tree until information\ngain is zero for all candidates will typically result in over\ufb01tting. Thus, some form of early stopping\nis common.\nIn this implementation expansions are only considered if they make a statistically\nsigni\ufb01cant change to the predictions (as measured by a likelihood ratio test). The statistical test\nrequires a signi\ufb01cance level, \u03b1, which controls the probability of detecting a spurious difference.\nApplying the test several times to the same data set compounds the danger of such an error, so \u03b1\nshould be set quite low. In the experiments below, \u03b1 = 10\u221210.\n\n4\n\n\f(a) Shooting Gallery\n\n(b) Three Card Monte\n\n(c) Snake\n\nFigure 2: Experiment Domains\n\n5 Experiments\n\nIn this section, timeline trees will be evaluated in three problems to which prediction pro\ufb01le models\nhave been previously applied. In each problem a set of features and a set of training trajectories are\nprovided. For various amounts of training trajectories, timeline trees are learned and their prediction\naccuracy is evaluated (as well as their usefulness for control). Results are averaged over 20 trials.\nNote that for prediction pro\ufb01le models a completely new model is learned for each batch of training\ntrajectories. For timeline trees, the new data is simply added to the existing tree and new splits are\nmade until the algorithm stops. This strategy is effective for timeline trees since the initial splits can\noften be made with relatively little data (this not possible for prediction pro\ufb01le models).\n\nIn addition to evaluating timeline trees, two variants will also be evaluated. One (labeled \u201cFinite\nSuf\ufb01x\u201d) does not use any timestamp features at all. Thus, it is similar to U-Tree (splitting on features\nof a \ufb01nite suf\ufb01x of history). The other (labeled \u201cNo Timestamps\u201d) includes timestamp features,\nbut does not use them to create new timestamps. This variant is meant to evaluate whether any\nperformance bene\ufb01t is due to the form of the features or due to the addition of new timestamps.\n\n5.1 Shooting Gallery\n\nIn this example, from Talvitie and Singh [4], the agent is in a shooting gallery (see Figure 2(a)). Its\ngun is aimed at a \ufb01xed position (marked by the \u201cX\u201d) and it must shoot a target that moves around\nthe grid, bouncing off the edges and obstacles (an example trajectory is pictured). If the target is in\nthe crosshairs in the step after the agent shoots, the agent gets a reward of 10. Otherwise it gets a\nreward of -5. Whenever the agent hits the target, the gallery resets (obstacles are placed randomly)\nand an special observation is emitted. The gallery may also reset on its own with a 0.01 chance.\n\nClearly the agent must predict whether the target will be in the crosshairs in the next timestep, but the\ntarget\u2019s movement is stochastic and partially observable. At every step it either moves in its current\ndirection with probability 0.7 or stays in place with probability 0.3. The agent must remember the\ndirection of the ball the last time it moved. This problem is also fairly high-dimensional. There are\nroughly 4,000,000 possible observations, and even more hidden states. Because of the large number\nof observations Talvitie and Singh [4] hand-crafted an observation abstraction and applied it to the\ntraining data before learning the prediction pro\ufb01le models. Their abstraction pays attention only to\nthe position of the target and the con\ufb01guration of the obstacles in its immediate neighborhood. By\nconstrast, timeline trees learn an abstraction over both observations and the history sequence.\n\nExperimental Setup: The prediction pro\ufb01le models were trained on trajectories of length 4, gener-\nated by the uniform random policy. Though short trajectories are necessary for training prediction\npro\ufb01le models, the timeline trees tended to over\ufb01t to the short trajectories. In short trajectories, a\nfeature like \u201cHas the target ever been in the crosshairs?\u201d might seem spuriously meaningful. Dur-\ning testing, which takes place on one long trajectory, this feature would be much less informative.\nTherefore, the tree models were trained on fewer, longer trajectories (of length 40).\n\nTo train the tree models, a binary feature was provided for each color (target, obstacle, background,\nor reset) for each pixel in the image. There was also a feature for each action. The maximum\ntemporal offset from a timestamp was set to 2.\n\nThe learned models are evaluated by using their predictions as features for a policy gradient algo-\nrithm, OLGARB [18]. Good predictions about the color under the cross-hairs should lead to a good\npolicy. For the details of how the predictions are encoded for OLGARB, see [4]. To evaluate the\nlearned models, OLGARB is run for 1,000,000 steps. The average reward obtained and the root\n\n5\n\n\fmean squared error (RMSE) of the probabilities provided by the model are reported (at each step,\nthe model\u2019s probability that the target will be in the crosshairs is compared to the true probability).\n\nResults: Figure 3 shows the results. The line marked \u201cPre-\ndiction Pro\ufb01le\u201d shows the best results reported by Talvitie and\nSingh [4]; the other curves show the performance of timeline\ntrees and the comparison variants. In the control performance\ngraph, the dashed line marked \u201cOptimal\u201d shows the average\nperformance of the optimal policy. The dashed line marked\n\u201cTrue\u201d shows the average performance of OLGARB when\ngiven the true predictions as features. This is the best perfor-\nmance a learned model could hope for.\n\nBoth the timeline trees and the prediction pro\ufb01le models are\nable to learn to make good predictions, but timeline trees do so\nwith less data. Remember that timeline trees are learning from\nraw images whereas the prediction pro\ufb01le models have been\nprovided a hand-crafted abstraction. The tree models without\ntimestamps are only able to make good predictions in histories\nwhere the target has recently moved, which limits their perfor-\nmance. The \u201cNo Timestamp\u201d variant is outperformed by the\n\u201cFinite Suf\ufb01x\u201d model, which indicates that, despite the longer\ntraining trajectories, it may still be over\ufb01tting.\n\n5.2 Three Card Monte\n\nControl Performance\n\nOptimal\nTrue\n\nTimelineTree\nPrediction Profile\nFinite Suffix\nNo Timestamps\n\n 0.025\n\n 0.02\n\n 0.015\n\n 0.01\n\n 0.005\n\n 0\n\nl\n\n)\ns\na\ni\nr\nT\n \n0\n2\n(\n \nd\nr\na\nw\ne\nR\n\n \n.\ng\nv\nA\n\n 0\n\n 1e+06\n\n 2e+06\n\n 3e+06\n\n 4e+06\n\n# Training Steps\n\nPrediction Performance\n\nTimelineTree\nPrediction Profile\nFinite Suffix\nNo Timestamps\n\nl\n\n)\ns\na\ni\nr\nT\n \n0\n2\n(\n \n\nE\nS\nM\nR\n\n \n.\n\ng\nv\nA\n\n 0.09\n\n 0.08\n\n 0.07\n\n 0.06\n\n 0.05\n\n 0.04\n\n 0.03\n\n 0.02\n\n 0.01\n\n 0\n\n 0\n\n 1e+06\n\n 2e+06\n\n 3e+06\n\n 4e+06\n\n# Training Steps\n\nFigure 3: Shooting gallery results.\n\nThe next example, also from Talvitie and Singh [4], is one for which the decision tree approach\nwould not be appropriate. While illustrating the limitations of timeline trees in comparison to more\nexpressive methods, it also demonstrates that they can represent useful knowledge that the simpler\ntree-based methods cannot. The problem is based on the simple game \u201cThree Card Monte\u201d. There\nare three face down cards on the table, one of which is the ace. A dealer repeatedly chooses two\ncards and swaps their positions. Eventually the dealer asks the agent to \ufb02ip over the ace. If the agent\nsucceeds, it gets a reward of 1; if it fails it gets a reward of -1. For a detailed speci\ufb01cation, see [4].\n\nNote that to do well in this game, the agent only needs to make the prediction, \u201cIf I \ufb02ip card 1, will\nit be the ace?\u201d (and the corresponding predictions for the other 2 cards) at any history. It does not,\nfor instance, need to predict which cards will be swapped in the next time step. A complete model\nwould attempt to make this prediction, which would mean not only modeling the movement of the\ncards, but also the decision making process of the dealer! The dealer in these experiments choses\nthe swap it has chosen least frequently so far with probability 0.5. With probability 0.4 it choses\nuniformly randomly between the other two swaps. With probability 0.1, it asks for a guess. Since\nmodeling the dealer\u2019s behavior requires counting the number of times each swap has been selected,\na complete POMDP model of this system would require in\ufb01nitely many states.\n\nFurther note that the entire sequence of swap observations since the last time the ace\u2019s position was\nobserved is important for predicting the ace\u2019s location. Since timeline trees\u2019 primary strength is\nignoring sections of history to focus on a few key events, they would not be expected to model this\nproblem well. Prediction pro\ufb01le models, on the other hand, are able to track the ace\u2019s location with\na 3-state machine (pictured in Figure 2(b)).\n\nExperimental Setup: Training and evaluation were the same as in the Shooting Gallery (above)\nexcept the prediction pro\ufb01le models were given length 10 trajectories and the tree-based models\nwere given length 100 trajectories. The features provided to the trees were encodings of the atomic\nactions and observations. There was a binary feature indicating each observation, action, and each\naction-observation pair. The maximum time offset from a timestamp was 10 steps (both positive\nand negative). The speci\ufb01cation of the prediction pro\ufb01le models implicitly encodes the fact that the\nagent\u2019s action at+1 is important to the predictions (i.e. which card it \ufb02ips). For fairness, the tree\nmodels were also seeded with these features (they were split on the agent\u2019s action before training).\n\nResults: Figure 4(a) presents the control performance results and Figure 4(b) shows the prediction\nerror results. The prediction pro\ufb01le models are able to perfectly track the ace\u2019s location after 100,000\n\n6\n\n\fControl Performance\n\nOptimal\n\nTrue\n\nPrediction Profile\nTimelineTree\nNo Timestamps\nFinite Suffix\n\n 1\n\n 0.8\n\n 0.6\n\n 0.4\n\n 0.2\n\n 0\n\n-0.2\n\n-0.4\n\nl\n\n)\ns\na\ni\nr\nT\n0\n2\n(\n \n\n \n\nd\nr\na\nw\ne\nR\n\n \n.\n\ng\nv\nA\n\nPrediction Performance\n\nControl (Expert Training)\n\nPrediction Profile\nTimelineTree\nNo Timestamps\nFinite Suffix\n\nl\n\n)\ns\na\ni\nr\nT\n0\n2\n(\n \n\n \n\nd\nr\na\nw\ne\nR\n\n \n.\n\ng\nv\nA\n\nTimelineTree (Expert)\nNo Timestamps (Expert)\nFinite Suffix (Expert)\nTimelineTree\nNo Timestamps\nFinite Suffix\n\n 0.15\n\n 0.1\n\n 0.05\n\n 0\n\n-0.05\n\n-0.1\n\n 0.7\n\n 0.6\n\n 0.5\n\n 0.4\n\n 0.3\n\n 0.2\n\n 0.1\n\n 0\n\n-0.1\n\n-0.2\n\nl\n\n)\ns\na\ni\nr\nT\n0\n2\n(\n \n\n \n\nE\nS\nM\nR\n\n \n.\n\ng\nv\nA\n\n 0\n\n 250000\n 500000\n# Training Steps\n\n 750000\n\n 0\n\n(a)\n\n 250000\n 500000\n# Training Steps\n\n(b)\n\n 750000\n\n 0\n\n 2.5e+07  5e+07  7.5e+07  1e+08\n\n# Training Steps\n\n(c)\n\nFigure 4: Results in Three Card Monte.\n\ntraining steps. As expected, the tree methods perform poorly (negative average reward indicates\nmore wrong guesses than right), though timeline trees have marginally better control performance.\n\nPart of the dif\ufb01culty is that randomly generated training data is quite different than what the agent\nwill encounter during testing (the random agent \ufb02ips cards over frequently, while the learning agent\neventually \ufb02ips a card only when prompted to). Figure 4(c) shows the control performance of the\ntree models trained with expert-generated data instead (generated by the optimal policy). The dashed\nlines show the results for random training data for comparison. Expert-trained timeline trees are\neventually good enough to allow the agent to achieve positive average reward, though they do require\na great deal of data to do so (note the changes to the axes). Though the expert training improves the\nperformance of the limited variants as well, neither achieves positive average reward. So, though\ntheir representational limitations do prevent all three tree-based methods from performing well in\nthis problem, timeline trees\u2019 ability to create new timestamps seems to allow them to make some\nmeaningful (and useful) predictions that the others cannot.\n\n5.3 Snake\n\nThe \ufb01nal example is an arcade game called \u201cSnake\u201d from Talvitie [15] (see Figure 2(c)). In this\nproblem, multiple partial models will be learned and combined to form a complete, structured model\nwhich can be used for planning. The agent controls the head of a snake. The snake\u2019s body trails\nbehind, and the tail does exactly what the head did, at a delay. There are 10 food pellets on the screen\nand the goal is to eat them in a particular order (indicated by the shades of grey in Figure 2(c)). If\nthe snake ever runs into the wrong pellet, its own body, or the edge of the screen, the game is over\nand the agent receives -0.01 reward. Whenever the snake eats a pellet, the agent gets 1 reward and\nthe tail stays still for 5 timesteps, making the snake\u2019s body longer. In addition, there is a 0.2 chance\neach step that the tail will not move, so the snake is always growing, imposing some time-pressure.\n\nThis version of Snake has two sources of partial observability. The tail shadows the head and, in\naddition, the pellet the snake must eat next is invisible. Initially, all 10 pellets are shown, but when\nthe \ufb01rst pellet is eaten, the next one disappears, and only reappears if the snake\u2019s head is adjacent\nto it. When that pellet is eaten, the next pellet disappears, and so on. To do well, the agent must\nremember the location of the next pellet. The observations are 20 \u00d7 20 images. There are over 1030\ndistinct possible observations and even more hidden states.\n\nExperimental Setup: For every location (x, y) and every color c, a timeline tree model was used\nto predict whether the pixel at (x, y) would next be color c. These models jointly predict the entire\nnext observation. The models were ordered, and each model could condition on outcomes predicted\nby the previous models (so \u03ba gave the portion of the image predicted by models earlier in the order).\nIn this case, the models were ordered by color, then by location (column-major order). The color\norder was head, tail, body, the 10 food pellet colors (in increasing order), and \ufb01nally background.\n\nFor each color, training data was pooled data across all locations (rather than learn a separate model\nfor each coordinate). However, features were provided that gave the position of the model (there\nwas a binary feature for each column and each row), so the tree could choose to attend to position\ninformation if necessary. There was also a binary feature for each action and several pixel-based\nfeatures. There was a feature for each color of each pixel in a 5 \u00d7 5 square around the pixel being\npredicted. There were also features for the same square of pixels indicating whether each pixel had\njust changed from one color to another color (for all pairs of colors). Finally, there were features\nindicating whether a particular color or particular color change existed anywhere in the image.\n\n7\n\n\fPrediction pro\ufb01le models were applied to this problem by Talvitie [15]. Similarly, multiple predic-\ntion pro\ufb01le models were learned, each responsible for a particular prediction in particular situations\n(called \u201chistories of interest\u201d). For instance, one model type predicted whether a particular pixel\nwould contain the head in the next timestep, but only when the head was in the immediate neighbor-\nhood of that pixel. Before training the prediction pro\ufb01le models, an abstraction learning process was\napplied to the data that mapped each action-observation sequence between histories of interest to a\nsingle abstract observation. Thus, even though the raw data might contain very long trajectories, the\nabstract data consisted of only short trajectories. The reader is referred to Talvitie [15] for a detailed\ndescription of this approach. The main thing to note is that the hand-crafted structure indicates to\neach model which key events it should attend to, and which stretches of history can be ignored. By\ncontrast, timeline trees learn this information.\n\nAs in [15], the training data was generated by running UCT [19] (a sample-based planning algo-\nrithm) on the true model (with a 0.25 chance of taking a random action). Each training trajectory\nis a full game (typically a few hundred steps long). The learned set of partial models was evaluated\ncollectively. The joint model was given to UCT and its average planning performance over 100 test\ngames was compared to that of UCT with a perfect model. The probability given by the model for\nthe observation at each step was compared to the true probability and the RMSE has been reported.\n\nResults: The results are shown in Figure 5. Despite the\nhand-crafted structure provided to the prediction pro\ufb01le mod-\nels, timeline trees learn higher quality models with less data.\nIn fact, UCT appears to perform better using the timeline tree\nmodel than the true model (marked \u201cTrue\u201d). This is due to\na coincidental interaction between the model and UCT. The\nlearned model mistakenly predicts that the snake may not die\nwhen it moves off the edge of the screen (a rare event in the\ntraining data). This emboldens UCT to consider staying near\nthe edges, which can be necessary to escape a tight spot.\n\nIn terms of control performance, the \u201cNo Timestamps\u201d vari-\nant performs nearly identically to the full timeline tree. This\nis because they are equally good at tracking the invisible food\npellet. The model checks if each pixel has ever contained a\nfood pellet and if it has ever contained the head. If \u201cyes\u201d and\n\u201cno,\u201d respectively then it must contain food. This can be ex-\npressed without creating new timestamps. However, the \u201cNo\nTimestamps\u201d model cannot fully represent a model of the tail\u2019s\nmovement (which requires remembering what the head did\nwhen it was in the tail\u2019s position). The timeline tree incurs\nsubstantially less prediction error, indicating that it is able to\nmodel the tail more accurately.\n\n6 Conclusions\n\nl\n\n)\ns\na\ni\nr\nT\n0\n2\n(\n \n\n \n\ne\nr\no\nc\nS\n\n \n.\n\ng\nv\nA\n\nl\n\n)\ns\na\ni\nr\nT\n0\n2\n(\n \n\n \n\nE\nS\nM\nR\n\n \n.\n\ng\nv\nA\n\nControl Performance\n\nTimelineTree\nNo Timestamps\nPrediction Profile\nFinite Suffix\n\nTrue\n\n 0\n\n 4000\n\n 2000\n# Training Trajectories\n\n 6000\n\n 8000  10000\n\nPrediction Performance\n\nTimelineTree\nNo Timestamps\nPrediction Profile\nFinite Suffix\n\n 12\n\n 10\n\n 8\n\n 6\n\n 4\n\n 2\n\n 0\n\n 0.5\n\n 0.4\n\n 0.3\n\n 0.2\n\n 0.1\n\n 0\n\n 0\n\n 4000\n\n 2000\n# Training Trajectories\n\n 6000\n\n 8000  10000\n\nFigure 5: Results in Snake.\n\nIn these experiments, timeline trees learned to capture long-range dependencies in complex, partially\nobservable systems with high-dimensional observations. The assumption that the predictions of\ninterest depend on only a few key events in the past is limiting in the sense that there are simple\npartial models that timeline trees cannot easily capture (e.g. Three Card Monte), but it does re\ufb02ect\na broad, natural class of partially observable phenomena (the examples here, for instance, were not\ndesigned with timeline trees in mind). In problems that do match timeline trees\u2019 inductive biases,\nthey have been shown to outperform the more expressive prediction pro\ufb01le models.\n\nThere are many possible directions in which to consider extending timeline trees. More sophisticated\ndecision tree induction methods could help with sample complexity and over\ufb01tting. Regression tree\nmethods could extend timeline trees into environments with continuous dimensions. The timestamp\nfeatures used here are only one of many possible types of temporally abstract features that could be\ndevised. Of particular interest is whether the ideas here can be combined with approaches described\nin Section 2.1 in order to increase expressive power, while retaining the bene\ufb01ts of timeline trees.\n\n8\n\n\fReferences\n\n[1] Craig Boutilier, Thomas Dean, and Steve Hanks. Decision-theoretic planning: Structural assumptions\n\nand computational leverage. Journal of Arti\ufb01cial Intelligence Research, 11:1\u201394, 1999.\n\n[2] Erik Talvitie and Satinder Singh. Simple local models for complex dynamical systems. In Advances in\n\nNeural Information Processing Systems 21 (NIPS), pages 1617\u20131624, 2009.\n\n[3] Andrew K. McCallum. Reinforcement Learning with Selective Perception and Hidden State. PhD thesis,\n\nRutgers University, 1995.\n\n[4] Erik Talvitie and Satinder Singh. Learning to make predictions in partially observable environments\n\nwithout a generative model. Journal of Arti\ufb01cial Intelligence Research (JAIR), 42:353\u2013392, 2011.\n\n[5] Michael Littman, Richard Sutton, and Satinder Singh. Predictive representations of state. In Advances in\n\nNeural Information Processing Systems 14 (NIPS), pages 1555\u20131561, 2002.\n\n[6] George E. Monahan. A survey of partially observable markov decisions processes: Theory, models, and\n\nalgorithms. Management Science, 28(1):1\u201316, 1982.\n\n[7] Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially ob-\nservable stochastic domains. In Proceedings of the Twelfth National Conference on Arti\ufb01cial Intelligence\n(AAAI), volume 2, pages 1023\u20131028, 1994.\n\n[8] Satinder Singh, Michael R. James, and Matthew R. Rudary. Predictive state representations: A new theory\nfor modeling dynamical systems. In Uncertainty in Arti\ufb01cial Intelligence: Proceedings of the Twentieth\nConference (UAI), pages 512\u2013519, 2004.\n\n[9] Alicia Peregrin Wolfe and Andrew G. Barto. Decision tree methods for \ufb01nding reusable MDP homo-\nmorphisms. In Proceedings of the Twenty-First National Conference on Arti\ufb01cial Intelligence (AAAI),\n2006.\n\n[10] Michael Holmes and Charles Isbell. Looping suf\ufb01x tree-based inference of partially observable hidden\nstate. In Proceedings of the Twenty-Third International Conference on Machine Learning (ICML), pages\n409\u2013416, 2006.\n\n[11] Dana Ron, Yoram Singer, and Naftali Tishby. The power of amnesia. In Advances in Neural Information\n\nProcessing Systems 6, pages 176\u2013183, 1994.\n\n[12] Monica Dinculescu and Doina Precup. Approximate predictive representations of partially observable\nsystems. In Proceedings of the Twenty-Seventh International Conference on Machine Learning (ICML),\npages 895\u2013902, 2010.\n\n[13] R. Andrew McCallum. Overcoming incomplete perception with utile distinction memory. In Proceedings\n\nof the Tenth International Conference on Machine Learning (ICML), pages 190\u2013196, 1993.\n\n[14] M. M. Hassan Mahmud. Constructing states for reinforcement learning. In Proceedings of the Twenty-\n\nSeventh International Conference on Machine Learning (ICML), pages 727\u2013734, 2010.\n\n[15] Erik Talvitie. Simple Partial Models for Complex Dynamical Systems. PhD thesis, University of Michigan,\n\nAnn Arbor, MI, 2010.\n\n[16] J. Ross Quinlan. Induction of decision trees. Machine Learning, 1:81\u2013106, 1986.\n\n[17] J. Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufman Publishers Inc., San Francisco,\n\nCA, 1993.\n\n[18] Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. In\nUncertainty in Arti\ufb01cial Intelligence: Proceedings of the Seventeenth Conference (UAI), pages 538\u2013545,\n2001.\n\n[19] Levente Kocsis and Csaba Szepesv\u00b4ari. Bandit based monte-carlo planning. In Proceedings of the Seven-\n\nteenth European Conference on Machine Learning (ECML), pages 282\u2013293, 2006.\n\n9\n\n\f", "award": [], "sourceid": 386, "authors": [{"given_name": "Erik", "family_name": "Talvitie", "institution": null}]}