{"title": "Hierarchical Decision Making by Generating and Following Natural Language Instructions", "book": "Advances in Neural Information Processing Systems", "page_first": 10025, "page_last": 10034, "abstract": "We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making.\nRather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Experiments show that models using natural language as a latent variable significantly outperform models that directly imitate human actions. The compositional structure of language proves crucial to its effectiveness for action representation. We also release our code, models and data.", "full_text": "Hierarchical Decision Making by Generating and\n\nFollowing Natural Language Instructions\n\nHengyuan Hu\u2217\n\nFacebook AI Research\n\nhengyuan@fb.com\n\nDenis Yarats\u2217\n\nNew York University & Facebook AI Research\n\ndenisyarats@cs.nyu.edu\n\nQucheng Gong\n\nFacebook AI Research\n\nqucheng@fb.com\n\nYuandong Tian\n\nFacebook AI Research\n\nyuandong@fb.com\n\nMike Lewis\n\nFacebook AI Research\nmikelewis@fb.com\n\nAbstract\n\nWe explore using natural language instructions as an expressive and compositional\nrepresentation of complex actions for hierarchical decision making. Rather than\ndirectly selecting micro-actions, our agent \ufb01rst generates a plan in natural language,\nwhich is then executed by a separate model. We introduce a challenging real-time\nstrategy game environment in which the actions of a large number of units must\nbe coordinated across long time scales. We gather a dataset of 76 thousand pairs\nof instructions and executions from human play, and train instructor and executor\nmodels. Experiments show that models generate intermediate plans in natural\nlangauge signi\ufb01cantly outperform models that directly imitate human actions. The\ncompositional structure of language is conducive to learning generalizable action\nrepresentations. We also release our code, models and data23.\n\n1\n\nIntroduction\n\nMany complex problems can be naturally decomposed into steps of high level planning and low level\ncontrol. However, plan representation is challenging\u2014manually specifying macro-actions requires\nsigni\ufb01cant domain expertise, limiting generality and scalability [18, 22], but learning composite\nactions from only end-task supervision can result in the hierarchy collapsing to a single action [3].\nWe explore representing complex actions as natural language instructions. Language can express\narbitrary goals, and has compositional structure that allows generalization across commands [14, 1].\nOur agent has a two-level hierarchy, where a high-level instructor model communicates a sub-goal in\nnatural language to a low-level executor model, which then interacts with the environment (Fig. 1).\nBoth models are trained to imitate humans playing the roles. This approach decomposes decision\nmaking into planning and execution modules, with a natural language interface between them.\nWe gather example instructions and executions from two humans collaborating in a complex game.\nBoth players have access to the same partial information about the game state. One player acts as the\ninstructor, and periodically issues instructions to the other player (the executor), but has no direct\ncontrol on the environment. The executor acts to complete the instruction. This setup forces the\ninstructor to focus on high-level planning, while the executor concentrates on low-level control.\n\n\u2217Equal Contribution.\n2A demo is available at www.minirts.net\n3Our code is open-sourced at www.github.com/facebookresearch/minirts\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Two agents, designated instructor and executor collaboratively play a real-time strategy\ngame (\u00a72). The instructor iteratively formulates plans and issues instructions in natural language to\nthe executor, who then executes them as a sequence of actions. We \ufb01rst gather a dataset of humans\nplaying each role (\u00a73). We then train models to imitate humans actions in each role (\u00a74).\n\nTo test our approach, we introduce a real-time strategy (RTS) game, developing an environment\nbased on [23]. A key property of our game is the rock-paper-scissors unit attack dynamic, which\nemphasises strategic planning over micro control. Our game environment is a challenging decision\nmaking task, because of exponentially large state-action spaces, partial observability, and the variety\nof effective strategies. However, it is relatively intuitive for humans, easing data collection.\nUsing this framework, we gather a dataset of 5392 games, where two humans (the instructor and\nexecutor) control an agent against rule-based opponents. The dataset contains 76 thousand pairs of\nhuman instructions and executions, spanning a wide range of strategies. This dataset poses challenges\nfor both instruction generation and execution, as instructions may apply to different subsets of units,\nand multiple instructions may apply at a given time. We design models for both problems, and\nextensive experiments show that planning in language signi\ufb01cantly improves performance.\nIn summary, we introduce a challenging RTS environment for sequential decision making, and a\ncorresponding dataset of instruction-execution mappings. We develop novel model architectures\nwith planning and control components, connected with a natural language interface. Agents that\ngenerate explicit natural language plans outperforms agents that directly imitate human actions, and\nwe show that exploiting the compositional structure of language improves performance by allowing\ngeneralization across a large instruction set. We also release our code, models and data.\n\n2 Task Environment\n\nWe implement our approach for an RTS game, which has several attractive properties compared to\ntraditional reinforcement learning environments, such as Atari [13] or grid worlds [19]. The large\nstate and action spaces mean that planning at different levels of abstraction is bene\ufb01cial for both\nhumans and machines. However, manually designed macro-actions typically do not match strong\nhuman performance, because of the unbounded space of possible strategies [21, 25]. Even with\nsimple rules, adversarial games have the scope for complex emergent behaviour.\nWe introduce a new RTS game environment, which distills the key features of more complex games\nwhile being faster to simulate and more tractable to learn. Current RTS environments, such as\nStarCraft, have dozens of unit types, adding large overheads for new players to learn the game. Our\nnew environment is based on MiniRTS [23]. It has a set of 7 unit types, designed with a rock-paper-\nscissors dynamic such that each has some units it is effective against and vulnerable to. Maps are\nrandomly generated each game to force models to adapt to their environment as well as their opponent.\nThe game is designed to be intuitive for new players (for example, catapults have long range and\nare effective against buildings). Numerous strategies are viable, and the game presents players with\ndilemmas such as whether to attack early or focus on resource gathering, or whether to commit to a\n\n2\n\nScreen FeaturesResult of Actions InstructorNon-spatial FeaturesInstructionObservationActionsour_army_attr\nenemy_army_attr\nresourses_attrselect_unit(peasant)\nbuild_building(tower)game_stateExecutorbuild a tower next to the base\fstrategy or to attempt to scout for the opponent\u2019s strategy \ufb01rst. Overall, the game is easy for humans\nto learn, but challenging for machines due to the large action space, imperfect information, and need\nto adapt strategies to both the map and opponent. See the Appendix for more details.\n\n3 Dataset\n\nTo learn to describe actions with natural language, we gather a dataset of two humans playing\ncollaboratively against a rule-based opponent. Both players have access to the same information\nabout the game state, but have different roles. One is designated the instructor, and is responsible for\ndesigning strategies and describing them in natural language, but has no direct control. The other\nplayer, the executor, must ground the instructions into low level control. The executor\u2019s goal is to\ncarry out commands, not to try to win the game. This setup causes humans to focus on either planning\nor control, and provides supervision for both generating and executing instructions.\nWe collect 5392 games of human teams against\nour bots.4 Qualitatively, we observe a wide vari-\nety of different strategies. An average game con-\ntains 14 natural language instructions and lasts\nfor 16 minutes. Each instruction corresponds to\nroughly 7 low-level actions, giving a challeng-\ning grounding problem (Table 1). The dataset\ncontains over 76 thousand instructions, most of\nwhich are unique, and their executions. The diver-\nsity of instructions shows the wide range of useful\nstrategies. The instructions contain a number of\nchallenging linguistic phenomena, particularly in\nterms of reference to locations and units in the\ngame, which often requires pragmatic inference.\nInstruction execution is typically highly depen-\ndent on context. Our dataset is available. For\nmore details, refer to the Appendix.\nAnalysing the list of instructions (see Appendix),\nwe see that the head of the distribution is dominated by straightforward commands to perform the\nmost frequent actions. However, samples from the complete instruction list reveal many complex\ncompositional instructions, such as Send one catapult to attack the northern guard tower [and] send\na dragon for protection. We see examples of challenging quanti\ufb01ers (Send all but 1 peasant to mine),\nanaphora (Make 2 more cavalry and send them over with the other ones), spatial references (Build a\nnew town hall between the two west minerals patches) and conditionals (If attacked retreat south).\n\nTable 1: We gather a large language dataset for\ninstruction generation and following. Major chal-\nlenges include the wide range of unique instruc-\ntions and the large number of low-level actions\nrequired to execute each instruction.\n\nStatistic\nTotal games\nWin rate\nTotal instructions\nUnique instructions\nTotal words\nUnique words\n# words per instruction\n# instructions per game\n\nValue\n5392\n58.6%\n76045\n50669\n483650\n5007\n9.54\n14.1\n\n4 Model\n\nWe factorize agent into an executor model (\u00a74.2), which maps instructions and the game states into\nunit-level actions of the environment, and an instructor model (\u00a74.3), which generates language\ninstructions given the game states. We train both models with human supervision (\u00a74.4).\n\n4.1 Game Observation Encoder\n\nWe condition both the instructor and executor models on a \ufb01xed-sized representation of the current\ngame state, which we construct from a spatial map observation, internal states of visible units, and\nseveral previous natural language instructions. (Fig. 2). We detail each individual encoder below.\n\n4.1.1 Spatial Inputs Encoder\n\nWe encode the spatial information of the game map using a convolutional network. We discretize\nthe map into a 32 \u00d7 32 grid and extract different bits of information from it using separate channels.\nFor example, three of those channels provide binary indication of a particular cell visibility, which\n\n4Using ParlAI [12]\n\n3\n\n\fFigure 2: At each time step of the environment we encode spatial observations (e.g. the game\nmap) and non-spatial internal states for each game object (e.g. units, buildings, or resources) via the\nobservation encoder, which produces separate feature vectors for each unit, resource, or discrete map\nlocations. We also embed each of the last K natural language instructions into individual instruction\nfeature vectors. Lastly, we learn features for all the other global game attributes by employing the\nauxiliary encoder. We then use these features for both the executor and instructor networks.\n\nindicates INVISIBLE, SEEN, and VISIBLE states. We also have a separate channel per unit type to\nrecord the number of units in each spatial position for both our and enemy units separately. Note that\ndue to \"fog-of-war\", not all enemy units are visible to the player. See the Appendix for more details.\nWe apply several 3 \u00d7 3 convolutional layers that preserve the spatial dimensions to the input tensor.\nThen we use 4 sets of different weights to project the shared convolutional features onto different 2D\nfeatures spaces, namely OUR UNITS, ENEMY UNITS, RESOURCES, and MAP CELLS. We then use\n(x, y) locations for units, resources, or map cells to extract their features vectors from corresponding\n2D features spaces.\n\n4.1.2 Non-spatial Inputs Encoder\n\nWe also take advantage of non-spatial attributes and internal state for game objects. Speci\ufb01cally, we\nimprove features vectors for OUR UNITS and ENEMY UNITS by adding encodings of units health\npoints, previous and current actions. If an enemy unit goes out the players visibility, we respect this\nby using the state of the unit\u2019s attributes from the last moment we saw it. We project attribute features\nonto the same dimensionality of the spatial features and do a element-wise multiplication to get the\n\ufb01nal set of OUR UNITS and ENEMY UNITS features vectors.\n\n4.1.3 Instruction Encoders\n\nThe state also contains a \ufb01xed-size representation of the current instruction. We experiment with:\n\n\u2022 An instruction-independent model (EXECUTORONLY), that directly mimics human actions.\n\u2022 A non-compositional encoder (ONEHOT) which embeds each instruction with no parameter\nsharing across instructions (rare instructions are represented with an unknown embedding).\n\u2022 A bag-of-words encoder (BOW), where an instruction encoding is a sum of word embed-\n\u2022 An RNN encoder (RNN), which is order-aware. Unlike BOW, this approach can differentiate\ninstructions such as attack the dragon with the archer and attack the archer with the dragon.\n\ndings. This model tests if the compositionality of language improves generalization.\n\n4.1.4 Auxiliary Encoder\n\nFinally, we encode additional game context, such as the amount of money the player has, through a\nsimple MLP to get the EXTRA features vector.\n\n4.2 Executor Model\n\nThe executor predicts an action for every unit controlled by the agent based on the global summary\nof the current observation. We predict an action for each of the player\u2019s units by choosing over an\n\n4\n\n{}dim{dim{dim{dim# our units}resources\nfeaturesmap cells\nfeaturesenemy units\nfeaturesour units\nfeatures# resources}(x,y){dim}(x,y){diminstructions\nfeatures}last K instructions{dim}}# enemy unitsobservation\nencoderinstruction\nencoderspatial inputsnon-spatial inputsunit_1{type, hp}\nunit_2{type, hp}\n...\nenemy_1{type, hp}\n...\nresource_1:{cap}last K instructionsinstruction_1\ninstruction_2\n...\ninstruction_K{dimextra\nfeaturesauxililary\nencoderextra inputsamount of\nresources\fACTION TYPE \ufb01rst, and then selecting the ACTION OUTPUT. There are 7 action types available:\nIDLE, CONTINUE, GATHER, ATTACK, TRAIN UNIT, BUILD BUILDING, MOVE. ACTION OUTPUT\nspeci\ufb01es the target output for the action, such as a target location for the MOVE action, or the unit\ntype for TRAIN UNIT. Fig. 3 gives an overview of the executor design, also refer to the Appendix.\nFor each unit, we consider a history of recent\nN instructions (N = 5 in all our experiments),\nbecause some units may still be focusing on\na previous instruction that has long term ef-\nfect like keep scouting or build 3 peasants.\nTo encode the N instructions, we \ufb01rst embed\nthem in isolation with the 4.1.3. We take K\nthat represents how many frames have passed\nsince that instruction gets issued and compute\nH = max(Hmax, K/B) where Hmax, B are\nconstants de\ufb01ning the number of bins and bin\nsize. We also take O = 1, 2, ..., N that repre-\nsents the temporal ordering of those instructions.\nWe embed O and H and concatenate the embed-\ndings with language embedding. Dot product\nattention is used to compute an attention score\nbetween a unit and recent instructions and then\na unit dependent instruction representation is\nobtained through a weighted sum of history in-\nstruction embeddings using attention score as\nweight.\nWe use the same observation encoder (\u00a74.1) to\nobtain the features mentioned above. To form\na global summary, we sum our unit features,\nenemy unit features, and resource features re-\nspectively and then concatenate together with\nEXTRA features.\nTo decide the action for each unit, we \ufb01rst feed\nthe concatenation of the unit feature, unit de-\npending instruction feature and the global sum-\nmary into a multi-layer neural classi\ufb01er to sample an ACTION TYPE. Depending on the action type,\nwe then feed inputs into different action-speci\ufb01c classi\ufb01ers to sample ACTION OUTPUT. In the action\nargument classi\ufb01er, the unit is represented by the concatenation of unit feature and instruction feature,\nand the targets are represented by different target embeddings. For ATTACK, the target embeddings\nare enemy features; for GATHER; the target embeddings are resource features; for MOVE, the target\nembeddings are map features; for TRAIN UNIT, the target embeddings are embeddings of unit types;\nfor BUILD BUILDING, the target embeddings are embeddings of unit types and map features, and we\nsample type and location independently. The distribution over targets for the action is computed by\ntaking the dot product between the unit representation and each target, followed by a softmax.\nWe add an additional binary classi\ufb01er, GLOBAL CONTINUE, that takes the global summary and\ncurrent instruction embedding as an input to predict whether all the agent\u2019s units should continue\nworking on their previous action.\n\nFigure 3: Modeling an action for an unit requires\npredicting an action type based on the global sum-\nmary of current observation, and then, depending\non the predicted action type, computing a probabil-\nity distribution over a set of the action targets. In\nthis case, the MOVE action is sampled, which uses\nthe map cells features as the action targets.\n\n4.3\n\nInstructor Model\n\nThe instructor maps the game state to instructions. It uses the game observation encoder (\u00a74.1) to\ncompute a global summary and current instruction embedding similar to the executor. We experiment\nwith two model types:\n\nDiscriminative Models These models operate on a \ufb01xed set of instructions. Each instruction is\nencoded as a \ufb01xed-size vector, and the dot product of this encoding and the game state encoding is\nfed into a softmax classi\ufb01er over the set of instructions. As in \u00a74.1.3, we consider non-compositional\n(ONEHOT), bag-of-words (BOW) and RNN DISCRIMINATIVE encoders.\n\n5\n\n}resource\nfeaturesenemy units\nfeaturesenemy units\nsummaryour units\nsummary}}resources\nfeaturesresources\nsummary}our units\nfeaturesinstructions\nfeaturesunit dependent\ninstructions featuresunit\nfeatures......extra\nfeaturessoftmax over\nunit actionsglobal\nsummaryMOVE~~map cells\nfeaturesattention over\neach cell (x,y)action outputaction type(x,y)(x,y)\fGenerative Model The discriminative models can only choose between a \ufb01xed set of instructions.\nWe also train a generative model, RNN GENERATIVE, which generates instructions autoregressively.\nTo compare likelihoods with the discriminative models, which consider a \ufb01xed set of instructions, we\nre-normalize the probability of an instruction over the space of instructions in the set.\nThe instructor model must also decide at each time-step whether to issue a new command, or leave\nthe executor to follow previous instructions. We add a simple binary classi\ufb01er that conditions on the\nglobal feature, and only sample a new instruction if the result is positive.\nBecause the game is only partially observable, it is important to consider historical information when\ndeciding an instruction. For simplicity, we add a running average of the number of enemy units\nof each type that have appeared in the visible region as an extra input to the model. To make the\ninstructor model aware of how long the current instruction has been executed, we add an extra input\nrepresenting number of time-step passed since the issuance of current instruction. As mentioned\nabove, these extra inputs are fed into separate MLPs and become part of the EXTRA feature.\n\n4.4 Training\n\nSince one game may last for tens of thousands of frames, it is not feasible nor necessary to use all\nframes for training. Instead, we take one frame every K frames to form the supervised learning\ndataset. To preserve unit level actions for the executor training, we put all actions that happen in\n[tK, (t + 1)K) frames onto the tKth frame if possible. For actions that cannot happen on the tKth\nframe, such as actions for new units built after tKth frame, we simply discard them.\nHumans players sometimes did not execute instructions immediately. To ensure our executor acts\npromptly, we \ufb01lter out action-less frames between a new instruction and \ufb01rst new actions.\n\n4.4.1 Executor Model\n\nThe executor is trained to minimize the following negative log-likelihood loss:\n\n|u|(cid:88)\n\nL = \u2212 log Pcont(c|s) \u2212 (1 \u2212 c) \u00b7\n\nlog PA(aui|s)\n\nwhere s represents game state and instruction, Pcont(\u00b7|s) is the executor GLOBAL CONTINUE\nclassi\ufb01er (see \u00a74.2), c is a binary label that is 1 if all units should continue their previous action,\nPA(aui|s) is the likelihood of unit i doing the correct action aui.\n\ni=1\n\n4.4.2 Instructor Model\n\nThe loss for the instructor model is the sum of a loss for deciding whether to issue a new instruction,\nand the loss for issuing the correct instruction:\n\nL = \u2212 log Pcont(c|s) \u2212 (1 \u2212 c) \u00b7 Llang\n\nwhere s represents game state and current instruction, Pcont(\u00b7|s) is the continue classi\ufb01er, and c is a\nbinary label with c = 1 indicating that no new instruction is issued. The language loss Llang is the\nloss for choosing the correct instruction, and is de\ufb01ned separately for each model.\nFor ONEHOT instructor, Llang is simply negative log-likelihood of a categorical classi\ufb01er over a pool\nof N instructions. If the true target is not in the candidate pool Llang is 0.\nBecause BOW and RNN DISCRIMINATIVE can compositionally encode any instruction (in contrast\nto ONEHOT), we can additionally train on instructions from outside the candidate pool. To do this,\nwe encode the true instruction, and discriminate against the N instructions in the candidate pool\nand another M randomly sampled instructions. The true target is forced to appear in the M + N\ncandidates. We then use the NLL of the true target as language loss. This approach approximates the\nexpensive softmax over all 40K unique instructions.\nFor RNN GENERATIVE, the language loss is the standard autoregressive loss.\n\n6\n\n\fExecutor Model\nEXECUTORONLY\nONEHOT\nBOW\nRNN\n\nNegative Log Likelihood Win/Lose/Draw Rate (%)\n\n3.15 \u00b1 0.0024\n3.05 \u00b1 0.0015\n2.89 \u00b1 0.0028\n2.88 \u00b1 0.0006\n\n41.2/40.7/18.1\n49.6/37.9/12.5\n54.2/33.9/11.9\n57.9/30.5/11.7\n\nTable 2: Negative log-likelihoods of human actions for executor models, and win-rates against\nEXECUTORONLY (which does not use an instructor model to generate natural language plans). We use\nthe RNN DISCRIMINATIVE instructor with 500 instructions. Modelling instructions compositionally\nimproves performance, showing linguistic structure enables generalization.\n\nInstructor Model\n(with N instructions)\nONEHOT\nBOW\nRNN DISCRIMINATIVE\nRNN GENERATIVE\n\nNegative Log Likelihood\n\nN=50\n\n0.662 \u00b1 0.005\n0.638 \u00b1 0.004\n0.618 \u00b1 0.005\n0.638 \u00b1 0.006\n\nN=250\n\n0.831 \u00b1 0.001\n0.792 \u00b1 0.001\n0.764 \u00b1 0.002\n0.794 \u00b1 0.006\n\nN=500\n\n0.911 \u00b1 0.005\n0.869 \u00b1 0.002\n0.826 \u00b1 0.002\n0.857 \u00b1 0.002\n\nWin/Lose/Draw rate (%)\n\nN=50\n\n44.6 / 43.4 / 12.0\n41.3 / 41.2 / 17.5\n47.8 / 36.5 / 15.7\n47.3 / 38.1 / 14.6\n\nN=250\n\n49.7 / 35.9 / 14.3\n51.5 / 33.3 / 15.3\n55.4 / 33.1 / 11.5\n51.1 / 33.7 / 15.2\n\nN=500\n\n43.1 / 41.1 / 15.7\n50.5 / 37.1 / 12.5\n57.9 / 30.5 / 11.7\n54.8 / 33.8 / 11.4\n\nTable 3: Win-rates and likelihoods for different instructor models, with the N most frequent\ninstructions. Win-rates are against a non-hierarchical executor model, and use the RNN executor.\nBetter results are achieved with larger instruction sets and more compositional instruction encoders.\n\n5 Experiments\n\nWe compare different executor (\u00a75.1) and instructor (\u00a75.2) models in terms of both likelihoods and\nend-task performance. We show that hierarchical models perform better, and that the compositional\nstructure of language improves results by allowing parameter sharing across many instructions.\n\n5.1 Executor Model\n\nThe executor model learns to ground pairs of states and instructions onto actions. With over 76\nthousand examples, a large action space, and multiple sentences of context, this problem in isolation\nis one of the largest and most challenging tasks currently available for grounding language in action.\nWe evaluate executor performance with different instruction encoding models (\u00a74.1.3). Results\nare shown in Table 2, and show that modelling instructions compositionally\u2014by encoding words\n(BOW) and word order (RNN)\u2014improves both the likelihoods of human actions, and win-rates over\nnon-compositional instructor models (ONEHOT). The gain increases with larger instruction sets,\ndemonstrating that a wide range of instructions are helpful, and that exploiting the compositional\nstructure of language is crucial for generalization across large instruction sets.\nWe additionally ablate the importance of considering multiple recent instructions during execution\n(our model performs attention over the most recent 5 commands \u00a74.2). When considering only the\ncurrent instruction with the RNN executor, we \ufb01nd performance drops to a win-rate of 52.9 (from\n57.9) and negative log likelihood worsens from 2.88 to 2.93.\n\n5.2\n\nInstructor Model\n\nWe compare different instructor models for mapping game states to instructions. As in \u00a75.1, we\nexperiment with non-compositional, bag-of-words and RNN models for instruction generation. For\nthe RNNs, we train both a discriminative model (which maps complete instructions onto vectors, and\nthen chooses between them) and a generative model that outputs words auto-regressively.\nEvaluating language generation quality is challenging, as many instructions may be reasonable in a\ngiven situation, and they may have little word overlap. We therefore compare the likelihood of the\nhuman instructions. Our models choose from a \ufb01xed set of instructions, so we measure the likelihood\nof choosing the correct instruction, normalized over all instructions in the set. Likelihoods across\ndifferent instructions sets are not comparable.\nTable 3 shows that, as \u00a75.1, more structured instruction models give better likelihoods\u2014particularly\nfor larger instruction sets, which are harder to model non-compositionally.\n\n7\n\n\fWe compare the win-rate of our models against a baseline which directly imitates human actions\n(without explicit plans). All models that plan by generating and executing instructions outperform\nthis baseline. More compositional instruction encoders improve performance, and can use more\ninstructions effectively. These results demonstrate the potential of language for compositionally\nrepresenting large spaces of complex plans.\n\n5.3 Qualitative Analysis\n\nObserving games played by our model, we \ufb01nd that most instructions are both generated and executed\nas humans plausibly would. The executor is often able to correctly count the number of units it should\ncreate in commands such as build 3 dragons.\nThere are several limitations. The executor sometimes acts without instructions\u2014partly due to\nmimicking some humans behaviour, but also indicating a failure to learn dependencies between\ninstructions and actions. The instructor sometimes issues commands which are impossible in its state\n(e.g. to attack with a unit that the it does not have)\u2014causing weak behaviour from executor model.\n\n6 Related work\n\nPrevious work has used language to specify exponentially many policies [14, 1, 27], allowing\nzero-shot generalization across tasks. We develop this work by generating instructions as well as\nexecuting them. We also show how complex tasks can be decomposed into a series of instructions\nand executions.\nExecuting natural language instructions has seen much attention. The task of grounding language\ninto an executable representation is sometimes called semantic parsing [29], and has been applied to\nnavigational instruction following, e.g. [2]. More recently, neural models instruction following have\nbeen developed for a variety of domains, for example [11] and [10]. Our dataset offers a challenging\nnew problem for instruction following, as different instructions will apply to different subsets of\navailable units, and multiple instructions may be apply at a given time.\nInstruction generation has been studied as a separate task. [7] map navigational paths onto instructions.\n[8] generate instructions for complex tasks that humans can follow, and [9] train a model for instruction\ngeneration, which is used both for data augmentation and for pragmatic inference when following\nhuman-generated instructions. We build on this work by also generating instructions at test time, and\nshowing that planning in language can improve end task performance.\nLearning to play a complete real-time strategy game, including unit building, resources gathering,\ndefence, invasion, scouting, and expansion, remains a challenging problem [15], in particular due\nto the complexity and variations of commercially successful games (e.g., StarCraft I/II), and its\ndemand of computational resources. Traditional approaches focus on sub-tasks with hand-crafted\nfeatures and value functions (e.g., building orders [5], spatial placement of building [4], attack\ntactics between two groups of units [6], etc). Inspired by the recent success of deep reinforcement\nlearning, more works focus on training a neural network to \ufb01nish sub-tasks [24, 17], some with strong\ncomputational requirement [28]. For full games, [23] shows that it is possible to train an end-to-end\nagent on a small-scaled RTS game with prede\ufb01ned macro actions, and TStarBot [20] applies this\nidea to StarCraft II and shows that the resulting agent can beat carefully-designed, and even cheating\nrule-based AI. By using human demonstrations, we hand crafting macro-actions.\nLearning an end-to-end agent that plays RTS games with unit-level actions is even harder. Progress is\nreported for MOBA games, a sub-genre of RTS games with fewer units\u2014for example, [16] shows\nthat achieving professional level of playing DoTA2 is possible with massive computation, and [26]\nshows that with supervised pre-training on unit actions, and hierarchical macro strategies, a learned\nagent on Honor of Kings is on par with a top 1% human player.\n\n7 Conclusion\n\nWe introduced a framework for decomposing complex tasks into steps of planning and execution,\nconnected with a natural language interface. We experimented with this approach on a new strategy\ngame which is simple to learn but features challenging strategic decision making. We collected a\n\n8\n\n\flarge dataset of human instruction generations and executions, and trained models to imitate each role.\nResults show that leveraging the compositional structure of natural language bene\ufb01ts generalization\nfor both the instructor and executor model, outperforming agents that do not plan with language.\nFuture work should use reinforcement learning to further improve the planning and execution models,\nand explore generating novel instructions.\n\nReferences\n[1] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with\npolicy sketches. In Proceedings of the 34th International Conference on Machine Learning,\nICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 166\u2013175, 2017.\n\n[2] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping\ninstructions to actions. Transactions of the Association of Computational Linguistics, 1:49\u201362,\n2013.\n\n[3] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. CoRR,\n\nabs/1609.05140, 2016.\n\n[4] Michal Certicky.\n\nImplementing a wall-in building placement in starcraft with declarative\n\nprogramming. arXiv preprint arXiv:1306.4460, 2013.\n\n[5] David Churchill and Michael Buro. Build order optimization in starcraft. In AIIDE, pages\n\n14\u201319, 2011.\n\n[6] David Churchill, Abdallah Saf\ufb01dine, and Michael Buro. Fast heuristic search for rts game\n\ncombat scenarios. In AIIDE, pages 112\u2013117, 2012.\n\n[7] Andrea F Daniele, Mohit Bansal, and Matthew R Walter. Navigational instruction generation\nas inverse reinforcement learning with neural machine translation. In Proceedings of the 2017\nACM/IEEE International Conference on Human-Robot Interaction, pages 109\u2013118. ACM, 2017.\n\n[8] Daniel Fried, Jacob Andreas, and Dan Klein. Uni\ufb01ed pragmatic models for generating and\n\nfollowing instructions. arXiv preprint arXiv:1711.04987, 2017.\n\n[9] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe\nMorency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-\nfollower models for vision-and-language navigation. arXiv preprint arXiv:1806.02724, 2018.\n\n[10] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer,\nDavid Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al.\nGrounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017.\n\n[11] Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping\n\nof navigational instructions to action sequences. In AAAI, volume 1, page 2, 2016.\n\n[12] A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. Parlai: A\n\ndialog research software platform. arXiv preprint arXiv:1705.06476, 2017.\n\n[13] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.\n\nPlaying atari with deep reinforcement learning. CoRR, 2013.\n\n[14] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization\n\nwith multi-task deep reinforcement learning. ICML, 2017.\n\n[15] Santiago Ontan\u00f3n, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, and\nMike Preuss. A survey of real-time strategy game ai research and competition in starcraft. IEEE\nTransactions on Computational Intelligence and AI in games, 5(4):293\u2013311, 2013.\n\n[16] OpenAI. Openai \ufb01ve. https://blog.openai.com/openai-five/, 2018.\n\n[17] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang.\nMultiagent bidirectionally-coordinated nets for learning to play starcraft combat games. CoRR,\nabs/1703.10069, 2017.\n\n9\n\n\f[18] Sutton Richard, Precup Doina, and Singh Satinder. Between mdps and semi-mdps: A framework\n\nfor temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2), 1999.\n\n[19] Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, and Rob Fergus.\n\nMazebase: A sandbox for learning from games. CoRR, abs/1511.07401, 2015.\n\n[20] Peng Sun, Xinghai Sun, Lei Han, Jiechao Xiong, Qing Wang, Bo Li, Yang Zheng, Ji Liu,\nYongsheng Liu, Han Liu, et al. Tstarbots: Defeating the cheating level builtin ai in starcraft ii in\nthe full game. arXiv preprint arXiv:1809.07193, 2018.\n\n[21] Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timoth\u00e9e Lacroix, Zeming\nLin, Florian Richoux, and Nicolas Usunier. Torchcraft: a library for machine learning research\non real-time strategy games. CoRR, abs/1611.00625, 2016.\n\n[22] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep\n\nhierarchical approach to lifelong learning in minecraft. CoRR, abs/1604.07255, 2016.\n\n[23] Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C Lawrence Zitnick. Elf: An\nextensive, lightweight and \ufb02exible research platform for real-time strategy games. In Advances\nin Neural Information Processing Systems, pages 2659\u20132669, 2017.\n\n[24] Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration\nfor deep deterministic policies: An application to starcraft micromanagement tasks. ICLR,\n2017.\n\n[25] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets,\nMichelle Yeo, Alireza Makhzani, Heinrich K\u00fcttler, John Agapiou, Julian Schrittwieser, John\nQuan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David\nSilver, Timothy P. Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence,\nAnders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft II: A new challenge for reinforcement\nlearning. CoRR, abs/1708.04782, 2017.\n\n[26] Bin Wu, Qiang Fu, Jing Liang, Peng Qu, Xiaoqian Li, Liang Wang, Wei Liu, Wei Yang,\nand Yongsheng Liu. Hierarchical macro strategy model for moba game ai. arXiv preprint\narXiv:1812.07887, 2018.\n\n[27] Denis Yarats and Mike Lewis. Hierarchical text generation and planning for strategic dia-\nlogue. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018,\nStockholmsm\u00e4ssan, Stockholm, Sweden, July 10-15, 2018, pages 5587\u20135595, 2018.\n\n[28] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl\nTuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement\nlearning. arXiv preprint arXiv:1806.01830, 2018.\n\n[29] Luke Zettlemoyer and Michael Collins. Online learning of relaxed ccg grammars for parsing to\nlogical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural\nLanguage Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007.\n\n10\n\n\f", "award": [], "sourceid": 5293, "authors": [{"given_name": "Hengyuan", "family_name": "Hu", "institution": "Facebook"}, {"given_name": "Denis", "family_name": "Yarats", "institution": "New York University"}, {"given_name": "Qucheng", "family_name": "Gong", "institution": "Facebook AI Research"}, {"given_name": "Yuandong", "family_name": "Tian", "institution": "Facebook AI Research"}, {"given_name": "Mike", "family_name": "Lewis", "institution": "Facebook AI Research"}]}