{"title": "Help or Hinder: Bayesian Models of Social Goal Inference", "book": "Advances in Neural Information Processing Systems", "page_first": 1874, "page_last": 1882, "abstract": "Everyday social interactions are heavily influenced by our snap judgments about others goals.  Even young infants can infer the goals of intentional agents from observing how they interact with objects and other agents in their environment: e.g., that one agent is `helping or `hindering anothers attempt to get up a hill or open a box. We propose a model for how people can infer these social goals from actions, based on inverse planning in multiagent Markov decision problems (MDPs).  The model infers the goal most likely to be driving an agents behavior by assuming the agent acts approximately rationally given environmental constraints and its model of other agents present.  We also present behavioral evidence in support of this model over a simpler, perceptual cue-based alternative.", "full_text": "Help or Hinder: Bayesian Models of\n\nSocial Goal Inference\n\nTomer D. Ullman, Chris L. Baker, Owen Macindoe, Owain Evans,\n\nNoah D. Goodman and Joshua B. Tenenbaum\n\n{tomeru, clbaker, owenm, owain, ndg, jbt}@mit.edu\n\nDepartment of Brain and Cognitive Sciences\n\nMassachusetts Institute of Technology\n\nAbstract\n\nEveryday social interactions are heavily in\ufb02uenced by our snap judgments about\nothers\u2019 goals. Even young infants can infer the goals of intentional agents from\nobserving how they interact with objects and other agents in their environment:\ne.g., that one agent is \u2018helping\u2019 or \u2018hindering\u2019 another\u2019s attempt to get up a hill\nor open a box. We propose a model for how people can infer these social goals\nfrom actions, based on inverse planning in multiagent Markov decision problems\n(MDPs). The model infers the goal most likely to be driving an agent\u2019s behav-\nior by assuming the agent acts approximately rationally given environmental con-\nstraints and its model of other agents present. We also present behavioral evidence\nin support of this model over a simpler, perceptual cue-based alternative.\n\n1 Introduction\n\nHumans make rapid, consistent intuitive inferences about the goals of agents from the most impover-\nished of visual stimuli. On viewing a short video of geometric shapes moving in a 2D world, adults\nspontaneously attribute to them an array of goals and intentions [7]. Some of these goals are simple,\ne.g. reaching an object at a particular location. Yet people also attribute complex social goals, such\nas helping, hindering or protecting another agent. Recent studies suggest that infants as young as\nsix months make the same sort of complex social goal attributions on observing simple displays of\nmoving shapes, or (at older ages) in displays of puppets interacting [6].\nHow do humans make these rapid social goal inferences from such impoverished displays? On\none approach, social goals are inferred directly from perceptual cues in a bottom-up fashion. For\nexample, infants in [6] may judge that a triangle pushing a circle up a hill is helping the circle get\nto the top of the hill simply because the circle is moving the triangle in the direction the triangle\nwas last observed moving on its own. This approach, which has been developed by Blythe et al. [3],\nseems suited to explain the rapidity of goal attribution, without the need for mediation from higher\ncognition. On an alternative approach, these inferences come from a more cognitive and top-down\nsystem for goal attribution. The inferences are based not just on perceptual evidence, but also on\nan intuitive theory of mind on which behavior results from rational plans in pursuit of goals. On\nthis approach, the triangle is judged to be helping the circle because in some sense he knows what\nthe circle\u2019s goal is, desires for the circle to achieve the goal, constructs a rational plan of action that\nhe expects will increase the probability of the circle realizing the goal. The virtue of this theory-\nof-mind approach is its generality, accounting for a much wider range of social goal inferences that\ncannot be reduced to simple perceptual cues. Our question here is whether the rapid goal inferences\nwe make in everyday social situations, and that both infants and adults have been shown to make\nfrom simple perceptual displays, require the sophistication of a theory-based approach or can be\nsuf\ufb01ciently explained in terms of perceptual cues.\n\n1\n\n\fThis paper develops the theory-based approach to intuitive social goal inference. There are two main\nchallenges for this approach. The \ufb01rst is to formalize social goals (e.g. helping or hindering) and\nto incorporate this formalization into a general computational framework for goal inference that is\nbased on theory of mind. This framework should enable the inference that agent A is helping or\nhindering agent B from a joint goal inference based on observing A and B interacting. Inference\nshould be possible even with minimal prior knowledge about the agents and without knowledge of\nB\u2019s goal. The second challenge is to show that this computational model provides a qualitative and\nquantitative \ufb01t to rapid human goal inferences from dynamic visual displays. Can inference based\non abstract criteria for goal attribution that draws on unobservable mental states (e.g. beliefs, goals,\nplanning abilities) explain fast human judgments from impoverished and unfamiliar stimuli?\nIn addressing the challenge of formalization, we present a formal account of social goal attribution\nbased on the abstract criterion of A helping (or hindering) B by acting to maximize (minimize)\nBs probability of realizing his goals. On this account, agent A rationally maximizes utility by\nmaximizing (minimizing) the expected utility of B, where this expectation comes from As model\nof Bs goals and plans of action. We incorporate this formalization of helping and hindering into an\nexisting computational framework for theory-based goal inference, on which goals are inferred from\nactions by inverting a generative rational planning (MDP) model [1]. The augmented model allows\nfor the inference that A is helping or hindering B from stimuli in which B\u2019s goal is not directly\nobservable. We test this Inverse Planning model of social goal attribution on a set of simple 2D\ndisplays, comparing its performance to that of an alternative model which makes inferences directly\nfrom visual cues, based on previous work such as that of Blythe et al. [3].\n\n2 Computational Framework\n\nOur framework assumes that people represent the causal role of agents\u2019 goals in terms of an intuitive\nprinciple of rationality [4]: the assumption that agents will tend to take ef\ufb01cient actions to achieve\ntheir goals, given their beliefs about the world. For agents with simple goals toward objects or states\nof the world, the principle of rationality can be formalized as probabilistic planning in Markov\ndecision problems (MDPs), and previous work has successfully applied inverse planning in MDPs\nto explain human inferences about the object-directed goals of maze-world agents [2]. Inferences of\nsimple relational goals between agents (such as chasing and \ufb02eeing) from maze-world interactions\nwere considered by Baker, Goodman and Tenenbaum [1], using multiagent MDP-based inverse\nplanning. In this paper, we present a framework for modeling inferences of more complex social\ngoals, such as helping and hindering, where an agent\u2019s goals depend on the goals of other agents.\nWe will de\ufb01ne two types of agents: simple agents, which have object-directed goals and do not\nrepresent other agents\u2019 goals, and complex agents, which have either social or object-directed goals,\nand represent other agents\u2019 goals and reason about their likely behavior. For each type of agent and\ngoal, we describe the multiagent MDPs they de\ufb01ne. We then describe joint inferences of object-\ndirected and social goals based on the Bayesian inversion of MDP models of behavior.\n\n2.1 Planning in multiagent MDPs\nAn MDP M = (S,A,T ,R, \u03b3) is a tuple that de\ufb01nes a model of an agent\u2019s planning process. S is an\nencoding of the world into a \ufb01nite set of mutually exclusive states, which speci\ufb01es the set of possible\ncon\ufb01gurations of all agents and objects. A is the set of actions and T is the transition function, which\nencodes the physical laws of the world, i.e. T (St+1, St, At) = P (St+1|St, At) is the marginal\ndistribution over the next state, given the current state and the agent\u2019s action (marginalizing over all\nother agents\u2019 actions). R : S \u00d7 A \u2192 R is the reward function, which provides agents with real-\nvalued rewards for each state-action pair, and \u03b3 is the discount factor. The following subsections\nwill describe how R depends on the agent\u2019s goal G (object-directed or social), and how T depends\non the agent\u2019s type (simple or complex). We then describe how agents plan over multiagent MDPs.\n\n2.1.1 Reward functions\n\nObject-directed rewards The reward function induced by an object-directed goal G is straight-\nforward. We assume that R is an additive function of state rewards and action costs, such that\nR(S, A) = r(S)\u2212c(S, A). We consider a two-parameter family of reward functions, parameterized\n\n2\n\n\fFigure 1: (a) Illustration of the state reward functions from the family de\ufb01ned by the parameters \u03c1g and \u03b4g.\nThe agent\u2019s goal is at (6,6), where the state reward is equal to \u03c1g. The state reward functions range from a unit\nreward in the goal location (row 1) to a \ufb01eld of reward that extends to every location in the grid (row 3). (b)\nBayes net generated by multiagent planning. In this \ufb01gure, we assume that there are two agents, i and j, with\ni simple and j complex. The parameters {\u03c1i\ng} and \u03b2 are omitted from the graphical model for\nreadability.\n\no, \u03c1j\n\ng, \u03b4j\n\ng, \u03c1i\n\ng, \u03b4i\n\nby \u03c1g and \u03b4g, which captures the intuition that different kinds of object goals induce different rewards\nin space. For instance, on a hot summer day in the park, a drinking fountain is only rewarding when\none is standing directly next to it. In contrast, a \ufb02ower\u2019s beauty is greatest from up close, but can\nalso be experienced from a range of distances and perspectives. Speci\ufb01cally, \u03c1g and \u03b4g determine the\nscale and shape of the state reward function, with ri(S) = max(\u03c1g(1 \u2212 distance(S, i, G)/\u03b4g), 0),\nwhere distance(S, i, G) is the geodesic distance between agent i and the goal. With \u03b4g \u2264 1, the\nreward function has a unit value of r(S) = \u03c1g when the agent and object goal occupy the same loca-\ntion, i.e. when distance(S, i, G) = 0, and r(S) = 0 otherwise (see Fig. 1(a), row 1). When \u03b4g > 1,\nthere is a \u201c\ufb01eld\u201d of positive reward around the goal, with a slope of \u2212\u03c1g/\u03b4g (see Fig. 1(a), rows\n2 and 3). The state reward is maximal at distance(S, i, G) = 0, where r(S) = \u03c1g, and decreases\nlinearly with the agent\u2019s geodesic distance from the goal, reaching a minimum of r(S) = 0 when\ndistance(S, i, G) \u2265 \u03b4g.\n\nSocial rewards for helping and hindering For complex agent j, the state reward function in-\nduced by a social goal Gj depends on the cost of j\u2019s action Aj, as well as the reward function\nRi of the agent that j wants to help or hinder. Speci\ufb01cally, j\u2019s reward function is the difference\nof the expectation of i\u2019s reward function and j\u2019s action cost function, such that Rj(S, Aj) =\n\u03c1oEAi[Ri(S, Ai)] \u2212 c(S, Aj). \u03c1o is the social agent\u2019s scaling of the expected reward of state S\nfor agent i, which determines how much j \u201ccares\u201d about i relative to its own costs. For helping\nagents, \u03c1o > 0, and for hindering agents, \u03c1o < 0. Computing the expectation EAi[Ri(S, Ai)] relies\non the social agent\u2019s model of i\u2019s planning process, which we will describe below.\n\n2.1.2 State-transition functions\nIn our interactive setting, T i depends not just on i\u2019s action, but on all other agents\u2019 actions as well.\nAgent i is assumed to compute T i(St+1, St, Ai\n\nt) by marginalizing over Aj\nP (St+1|St, A1:n\n\nP (Aj\n\n)\n\nt\n\nt for all j (cid:54)= i:\nt \u2208 Aj(cid:54)=i\n\nt\n\n|St, G1:n)\n\nT i(St+1, St, Ai\n\nt) = P (St+1|St, Ai\n\nt) =\n\n(cid:88)\n\nAj(cid:54)=i\n\nt\n\n(cid:89)\n\nj\n\nwhere n is the number of agents. This computation requires that an agent have a model of all other\nagents, whether simple or complex.\n\nSimple agents We assume that the simple agents model other agents as randomly selecting actions\nin proportion to the softmax of their expected cost, i.e. for agent j, P (Aj|S) \u221d exp(\u03b2 \u00b7 c(S, Aj)).\nComplex agents We assume that the social agent j uses its model of other agents\u2019 planning pro-\ncess to compute P (Ai|S, Gi), for i (cid:54)= j, allowing for accurate prediction of other agents\u2019 actions.\nWe assume agents have access to the true environment dynamics. This is a simpli\ufb01cation of a more\nrealistic framework in which agents have only partial or false knowledge about the environment.\n\n3\n\nGiStSt+1GjAitAjtAit+1Ajt+1  1234567765432100.10.20.30.40.50.60.70.80.91.0  12345677654321  12345677654321\u03c1g = 1.0\u03b4g = 0.5\u03c1g = 1.0\u03b4g = 2.5\u03c1g = 1.0\u03b4g = 10.5(a)(b)\f2.1.3 Multiagent planning\nGiven the variables of MDP M, we can compute the optimal state-action value function Q\u2217 :\nS\u00d7A \u2192 R, which determines the expected in\ufb01nite-horizon reward of taking an action in each state.\nWe assume that agents have softmax-optimal policies, such that P (A|S, G) \u221d exp(\u03b2Q\u2217(S, A)),\nallowing occasional deviations from the optimal action depending on the parameter \u03b2, which deter-\nmines agents\u2019 level of determinism (higher \u03b2 implies higher determinism, or less randomness). In\na multiagent setting, joint value functions can be optimized recursively, with one agent representing\nthe value function of the other, and the other representing the representation of the \ufb01rst, and so on\nto an arbitrarily high order [10]. Here, we restrict ourselves to the \ufb01rst level of this reasoning hier-\narchy. That is, an agent A can at most represent an agent B\u2019s reasoning about A\u2019s goals and actions,\nbut not a deeper recursion in which B reasons about A reasoning about B.\n\nInverse planning in multiagent MDPs\n\n2.2\nOnce we have computed P (Ai|S, Gi) for agents 1 through n using multiagent planning, we use\nBayesian inverse planning to infer agents\u2019 goals, given observations of their behavior. Fig. 1(b)\nshows the structure of the Bayes net generated by multiagent planning, and over which goal infer-\no}1:n be a vector of the parameters of the agents\u2019 reward\nences are performed. Let \u03b8 = {\u03c1i\n(cid:88)\nfunctions. We compute the joint posterior marginal of agent i\u2019s goal Gi and \u03b8, given the observed\nstate-sequence S1:T and the action-sequences A1:n\n\ng, \u03c1i\n\ng, \u03b4i\n\nP (Gi, \u03b8|S1:T , A1:n\n\n1:T\u22121, \u03b2) \u221d\n\nGj(cid:54)=i\n\nP (A1:n\n\n1:T\u22121 of agents 1:n using Bayes\u2019 rule:\n1:T\u22121|S1:T , G1:n, \u03b8, \u03b2)P (G1:n)P (\u03b8)\n(cid:88)\n\nP (Gi, \u03b8|S1:T , A1:n\n\n1:T\u22121, \u03b2)\n\n(1)\n\n(2)\n\nTo generate goal inferences for our experimental stimuli to compare with people\u2019s judgments, we\nintegrate Eq. 1 over a range of \u03b8 values for each stimulus trial:\n\nP (Gi|S1:T , A1:n\n\n1:T\u22121, \u03b2) =\n\nThis allows our models to infer the combination of goals and reward functions that best explains the\nagents\u2019 behavior for each stimulus.\n\n\u03b8\n\n3 Experiment\n\nWe designed an experiment to test the Inverse Planning model of social goal attributions in a simple\n2D maze-world domain, inspired by the stimuli of many previous studies involving children and\nadults [7, 5, 8, 6, 9, 12]. We created a set of videos which depicted agents interacting in a maze.\nEach video contained one \u201csimple agent\u201d and one \u201ccomplex agent\u201d, as described in the Computa-\ntional Framework section. Subjects were asked to attribute goals to the agents after viewing brief\nsnippets of these videos. Many of the snippets showed agent behavior consistent with more than\none hypothesis about the agents\u2019 goals. Data from subjects was compared to the predictions of the\nInverse Planning model and a model based on simple visual cues that we describe in the Modeling\nsubsection below.\n\n3.1 Participants\n\nParticipants were 20 adults, 8 female and 12 male. Mean age was 31 years.\n\n3.2 Stimuli\n\nWe constructed 24 scenarios in which two agents moved around a 2D maze (shown in Fig. 2).\nThe maze always contained two potential object goals (a \ufb02ower and a tree), and on 12 of the 24\nscenarios it also contained a movable obstacle (a boulder). The scenarios were designed to satisfy\ntwo criteria. First, scenarios were to have agents acting in ways that were consistent with more\nthan one hypothesis concerning their goals, with these ambiguities between goals sometimes being\nresolved as the scenario developed (see Fig. 2(a)). This criterion was included to test our model\u2019s\npredictions based on ambiguous action sequences. Second, scenarios were to involve a variety of\n\n4\n\n\fperceptually distinct plans of action that might be interpreted as issuing from helping or hindering\ngoals. For example, one agent pushing another toward an object goal, removing an obstacle from\nthe other agent\u2019s path, and moving aside for the other agent (all of which featured in our scenarios)\ncould all be interpreted as helping. This criterion was included to test our formalization of social\ngoals as based on an abstract relation between reward functions. In our model, social agents act to\nmaximize or minimize the reward of the other agent, and the precise manner in which they do so\nwill vary depending on the structure of the environment and their initial positions.\n\nFigure 2: Example interactions between Small and Large agents. Agents start as in Frame 1 and progress\nthrough the sequence along the corresponding colored paths. Each frame after Frame 1 corresponds to a probe\npoint at which the video was cut off and subjects were asked to judge the agents\u2019 goals. (a) The Large agent\nmoves over each of the goal objects (Frames 1-7) and so the video is initially ambiguous between his having\nan object goal and a social goal. Disambiguation occurs from Frame 8, when the Large agent moves down and\nblocks the Small agent from continuing his path up to the object goal. (b) The Large agent moves the boulder,\nunblocking the Small agent\u2019s shortest path to the \ufb02ower (Frames 1-6). Once the Small agent moves into the\nsame room (6), the Large agent pushes him onto the \ufb02ower and allows him to rest there (8-16).\n\nEach scenario featured two different agents, which we call \u201cSmall\u201d and \u201cLarge\u201d. Large agents were\nvisually bigger and are able to shift both movable obstacles and Small agents by moving directly\ninto them. Large agents never fail in their actions, e.g. when they try to move left, they indeed move\nleft. Small agents were visually smaller, and could not shift agents or boulders. In our scenarios,\nthe actions of Small agents failed with a probability of about 0.4. Large agents correspond to the\n\u201ccomplex agents\u201d introduced in Section 2, in that they could have either object-directed goals or\nsocial goals (helping or hindering the Small agent). Small agents correspond to \u201csimple agents\u201d and\ncould have only object goals.\nWe produced videos of 16 frames in length, displaying each scenario. We showed three snippets\nfrom each video, which stopped some number of frames before the end. For example, the three\nsnippets of scenario 6 were cut off at frames 4, 7, and 8 respectively (see Fig. 2(a)). Subjects were\nasked to make goal attributions at the end of both the snippets and the full 16-frame videos. Asking\nsubjects for goal attributions at multiple points in a sequence allowed us to track the change in their\njudgments as evidence for particular goals accumulated. These cut-off or probe points were selected\nto try to capture key events in the scenarios and so occurred before and after crucial actions that\ndisambiguated between different goals. Since each scenario was used to create 4 stimuli of varying\nlength, there was a total of 96 stimuli.\n\n3.3 Procedure\n\nSubjects were initially shown a set of familiarization videos of agents interacting in the maze, illus-\ntrating the structural properties of the maze-world e.g. the actions available to agents and the possi-\nbility of moving obstacles) and the differences between Small and Large agents. The experimental\nstimuli were then presented in four blocks, each containing 24 videos. Scenarios were randomized\nwithin blocks across subjects. The left-right orientation of agents and goals was counterbalanced\nacross subjects. Subjects were told that each snippet would contain two new agents (one Small and\none Large) and this was highlighted in the stimuli by randomly varying the color of the agents for\neach snippet. Subjects were told that agents had complete knowledge of the physical structure of the\nmaze, including the position of all goals, agents and obstacles. After each snippet, subjects made\n\n5\n\nScenario 6Frame 1Frame 4Frame 7Frame 8Frame 16Scenario 19Frame 1Frame 4Frame 6Frame 8Frame 16(a)(b)\fa forced-choice for the goal of each agent. For the Large agent, they could select either of the two\nsocial goals and either of the two object goals. For the Small agent, they could choose only from the\nobject goals. Subjects also rated their con\ufb01dence on a 3-point scale.\n\n3.4 Modeling\n\nModel predictions were generated using Eq. 2, assuming uniform priors on goals, and were com-\npared directly to subjects\u2019 judgments. In our experiments, the world was given by a 2D maze-world,\nand the state space included the set of positions that agents and objects can jointly occupy without\noverlapping. The set of actions included U p, Down, Lef t, Right and Stay and we assume that\nc(S, A \u2208 {U p, Down, Lef t, Right}) = 1, and c(S, Stay) = 0.1 to re\ufb02ect the greater cost of\nmoving than staying put. We set \u03b2 to 2 and \u03b3 to 0.99, following [2].\nFor the other parameters (namely \u03c1g, \u03b4g and \u03c1o) we integrated over a range of values that provided a\ngood statistical \ufb01t to our stimuli. For instance, some stimuli were suggestive of \u201c\ufb01eld\u201d goals rather\nthan point goals, and marginalizing over \u03b4g allowed our model to capture this. Values for \u03c1g ranged\nfrom 0.5 to 2.5, going from a weak to a strong reward. For \u03b4g we integrated over three possible\nvalues: 0.5, 2.5 and 10.5. These corresponded to \u201cpoint\u201d object goals (agent receives reward for\nbeing on the goal only), \u201croom\u201d object goals (agent receives the most reward for being on the goal\nand some reward for being in the same room as the goal) and \u201cfull space\u201d object goals (agent receives\nreward at any point in proportion to distance from goal). Values for \u03c1o ranged from 1 to 9, from\ncaring weakly about the other agent to caring about it to a high degree.\nWe compared the Inverse Planning model to a model that made inferences about goals based on\nsimple visual cues, inspired by previous heuristic- or perceptually-based accounts of human action\nunderstanding of similar 2D animated displays [3, 11]. Our aim was to test whether accurate goal\ninferences could be made simply by recognizing perceptual cues that correlate with goals, rather than\nby inverting a rational model. We constructed our \u201cCue-based\u201d model by selecting ten visual cues\n(listed below), including nearly all the applicable cues from the existing cue-based model described\nin [3], leaving out those that do not apply to our stimuli, such as heading, angle and acceleration.\nWe then formulated an inference model based on these cues by using multinomial logistic regression\nto subjects\u2019 average judgments. The set of cues was as following: (1) the distance moved on the last\ntimestep, (2) the change in movement distance between successive timesteps, (3+4) the geodesic\ndistance to goals 1 and 2, (5+6) the change in distance to goals 1 and 2 (7) the distance to Small, (8)\nthe change in distance to Small, (9+10) the distance of Small to goals 1 and 2.\n\n3.5 Results\n\nBecause our main interest is in judgments about the social goals of representationally complex\nagents, we analzyed only subjects\u2019 judgments about the Large agents. Each subject judged a to-\ntal of 96 stimuli, corresponding to 4 time points along each of 24 scenarios. For each of these 96\nstimuli, we computed an empirical probability distribution representing how likely a subject was\nto believe that the Large agent had each of the four goals \u2018\ufb02ower\u2019, \u2018tree\u2019, \u2018help\u2019, or \u2018hinder\u2019, by\naveraging judgments for that stimulus across subjects, weighted by subjects\u2019 con\ufb01dence ratings. All\nanalyses then compared these average human judgments to the predictions of the Inverse Planning\nand Cue-based models.\nAcross all goal types, the overall linear correlations between human judgments and predictions\nfrom the two models appear similar: r = 0.83 for the Inverse Planning model, and r = 0.77\nfor the Cue-based model. Fig. 3 shows these correlations broken down by goal type, and reveals\nsigni\ufb01cant differences between the models on social versus object goals. The Inverse Planning\nmodel correlates well with judgments for all goal types: r = 0.79, 0.77, 0.86, 0.81 for \ufb02ower, tree,\nhelping, and hindering respectively. The Cue-based model correlates well with judgments for object\ngoals (r = 0.85, 0.90 for \ufb02ower, tree) \u2013 indeed slightly better the Inverse Planning model \u2013 but much\nless well for social goals (r = 0.67, 0.66 for helping, hindering). The most notable differences come\non the left-hand sides of the bottom panels in Fig. 3. There are many stimuli for which people are\nvery con\ufb01dent that the Large agent is either helping or hindering, and the Inverse Planning model\nis similarly con\ufb01dent (bar heights near 1). The Cue-based model, in contrast, is unsure: it assigns\nroughly equal probabilities of helping or hindering to these cases (bar heights near 0.5). In other\nwords, the Cue-based model is effective at inferring simple object goals of maze-world agents, but\n\n6\n\n\fis generally unable to distinguish between the more complex goals of helping and hindering. When\nconstrained to simply differentiating between social and object goals both models succeed equally\n(r = 0.84), where in the Cue-based model this is probably because moving away from the object\ngoals serves as a good cue to separate these categories. However, the Inverse Planning model is more\nsuccessful in differentiating the right goal within social goals (r = 0.73 for the Inverse Planning\nmodel vs. r = 0.44 for the Cue-based model).\nSeveral other general trends in the results are worth noting. The Inverse Planning model \ufb01ts very\nclosely with the judgments subjects make after the full 16-frame videos. On 23 of the 24 scenarios,\nhumans and the Inverse Planning model have the highest posterior / rating in the same goal (r =\n0.97, contrasted with r = 0.77 for the Cue-based model). Note that in the one scenario for which\nhumans and the Inverse Planning model disagreed after observing the full sequence, both humans\nand the model were close to being ambivalent whether the Large agent was hindering or interested\nin the \ufb02ower. There is also evidence that the reasonably good overall correlation for the Cue-based\nmodel is partially due to over\ufb01tting; this should not be surprising given how many free parameters\nthe model has. We divided scenarios into two groups depending on whether a boulder was moved\naround in the scenario, as movable boulders increase the range of variability in helping and hindering\naction sequences. When trained on the \u2018no boulder\u2019 cases, the Cue-based model correlates poorly\nwith subjects\u2019 average judgments on the \u2018boulder\u2019 cases: r = 0.42. The same failure of transfer\noccurs when the Cue-based model is trained on the \u2018boulder\u2019 cases and tested on the \u2018no boulder\u2019\ncases: r = 0.36. This is consistent with our general concern that a Cue-based model incorporating\nmany free parameters may do well when tailored to a particular environment, but is not likely to\ngeneralize well to new environments.\nIn contrast, the Inverse Planning model captures abstract\nrelations between the agents and their possible goal and so lends itself to a variety of environments.\n\nFigure 3: Correlations between human goal judgments and predictions of the Inverse Planning model (a) and\nthe Cue-based model (b), broken down by goal type. Bars correspond to bins of stimuli (out of 96 total) on\nwhich the average human judgment for the probability of that goal was within a particular range; the midpoint\nof each bin\u2019s range is shown on the x-axis labels. The height of each bar shows the model\u2019s average probability\njudgment for all stimuli in that bin. Linear correlations between the model\u2019s goal probabilities and average\nhuman judgments for all 96 stimuli are given in the y-axis labels.\n\nThe inability of the heuristic model to distinguish between helping and hindering is illustrated by\nthe plots in Fig. 4. In contrast, both the Inverse Planning model and the human subjects are often\nvery con\ufb01dent that an agent is helping and not hindering (or vice versa).\nFig. 4 also illustrates a more general \ufb01nding, that the Inverse Planning model captures most of\nthe major qualitative shifts (e.g. shifts resulting from disambiguating sequences) in subjects\u2019 goal\nattribution. Figure 4 displays mean human judgments on four scenarios. Probe points (i.e. points\nwithin the sequences at which subjects made judgments) are indicated on the plots and human data\nis compared with predictions from the Inverse Planning model and the Cue-based model.\nOn scenario 6 (depicted in Fig. 2(a) but with goals switched), both the Inverse Planning model and\nhumans subjects recognize the movement of the Large agent one step off the \ufb02ower (or the tree in\nFig. 2(b)) as strong evidence that Large has a hindering goal. The Cue-based model responds in the\nsame way but with much less con\ufb01dence in hindering. Even after 8 subsequent frames of action it\nis unable to decide in favor of hindering over helping.\n\n7\n\n(r = 0.79)(r = 0.77)0.0250.10.250.50.750.90.975(r = 0.86)(r = 0.81)Human judgmentsInverse planning model0.0250.10.250.50.750.90.9750.0250.10.250.50.750.90.9750.0250.10.250.50.750.90.975FlowerTreeHelpHinder00.20.40.60.81(r = 0.85)(r = 0.90)00.20.40.60.81Cue\u2212based model(r = 0.67)(r = 0.66)Human judgments0.0250.10.250.50.750.90.9750.0250.10.250.50.750.90.9750.0250.10.250.50.750.90.9750.0250.10.250.50.750.90.97500.20.40.60.8100.20.40.60.8100.20.40.60.8100.20.40.60.8100.20.40.60.8100.20.40.60.81(b)(a)\fWhile the Inverse Planning model and subjects almost always agree by the end of a sequence, they\nsometimes disagree at early probe points. In scenario 5, both agents start off in the bottom-left room,\nbut with the Small agent right at the entrance to the top-left room. As the Small agent tries to move\ntowards the \ufb02ower (the top-left goal), the Large agent moves up from below and pushes Small one\nstep towards the \ufb02ower before moving off to the right to the tree. People interpret the Large agent\u2019s\naction as strong evidence for helping, in contrast with the Inverse Planning model. For the model,\nbecause Small is so close to his goal, Large could just as well stay put and save his own action costs.\nTherefore his movement upwards is not evidence of helping.\n\nFigure 4: Example data and model predictions. Probe points are marked as black circles. (a) Average subject\nratings with standard error bars. (b) Predictions of Inverse Planning model interpolated from cut points. (c)\nPredictions of Inverse Planning model for all points in the sequence. (d) Predictions of Cue-based model.\n4 Conclusion\nOur goal in this paper was to address two challenges. The \ufb01rst was to provide a formalization of\nsocial goal attribution incorporated into a general theory-based model for goal attribution. This\nmodel had to enable the inference that A is helping or hindering B from interactions between A\nand B but without prior knowledge of either agent\u2019s goal, and to account for the range of behaviors\nthat humans judge as evidence of helping or hindering. The second challenge was for the model to\nperform well on a demanding inference task in which social goals must be inferred from very few\nobservations without directly observable evidence of agents\u2019 goals.\nThe experimental results presented here go some way to meeting these challenges. The Inverse\nPlanning model classi\ufb01ed a diverse range of agent interactions as helping or hindering in line with\nhuman judgments. This model also distinguished itself against a model based solely on simple\nperceptual cues. It produced a closer \ufb01t to humans for both social and nonsocial goal attributions,\nand was far superior to the visual cue model in discriminating between helping and hindering.\nThese results suggest various lines of further research. One task is to augment this formal model of\nhelping and hindering to capture more of the complexity behind human judgments. On the Inverse\nPlanning model, A will act to advance B\u2019s progress only if there is some chance of B actually\nreceiving a nontrivial amount of reward in a future state. However, people often help others towards\na goal even if they think it very unlikely that the goal will be achieved. This aspect of helping could\nbe explored by supposing that the utility of a helping agent depends not just on another agent\u2019s\nreward function but also his value function.\nAcknowledgments: This work was supported by the James S. McDonnell Foundation Causal Learning Col-\nlaborative Initiative, ARO MURI grant W911NF-08-1-0242, AFOSR MURI grant FA9550-07-1-0075 and the\nNSF Graduate Fellowship (CLB).\n\n8\n\nflowertreehelphinderFrame24681012141600.51People 24681012141600.51Inverse Planning (probe points) 24681012141600.51 24681012141600.51Visual Cue Scenario 19Frame24681012141600.51People 24681012141600.51Inverse Planning(probe points) 24681012141600.51 24681012141600.51Visual Cue Scenario 6Frame24681012141600.51People 24681012141600.51Inverse Planning(probe points) 24681012141600.51 24681012141600.51Visual Cue Scenario 24Frame24681012141600.51People 24681012141600.51Inverse Planning(probe points)Inverse Planning (all points)Inverse Planning(all points)Inverse Planning(all points)Inverse Planning(all points) 24681012141600.51 24681012141600.51Visual Cue Scenario 12AverageratingP(Goal|Trial)P(Goal|Trial)P(Goal|Trial)(a)(b)(c)(d)\fReferences\n[1] C. L. Baker, N. D. Goodman, and J. B. Tenenbaum. Theory-based social goal inference. In Proceedings\n\nof the Thirtieth Annual Conference of the Cognitive Science Society, 2008.\n\n[2] C. L. Baker, J. B. Tenenbaum, and R. R. Saxe. Bayesian models of human action understanding.\n\nAdvances in Neural Information Processing Systems, volume 18, pages 99\u2013106, 2006.\n\nIn\n\n[3] P. W. Blythe, P. M. Todd, and G. F. Miller. How motion reveals intention: categorizing social interactions.\nIn G. Gigerenzer, P. M. Todd, and the ABC Research Group, editors, Simple heuristics that make us smart,\npages 257\u2013286. Oxford University Press, New York, 1999.\n\n[4] D. C. Dennett. The Intentional Stance. MIT Press, Cambridge, MA, 1987.\n[5] G. Gergely, Z. N\u00b4adasdy, G. Csibra, and S. Bir\u00b4o. Taking the intentional stance at 12 months of age.\n\nCognition, 56:165\u2013193, 1995.\n\n[6] J. K. Hamlin, Karen Wynn, and Paul Bloom. Social evaluation by preverbal infants. Nature, 450:557\u2013560,\n\n2007.\n\n[7] F. Heider and M. A. Simmel. An experimental study of apparent behavior. American Journal of Psychol-\n\nogy, 57:243\u2013249, 1944.\n\n[8] V. Kuhlmeier, Karen Wynn, and Paul Bloom. Attribution of dispositional states by 12-month-olds. Psy-\n\nchological Science, 14(5):402\u2013408, 2003.\n\n[9] J. Schultz, K. Friston, D. M. Wolpert, and C. D. Frith. Activation in posterior superior temporal sulcus\n\nparallels parameter inducing the percept of animacy. Neuron, 45:625\u2013635, 2005.\n\n[10] Wako Yoshida, Ray J. Dolan, and Karl J. Friston. Game theory of mind. PLoS Computational Biology,\n\n4(12):1\u201314, 2008.\n\n[11] Jeffrey M. Zacks. Using movement and intentions to understand simple events. Cognitive Science,\n\n28:979\u20131008, 2004.\n\n[12] P. D. Tremoulet and J. Feldman The in\ufb02uence of spatial context and the role of intentionality in the\n\ninterpretation of animacy from motion. Perception and Psychophysics, 29:943\u2013951, 2006.\n\n9\n\n\f", "award": [], "sourceid": 1192, "authors": [{"given_name": "Tomer", "family_name": "Ullman", "institution": null}, {"given_name": "Chris", "family_name": "Baker", "institution": null}, {"given_name": "Owen", "family_name": "Macindoe", "institution": null}, {"given_name": "Owain", "family_name": "Evans", "institution": null}, {"given_name": "Noah", "family_name": "Goodman", "institution": null}, {"given_name": "Joshua", "family_name": "Tenenbaum", "institution": null}]}