{"title": "Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 127, "page_last": 135, "abstract": "Humans demonstrate remarkable abilities to predict physical events in dynamic scenes, and to infer the physical properties of objects from static images. We propose a generative model for solving these problems of physical scene understanding from real-world videos and images. At the core of our generative model is a 3D physics engine, operating on an object-based representation of physical properties, including mass, position, 3D shape, and friction. We can infer these latent properties using relatively brief runs of MCMC, which drive simulations in the physics engine to fit key features of visual observations. We further explore directly mapping visual inputs to physical properties, inverting a part of the generative process using deep learning. We name our model Galileo, and evaluate it on a video dataset with simple yet physically rich scenarios. Results show that Galileo is able to infer the physical properties of objects and predict the outcome of a variety of physical events, with an accuracy comparable to human subjects. Our study points towards an account of human vision with generative physical knowledge at its core, and various recognition models as helpers leading to efficient inference.", "full_text": "Galileo: Perceiving Physical Object Properties by\nIntegrating a Physics Engine with Deep Learning\n\nJiajun Wu\u2217\nEECS, MIT\n\njiajunwu@mit.edu\n\nJoseph J. Lim\nEECS, MIT\n\nlim@csail.mit.edu\n\nIlker Yildirim\u2217\n\nBCS MIT, The Rockefeller University\n\nilkery@mit.edu\n\nWilliam T. Freeman\n\nEECS, MIT\n\nbillf@mit.edu\n\nJoshua B. Tenenbaum\n\nBCS, MIT\n\njbt@mit.edu\n\nAbstract\n\nHumans demonstrate remarkable abilities to predict physical events in dynamic\nscenes, and to infer the physical properties of objects from static images. We\npropose a generative model for solving these problems of physical scene under-\nstanding from real-world videos and images. At the core of our generative model\nis a 3D physics engine, operating on an object-based representation of physical\nproperties, including mass, position, 3D shape, and friction. We can infer these\nlatent properties using relatively brief runs of MCMC, which drive simulations in\nthe physics engine to \ufb01t key features of visual observations. We further explore\ndirectly mapping visual inputs to physical properties, inverting a part of the gener-\native process using deep learning. We name our model Galileo, and evaluate it on a\nvideo dataset with simple yet physically rich scenarios. Results show that Galileo\nis able to infer the physical properties of objects and predict the outcome of a vari-\nety of physical events, with an accuracy comparable to human subjects. Our study\npoints towards an account of human vision with generative physical knowledge at\nits core, and various recognition models as helpers leading to ef\ufb01cient inference.\n\n1\n\nIntroduction\n\nOur visual system is designed to perceive a physical world that is full of dynamic content. Consider\nyourself watching a Rube Goldberg machine unfold: as the kinetic energy moves through the ma-\nchine, you may see objects sliding down ramps, colliding with each other, rolling, entering other\nobjects, falling \u2014 many kinds of physical interactions between objects of different masses, mate-\nrials and other physical properties. How does our visual system recover so much content from the\ndynamic physical world? What is the role of experience in interpreting a novel dynamical scene?\nRecent behavioral and computational studies of human physical scene understanding push forward\nan account that people\u2019s judgments are best explained as probabilistic simulations of a realistic, but\nmental, physics engine [2, 8]. Speci\ufb01cally, these studies suggest that the brain carries detailed but\nnoisy knowledge of the physical attributes of objects and the laws of physical interactions between\nobjects (i.e., Newtonian mechanics). To understand a physical scene, and more crucially, to predict\nthe future dynamical evolution of a scene, the brain relies on simulations from this mental physics\nengine. Even though the probabilistic simulation account is very appealing, there are missing practi-\ncal and conceptual leaps. First, as a practical matter, the probabilistic simulation approach is shown\nto work only with synthetically generated stimuli: either in 2D worlds, or in 3D worlds but each\n\n\u2217 Indicates equal contribution. The authors are listed in the alphabetical order.\n\n1\n\n\fobject is constrained to be a block and the joint inference of the mass and friction coef\ufb01cient is not\nhandled [2]. Second, as a conceptual matter, previous research rarely clari\ufb01es how a mental physics\nengine could take advantage of previous experience of the agent [11]. It is the case that humans\nhave a life long experience with dynamical scenes, and a fuller account of human physical scene\nunderstanding should address it.\nHere, we build on the idea that humans utilize a realistic physics engine as part of a generative\nmodel to interpret real-world physical scenes. We name our model Galileo. The \ufb01rst component of\nour generative model is the physical object representations, where each object is a rigid body and\nrepresented not only by its 3D geometric shape (or volume) and its position in space, but also by its\nmass and its friction. All of these object attributes are treated as latent variables in the model, and\nare approximated or estimated on the basis of the visual input.\nThe second part is a fully-\ufb02edged realistic physics engine \u2014 in this paper, speci\ufb01cally the Bullet\nphysics engine [4]. The physics engine takes a scene setup as input (e.g., speci\ufb01cation of each of the\nphysical objects in the scene, which constitutes a hypothesis in our generative model), and physically\nsimulates it forward in time, generating simulated velocity pro\ufb01les and positions for each object.\nThe third part of Galileo is the likelihood function. We evaluate the observed real-world videos\nwith respect to the model\u2019s hypotheses using the velocity vectors of objects in the scene. We use a\nstandard tracking algorithm to map the videos to the velocity space.\nNow, given a video as observation to the model, physical scene understanding in the model corre-\nsponds to inverting the generative model by probabilistic inference to recover the underlying physi-\ncal object properties in the scene. Here, we build a video dataset to evaluate our model and humans\non real-world data, which contains 150 videos of different objects with a range of materials and\nmasses over a simple yet physically rich scenario: an object sliding down an inclined surface, and\npotentially collide with another object on the ground. Note that in the \ufb01elds of computer vision\nand robotics, there have been studies on predicting physical interactions or inferring 3D properties\nof objects for various purposes including 3D reasoning [6, 13] and tracking [9]. However, none\nof them focused on learning physical properties directly, and nor they have incorporated a physics\nengine with representation learning.\nBased on the estimates we derived from visual input with a physics engine, a natural extension is\nto generate or synthesize training data for any automatic learning systems by bootstrapping from\nthe videos already collected, and labeling them with estimates of Galileo. This is a self-supervised\nlearning algorithm for inferring generic physical properties, and relates to the wake/sleep phases in\nHelmholtz machines [5], and to the cognitive development of infants. Extensive studies suggest that\ninfants either are born with or can learn quickly physical knowledge about objects when they are very\nyoung, even before they acquire more advanced high-level knowledge like semantic categories of\nobjects [3, 1]. Young babies are sensitive to physics of objects mainly from the motion of foreground\nobjects from background [1]; in other words, they learn by watching videos of moving objects. But\nlater in life, and clearly in adulthood, we can perceive physical attributes in just static scenes without\nany motion.\nHere, building upon the idea of Helmholtz machiness [5], our approach suggests one potential com-\nputational path to the development of the ability to perceive physical content in static scenes. Fol-\nlowing the recent work [12], we train a recognition model (i.e., sleep cycle) that is in the form of a\ndeep convolutional network, where the training data is generated in a self-supervised manner by the\ngenerative model itself (i.e., wake cycle: real-world videos observed by our model and the resulting\nphysical inferences).\nOur work makes three contributions. First, we propose Galileo, a novel model for estimating phys-\nical properties of objects from visual inputs by incorporating the feedback of a physics engine in\nthe loop. We demonstrate that it achieves encouraging performance on a real-world video dataset.\nSecond, we train a deep learning based recognition model that leads to ef\ufb01cient inference in the\ngenerative model, and enables the generative model to predict future dynamical evolution of static\nscenes (e.g., how would that scene unfold in time). Third, we test our model and compare it to hu-\nmans on a variety of physical judgment tasks. Our results indicate that humans are quite successful\nin these tasks, and our model closely matches humans in performance, but also consistently makes\nsimilar errors as humans do, providing further evidence in favor of the probabilistic simulation ac-\ncount of human physical scene understanding.\n\n2\n\n\f(a)\n\n(b)\n\nFigure 1: (a) Snapshots of the dataset. (b) Overview of the model. Our model formalizes a hy-\npothesis space of physical object representations, where each object is de\ufb01ned by its mass, friction\ncoef\ufb01cient, 3D shape, and a positional offset w.r.t. an origin. To model videos, we draw exactly two\nobjects from that hypothesis space into the physics engine. The simulations from the physics engine\nare compared to observations in the velocity space, a much \u201cnicer\u201d space than pixels.\n\n2 Scenario\n\nWe seek to learn physical properties of objects by observing videos. Among many scenarios, we\nconsider an introductory setup: an object is put on an inclined surface; it may either slide down or\nkeep static due to gravity and friction, and may hit another object if it slides down.\nThis seemingly simple scenario is physically highly involved. The observed outcome of these sce-\nnario are physical values which help to describe the scenario, such as the velocity and moving\ndistance of objects. Causally underlying these observations are the latent physical properties of ob-\njects such as the material, density, mass and friction coef\ufb01cient. As shown in Section 3, our Galileo\nmodel intends to model the causal generative relationship between these observed and unobserved\nvariables.\nWe collect a real-world video dataset of around 100 objects sliding down a ramp, possibly hitting\nanother object. Figure 1a provides some exemplar videos in the dataset. The results of collisions,\nincluding whether it will happen or not, are determined by multiple factors, such as material (density\nand friction coef\ufb01cient), size and shape (volume), and slope of surface (gravity). Videos in our\ndataset vary in all these parameters.\nSpeci\ufb01cally, there are 15 different materials \u2014 cardboard, dough, foam, hollow rubber, hollow\nwood, metal coin, metal pole, plastic block, plastic doll, plastic ring, plastic toy, porcelain, rubber,\nwooden block, and wooden pole. For each material, there are 4 to 12 objects of different sizes and\nshapes. The angle between the inclined surface and the ground is either 10o or 20o. When an object\nslides down, it may hit either a cardboard box, or a piece of foam, or neither.\n\n3 Galileo: A Physical Object Model\n\nThe gist of our model can be summarized as probabilistically inverting a physics engine in order\nto recover unobserved physical properties of objects. We collectively refer to the unobserved latent\nvariables of an object as its physical representation T . For each object i, Ti consists of its mass mi,\nfriction coef\ufb01cient ki, 3D shape Vi, and position offset pi w.r.t. an origin in 3D space.\n\n3\n\nIB A IA BGANARANBGBGANARANBGBGANANBGB1Physical object i- Mass (m)- Friction coefficient (k)- 3D shape (S)- Position offset (x)Draw two physical objects23D Physics engine...Tracking algorithmObserved velocitiesSimulated velocitiesLikelihood function\fWe place uniform priors over the mass and the friction coef\ufb01cient for each object: mi \u223c\nUniform(0.001, 1) and ki \u223c Uniform(0, 1), respectively. For 3D shape Vi, we have four variables: a\nshape type ti, and the scaling factors for three dimensions xi, yi, zi. We simplify the possible shape\nspace in our model by constraining each shape type ti to be one of the three with equal probability: a\nbox, a cylinder, and a torus. Note that applying scaling differently on each dimension to these three\nbasic shapes results in a large space of shapes.1 The scaling factors are chosen to be uniform over\nthe range of values to capture the extent of different shapes in the dataset.\nRemember that our scenario consists of an object on the ramp and another on the ground. The\nposition offset, pi, for each object is uniform over the set {0,\u00b11,\u00b12,\u00b7\u00b7\u00b7 ,\u00b15}. This indicates that\nfor the object on the ramp, its position can be perturbed along the ramp (i.e., in 2D) at most 5 units\nupwards or downwards from its starting position, which is 30 units upwards on the ramp from the\nground.\nThe next component of our generative model is a fully-\ufb02edged realistic physics engine that we\ndenote as \u03c1. Speci\ufb01cally we use the Bullet physics engine [4] following the earlier related work.\nThe physics engine takes a speci\ufb01cation of each of the physical objects in the scene within the\nbasic ramp setting as input, and simulates it forward in time, generating simulated velocity vectors\nfor each object in the scene, vs1 and vs2 respectively \u2013 among other physical properties such as\nposition, rendered image of each simulation step, etc.\nIn light of initial qualitative analysis, we use velocity vectors as our feature representation in evalu-\nating the hypothesis generated by the model against data. We employ a standard tracking algorithm\n(KLT point tracker [10]) to \u201clift\u201d the visual observations to the velocity space. That is, for each\nvideo, we \ufb01rst run the tracking algorithm, and we obtain velocities by simply using the center loca-\ntions of each of the tracked moving objects between frames. This gives us the velocity vectors for\nthe object on the ramp and the object on the ground, vo1 and vo2, respectively. Note that we could\nreplace the KLT tracker with state-of-the-art tracking algorithms for more complicated scenarios.\nGiven a pair of observed velocity vectors, vo1 and vo2, the recovery of the physical object represen-\ntations T1 and T2 for the two objects via physics-based simulation can be formalized as:\n\nP (T1, T2|vo1 , vo2 , \u03c1(\u00b7)) \u221d P (vo1, vo2|vs1, vs2 ) \u00b7 P (vs1, vs2|T1, T2, \u03c1(\u00b7)) \u00b7 P (T1, T2).\n\n(1)\nwhere we de\ufb01ne the likelihood function as P (vo1 , vo2|vs1, vs2) = N (vo|vs, \u03a3), where vo is the\nconcatenated vector of vo1 , vo2, and vs is the concatenated vector of vs1, vs2. The dimensionality\nof vo and vs are kept the same for a video by adjusting the number of simulation steps we use to\nobtain vo according to the length of the video. But from video to video, the length of these vectors\nmay vary. In all of our simulations, we \ufb01x \u03a3 to 0.05, which is the only free parameter in our model.\nExperiments show that the value of \u03a3 does not change our results signi\ufb01cantly.\n\n3.1 Tracking as Recognition\n\nThe posterior distribution in Equation 1 is intractable. In order to alleviate the burden of posterior\ninference, we use the output of our recognition model to predict and \ufb01x some of the latent variables\nin the model.\nSpeci\ufb01cally, we determine the Vi, or {ti, xi, yi, zi}, using the output of the tracking algorithm, and\n\ufb01x these variables without further sampling them. Furthermore, we \ufb01x values of pis also on the\nbasis of the output of the tracking algorithm.\n\n3.2\n\nInference\n\nOnce we initialize and \ufb01x the latent variables using the tracking algorithm as our recognition model,\nwe then perform single-site Metropolis Hasting updates on the remaining four latent variables,\nm1, m2, k1 and k2. At each MCMC sweep, we propose a new value for one of these random\nvariables, where the proposal distribution is Uniform(\u22120.05, 0.05). In order to help with mixing,\nwe also use a broader proposal distribution, Uniform(\u22120.5, 0.5) at every 20 MCMC sweeps.\n\n1For shape type box, xi, yi, and zi could all be different values; for shape type torus, we constrained the\nscaling factors such that xi = zi; and for shape type cylinder, we constrained the scaling factors such that\nyi = zi.\n\n4\n\n\fFigure 2: Simulation results. Each row represents one video in the data: (a) the \ufb01rst frame of the\nvideo, (b) the last frame of the video, (c) the \ufb01rst frame of the simulated scene generated by Bullet,\n(d) the last frame of the simulated scene, (e) the estimated object with larger mass, (f) the estimated\nobject with larger friction coef\ufb01cient.\n\n4 Simulations\n\nFor each video, as mentioned earlier, we use the tracking algorithm to initialize and \ufb01x the shapes\nof the objects, S1 and S2, and the position offsets, p1 and p2. We also obtain the velocity vector for\neach object using the tracking algorithm. We determine the length of the physics engine simulation\nby the length of the observed video \u2014 that is, the simulation runs until it outputs a velocity vector\nfor each object that is as long as the input velocity vector from the tracking algorithm.\nAs mentioned earlier, we collect 150 videos, uniformly distributed across different object categories.\nWe perform 16 MCMC simulations for a single video, each of which was 75 MCMC sweeps long.\nWe report the results with the highest log-likelihood score across the 16 chains (i.e., the MAP esti-\nmate).\nIn Figure 2, we illustrate the results for three individual videos. Every two frame of the top row\nshows the \ufb01rst and the last frame of a video, and the bottom row images show the corresponding\nframes from our model\u2019s simulations with the MAP estimate. We quantify different aspects of\nour model in the following behavioral experiments, where we compare our model against human\nsubjects\u2019 judgments. Furthermore, we use the inferences made by our model here on the 150 videos\nto train a recognition model to arrive at physical object perception in static scenes with the model.\nImportantly, note that our model can generalize across a broad range of tasks beyond the ramp\nscenario. For example, once we infer the coef\ufb01cient friction of an object, we can make a prediction\non whether it will slide down a ramp with a different slope by doing simulation. We test some of the\ngeneralizations in Section 6.\n\n5 Bootstrapping as Ef\ufb01cient Perception in Static Scenes\n\nBased on the estimates we derived from the visual input with a physics engine, we bootstrap from the\nvideos already collected, by labeling them with estimates of Galileo. This is a self-supervised learn-\ning algorithm for inferring generic physical properties. As discussed in Section 1, this formulation\nis also related to the wake/sleep phases in Helmholtz machines, and to the cognitive development of\ninfants.\nHere we focus on two physical properties: mass and friction coef\ufb01cient. To do this, we \ufb01rst estimate\nthese physical properties using the method described in earlier sections. Then, we train LeNet [7], a\nwidely used deep neural network for small-scale datasets, using image patches cropped from videos\nbased on the output of the tracker as data, and estimated physical properties as labels. The trained\nmodel can then be used to predict these physical properties of objects based on purely visual cues,\neven though they might have never appeared in the training set.\nWe also measure masses of all objects in the dataset, which makes it possible for us to quantitatively\nevaluate the predictions of the deep network. We choose one object per material as our test cases,\n\n5\n\nCardboardDoughPole(a)(b)(c)(d)(e)(f)\fMass\n\nMethods\n\nMSE Corr\n0.71\nOracle\n0.042\nGalileo\n0.052\n0.44\nUniform 0.081\n\n0\n\nFigure 3: Mean squared errors of oracle es-\ntimation, our estimation, and uniform esti-\nmations of mass on a log-normalized scale,\nand the correlations between estimations and\nground truths\n\nFigure 4: The log-likelihood traces of sev-\neral chains with and without recognition-model\n(LeNet) based initializations.\n\nuse all data of those objects as test data, and the others as training data. We compare our model\nwith a baseline, which always outputs a uniform estimate calculated by averaging the masses of\nall objects in the test data, and with an oracle algorithm, which is a LeNet trained using the same\ntraining data, but has access to the ground truth masses of training objects as labels. Apparently, the\nperformance of the oracle model can be viewed as an upper bound of our Galileo system.\nTable 3 compares the performance of Galileo, the oracle algorithm, and the baseline. We can observe\nthat Galileo is much better than baseline, although there is still some space for improvement.\nBecause we trained LeNet using static images to predict physical object properties such as friction\nand mass ratios, we can use it to recognize those attributes in a quick bottom-up pass at the very \ufb01rst\nframe of the video. To the extent that the trained LeNet is accurate, if we initialize the MCMC chains\nwith these bottom-up predictions, we expect to see an overall boost in our log-likelihood traces. We\ntest by running several chains with and without LeNet-based initializations. Results can be seen in\nFigure 4. Despite the fact that LeNet is not achieving perfect performance by itself, we indeed get a\nboost in speed and quality in the inference.\n\n6 Experiments\n\nIn this section, we conduct experiments from multiple perspectives to evaluate our model. Speci\ufb01-\ncally, we use the model to predict how far objects will move after the collision; whether the object\nwill remain stable in a different scene; and which of the two objects is heavier based on observations\nof collisions. For every experiment, we also conduct behavioral experiments on Amazon Mechanical\nTurk so that we may compare the performance of human and machine on these tasks.\n\n6.1 Outcome Prediction\n\nIn the outcome prediction experiment, our goal is to measure and compare how well human and\nmachines can predict the moving distance of an object if only part of the video can be observed.\nSpeci\ufb01cally, for behavioral experiments on Amazon Mechanical Turk, we \ufb01rst provide users four full\nvideos of objects made of a certain material, which contain complete collisions. In this way, users\nmay infer the physical properties associated with that material in their mind. We select a different\nobject, but made of the same material, show users a video of the object, but only to the moment\nof collision. We \ufb01nally ask users to label where they believe the target object (either cardboard or\nfoam) will be after the collision, i.e., how far the target will move. We tested 30 users per case.\nGiven a partial video, for Galileo to generate predicted destinations, we \ufb01rst run it to \ufb01t the part of\nthe video to derive our estimate of its friction coef\ufb01cient. We then estimate its density by averaging\n\n6\n\n-2e+05-1e+050e+000204060Number of MCMC sweepsLog Likelihoodinitialization with recognition modelrandom initialization\fFigure 5: Mean errors in numbers of pixels of human predictions, Galileo outputs, and a uniform\nestimate calculated by averaging ground truth ending points over all test cases. As the error patterns\nare similar for both target objects (foam and cardboard), the errors here are averaged across target\nobjects for each material.\n\nFigure 6: Heat maps of user predictions, Galileo outputs (orange crosses), and ground truths (white\ncrosses).\n\nthe density values we derived from other objects with that material by observing collisions that they\nare involved. We further estimate the density (mass) and friction coef\ufb01cient of the target object by\naveraging our estimates from other collisions. We now have all required information for the model\nto predict the ending point of the target after the collision. Note that the information available to\nGalileo is exactly the same as that available to humans.\nWe compare three kinds of predictions: human feedback, Galileo output, and, as a baseline, a uni-\nform estimate calculated by averaging ground truth ending points over all test cases. Figure 5 shows\nthe Euclidean distance in pixels between each of them and the ground truth. We can see that human\npredictions are much better than the uniform estimate, but still far from perfect. Galileo performs\nsimilar to human in the average on this task. Figure 6 shows, for some test cases, heat maps of user\npredictions, Galileo outputs (orange crosses), and ground truths (white crosses). The error correla-\ntion between human and POM is 0.70. The correlation analysis for the uniform model is not useful\nbecause the correlation is a constant independent of the uniform prediction value.\n\n6.2 Mass Prediction\n\nThe second experiment is to predict which of two objects is heavier, after observing a video of a\ncollision of them. For this task, we also randomly choose 50 objects, we test each of them on 50\nusers. For Galileo, we can directly obtain its guess based on the estimates of the masses of the\nobjects.\nFigure 7 demonstrates that human and our model achieve about the same accuracy on this task.\nWe also calculate correlations between different outputs. Here for correlation analysis, we use the\nratio of the masses of the two objects estimated by Galileo as its predictor. Human responses are\naggregated for each trial to get the proportion of people making each decision. As the relation\nis highly nonlinear, we calculate Spearman\u2019s coef\ufb01cients. From Table 1, we notice that human\nresponses, machine outputs, and ground truths are all positively correlated.\n\n7\n\ncardboarddoughhollow woodmetal coinmetal poleplastic blockplastic dollplastic toyporcelainwooden blockwooden pole Mean050100150200250 Error in pixelsHumanGalileoUniform\fMass\nHuman vs Galileo\nHuman vs Truth\nGalileo vs Truth\n\n\u201cWill it move\u201d\nHuman vs Galileo\nHuman vs Truth\nGalileo vs Truth\n\nSpearman\u2019s Coeff\n\n0.51\n0.68\n0.52\n\nPearson\u2019s Coeff\n\n0.56\n0.42\n0.20\n\nFigure 7: Average accuracy of human predic-\ntions and Galileo outputs on the tasks of mass\nprediction and \u201cwill it move\u201d prediction. Error\nbars indicate standard deviations of human ac-\ncuracies.\n\nTable 1: Correlations between pairs of outputs in\nthe mass prediction experiment (in Spearman\u2019s\ncoef\ufb01cient) and in the \u201cwill it move\u201d prediction\nexperiment (in Pearson\u2019s coef\ufb01cient).\n\n6.3\n\n\u201cWill it move\u201d Prediction\n\nOur third experiment is to predict whether a certain object will move in a different scene, after\nobserving one of its collisions. On Amazon Mechanical Turk, we show users a video containing a\ncollision of two objects. In this video, the angle between the inclined surface and the ground is 20\ndegrees. We then show users the \ufb01rst frame of a 10-degree video of the same object, and ask them to\npredict whether the object will slide down the surface in this case. We randomly choose 50 objects\nfor the experiment, and divide them into lists of 10 objects per user, and get each of the item tested\non 50 users overall.\nFor Galileo, it is straightforward to predict the stability of an object in the 10-degree case using\nestimates from the 20-degree video. Interestingly, both humans and the model are at chance on this\ntask (Figure 7), and their responses are reasonably correlated (Table 1). Again, here we aggregate\nhuman responses for each trial to get the proportion of people making each decision. Moreover, both\nsubjects and the model show a bias towards saying \u201cit will move.\u201d Future controlled experimentation\nand simulations will investigate what underlies this correspondence.\n\n7 Conclusion\n\nThis paper accomplishes three goals: \ufb01rst, it shows that a generative vision system with physical\nobject representations and a realistic 3D physics engine at its core can ef\ufb01ciently deal with real-world\ndata when proper recognition models and feature spaces are used. Second, it shows that humans\u2019\nintuitions about physical outcomes are often accurate, and our model largely captures these intuitions\n\u2014 but crucially, humans and the model make similar errors. Lastly, the experience of the model,\nthat is, the inferences it makes on the basis of dynamical visual scenes, can be used to train a deep\nlearning model, which leads to more ef\ufb01cient inference and to the ability to see physical properties\nin the static images. Our study points towards an account of human vision with generative physical\nknowledge at its core, and various recognition models as helpers to induce ef\ufb01cient inference.\n\nAcknowledgements\n\nThis work was supported by NSF Robust Intelligence 1212849 Reconstructive Recognition and the\nCenter for Brains, Minds, and Machines (funded by NSF STC award CCF-1231216).\n\n8\n\nMass\"Will it move\"00.20.40.60.81HumanGalileo\fReferences\n[1] Ren\u00b4ee Baillargeon.\n13(3):89\u201394, 2004.\n\nInfants\u2019 physical world. Current directions in psychological science,\n\n[2] Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of\n\nphysical scene understanding. PNAS, 110(45):18327\u201318332, 2013.\n\n[3] Susan Carey. The origin of concepts. Oxford University Press, 2009.\n[4] Erwin Coumans. Bullet physics engine. Open Source Software: http://bulletphysics. org, 2010.\n[5] Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz\n\nmachine. Neural computation, 7(5):889\u2013904, 1995.\n\n[6] Zhaoyin Jia, Andy Gallagher, Ashutosh Saxena, and Tsuhan Chen. 3d reasoning from blocks\n\nto stability. IEEE TPAMI, 2014.\n\n[7] Yann LeCun, L\u00b4eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning\n\napplied to document recognition. Proceedings of the IEEE, 86(11):2278\u20132324, 1998.\n\n[8] Adam N Sanborn, Vikash K Mansinghka, and Thomas L Grif\ufb01ths. Reconciling intuitive\nphysics and newtonian mechanics for colliding objects. Psychological review, 120(2):411,\n2013.\n\n[9] John Schulman, Alex Lee, Jonathan Ho, and Pieter Abbeel. Tracking deformable objects with\npoint clouds. In Robotics and Automation (ICRA), 2013 IEEE International Conference on,\npages 1130\u20131137. IEEE, 2013.\n\n[10] Carlo Tomasi and Takeo Kanade. Detection and tracking of point features.\n\nJournal of Computer Vision, 1991.\n\nInternational\n\n[11] Tomer Ullman, Andreas Stuhlm\u00a8uller, Noah Goodman, and Josh Tenenbaum. Learning physics\n\nfrom dynamical scenes. In CogSci, 2014.\n\n[12] Ilker Yildirim, Tejas D Kulkarni, Winrich A Freiwald, and Joshua B Tenenbaum. Ef\ufb01cient\nanalysis-by-synthesis in vision: A computational framework, behavioral tests, and modeling\nneuronal representations. In Thirty-Seventh Annual Conference of the Cognitive Science Soci-\nety, 2015.\n\n[13] Bo Zheng, Yibiao Zhao, Joey C Yu, Katsushi Ikeuchi, and Song-Chun Zhu. Detecting potential\n\nfalling objects by inferring human action and natural disturbance. In ICRA, 2014.\n\n9\n\n\f", "award": [], "sourceid": 75, "authors": [{"given_name": "Jiajun", "family_name": "Wu", "institution": "MIT"}, {"given_name": "Ilker", "family_name": "Yildirim", "institution": "MIT"}, {"given_name": "Joseph", "family_name": "Lim", "institution": "MIT"}, {"given_name": "Bill", "family_name": "Freeman", "institution": "MIT"}, {"given_name": "Josh", "family_name": "Tenenbaum", "institution": "MIT"}]}