{"title": "Sim2real transfer learning for 3D human pose estimation: motion to the rescue", "book": "Advances in Neural Information Processing Systems", "page_first": 12949, "page_last": 12961, "abstract": "Synthetic visual data can provide practicically infinite diversity and rich labels,\nwhile avoiding ethical issues with privacy and bias. However, for many tasks,\ncurrent models trained on synthetic data generalize poorly to real data. The task of\n3D human pose estimation is a particularly interesting example of this sim2real\nproblem, because learning-based approaches perform reasonably well given real\ntraining data, yet labeled 3D poses are extremely difficult to obtain in the wild,\nlimiting scalability. In this paper, we show that standard neural-network approaches,\nwhich perform poorly when trained on synthetic RGB images, can perform well\nwhen the data is pre-processed to extract cues about the person\u2019s motion, notably\nas optical flow and the motion of 2D keypoints. Therefore, our results suggest\nthat motion can be a simple way to bridge a sim2real gap when video is available.\nWe evaluate on the 3D Poses in the Wild dataset, the most challenging modern\nbenchmark for 3D pose estimation, where we show full 3D mesh recovery that is\non par with state-of-the-art methods trained on real 3D sequences, despite training\nonly on synthetic humans from the SURREAL dataset.", "full_text": "Sim2real transfer learning for 3D human pose\n\nestimation: motion to the rescue\n\n\u2217 Deepmind, London\n\n\u2020 VGG, Department of Engineering Science, University of Oxford\n\nCarl Doersch\u2217\n\nAndrew Zisserman\u2217\u2020\n\nAbstract\n\nSynthetic visual data can provide practically in\ufb01nite diversity and rich labels,\nwhile avoiding ethical issues with privacy and bias. However, for many tasks,\ncurrent models trained on synthetic data generalize poorly to real data. The task of\n3D human pose estimation is a particularly interesting example of this sim2real\nproblem, because learning-based approaches perform reasonably well given real\ntraining data, yet labeled 3D poses are extremely dif\ufb01cult to obtain in the wild,\nlimiting scalability. In this paper, we show that standard neural-network approaches,\nwhich perform poorly when trained on synthetic RGB images, can perform well\nwhen the data is pre-processed to extract cues about the person\u2019s motion, notably\nas optical \ufb02ow and the motion of 2D keypoints. Therefore, our results suggest\nthat motion can be a simple way to bridge a sim2real gap when video is available.\nWe evaluate on the 3D Poses in the Wild dataset, the most challenging modern\nbenchmark for 3D pose estimation, where we show full 3D mesh recovery that is\non par with state-of-the-art methods trained on real 3D sequences, despite training\nonly on synthetic humans from the SURREAL dataset.\n\n1\n\nIntroduction\n\n3D pose estimation, especially for humans, is a classic computer vision problem, with applications\nin imitation learning, robotic interaction, and activity understanding. Pose estimation is extremely\nchallenging with objects that are articulated, deformable, or have wide intra-class variation, as is\nthe case with humans. Therefore, state-of-the-art approaches rely on neural networks and learning.\nHowever, learning-based methods are extremely data hungry, and acquiring suf\ufb01cient data in the real\nworld is dif\ufb01cult. First, there is no straightforward way for people to annotate 3D ground truth poses.\nWorse, in settings like industrial warehouses or homes, hundreds of thousands of different object\ntypes may appear, and new objects may arrive at random. Here, even simple labeling will generally\nbe impractical, much less 3D poses. And any time data involves real humans, issues with privacy,\nintellectual property, and bias can become serious obstacles [33, 36, 92].\nSynthetic data, however, provides an answer to all these problems, providing a potentially in\ufb01nite\ndataset where ground-truth properties are easily accessible. In domains with many objects where\nlabeling is impractical, scanning and simulating objects may not be [25, 26]. Furthermore, synthetic\nhumans do not have any privacy or intellectual property concerns [33], and datasets can be balanced\nexactly with respect to sensitive attributes like race, gender, and other physical characteristics,\nminimizing the problems algorithms currently have with bias [36, 92]. Even better, simulations can\nbe made interactive for training robotic policies.\nConsidering all the advantages, why isn\u2019t simulation the dominant approach in computer vision? One\nproblem is that neural networks trained on synthetic data do not necessarily work on real data as well\nas methods trained directly on real data. Thus, even though such \u201csim2real\u201d transfer has performed\nwell in some domains, such as hand tracking [48] or text detection [23], it is rare on the most popular\nbenchmarks like human pose estimation, object classi\ufb01cation, or object detection. Curiously, the\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Given a detected person in a video sequence, our algorithm estimates 3D pose by \ufb01rst\nextracting optical \ufb02ow and keypoint location estimates. We \ufb01nd that neural networks trained on such\ninput can generalize well even when trained purely on synthetic inputs of this type, far better than\nequivalent networks trained on only RGB synthetic images.\n\ncommunity\u2019s gold standard for popular vision algorithms is generally quantitative performance on\nlarge evaluation datasets. In practice, this kind of evaluation is limited to tasks where labels are\neasy to obtain, and for such tasks, it is equally straightforward to create large training sets with\nmatching statistics. In contrast, vision problems like robotic manipulation\u2014where simulation has\nbeen in\ufb02uential\u2014are less popular not because they are unimportant, but because they exist in a sort of\n\u201cevaluation blind spot,\u201d due to the lack of standard benchmarks that can boil real-world performance\ndown to a number. 3D human pose estimation is a rare exception to this trend: manual annotation is\nalmost impossible, yet a small dataset of in-the-wild data is available for evaluation due to clever use\nof external sensors [87], called 3D Poses in the Wild (3DPW). Thus, we set our sights speci\ufb01cally on\nthis problem, as a testbed for understanding how to design algorithms that can learn real-world pose\nestimation from simulation.\nTo train a 3D human pose estimation network, we must \ufb01rst confront the domain gap posed by\nstandard datasets. The synthetic humans video dataset SURREAL [86], for example, lacks deformable\nclothing meshes, realistic lighting, and environmental interaction. While these aspects could be\nimproved via better simulators, the problems with SURREAL are representative of the problems that\nrapidly-scanned real-world objects have: small features may be lost, and physical properties will be\napproximate. Humans have little dif\ufb01culty understanding the 3D structure of SURREAL without any\nprior experience, giving hope that computers can transfer between the domains as well. However, we\n\ufb01nd empirically that na\u00efve transfer for computers is poor from SURREAL to 3DPW.\nOur key insight is that motion, extracted from video sequences, can be a better cue for enabling transfer.\nOur intuition is grounded in the psychology literature, where humans have been shown to extract\nremarkably detailed 3D interpretations when only simple point-light motion is visible [13, 32, 39].\nModern simulations (including SURREAL) explicitly use 3D models which match the 3D geometry\nof real humans, and therefore hypothetically match well in terms of plausible motion.\nArmed with this intuition, we build a system to estimate 3D human poses in real videos. Our core\ncontributions are relatively simple modi\ufb01cations to a standard 3D human pose estimation algorithm\u2014\nHuman Mesh Recovery (HMR) [34]\u2014which greatly improve transfer from simulation to reality.\nSpeci\ufb01cally, we \ufb01rst modify SURREAL to contain more realistic overall motion, for example, by\ncompositing SURREAL humans onto real backgrounds from videos collected in-the-wild. Then we\nadd explicit motion cues, including optical \ufb02ow from FlowNet [15] (also trained with synthetic data),\nand 2D keypoint tracks obtained from an off-the-shelf 2D detector (such as [53], which is trained on\nreal 2D keypoints, and therefore used only at test time, while at train time the 2D keypoints come\nfrom the simulator). We \ufb01nd that both modi\ufb01cations substantially improve performance on 3DPW,\ntracking close to state-of-the-art performance, while adding synthetic RGB inputs can actually harm\nperformance. We also compare to the standard Domain Adversarial Neural Network approach to\ndomain transfer [18], and \ufb01nd relatively marginal bene\ufb01ts compared to motion cues.\n\n2 Related Work\n\nOur work is part of a long line of research that has attempted to use simulation for human 3D pose\nestimation. Principal among these is work on using datasets of synthetic humans for human pose\nestimation [11, 16, 20, 51, 67, 75, 84, 86, 97]. These works generally note that transfer is a challenge,\nand therefore the majority train on real data as well as synthetic using a variety of strategies. For\n\n2\n\nInputFlowKeypointsEstimateRotated Estimate\finstance, some work constructs 3D datasets entirely by stitching together 2D images [67]; other\nworks use feature selection [51] and stage-wise training [16] to improve transfer. Algorithms trained\nentirely on synthetic data often underperform those trained entirely on real data, even when the real\ndatasets are small [86]. An interesting exception is work that uses depth images [72], where sim2real\n3D human pose estimation is effective, although depth cameras are required.\nNumerous other areas of computer vision have made use of synthetic humans with varying success.\nAmong work on 2D pose estimation [59, 61, 69], FlowCap [69] is particularly relevant due to its\nreliance on \ufb02ow, although its model-based optimization renders the algorithm somewhat brittle.\nOther works consider pedestrian detection [59\u201361] and action recognition [62, 63]. 3D hand pose\nestimation [48, 99] and eye tracking [73] are particularly promising, as neural networks trained on\npurely synthetic data are effective, perhaps because the lack of clothing makes appearance easier\nto model. Again, depth has proven useful [49, 76, 80], where state-of-the-art algorithms typically\nincorporate some form of generative model in-the-loop at test time.\nRobotics is a particularly inspiring domain for sim2real research, and here it has been again found that\nmore abstract representations than RGB, such as segmentations [50, 94], can improve performance.\nIn some cases, these abstractions can be obtained automatically from a simulator alone [31]. Other\nworks use generative models that make simulation look more like reality [9, 10], or randomize the\nsimulator to increase the distribution overlap [70, 81]. These works emphasize that sim2real is\nessential: real-world data is impossible to annotate at the level of desired robot commands, and\neven unlabeled data is expensive since a robot can break itself or its environment. These sim2real\nworks in robotics build on a long tradition of domain adaptation for visual data, which can involve\nlearning maps between feature spaces [21], invariant feature extractors [18], or image-to-image\ntranslation [98]; for a review, see [12, 54].\nWe are also not the \ufb01rst to note that optical \ufb02ow can be useful preprocessing for human pose\nestimation: optical \ufb02ow has been used for 2D keypoint estimation [59, 69], part segmentation [38],\nand even 3D pose [4], although the latter work involves \ufb01tting a 3D model to optical \ufb02ow, which is\npotentially slow, sensitive to initialization, and limits robustness. Similarly, some works have noted\nthat 2D keypoints can be useful in 3D interpretation of humans [44] and objects [89]. There is also\nevidence that \ufb02ow can aid sim2real transfer for foreground/background segmentation [82, 83].\nFinally, our work is related to a long tradition of 3D human pose estimation, where learning-based\nmethods have grown recently due to the emergence of motion-capture datasets. One straightforward\napproach is to \u2018lift\u2019 2D poses into 3D, using either dictionaries or direct regression [3, 44, 47, 64,\n79, 85, 88, 93]. Other works regress poses directly from pixels [46, 55, 56, 68, 71, 77, 78, 96],\nwhich generally relies on having a good match between training and testing. Similar to our work,\nstate-of-the-art approaches often rely on parametric body models to incorporate strong priors on 3D\nposes [5, 7, 8, 22, 24, 34, 40, 43, 52, 57, 74]. Among these, we are particularly related to recent\nworks which leverage temporal cues from videos to gain extra information about depth [6, 14, 27, 28,\n35, 41, 58, 90, 91].\n\n3 Algorithm\n\nOur goal is to train a network which can predict N 3D poses given a sequence of N video frames.\nThis means we \ufb01rst require a synthetic dataset with reasonably realistic sequences of human poses,\nwhich we obtain by compositing SURREAL renders onto real, unlabeled scenes. We also need a pose\nestimation model which can properly exploit motion information and, importantly, propagate this\ninformation across frames where no motion is available. For this purpose, we augment the Human\nMesh Recovery (HMR) algorithm to operate on motion, and add memory in the form of a LSTM.\n\n3.1 Dataset Construction\n\nWe require a dataset which captures complex human motion, provides ground-truth 3D poses for each\nframe, and also has all the distractors that are likely to cause problems in real data. Distractors can\ninclude background motion, occluders covering the person, and frames where the person is completely\nmissing. While it is straightforward to get complex human motion following previous datasets such\nas SURREAL [86], which take pose sequences from the CMU motion capture dataset [1] and render\nthem using the SMPL mesh [43], real videos are harder than this. SURREAL people are composited\n\n3\n\n\fFigure 2: Procedure for generating a synthetic training example. Top left: a randomly-chosen Kinetics\nvideo. Bottom left: a SURREAL person with the background removed, to be composited onto the\nKinetics video. Top right: a SLIC superpixel which was generated from the Kinetics video as a\nsynthetic \u2018occluder\u2019. Bottom middle: the composited video. Note that the person\u2019s legs are missing\nbecause they overlapped with the superpixel. Bottom right: we obtain the \ufb01nal training example by\ncropping the composited video around the person, simulating the behavior of a person detector.\n\non static backgrounds, which means that the video version allows for a shortcut for pose estimation:\nit\u2019s straightforward to segment the person from the background by identifying moving pixels. There\nare also no occlusions or missing frames in this dataset.\nTherefore, to construct our dataset, we take humans from the SURREAL dataset and re-composite\nthem onto a background from the large-scale Kinetics dataset [37]. Kinetics videos and SURREAL\nvideos are sampled independently at training time: thus, 68K SURREAL videos may be composited\non roughly 300K kinetics videos for around 20 billion possible combinations. Howeve, na\u00efve\ncompositing following SURREAL\u2014i.e., simply removing the static background and replacing it\nwith a kinetics video\u2014still makes the task too easy, both because there are no occlusions or missing\nframes, and also because the motion of the person won\u2019t match the motion of the camera. Therefore,\nwe modify the SURREAL video before compositing. To solve the motion discrepancy problem, our\n\ufb01rst step is to estimate the camera motion, using off-the-shelf procedures. We then translate the\nperson to follow the camera motion, by offsetting each frame of the SURREAL video by a vector\nequal to the estimated camera motion of the corresponding Kinetics frame.\nWe then follow the procedure shown in Figure 2 to actually construct the video. Our \ufb01rst challenge\nis to simulate occlusions. One approach might be to take occluders from standard segmentation\ndatasets like COCO [42], but this approach has some of the same problems as the SURREAL static\nbackgrounds did. Speci\ufb01cally, COCO consists of static images, and so they don\u2019t have any associated\nmotion information. We might apply random, smooth trajectories to get some motion, but occlusions\nin real video often have internal motion on top of global translation: i.e., the occluding objects may\nbe deformable and have 3D structure. Furthermore, occluders in real videos tend to move with the\nscene. Missing either of these properties will lead to occlusions that are arti\ufb01cially easily identi\ufb01ed\nin synthetic data, which may lead to detectors which generalize poorly on real videos. Therefore,\nour approach is to extract occluders from the Kinetics video itself. We use standard superpixel\nsegmentation (speci\ufb01cally SLIC [2]) to rapidly extract segments from the video, which are generally\nblobs of roughly uniform color tracked throughout the video, resulting in binary masks as shown in\nFigure 2. One superpixel is chosen at random, and then all pixels that overlap with the SURREAL\nperson are removed from the person, resulting in a synthetic occlusion.\nAs a \ufb01nal step, for some videos we randomly occlude the entire person for a small number of frames,\nwhich we call a \u2018total\u2019 occlusion. This simulates a detector failure, which is common in real-world\ndatasets like 3DPW where the person may move out of the frame or be otherwise totally hidden. To\ndo this, we select a continuous chunk of frames and set all feature channels to 0. We add an extra\nchannel to the input which is 1 if the frame is totally occluded in this way, and 0 for un-occluded\nframes. Further implementation details on the dataset construction are given in the supplementary\nmaterial.\n\n4\n\nKinetics VideoSURREAL PersonRandom SLIC SuperpixelCompositeTraining ExampleCrop\fFigure 3: Network architecture for the Motion HMR model. The input is 2D keypoint heatmaps and\noptical \ufb02ow computed on a sequence of human detections (left). These inputs are passed to a Motion\nHMR module (right), which applies a single-frame CNN followed by an LSTM to integrate over\ntime, before independently estimating a pose for each frame. For baseline experiments, the inputs\nmay be removed or replaced with RGB videos. Note, only 4 frames are shown in the schematic for\nclarity, but in practice 31 frames are used.\n\nFigure 4: Our training (left) and evaluation (right) pipelines for the Motion HMR model: each takes a\nstack of person detections and \ufb02ow from a pre-trained module (FlowNet). 2D keypoints come from\nsimulation at train time, vs. a pretrained 2D keypoint detector during evaluation (provided by 3DPW).\n\n3.2 Network Architecture\n\nOur hypothesis is that easily-accessible motion information will be useful for bridging the sim2real\ngap in 3D pose estimation. Therefore, we seek a method to provide motion information to a pose\nestimation model, while otherwise staying close to existing pipelines for comparability. Our starting\npoint is the Human Mesh Recovery (HMR) pipeline [34]. This recent algorithm directly regresses\n3D SMPL poses from pixels, using \ufb01rst a ConvNet (ResNet-50) to obtain a feature vector, and then\napplying an iterative re\ufb01nement algorithm on top of that feature vector to infer the pose.\nA \ufb01rst modi\ufb01cation is required to extend HMR to video. Our input, both at training and test time, is\nshort clips (31 frames in most of our experiments). These may be the raw RGB videos, or the videos\nmay be preprocessed to include other features like 2D keypoints or optical \ufb02ow as described below.\nWe assume that a person detector has already been run, meaning that the sequence tracks a single\nperson whose pose needs to be estimated. A scalable memory architecture is important, because\nnot every frame will be equally discriminative, especially when using motion features on frames\nthat contain little motion. Thus, we need an architecture which can update its beliefs when the pose\nis easily identi\ufb01ed, and otherwise leave them unchanged. We use an LSTM for this purpose (in a\nsimilar manner to [82]). Our architecture, which we call Motion HMR, is shown on the right hand\nside of Figure 3. This architecture applies a CNN, which is a standard ResNet-50, independently\non each frame, average-pooling at the end to obtain a single feature vector per frame. We then pass\nthese features into a bi-directional LSTM that operates in time over the short clips. Finally, we\napply HMR\u2019s iterative pose re\ufb01nement on the output feature vectors from the LSTM for each frame\nindependently. The result is a pose estimate for each frame in the sequence. At training time, we use\nthe simpli\ufb01ed version of the HMR loss function that was proposed for training from Kinetics pseudo\nground truth [6]. That is, we train directly for Procrustes-aligned 3D keypoint location error (rather\nthan SMPL joint angles/absolute 3D keypoint positions), and 2D reprojection error of the 3D pose.\n\n5\n\nMotion HMR2D ConvNetLSTMPosetimeGround Truth2D KeypointsFlowNetMotionHMRPoseTraining Time2DKeypointsFlowNetMotionHMRPoseEvaluation Time\fProviding motion inputs. Given a video-based architecture, we next add motion information. Our\n\ufb01rst strategy is to use optical \ufb02ow. Optical \ufb02ow is already known to transfer well across domains,\nbecause it relies more on similarities between frames than on recognizing speci\ufb01c patterns [15, 17,\n29, 45, 65, 66]. Furthermore, optical \ufb02ow can have strong cues for depth: for example, if one end of a\nrigid body is stationary, but the other end is moving toward the \ufb01rst, then this indicates an out-of-plane\nrotation. We implement this as a simple preprocessing step. That is, we use an off-the-shelf optical\n\ufb02ow algorithm FlowNet [15] as a frozen module, which produces an estimate of optical \ufb02ow at the\nfull resolution of the input sequence.\nOne disadvantage of optical \ufb02ow is that it can become dif\ufb01cult to distinguish body parts from\nbackground. Especially for frames with little motion, the movement of individual limbs will be\nsimply blobs of smooth motion, much like blobs of background. We hypothesize that this is mostly\na problem of 2D part detection, and we note that 2D keypoint detection is a well-studied \ufb01eld. 2D\nkeypoint detection is far easier to annotate than 3D, and even when 2D keypoints are not available,\nsome recent works have argued that 2D correspondence and keypoints can even be obtained in a\nself-supervised manner [30, 95]. 2D keypoints alone do contain some information about 3D pose [44],\nalthough follow-up work has suggested that this approach to using 2D keypoints by itself performs\npoorly for 3D pose estimation on 3DPW [35]. This leads to an interesting research question: if neither\n\ufb02ow information nor 2D keypoints are enough to perform 3D pose estimation, then is it suf\ufb01cient to\nidentify the 2D keypoints, and then use the \ufb02ow and keypoint motion to estimate 3D structure?\nTo answer this question, we provide 2D keypoints as another input to Motion HMR. At training time,\nthese are obtained automatically from the known synthetic pose; at test time, these are the detections\nfrom an automatic 2D keypoint detector (for reproducibility, we use the automatic 2D keypoints\nprovided with the 3DPW dataset). After computing optical \ufb02ow, we concatenate an additional set of\n12 channels to the input image which are keypoint heatmaps. That is, each channel contains zeros\neverywhere except near the keypoint associated with the channel; they are 1 at the keypoint location\nand fall off with a Gaussian distribution with a standard deviation of 10 pixels. For more details on\nthe architecture and training, see the supplementary material.\n\n4 Results\n\nWe apply our trained models to the 3D Poses in the Wild dataset. This dataset is challenging because\nit is shot in real-world environments using handheld cameras, rather than the motion-capture rigs\nof prior 3D pose estimation work. There is non-trivial camera motion, strong lighting variations\n(indoor and outdoor scenes), and substantial clutter, including objects moving in the background and\nocclusions by both objects and humans.\nFollowing prior work [6], we evaluate only on sequences in the test set, and among these, only\non frames where at least 7 keypoints are visible (although all frames are visible to the algorithm).\nWe pass 31-frame clips to the algorithm, the same as at training time, and evaluate the predicted\nposes using the 14 joints that are common to the SMPL and COCO models. We use the standard\nperformance metric PA-MPJPE, which uses the procrustes algorithm to align the poses in 3D before\ncomputing squared error, and we average across each individual person before \ufb01nally averaging\nacross the entire dataset.\nAs a baseline, we also ran Domain Adversarial Neural Networks (DANN) [18], a mainstay of domain\nadaptation, which uses an adversarial network trained to distinguish between the representations\nof real and synthetic images. The trunk of the network is then trained with the negative of this\ndiscriminator loss, resulting in representations that are indistinguishable across domains. While\nstraightforward to implement, DANN is challenging to tune: synthetic images and real ones are\nmapped to overlapping distributions, there is no way to guarantee that the mapping preserves\nsemantics. We apply DANN to the per-frame representations directly before the LSTM.\nTable 1 shows our results, comparing training with and without motion information as input to the\nnetwork. RGB alone performs poorly: a network trained only on short RGB video clips fares worse\nthan one trained on \ufb02ow, despite the relatively uninformative \ufb02ow images. This relationship holds\neven when the RGB-only model is trained with DANN. One possible explanation is that neural\nnetworks tend to rely heavily on texture cues [19]. Synthetic textures are not similar to real ones;\ntherefore the network can localize a synthetic person by distinguishing between sim and real textures.\n\n6\n\n\fTable 1: Results for sim2real transfer for the 3DPW dataset. Lower is better. Methods above the\ndouble line do not train the 3D pose estimator on any real RGB data. We see a substantial boost for\nusing 2D keypoints (also used in Martinez et al.), and non-trivial improvements for using \ufb02ow rather\nthan RGB. For DANN, we ran 5 seeds with varying weight for the DANN loss, and report the best\nseed with the mean in parentheses.\n\nTraining:\n\n3D poses only\n\nfor synthetic data\n\nAlgorithm\nMartinez et al. [44] (from [35])\nRGB Only\nRGB + DANN [18]\nFlow Only (proposed)\nRGB + Keypoints (proposed)\nKeypoints Only (proposed)\nFlow + Keypoints (proposed)\n\nPA-MPJPE\n\n103.0 (107.5)\n\n157.0\n105.6\n\n100.1\n82.4\n77.6\n74.7\n\nTraining:\n3D poses\non real data\n\nHMR [34] (from [6])\nTemporal HMR [35]\nTemporal HMR + InstaVariety [35]\nHMR + Kinetics [6]\n\n77.2\n80.1\n72.4\n72.2\n\nAt test time this is impossible, and failures to identify parts early on can amplify at later layers which\ndo detailed depth estimation.\nKeypoints, on the other hand, perform surprisingly well, validating the intuitions from psychology\nthat 2D motion encapsulates substantial information about 3D activities [32]. It is interesting to\ncompare our keypoints-only results to Martinez et al. [44], which is a comparable algorithm in that it\nonly uses 2D keypoints. There are a number of factors which may explain the relatively poor results\nof [44]. Primarily, [44] is trained on single frames, whereas we use sequences. Furthermore, Martinez\net al. gives relatively little attention to occlusions, resulting in a domain gap relative to 3DPW.\nAnother interesting result is that the network which incorporates RGB on top of keypoints actually\nperforms worse than one that uses only keypoints, further emphasizing the size of the domain gap.\nIt\u2019s likely the simple presence of synthetic textures causes the network to rely on them, and ignore 2D\nmotion cues that are more reliable out-of-domain, but harder to learn. On the other hand, adding \ufb02ow\nto keypoints yields substantial improvements. This suggests that optical \ufb02ow contains information\nabout motion and silhouettes that pure 2D keypoints do not, and furthermore, that the optical \ufb02ow\nestimated on synthetic and real images are a reasonably good match. We conjecture that the \ufb02ow\nfeatures lose low-level texture information that neural networks can easily over\ufb01t to, replacing it with\nsimple piecewise-smooth regions that capture only shape and motion.\nOur \ufb01nal result, of 74.7, is comparable to state-of-the-art works that use similar training pipelines.\nWe outperform HMR [34], which trains on real-world motion capture datasets as well as real-world\n2D images as a regularizer (ensuring that 3D poses are consistent with 2D annotations). In contrast,\nour network is trained only on annotated synthetic images from SURREAL. Even more interesting is\nextensions to HMR that use temporal sequences [35]. While [35] reports pose sequences that are\nsubstantially more coherent, they \ufb01nd that adding temporal information harms the method\u2019s absolute\npose accuracy. While counter-intuitive, we hypothesize that this is due to another form of domain gap:\nspeci\ufb01cally, the 3D datasets that this algorithm was trained on all contain static backgrounds. Thus,\nin the training set, motion is a very strong cue for the person\u2019s location. At test time, however, 3DPW\ncontains substantial background motion, which can confuse the algorithm. Our synthetic pipeline,\non the other hand, allows us to provide realistic background motion. Overall, the only algorithms\nwhich currently outperform us are trained on large, weakly-labeled video datasets [6, 35]. This sort\nof semi-supervised learning on real videos is an interesting avenue for future research, since it would\nallow us to add real videos to our synthetic training set without any manual annotation, and therefore\npotentially boost the performance of our algorithm even further.\nFigure 5 shows qualitative results of the performance of our algorithm on 3DPW scenes. Our\nalgorithm is often robust to both unusual poses and to occlusion, even when the occluders are other\npeople (bottom left). For a qualitative comparison to an RGB baseline, as well as outputs on selected\nwhole videos, see the supplementary material.\n\n7\n\n\fFigure 5: Qualitative results. The input is shown with a magenta detection box identifying the person\nat its center. Blue boxes show the output of our Motion HMR model. We show both the image-aligned\nestimated mesh, and the same mesh rotated by 60 degrees. Note that our algorithm is robust to both\nunusual poses (top left; bottom right) and occlusions (top right; bottom left).\n\nTable 2: Effects of the stages of our dataset generation\npipeline.\n\nTable 3: Effects of changing the clip\nlength used for training and validation.\n\nDataset construction approach\nFull Model\nNo occlusions\nNo background tracking, no occlusions\nStatic background, no occlusions\n\nPA-MPJPE\n\n74.7\n77.2\n80.3\n88.9\n\nClip length\n8\n16\n31\n56\n\nPA-MPJPE\n\n79.0\n77.8\n74.7\n76.2\n\n4.1 Ablations\n\nThe ablation results for the dataset preprocessing steps are given in Table 2. In all cases, we train a\nmodel from scratch using both optical \ufb02ow and 2D keypoints as input. We begin with a model that\nis composited as SURREAL was: we select a single Kinetics frame as background, and composite\nall of the SURREAL images from a sequence onto it. Replacing static backgrounds with moving\nones gives a substantial boost, con\ufb01rming that networks which segment a moving person from a\nstatic background may fail to generalize to dynamic backgrounds. Tracking the background also\nhelps, suggesting that camera motion is a non-trivial artifact in 3DPW. Finally, the boost from using\nocclusions validates that superpixels are a good approximation to the occlusions seen in real videos.\nFinally, we consider the importance of long-term versus short-term motion for our network by varying\nthe length of the clips that were fed into the network and retraining from scratch. Table 3 shows the\nresults. We can see that performance improves until 31 frames, corresponding to roughly 1 second of\nvideo. This isn\u2019t surprising because 3DPW contains clips where people are occasionally standing still.\nHowever, we don\u2019t see any improvement moving to 2 seconds of video. One possible explanation is\nthat errors are accumulating in the LSTM as the sequence length increases, indicating a potential area\nfor future research in architectures. Furthermore, with a batch size of 2, we could only \ufb01t 2 clips in\nGPU memory simultaneously, which reduces the stability of the Batch Norm required by HMR (31\nframes uses batch size 3; 8 and 16 used batch size 6).\n\n5 Conclusions\n\nOur results show that motion information can help neural networks learn 3D human pose estimation\nfrom synthetic images. Human pose estimation is challenging because humans are articulated and\ndeformable, with wide appearance variation, yet they are far from the only thing in the visual world\nlike this. Our results may have wide-ranging applications in, for example, robotics, where both\ncamera motion and object motion (via manipulation) can provide strong cues for object pose. While\nit is somewhat disappointing that neural networks over\ufb01t to the RGB appearance of synthetic images,\nand therefore our \ufb01nal model loses out on cues like shading, it is possible that the advantages of RGB\nmight be recovered through self-supervised learning. That is, we can estimate poses in video using\nsim2real, potentially \ufb01x errors using e.g. bundle adjustment [6], and then train a single-frame RGB\nmodel on the result. Overall, we believe motion information, and the sim2real transfer that it enables,\nmay become an essential component of pose estimation systems whenever video is available.\n\n8\n\n\fAcknowledgements: We thank Konstantinos Bousmalis, Jo\u00e3o Carreira, Ankush Gupta, Mateusz\nMalinowski, Relja Arandjelovi\u00b4c, Jean-Baptiste Alayrac, Viorica P\u02d8atr\u02d8aucean, Jacob Walker, Yuxiang\nZhou, and Anurag Arnab for helpful discussions.\n\nReferences\n\n[1] Carnegie-mellon mocap database, http://mocap.cs.cmu.edu/.\n[2] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. S\u00fcsstrunk. Slic superpixels compared\nto state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine\nintelligence, 34(11):2274\u20132282, 2012.\n\n[3] I. Akhter and M. J. Black. Pose-conditioned joint angle limits for 3D human pose reconstruction.\n\nIn CVPR, 2015.\n\n[4] T. Alldieck, M. Kassubeck, B. Wandt, B. Rosenhahn, and M. Magnor. Optical \ufb02ow-based 3D\nhuman motion estimation from monocular video. In German Conference on Pattern Recognition,\npages 347\u2013360. Springer, 2017.\n\n[5] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. Scape: shape\n\ncompletion and animation of people. ACM TOG, 24(3):408\u2013416, 2005.\n\n[6] A. Arnab, C. Doersch, and A. Zisserman. Exploiting temporal context for 3D human pose\n\nestimation in the wild. In CVPR, 2019.\n\n[7] A. O. Balan, L. Sigal, M. J. Black, J. E. Davis, and H. W. Haussecker. Detailed human shape\n\nand pose from images. In CVPR, pages 1\u20138, 2007.\n\n[8] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black. Keep it SMPL:\nAutomatic estimation of 3D human pose and shape from a single image. In ECCV, Oct. 2016.\n[9] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz,\nP. Pastor, K. Konolige, et al. Using simulation and domain adaptation to improve ef\ufb01ciency of\ndeep robotic grasping. In 2018 IEEE International Conference on Robotics and Automation\n(ICRA), pages 4243\u20134250. IEEE, 2018.\n\n[10] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level\ndomain adaptation with generative adversarial networks. In Proceedings of the IEEE conference\non computer vision and pattern recognition, pages 3722\u20133731, 2017.\n\n[11] W. Chen, H. Wang, Y. Li, H. Su, Z. Wang, C. Tu, D. Lischinski, D. Cohen-Or, and B. Chen. Syn-\nthesizing training images for boosting human 3D pose estimation. In 2016 Fourth International\nConference on 3D Vision (3DV), pages 479\u2013488. IEEE, 2016.\n\n[12] G. Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint\n\narXiv:1702.05374, 2017.\n\n[13] J. E. Cutting and L. T. Kozlowski. Recognizing friends by their walk: Gait perception without\n\nfamiliarity cues. Bulletin of the psychonomic society, 9(5):353\u2013356, 1977.\n\n[14] R. Dabral, A. Mundhada, U. Kusupati, S. Afaque, and A. Jain. Structure-aware and temporally\n\ncoherent 3D human pose estimation. In ECCV, 2018.\n\n[15] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt,\nD. Cremers, and T. Brox. Flownet: Learning optical \ufb02ow with convolutional networks. In\nProceedings of the IEEE international conference on computer vision, pages 2758\u20132766, 2015.\n[16] Y. Du, Y. Wong, Y. Liu, F. Han, Y. Gui, Z. Wang, M. Kankanhalli, and W. Geng. Marker-less\n3D human motion capture with monocular image sequence and height-maps. In European\nConference on Computer Vision, pages 20\u201336. Springer, 2016.\n\n[17] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig. Virtual worlds as proxy for multi-object tracking\nanalysis. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 4340\u20134349, 2016.\n\n[18] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand,\nand V. Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine\nLearning Research, 17(1):2096\u20132030, 2016.\n\n[19] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet-\ntrained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.\nInternational Conference on Learning Representations, 2018.\n\n[20] M. F. Ghezelghieh, R. Kasturi, and S. Sarkar. Learning camera viewpoint using cnn to improve\n3D body pose estimation. In 2016 Fourth International Conference on 3D Vision (3DV), pages\n685\u2013693. IEEE, 2016.\n\n[21] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic \ufb02ow kernel for unsupervised domain\nadaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages\n2066\u20132073. IEEE, 2012.\n\n9\n\n\f[22] P. Guan, A. Weiss, A. O. Balan, and M. J. Black. Estimating human shape and pose from a\n\nsingle image. In ICCV, pages 1381\u20131388. IEEE, 2009.\n\n[23] A. Gupta, A. Vedaldi, and A. Zisserman. Synthetic data for text localisation in natural images.\nIn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n2315\u20132324, 2016.\n\n[24] N. Hasler, H. Ackermann, B. Rosenhahn, T. Thorm\u00e4hlen, and H.-P. Seidel. Multilinear pose and\nbody shape estimation of dressed subjects from image sets. In CVPR, pages 1823\u20131830, 2010.\n[25] S. Hinterstoisser, V. Lepetit, P. Wohlhart, and K. Konolige. On pre-trained image features and\nsynthetic images for deep learning. In Proceedings of the European Conference on Computer\nVision (ECCV), pages 0\u20130, 2018.\n\n[26] S. Hinterstoisser, O. Pauly, H. Heibel, M. Marek, and M. Bokeloh. An annotation saved is an\nannotation earned: Using fully synthetic training for object instance detection. arXiv preprint\narXiv:1902.09967, 2019.\n\n[27] M. R. I. Hossain and J. J. Little. Exploiting temporal information for 3D pose estimation. In\n\nECCV, 2018.\n\n[28] Y. Huang, F. Bogo, C. Classner, A. Kanazawa, P. V. Gehler, I. Akhter, and M. J. Black. Towards\naccurate marker-less human shape and pose estimation over time. In 3DV, pages 421\u2013430,\n2017.\n\n[29] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of\noptical \ufb02ow estimation with deep networks. In Proceedings of the IEEE conference on computer\nvision and pattern recognition, pages 2462\u20132470, 2017.\n\n[30] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks\nthrough conditional image generation. In Advances in Neural Information Processing Systems,\npages 4016\u20134027, 2018.\n\n[31] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell,\nand K. Bousmalis. Sim-to-real via sim-to-sim: Data-ef\ufb01cient robotic grasping via randomized-\nto-canonical adaptation networks. arXiv preprint arXiv:1812.07252, 2018.\n\n[32] G. Johansson. Visual perception of biological motion and a model for its analysis. Perception\n\n& psychophysics, 14(2):201\u2013211, 1973.\n\n[33] M. I. Jordan and T. M. Mitchell. Machine learning: Trends, perspectives, and prospects. Science,\n\n[34] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-to-end recovery of human shape\n\n[35] A. Kanazawa, J. Zhang, P. Felsen, and J. Malik. Learning 3D human dynamics from video. In\n\n349(6245):255\u2013260, 2015.\n\nand pose. In CVPR, 2018.\n\nCVPR, 2019.\n\n[36] M. Kay, C. Matuszek, and S. A. Munson. Unequal representation and gender stereotypes in\nimage search results for occupations. In Proceedings of the 33rd Annual ACM Conference on\nHuman Factors in Computing Systems, pages 3819\u20133828. ACM, 2015.\n\n[37] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green,\nT. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset.\narXiv preprint arXiv:1705.06950, 2017.\n\n[38] J. Kim, K. Joo, T.-H. Oh, and I. S. Kweon. Human body part classi\ufb01cation from optical \ufb02ow. In\n2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI),\npages 903\u2013904. IEEE, 2016.\n\n[39] L. T. Kozlowski and J. E. Cutting. Recognizing the sex of a walker from a dynamic point-light\n\ndisplay. Perception & psychophysics, 21(6):575\u2013580, 1977.\n\n[40] C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J. Black, and P. V. Gehler. Unite the people:\n\nClosing the loop between 3D and 2D human representations. In CVPR, 2017.\n\n[41] Z. Li, T. Dekel, F. Cole, R. Tucker, N. Snavely, C. Liu, and W. T. Freeman. Learning the depths\n\nof moving people by watching frozen people. arXiv preprint arXiv:1904.11111, 2019.\n\n[42] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll\u00e1r, and C. L. Zitnick.\nMicrosoft coco: Common objects in context. In European conference on computer vision, pages\n740\u2013755. Springer, 2014.\n\n[43] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl: A skinned multi-\n\nperson linear model. ACM TOG, 34(6):248:1\u2013248:16, Oct. 2015.\n\n[44] J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective baseline for 3D\n\nhuman pose estimation. In ICCV, 2017.\n\n[45] N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large\ndataset to train convolutional networks for disparity, optical \ufb02ow, and scene \ufb02ow estimation.\nIn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n4040\u20134048, 2016.\n\n10\n\n\f[46] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt. Monocular\n3D human pose estimation in the wild using improved cnn supervision. In 3DV. IEEE, 2017.\n[47] F. Moreno-Noguer. 3D human pose estimation from a single image via distance matrix regres-\n\nsion. In CVPR, pages 1561\u20131570. IEEE, 2017.\n\n[48] F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, and C. Theobalt.\nGanerated hands for real-time 3D hand tracking from monocular rgb. In Proceedings of the\nIEEE Conference on Computer Vision and Pattern Recognition, pages 49\u201359, 2018.\n\n[49] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and C. Theobalt. Real-time\nhand tracking under occlusion from an egocentric rgb-d sensor. In Proceedings of the IEEE\nInternational Conference on Computer Vision, pages 1284\u20131293, 2017.\n\n[50] M. M\u00fcller, A. Dosovitskiy, B. Ghanem, and V. Koltun. Driving policy transfer via modularity\n\nand abstraction. arXiv preprint arXiv:1804.09364, 2018.\n\n[51] R. Okada and S. Soatto. Relevant feature selection for human pose estimation and localization\nin cluttered images. In European Conference on Computer Vision, pages 434\u2013445. Springer,\n2008.\n\n[52] M. Omran, C. Lassner, G. Pons-Moll, P. V. Gehler, and B. Schiele. Neural body \ufb01tting: Unifying\ndeep learning and model-based human pose and shape estimation. In 3DV, Verona, Italy, 2018.\n[53] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy.\nIn Proceedings of the IEEE\n\nTowards accurate multi-person pose estimation in the wild.\nConference on Computer Vision and Pattern Recognition, pages 4903\u20134911, 2017.\n\n[54] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa. Visual domain adaptation: A survey of recent\n\nadvances. IEEE signal processing magazine, 32(3):53\u201369, 2015.\n\n[55] G. Pavlakos, X. Zhou, and K. Daniilidis. Ordinal depth supervision for 3D human pose\n\nestimation. In CVPR, 2018.\n\n[56] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-\ufb01ne volumetric prediction\n\nfor single-image 3D human pose. In CVPR, pages 1263\u20131272, 2017.\n\n[57] G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis. Learning to estimate 3D human pose and\n\nshape from a single color image. In CVPR, 2018.\n\n[58] X. B. Peng, A. Kanazawa, J. Malik, P. Abbeel, and S. Levine. Sfv: Reinforcement learning of\n\nphysical skills from videos. arXiv preprint arXiv:1810.03599, 2018.\n\n[59] L. Pishchulin, A. Jain, M. Andriluka, T. Thorm\u00e4hlen, and B. Schiele. Articulated people\ndetection and pose estimation: Reshaping the future. In 2012 IEEE Conference on Computer\nVision and Pattern Recognition, pages 3178\u20133185. IEEE, 2012.\n\n[60] L. Pishchulin, A. Jain, C. Wojek, M. Andriluka, T. Thorm\u00e4hlen, and B. Schiele. Learning\npeople detection models from few training samples. In CVPR 2011, pages 1473\u20131480. IEEE,\n2011.\n\n[61] W. Qiu. Generating human images and ground truth using computer graphics. PhD thesis,\n\nUCLA, 2016.\n\n[62] H. Rahmani and A. Mian. Learning a non-linear knowledge transfer model for cross-view\naction recognition. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, pages 2458\u20132466, 2015.\n\n[63] H. Rahmani and A. Mian. 3D action recognition from novel viewpoints. In Proceedings of the\n\nIEEE Conference on Computer Vision and Pattern Recognition, pages 1506\u20131515, 2016.\n\n[64] V. Ramakrishna, T. Kanade, and Y. Sheikh. Reconstructing 3D human pose from 2D image\n\nlandmarks. In ECCV, pages 573\u2013586. Springer, 2012.\n\n[65] A. Ranjan and M. J. Black. Optical \ufb02ow estimation using a spatial pyramid network.\n\nIn\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n4161\u20134170, 2017.\n\n[66] A. Ranjan, J. Romero, and M. J. Black. Learning human optical \ufb02ow. British Machine Vision\n\nConference, 2018.\n\n[67] G. Rogez and C. Schmid. Mocap-guided data augmentation for 3D pose estimation in the wild.\n\nIn Advances in neural information processing systems, pages 3108\u20133116, 2016.\n\n[68] G. Rogez, P. Weinzaepfel, and C. Schmid. LCR-net: Localization-classi\ufb01cation-regression for\n\nhuman pose. In CVPR, 2017.\n\n[69] J. Romero, M. Loper, and M. J. Black. Flowcap: 2d human pose from optical \ufb02ow. In German\n\nconference on pattern recognition, pages 412\u2013423. Springer, 2015.\n\n[70] F. Sadeghi and S. Levine. CAD2RL: Real single-image \ufb02ight without a single real image. In\n\nRobotics: Science and Systems Conference, 2016.\n\n[71] I. S\u00e1r\u00e1ndi, T. Linder, K. O. Arras, and B. Leibe. How robust is 3D human pose estimation to\n\nocclusion? In arXiv preprint arXiv:1808.09316, 2018.\n\n11\n\n\f[72] J. Shotton, A. W. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and\nA. Blake. Real-time human pose recognition in parts from single depth images. In Cvpr,\nvolume 2, page 3, 2011.\n\n[73] A. Shrivastava, T. P\ufb01ster, O. Tuzel, J. Susskind, W. Wang, and R. Webb. Learning from\nsimulated and unsupervised images through adversarial training. In Proceedings of the IEEE\nconference on computer vision and pattern recognition, pages 2107\u20132116, 2017.\n\n[74] L. Sigal, A. Balan, and M. J. Black. Combined discriminative and generative articulated pose\n\nand non-rigid shape estimation. In NIPS, pages 1337\u20131344, 2008.\n\n[75] C. Sminchisescu, A. Kanaujia, and D. Metaxas. Learning joint top-down and bottom-up\nprocesses for 3D visual inference. In 2006 IEEE Computer Society Conference on Computer\nVision and Pattern Recognition (CVPR\u201906), volume 2, pages 1743\u20131752. IEEE, 2006.\n\n[76] S. Sridhar, F. Mueller, A. Oulasvirta, and C. Theobalt. Fast and robust hand tracking using\ndetection-guided optimization. In Proceedings of the IEEE Conference on Computer Vision and\nPattern Recognition, pages 3213\u20133221, 2015.\n\n[77] X. Sun, J. Shang, S. Liang, and Y. Wei. Compositional human pose regression. In ICCV, 2017.\n[78] X. Sun, B. Xiao, S. Liang, and Y. Wei. Integral human pose regression. In ECCV, 2018.\n[79] C. J. Taylor. Reconstruction of articulated objects from point correspondences in a single\n\nuncalibrated image. CVIU, 80(3):349\u2013363, 2000.\n\n[80] J. Taylor, L. Bordeaux, T. Cashman, B. Corish, C. Keskin, T. Sharp, E. Soto, D. Sweeney,\nJ. Valentin, B. Luff, et al. Ef\ufb01cient and precise interactive hand tracking through joint, continuous\noptimization of pose and correspondences. ACM Transactions on Graphics (TOG), 35(4):143,\n2016.\n\n[81] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization\nfor transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ\nInternational Conference on Intelligent Robots and Systems (IROS), pages 23\u201330. IEEE, 2017.\n[82] P. Tokmakov, K. Alahari, and C. Schmid. Learning video object segmentation with visual\nmemory. In Proceedings of the IEEE International Conference on Computer Vision, pages\n4481\u20134490, 2017.\n\n[83] P. Tokmakov, C. Schmid, and K. Alahari. Learning to segment moving objects. International\n\nJournal of Computer Vision, 127(3):282\u2013301, 2019.\n\n[84] H.-Y. Tung, H.-W. Tung, E. Yumer, and K. Fragkiadaki. Self-supervised learning of motion\n\ncapture. In Advances in Neural Information Processing Systems, pages 5236\u20135246, 2017.\n\n[85] J. Valmadre and S. Lucey. Deterministic 3D human pose estimation using rigid structure. In\n\nECCV, 2010.\n\n[86] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev, and C. Schmid. Learning\n\nfrom synthetic humans. In CVPR, 2017.\n\n[87] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate\n\n3D human pose in the wild using imus and a moving camera. In ECCV, 2018.\n\n[88] C. Wang, Y. Wang, Z. Lin, A. L. Yuille, and W. Gao. Robust estimation of 3D human poses\n\nfrom a single image. In CVPR, pages 2361\u20132368, 2014.\n\n[89] J. Wu, T. Xue, J. J. Lim, Y. Tian, J. B. Tenenbaum, A. Torralba, and W. T. Freeman. Single\nimage 3D interpreter network. In European Conference on Computer Vision, pages 365\u2013382.\nSpringer, 2016.\n\n[90] A. Zan\ufb01r, E. Marinoiu, and C. Sminchisescu. Monocular 3D pose and shape estimation of\nmultiple people in natural scenes\u2013the importance of multiple scene constraints. In CVPR, 2018.\n[91] X. Zhang, T. Dekel, T. Xue, A. Owens, Q. He, J. Wu, S. Mueller, and W. T. Freeman. Mosculp:\n\nInteractive visualization of shape and time. In arXiv preprint arXiv:1809.05491, 2018.\n\n[92] J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang. Men also like shopping: Reducing\ngender bias ampli\ufb01cation using corpus-level constraints. arXiv preprint arXiv:1707.09457,\n2017.\n\n[93] R. Zhao, Y. Wang, and A. M. Martinez. A simple, fast and highly-accurate algorithm to recover\n\n3D shape from 2D landmarks on a single image. PAMI, 2017.\n\n[94] B. Zhou, P. Kr\u00e4henb\u00fchl, and V. Koltun. Does computer vision matter for action? arXiv preprint\n\narXiv:1905.12887, 2019.\n\n[95] T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, and A. A. Efros. Learning dense correspondence\nvia 3D-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision\nand Pattern Recognition, pages 117\u2013126, 2016.\n\n[96] X. Zhou, Q. Huang, X. Sun, X. Xue, and Y. Wei. Towards 3D human pose estimation in the\n\nwild: a weakly-supervised approach. In ICCV, 2017.\n\n[97] X. Zhou, M. Zhu, S. Leonardos, K. G. Derpanis, and K. Daniilidis. Sparseness meets deepness:\n3D human pose estimation from monocular video. In Proceedings of the IEEE conference on\n\n12\n\n\fcomputer vision and pattern recognition, pages 4966\u20134975, 2016.\n\n[98] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using\ncycle-consistent adversarial networks. In Proceedings of the IEEE international conference on\ncomputer vision, pages 2223\u20132232, 2017.\n\n[99] C. Zimmermann and T. Brox. Learning to estimate 3D hand pose from single rgb images.\nIn Proceedings of the IEEE International Conference on Computer Vision, pages 4903\u20134911,\n2017.\n\n13\n\n\f", "award": [], "sourceid": 7092, "authors": [{"given_name": "Carl", "family_name": "Doersch", "institution": "DeepMind"}, {"given_name": "Andrew", "family_name": "Zisserman", "institution": "DeepMind & University of Oxford"}]}