{"title": "MarrNet: 3D Shape Reconstruction via 2.5D Sketches", "book": "Advances in Neural Information Processing Systems", "page_first": 540, "page_last": 550, "abstract": "3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenge for learning-based approaches, as 3D object annotations in real images are scarce. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from the domain adaptation issue when tested on real data.  In this work, we propose an end-to-end trainable framework, sequentially estimating 2.5D sketches and 3D object shapes. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image, and to transfer from synthetic to real data. Second, for 3D reconstruction from the 2.5D sketches, we can easily transfer the learned model on synthetic data to real images, as rendered 2.5D sketches are invariant to object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches, making the framework end-to-end trainable on real images, requiring no real-image annotations. Our framework achieves state-of-the-art performance on 3D shape reconstruction.", "full_text": "MarrNet: 3D Shape Reconstruction via 2.5D Sketches\n\nJiajun Wu*\nMIT CSAIL\n\nYifan Wang*\n\nShanghaiTech University\n\nTianfan Xue\nMIT CSAIL\n\nXingyuan Sun\n\nShanghai Jiao Tong University\n\nWilliam T. Freeman\n\nMIT CSAIL, Google Research\n\nJoshua B. Tenenbaum\n\nMIT CSAIL\n\nAbstract\n\n3D object reconstruction from a single image is a highly under-determined prob-\nlem, requiring strong prior knowledge of plausible 3D shapes. This introduces\nchallenges for learning-based approaches, as 3D object annotations are scarce in\nreal images. Previous work chose to train on synthetic data with ground truth 3D\ninformation, but suffered from domain adaptation when tested on real data.\nIn this work, we propose MarrNet, an end-to-end trainable model that sequentially\nestimates 2.5D sketches and 3D object shape. Our disentangled, two-step formula-\ntion has three advantages. First, compared to full 3D shape, 2.5D sketches are much\neasier to be recovered from a 2D image; models that recover 2.5D sketches are also\nmore likely to transfer from synthetic to real data. Second, for 3D reconstruction\nfrom 2.5D sketches, systems can learn purely from synthetic data. This is because\nwe can easily render realistic 2.5D sketches without modeling object appearance\nvariations in real images, including lighting, texture, etc. This further relieves the\ndomain adaptation problem. Third, we derive differentiable projective functions\nfrom 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on\nreal images, requiring no human annotations. Our model achieves state-of-the-art\nperformance on 3D shape reconstruction.\n\n1\n\nIntroduction\n\nHumans quickly recognize 3D shapes from a single image. Figure 1a shows a number of images\nof chairs; despite their drastic difference in object texture, material, environment lighting, and\nbackground, humans easily recognize they have very similar 3D shapes. What is the most essential\ninformation that makes this happen?\nResearchers in human perception argued that our 3D perception could rely on recovering 2.5D\nsketches [Marr, 1982], which include intrinsic images [Barrow and Tenenbaum, 1978, Tappen et al.,\n2003] like depth and surface normal maps (Figure 1b). Intrinsic images disentangle object appearance\nvariations in texture, albedo, lighting, etc., with its shape, which retains all information from the\nobserved image for 3D reconstruction. Humans further combine 2.5D sketches and a shape prior\nlearned from past experience to reconstruct a full 3D shape (Figure 1c). In the \ufb01eld of computer\nvision, there have also been abundant works exploiting the idea for reconstruction 3D shapes of\nfaces [Kemelmacher-Shlizerman and Basri, 2011], objects [Tappen et al., 2003], and scenes [Hoiem\net al., 2005, Saxena et al., 2009].\nRecently, researchers attempted to tackle the problem of single-image 3D reconstruction with deep\nlearning. These approaches usually regress a 3D shape from a single RGB image directly [Tulsiani\net al., 2017, Choy et al., 2016, Wu et al., 2016b]. In contrast, we propose a two-step while end-to-end\ntrainable pipeline, sequentially recovering 2.5D sketches (depth and normal maps) and a 3D shape.\n\n\u2217 indicates equal contributions.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fFigure 1: Objects in real images (a) are subject to appearance variations regarding color, texture,\nlighting, material, background, etc. Despite this, their 2.5D sketches like surface normal and depth\nmaps remain constant (b). The 2.5D sketches can be seen as an abstraction of the image, retaining all\ninformation about the 3D shape of the object inside. We combine the sketches with learned shape\npriors to reconstruct the full 3D shape (c).\n\nWe use an encoder-decoder structure for each component of the framework, and also enforce the\nreprojection consistency between the estimated sketch and the 3D shape. We name it MarrNet, for its\nclose resemblance to David Marr\u2019s theory of perception [Marr, 1982].\nOur approach offers several unique advantages. First, the use of 2.5D sketches releases the burden on\ndomain transfer. As single-image 3D reconstruction is a highly under-constrained problem, strong\nprior knowledge of object shapes is needed. This poses challenges to learning-based methods, as\naccurate 3D object annotations in real images are rare. Most previous methods turned to training\npurely on synthetic data [Tulsiani et al., 2017, Choy et al., 2016, Girdhar et al., 2016]. However, these\napproaches often suffer from the domain adaption issue due to imperfect rendering. Learning 2.5D\nsketches from images, in comparison, is much easier and more robust to transfer from synthetic to\nreal images, as shown in Section 4.\nFurther, as our second step recovers 3D shape from 2.5D sketches \u2014 an abstraction of the raw input\nimage, it can be trained purely relying on synthetic data. Though rendering diverse realistic images is\nchallenging, it is straightforward to obtain almost perfect object surface normals and depths from a\ngraphics engine. This further relieves the domain adaptation issue.\nWe also enforce differentiable constraints between 2.5D sketches and 3D shape, making our system\nend-to-end trainable, even on real images without any annotations. Given a set of unlabeled images,\nour algorithm, pre-trained on synthetic data, can infer the 2.5D sketches of objects in the image,\nand use it to re\ufb01ne its estimation of objects\u2019 3D shape. This self-supervised feature enhances its\nperformance on images from different domains.\nWe evaluate our framework on both synthetic images of objects from ShapeNet [Chang et al., 2015],\nand real images from the PASCAL 3D+ dataset [Xiang et al., 2014]. We demonstrate that our\nframework performs well on 3D shape reconstruction, both qualitatively and quantitatively.\nOur contributions are three-fold: inspired by visual cognition theory, we propose a two-step, dis-\nentangled formulation for single-image 3D reconstruction via 2.5D sketches; we develop a novel,\nend-to-end trainable model with a differentiable projection layer that ensures consistency between\n3D shape and mid-level representations; we demonstrate its effectiveness on 2.5D sketch transfer and\n3D shape reconstruction on both synthetic and real data.\n\n2 Related Work\n\n2.5D Sketch Recovery Estimating 2.5D sketches has been a long-standing problem in computer\nvision. In the past, researchers have explored recovering 2.5D shape from shading, texture, or color\nimages [Horn and Brooks, 1989, Zhang et al., 1999, Tappen et al., 2003, Barron and Malik, 2015,\nWeiss, 2001, Bell et al., 2014]. With the development of depth sensors [Izadi et al., 2011] and larger\nscale RGB-D datasets [Silberman et al., 2012, Song et al., 2017, McCormac et al., 2017], there\nhave also been papers on estimating depth [Chen et al., 2016, Eigen and Fergus, 2015], surface\nnormals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Shi et al., 2017,\n\n2\n\n(a) Images(b) 2.5D Sketches(c) 3D Shape\fFigure 2: Our model (MarrNet) has three major components: (a) 2.5D sketch estimation, (b) 3D\nshape estimation, and (c) a loss function for reprojection consistency. MarrNet \ufb01rst recovers object\nnormal, depth, and silhouette images from an RGB image. It then regresses the 3D shape from the\n2.5D sketches. In both steps, it uses an encoding-decoding network. It \ufb01nally employs a reprojection\nconsistency loss to ensure the estimated 3D shape aligns with the 2.5D sketches. The entire framework\ncan be trained end-to-end.\n\nJanner et al., 2017] with deep networks. Our method employs 2.5D estimation as a component, but\ntargets reconstructing full 3D shape of an object.\n\nSingle-Image 3D Reconstruction The problem of recovering object shape from a single image\nis challenging, as it requires both powerful recognition systems and prior shape knowledge. With\nthe development of large-scale shape repository like ShapeNet [Chang et al., 2015], researchers\ndeveloped models encoding shape prior for this task [Girdhar et al., 2016, Choy et al., 2016, Tulsiani\net al., 2017, Wu et al., 2016b, Kar et al., 2015, Kanazawa et al., 2016, Soltani et al., 2017], with\nextension to scenes [Song et al., 2017]. These methods typically regress a voxelized 3D shape\ndirectly from an input image, and rely on synthetic data or 2D masks for training. In comparison, our\nformulation tackles domain difference better, as it can be end-to-end \ufb01ne-tuned on images without\nany annotations.\n\n2D-3D Consistency It is intuitive and practically helpful to constrain the reconstructed 3D shape\nto be consistent with 2D observations. Researchers have explored this idea for decades [Lowe, 1987].\nThis idea is also widely used in 3D shape completion from depths or silhouettes [Firman et al.,\n2016, Rock et al., 2015, Dai et al., 2017]. Recently, a few papers discussed enforcing differentiable\n2D-3D constraints between shape and silhouettes, enabling joint training of deep networks for 3D\nreconstruction [Wu et al., 2016a, Yan et al., 2016, Rezende et al., 2016, Tulsiani et al., 2017]. In our\npaper, we exploit this idea to develop differentiable constraints on the consistency between various\n2.5D sketches and 3D shape.\n\n3 Approach\n\nTo recover the 3D structure from a single view RGB image, our MarrNet contains three parts: \ufb01rst, a\n2.5D sketch estimator, which predicts the depth, surface normal, and silhouette images of the object\n(Figure 2a); second, a 3D shape estimator, which infers 3D object shape using a voxel representation\n(Figure 2b); third, a reprojection consistency function, enforcing the alignment between the estimated\n3D structure and inferred 2.5D sketches (Figure 2c).\n\n3.1\n\n2.5D Sketch Estimation\n\nThe \ufb01rst component of our network (Figure 2a) takes a 2D RGB image as input, and predicts its 2.5D\nsketch: surface normal, depth, and silhouette. The goal of the 2.5D sketch estimation step is to distill\nintrinsic object properties from input images, while discarding properties that are non-essential for\nthe task of 3D reconstruction, such as object texture and lighting.\nWe use an encoder-decoder network architecture for 2.5D sketch estimation. Our encoder is a\nResNet-18 [He et al., 2015], encoding a 256\u00d7256 RGB image into 512 feature maps of size 8\u00d78.\n\n3\n\n3D Shape(a) 2.5D Sketch Estimation(b) 3D Shape Estimation(c) ReprojectionConsistency\u2026\u20262.5D Sketches2D ImagenormaldepthsilhouetteNormal Ball\fFigure 3: Reprojection consistency between 2.5D sketches and 3D shape. Left and middle: the\ncriteria for depths and silhouettes; right: the criterion for surface normals. See Section 3.3 for details.\n\nThe decoder contains four sets of 5\u00d75 fully convolutional and ReLU layers, followed by four sets\nof 1\u00d71 convolutional and ReLU layers. It outputs the corresponding depth, surface normal, and\nsilhouette images, also at the resolution of 256\u00d7256.\n\n3.2\n\n3D Shape Estimation\n\nThe second part of our framework (Figure 2b) infers 3D object shape from estimated 2.5D sketches.\nHere, the network focuses on learning the shape prior that explains input well. As it takes only surface\nnormal and depth images as input, it can be trained on synthetic data, without suffering from the\ndomain adaption problem: it is straightforward to render nearly perfect 2.5D sketches, but much\nharder to render realistic images.\nThe network architecture is inspired by the TL network [Girdhar et al., 2016], and the 3D-VAE-\nGAN [Wu et al., 2016b], again with an encoding-decoding style. It takes a normal image and a depth\nimage as input (both masked by the estimated silhouette), maps them to a 200-dim vector via \ufb01ve sets\nof convolutional, ReLU, and pooling layers, followed by two fully connected layers. The detailed\nencoder structure can be found in Girdhar et al. [2016]. The vector then goes through a decoder,\nwhich consists of \ufb01ve fully convolutional and ReLU layers to output a 128\u00d7128\u00d7128 voxel-based\nreconstruction of the input. The detailed encoder structure can be found in Wu et al. [2016b].\n\n3.3 Reprojection Consistency\n\nThere have been works attempting to enforce the consistency between estimated 3D shape and 2D\nrepresentations in a neural network [Yan et al., 2016, Rezende et al., 2016, Wu et al., 2016a, Tulsiani\net al., 2017]. Here, we explore novel ways to include a reprojection consistency loss between the\npredicted 3D shape and the estimated 2.5D sketch, consisting of a depth reprojection loss and a\nsurface normal reprojection loss.\nWe use vx,y,z to represent the value at position (x, y, z) in a 3D voxel grid, assuming that vx,y,z \u2208\n[0, 1], \u2200x, y, z. We use dx,y to denote the estimated depth at position (x, y), and nx,y = (na, nb, nc)\nto denote the estimated surface normal. We assume orthographic projection in this work.\nDepths The projected depth loss tries to guarantee that the voxel with depth vx,y,dx,y should be 1,\nand all voxels in front of it should be 0. This ensures the estimated 3D shape matches the estimated\ndepth values.\nAs illustrated in Figure 3a, we de\ufb01ne projected depth loss as follows:\n\nLdepth(x, y, z) =\n\nThe gradients are\n\n\u2202Ldepth(x, y, z)\n\n\u2202vx,y,z\n\n=\n\nx,y,z,\n(1 \u2212 vx,y,z)2,\n0,\n\n\uf8f1\uf8f2\uf8f3v2\n\uf8f1\uf8f2\uf8f32vx,y,z,\n\n2(vx,y,z \u2212 1),\n0,\n\n4\n\nz < dx,y\nz = dx,y\nz > dx,y\n\n.\n\nz < dx,y\nz = dx,y\nz > dx,y\n\n.\n\n(1)\n\n(2)\n\nVoxels that should be 1 for reprojecteddepth consistencyVoxels that should be 0 for reprojecteddepth consistencyVoxels that should be 1 for reprojectedsurface normal consistency\ud835\udc67=4\ud835\udc67=\u221e\ud835\udc5b=\ud835\udc67\ud835\udc67\ud835\udc67\fWhen dx,y = \u221e, our depth criterion reduces to a special case \u2014 the silhouette criterion. As shown\nin Figure 3b, for a line that has no intersection with the shape, all voxels in it should be 0.\nSurface Normals As vectors nx = (0,\u2212nc, nb) and ny = (\u2212nc, 0, na) are orthogonal to the nor-\nx = (0,\u22121, nb/nc)\nmal vector nx,y = (na, nb, nc), we can normalize them to obtain two vectors, n(cid:48)\ny = (\u22121, 0, na/nc), both on the estimated surface plane at (x, y, z). The projected surface\nand n(cid:48)\nnormal loss tries to guarantee that the voxels at (x, y, z) \u00b1 n(cid:48)\nx and (x, y, z) \u00b1 n+y should be 1 to\nmatch the estimated surface normals. These constraints only apply when the target voxels are inside\nthe estimated silhouette.\nAs shown in Figure 3c, let z = dx,y, the projected surface normal loss is de\ufb01ned as\n\n(cid:17)2\n(cid:17)2\n\n+\n\n+\n\n(cid:17)2\n(cid:17)2\n\n+\n\n.\n\n(cid:16)\n(cid:16)\n1 \u2212 vx,y+1,z\u2212 nb\n1 \u2212 vx+1,y,z\u2212 na\n(cid:16)\n\nnc\n\nnc\n\nand\n\n\u2202Lnormal(x, y, z)\n\u2202vx+1,y,z\u2212 na\n\nnc\n\n= 2\n\nvx+1,y,z\u2212 na\n\nnc\n\n(3)\n\n(cid:17)\n\n.\n\n\u2212 1\n\n(4)\n\nLnormal(x, y, z) =\n\n(cid:16)\n(cid:16)\n\n1 \u2212 vx,y\u22121,z+ nb\n1 \u2212 vx\u22121,y,z+ na\n\nnc\n\nnc\n\nThen the gradients along the x direction are\n\u2212 1\n\nvx\u22121,y,z+ na\n\n\u2202Lnormal(x, y, z)\n\u2202vx\u22121,y,z+ na\n\n= 2\n\nnc\n\nnc\n\n(cid:16)\n\n(cid:17)\n\nThe gradients along the y direction are similar.\n\n3.4 Training Paradigm\n\nWe employ a two-step training paradigm. We \ufb01rst train the 2.5D sketch estimation and the 3D shape\nestimation components separately on synthetic images; we then \ufb01ne-tune the network on real images.\nFor pre-training, we use synthetic images of ShapeNet objects. The 2.5D sketch estimator is trained\nusing the ground truth surface normal, depth, and silhouette images with a L2 loss. The 3D interpreter\nis trained using ground truth voxels and a cross-entropy loss. Please see Section 4.1 for details on\ndata preparation.\nThe reprojection consistency loss is used to \ufb01ne-tune the 3D estimation component of our model on\nreal images, using the predicted normal, depth, and silhouette. We observe that a straightforward\nimplementation leads to shapes that explain 2.5D sketches well, but with unrealistic appearance. This\nis because the 3D estimation module over\ufb01ts the images without preserving the learned 3D shape\nprior. See Figure 5 for examples, and Section 4.2 for more details.\nWe therefore choose to \ufb01x the decoder of the 3D estimator and only \ufb01ne-tune the encoder. During\ntesting, our method can be self-supervised, i.e., we can \ufb01ne-tune even on a single image without any\nannotations. In practice, we \ufb01ne-tune our model separately on each image for 40 iterations. For each\ntest image, \ufb01ne-tuning takes up to 10 seconds on a modern GPU; without \ufb01ne-tuning, testing time\nis around 100 milliseconds. We use SGD for optimization with a batch size of 4, a learning rate of\n0.001, and a momentum of 0.9. We implemented our framework in Torch7 [Collobert et al., 2011].\n\n4 Evaluation\n\nIn this section, we present both qualitative and quantitative results on single-image 3D reconstruction\nusing variants of our framework. We evaluate our entire framework on both synthetic and real-life\nimages on three datasets.\n\n4.1\n\n3D Reconstruction on ShapeNet\n\nData We start with experiments on synthesized images of ShapeNet chairs [Chang et al., 2015]. We\nput objects in front of random backgrounds from the SUN database [Xiao et al., 2010], and render the\ncorresponding RGB, depth, surface normal, and silhouette images. We use a physics-based renderer,\nMitsuba [Jakob, 2010], to obtain more realistic images. For each of the 6,778 ShapeNet chairs, we\nrender 20 images of random viewpoints.\n\n5\n\n\fFigure 4: Results on rendered images of ShapeNet objects [Chang et al., 2015]. From left to right:\ninput, estimated normal map, estimated depth map, our prediction, a baseline algorithm that predicts\n3D shape directly from RGB input without modeling 2.5D sketch, and ground truth. Both normal and\ndepth maps are masked by predicted silhouettes. Our method is able to recover shapes with smoother\nsurfaces and \ufb01ner details.\n\nMethods We follow the training paradigm described in Section 3.4, but without the \ufb01nal \ufb01ne-tuning\nstage, as ground truth 3D shapes are available on this synthetic dataset. Speci\ufb01cally, the 2.5D sketch\nestimator is trained using ground truth depth, normal and silhouette images and a L2 reconstruction\nloss. The 3D shape estimation module takes in the masked ground truth depth and normal images as\ninput, and predicts 3D voxels of size 128\u00d7128\u00d7128 with a binary cross entropy loss.\nWe compare MarrNet with a baseline that predicts 3D shape directly from an RGB image, without\nmodeling 2.5D sketches. The baseline employs the same architecture as our 3D shape estimator\n(Section 3.2). We show qualitative results in Figure 4. Our estimated surface normal and depth images\nabstract out non-essential information like textures and lighting in the RGB image, while preserving\nintrinsic information about object shape. Compared with the direct prediction baseline, our model\noutputs objects with more details and smoother surfaces. For quantitative evaluation, previous works\nusually compute the Intersection-over-Union (IoU) [Tulsiani et al., 2017, Choy et al., 2016]. Our full\nmodel achieves a higher IoU (0.57) than the direct prediction baseline (0.52).\n\n4.2\n\n3D Reconstruction on Pascal 3D+\n\nData PASCAL 3D+ dataset [Xiang et al., 2014] provides (rough) 3D models for objects in real-life\nimages. Here, we use the same test set of PASCAL 3D+ with earlier works [Tulsiani et al., 2017].\n\nMethods We follow the paradigm described in Section 3.4: we \ufb01rst train each module separately\non the ShapeNet dataset, and then \ufb01ne-tune them on the PASCAL 3D+ dataset. Unlike previous\nworks [Tulsiani et al., 2017], our model requires no silhouettes as input during \ufb01ne-tuning; it instead\nestimates silhouette jointly.\nAs an ablation study, we compare three variants of our model: \ufb01rst, the model trained using ShapeNet\ndata only, without \ufb01ne-tuning; second, the \ufb01ne-tuned model whose decoder is not \ufb01xed during\n\n6\n\nEstimated normalsEstimated depthsGround truthDirect predictionsImagesMarrNet\fFigure 5: We present an ablation study, where we compare variants of our models. From left to\nright: input, estimated normal, estimated depth, 3D prediction before \ufb01ne-tuning, two views of the\n3D prediction after \ufb01ne-tuning without \ufb01xing decoder, and two views of the 3D prediction after\n\ufb01ne-tuning with the decoder \ufb01xed. When the decoder is not \ufb01xed, the model explains the 2.5D sketch\nwell, but fails to preserve the learned shape prior. Fine-tuning with a \ufb01xed decoder resolves the issue.\n\n\ufb01ne-tuning; and third, the full model whose decoder is \ufb01xed during \ufb01ne-tuning. We also compare\nwith the state-of-the-art method (DRC) [Tulsiani et al., 2017], and the provided ground truth shapes.\n\nResults The results of our ablation study are shown in Figure 5. The model trained on synthetic\ndata provides a reasonable shape estimate. If we \ufb01ne-tune the model on Pascal 3D+ without \ufb01xing\nthe decoder, the output voxels explain the 2.5D sketch data well, but fail to preserve the learned shape\nprior, leading to impossible shapes from certain views. Our \ufb01nal model, \ufb01ne-tuned with the decoder\n\ufb01xed, keeps the shape prior and provides more details of the shape.\nWe show more results in Figure 6, where we compare with the state-of-the-art (DRC) [Tulsiani et al.,\n2017], and the provided ground truth shapes. Quantitatively, our algorithm achieves a higher IoU\nover these methods (MarrNet 0.39 vs. DRC 0.34). However, we \ufb01nd the IoU metric sub-optimal\nfor three reasons. First, measuring 3D shape similarity is a challenging yet unsolved problem, and\nIoU prefers models that predict mean shapes consistently, with no emphasize on details. Second, as\nobject shape can only be reconstructed up to scale from a single image, it requires searching over all\npossible scales during the computation of IoU, making it less ef\ufb01cient. Third, as discussed in Tulsiani\net al. [2017], PASCAL 3D+ has only rough 3D annotations (10 CAD chair models for all images).\nComputing IoU with these shapes would thus not be the most informative evaluation metric.\nWe instead conduct human studies, where we show users the input image and two reconstructions,\nand ask them to choose the one that looks closer to the shape in the image. We show each test image\nto 10 human subjects. As shown in Table 1, our reconstruction is preferred 74% of the time to DRC,\nand 42% of the time to ground truth, showing a clear advantage.\nWe present some failure cases in Figure 7. Our algorithm does not perform well on recovering\ncomplex, thin structure, and sometimes fails when the estimated mask is very inaccurate. Also, while\nDRC may bene\ufb01t from multi-view supervision, we have only evaluated MarrNet given a single view\nof the shape, though adapting our formulation to multi-view data should be straightforward.\n\n7\n\nEstimated normalsand silhouettesEstimated depths and silhouettesEstimated 3Dnot fine-tunedTwo views of refined 3D shapefine-tuned, not fixing decoderTwo views of refined 3D shapefine-tuned, fixing decoderImages\fFigure 6: 3D reconstructions of chairs on the Pascal 3D+ [Xiang et al., 2014] dataset. From left to\nright: input, the ground truth shape from the dataset, 3D estimation by DRC [Tulsiani et al., 2017],\nand two views of MarrNet predictions. Our model recovers more accurate 3D shapes.\n\nDRC\nMarrNet\nGround truth\n\nDRC MarrNet GT\n17\n50\n42\n74\n83\n50\n\n26\n50\n58\n\nTable 1: Human preferences on chairs in PAS-\nCAL 3D+ [Xiang et al., 2014]. We compare\nMarrNet with the state-of-the-art (DRC) [Tul-\nsiani et al., 2017], and the ground truth provided\nby the dataset. Each number shows the percent-\nage of humans prefer the left method to the top\none. MarrNet is preferred 74% of the time to\nDRC, and 42% of the time to ground truth.\n\n4.3\n\n3D Reconstruction on IKEA\n\nFigure 7: Failure cases on Pascal 3D+. Our algo-\nrithm does not perform well on recovering com-\nplex, thin structure, and sometimes fails when\nthe estimated mask is very inaccurate.\n\nData The IKEA dataset [Lim et al., 2013] contains images of IKEA furniture, along with accurate\n3D shape and pose annotations. These images are challenging, as objects are often heavily occluded\nor cropped. We also evaluate our model on the IKEA dataset.\n\nResults We show qualitative results in Figure 8, where we compare with estimations by 3D-VAE-\nGAN [Wu et al., 2016b] and the ground truth. As shown in the \ufb01gure, our model can deal with mild\nocclusions in real life scenarios. We also conduct human studies on the IKEA dataset. Results show\nthat 61% of the subjects prefer our reconstructions to those of 3D-VAE-GAN.\n\n4.4 Extensions\n\nWe also apply our framework on cars and airplanes. We use the same setup as that for chairs. As\nshown in Figure 9, shape details like the horizontal stabilizer and rear-view mirrors are recovered\n\n8\n\nImagesGround truthMarrNetDRCImagesGround truthMarrNetDRCEstimated normalsEstimated depthsMarrNetImages\fFigure 8: 3D reconstruction of chairs on the IKEA [Lim et al., 2013] dataset. From left to right:\ninput, ground truth, 3D estimation by 3D-VAE-GAN [Wu et al., 2016b], and two views of MarrNet\npredictions. Our model recovers more details compared to 3D-VAE-GAN.\n\nFigure 9: 3D reconstructions of airplanes and cars from PASCAL 3D+. From left to right: input, the\nground truth shape from the dataset, 3D estimation by DRC [Tulsiani et al., 2017], and two views of\nMarrNet predictions.\n\nFigure 10: 3D reconstruction of objects from multiple categories on the PASCAL 3D+ [Xiang et al.,\n2014] dataset. MarrNet also recovers 3D shape well when it is trained on multiple categories.\n\nby our model. We further train MarrNet jointly on all three object categories, and show results in\nFigure 10. Our model successfully recovers shapes of different categories.\n\n5 Conclusion\n\nWe proposed MarrNet, a novel model that explicitly models 2.5D sketches for single-image 3D\nshape reconstruction. The use of 2.5D sketches enhanced the model\u2019s performance, and made it\neasily adaptive to images across domains or even categories. We also developed differentiable loss\nfunctions for the consistency between 3D shape and 2.5D sketches, so that MarrNet can be end-to-end\n\ufb01ne-tuned on real images without annotations. Experiments demonstrated that our model performs\nwell, and is preferred by human annotators over competitors.\n\n9\n\nImagesGround truthMarrNet3D-VAE-GANImagesGround truthMarrNet3D-VAE-GANImagesGround truthMarrNetDRCImagesGround truthMarrNetDRCGround truthMarrNetImagesGround truthMarrNetImages\fAcknowledgements\n\nWe thank Shubham Tulsiani for sharing the DRC results, and Chengkai Zhang for the help on shape\nvisualization. This work is supported by NSF #1212849 and #1447476, ONR MURI N00014-16-\n1-2007, the Center for Brain, Minds and Machines (NSF #1231216), Toyota Research Institute,\nSamsung, and Shell.\n\nReferences\n\nAayush Bansal and Bryan Russell. Marr revisited: 2d-3d alignment via surface normal prediction. In CVPR,\n\n2016. 2\n\nJonathan T Barron and Jitendra Malik. Shape, illumination, and re\ufb02ectance from shading. IEEE TPAMI, 37(8):\n\n1670\u20131687, 2015. 2\n\nHarry G Barrow and Jay M Tenenbaum. Recovering intrinsic scene characteristics from images. Computer\n\nvision systems, 1978. 1\n\nSean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM TOG, 33(4):159, 2014. 2\n\nAngel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio\nSavarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repos-\nitory. arXiv:1512.03012, 2015. 2, 3, 5, 6\n\nWeifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. In NIPS, 2016.\n\n2\n\nChristopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A uni\ufb01ed approach\n\nfor single and multi-view 3d object reconstruction. In ECCV, 2016. 1, 2, 3, 6\n\nRonan Collobert, Koray Kavukcuoglu, and Cl\u00e9ment Farabet. Torch7: A matlab-like environment for machine\n\nlearning. In BigLearn, NIPS Workshop, 2011. 5\n\nAngela Dai, Charles Ruizhongtai Qi, and Matthias Nie\u00dfner. Shape completion using 3d-encoder-predictor cnns\n\nand shape synthesis. In CVPR, 2017. 3\n\nDavid Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale\n\nconvolutional architecture. In ICCV, 2015. 2\n\nMichael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel J Brostow. Structured prediction of unobserved\n\nvoxels from a single depth image. In CVPR, 2016. 3\n\nRohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative\n\nvector representation for objects. In ECCV, 2016. 2, 3, 4\n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In\n\nCVPR, 2015. 3\n\nDerek Hoiem, Alexei A Efros, and Martial Hebert. Automatic photo pop-up. ACM TOG, 24(3):577\u2013584, 2005.\n\n1\n\nBerthold KP Horn and Michael J Brooks. Shape from shading. MIT press, 1989. 2\n\nShahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard A. Newcombe, Pushmeet Kohli, Jamie\nShotton, Steve Hodges, Dustin Freeman, Andrew J. Davison, and Andrew W. Fitzgibbon. Kinectfusion:\nreal-time 3d reconstruction and interaction using a moving depth camera. In UIST, 2011. 2\n\nWenzel Jakob. Mitsuba renderer, 2010. http://www.mitsuba-renderer.org. 5\n\nMichael Janner, Jiajun Wu, Tejas D Kulkarni, Ilker Yildirim, and Joshua B Tenenbaum. Self-supervised intrinsic\n\nimage decomposition. In NIPS, 2017. 3\n\nAngjoo Kanazawa, David W Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching for\n\nsingle-view reconstruction. In CVPR, 2016. 3\n\nAbhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Category-speci\ufb01c object reconstruction\n\nfrom a single image. In CVPR, 2015. 3\n\n10\n\n\fIra Kemelmacher-Shlizerman and Ronen Basri. 3d face reconstruction from a single image using a single\n\nreference face shape. IEEE TPAMI, 33(2):394\u2013405, 2011. 1\n\nJoseph J. Lim, Hamed Pirsiavash, and Antonio Torralba. Parsing ikea objects: Fine pose estimation. In ICCV,\n\n2013. 8, 9\n\nDavid G Lowe. Three-dimensional object recognition from single two-dimensional images. Artif. Intell., 31(3):\n\n355\u2013395, 1987. 3\n\nDavid Marr. Vision: A computational investigation into the human representation and processing of visual\n\ninformation. W. H. Freeman and Company, 1982. 1, 2\n\nJohn McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison. Scenenet rgb-d: Can 5m synthetic\n\nimages beat generic imagenet pre-training on indoor segmentation? In ICCV, 2017. 2\n\nDanilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess.\n\nUnsupervised learning of 3d structure from images. In NIPS, 2016. 3, 4\n\nJason Rock, Tanmay Gupta, Justin Thorsen, JunYoung Gwak, Daeyun Shin, and Derek Hoiem. Completing 3d\n\nobject shape from one depth image. In CVPR, 2015. 3\n\nAshutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image.\n\nIEEE TPAMI, 31(5):824\u2013840, 2009. 1\n\nJian Shi, Yue Dong, Hao Su, and Stella X Yu. Learning non-lambertian object intrinsics across shapenet\n\ncategories. In CVPR, 2017. 2\n\nNathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference\n\nfrom rgbd images. In ECCV, 2012. 2\n\nAmir Arsalan Soltani, Haibin Huang, Jiajun Wu, Tejas D Kulkarni, and Joshua B Tenenbaum. Synthesizing 3d\nshapes via modeling multi-view depth maps and silhouettes with deep generative networks. In CVPR, 2017. 3\n\nShuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene\n\ncompletion from a single depth image. In CVPR, 2017. 2, 3\n\nMarshall F Tappen, William T Freeman, and Edward H Adelson. Recovering intrinsic images from a single\n\nimage. In NIPS, 2003. 1, 2\n\nShubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view\n\nreconstruction via differentiable ray consistency. In CVPR, 2017. 1, 2, 3, 4, 6, 7, 8, 9\n\nXiaolong Wang, David Fouhey, and Abhinav Gupta. Designing deep networks for surface normal estimation. In\n\nCVPR, 2015. 2\n\nYair Weiss. Deriving intrinsic images from image sequences. In ICCV, 2001. 2\n\nJiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T\n\nFreeman. Single image 3d interpreter network. In ECCV, 2016a. 3, 4\n\nJiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. Learning a Proba-\nbilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In NIPS, 2016b. 1, 3, 4, 8,\n9\n\nYu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the\n\nwild. In WACV, 2014. 2, 6, 8, 9\n\nJianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale\n\nscene recognition from abbey to zoo. In CVPR, 2010. 5\n\nXinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning\n\nsingle-view 3d object reconstruction without 3d supervision. In NIPS, 2016. 3, 4\n\nRuo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. Shape-from-shading: a survey. IEEE\n\nTPAMI, 21(8):690\u2013706, 1999. 2\n\n11\n\n\f", "award": [], "sourceid": 374, "authors": [{"given_name": "Jiajun", "family_name": "Wu", "institution": "MIT"}, {"given_name": "Yifan", "family_name": "Wang", "institution": "ShanghaiTech University"}, {"given_name": "Tianfan", "family_name": "Xue", "institution": "MIT CSAIL"}, {"given_name": "Xingyuan", "family_name": "Sun", "institution": "Shanghai Jiao Tong University"}, {"given_name": "Bill", "family_name": "Freeman", "institution": "MIT/Google"}, {"given_name": "Josh", "family_name": "Tenenbaum", "institution": "MIT"}]}