{"title": "Surface Reconstruction using Learned Shape Models", "book": "Advances in Neural Information Processing Systems", "page_first": 1305, "page_last": 1312, "abstract": null, "full_text": "                  Surface Reconstruction using\n                        Learned Shape Models\n\n\n               Jan Erik Solem                                Fredrik Kahl\n     School of Technology and Society           RSISE, Australian National University\n         Malmo University, Sweden                       ACT 0200, Australia\n             jes@ts.mah.se                           fredrik@maths.lth.se\n\n\n                                         Abstract\n         We consider the problem of geometrical surface reconstruction from one\n         or several images using learned shape models. While humans can effort-\n         lessly retrieve 3D shape information, this inverse problem has turned out\n         to be difficult to perform automatically. We introduce a framework based\n         on level set surface reconstruction and shape models for achieving this\n         goal. Through this merging, we obtain an efficient and robust method for\n         reconstructing surfaces of an object category of interest.\n         The shape model includes surface cues such as point, curve and silhou-\n         ette features. Based on ideas from Active Shape Models, we show how\n         both the geometry and the appearance of these features can be modelled\n         consistently in a multi-view context. The complete surface is obtained by\n         evolving a level set driven by a PDE, which tries to fit the surface to the\n         inferred 3D features. In addition, an a priori 3D surface model is used to\n         regularize the solution, in particular, where surface features are sparse.\n         Experiments are demonstrated on a database of real face images.\n\n\n1 Introduction\n\nHumans have a remarkable ability of perceiving 3D shape information - even from a single\nphotograph. Exactly how the human visual system works or how shape is represented is to\na large extent unknown. It is clear that the capabilities rely on a strong prior model and the\nefficient use of different surface cues. The corresponding task of automatically recovering\na 3D surface model for a computer has turned out to be a challenging problem, even with\nthe addition of multiple images. In the present paper, we try to solve the problem in the\ncase of a single object category, more specifically, faces. At the heart of our approach lies\nthe combination of sophisticated surface reconstruction techniques and a strong statistical\nmodel of surface features.\nThe first part concerns the statistical model of surface features for inferring 3D shape. The\nfeatures are primarily geometric ones, such as point, curve and silhouette features. Both the\ngeometric relations and their appearances (in terms of image intensities) are modelled. The\ndistributions are learned from real data. Also, a 3D model of the complete surface is used as\na weak regularizer where surface features are sparse. The motivations for introducing such\na model are several. We are interested in automatically recovering a surface model given\nnew image data of the object of interest. It is a hard problem to robustly extract curves and\napparent contours (i.e. silhouettes) without any a priori model. Moreover, many objects are\n\n\f\nhard to reliably reconstruct due to specularities and illumination effects. By using distinct\ngeometric features and strong priors, we will still be able to obtain reliable results. Another\nproblem is textureless areas, and more generally, lack of information in the input. Our a\npriori model will work as a domain-specific regularizer.\nThe second part of this work deals with fitting surfaces to points and curves in 3D space,\nand at the same time, fitting the projections of surface contours to apparent contours in the\nimages. The approach taken here is a variational one - we define a functional which we will\ntry to minimize with respect to some parameters describing the geometry of the surfaces.\nThis variational problem leads to a surface evolution, driven by a PDE. The surface is\nrepresented implicitly as the level set of a real-valued function [1].\n\n1.1 Related Work\n\nIn the area of statistical shape models, our work is related to and inspired by Active Shape\nModels (ASM) [2]. One distinction is that we will model both 2D and 3D data, while\nASM have mainly have been applied to 2D objects. In contrast to standard ASM, our\nobservations are from multiple views. In [3] a multi-view model is utilized, but no explicit\n(or even consistent) 3D data is maintained within the model. In order to do inference on\nour model, we have adopted the ideas of Probabilistic PCA (PPCA) [4].\nIn the seminal work [5], a complete 3D model is built from a database of 200 laser scans\nof faces. The so-called morphable model can be fitted to an image with very impressive\nresults. The model itself is quite complex (approximately 70000 vertices), resulting in long\ncomputation times. Although recent advances, their method still requires manual interven-\ntion [6]. A generic face model has also been used in [7] for computing regularized structure\nand motion as well as in [8] based on silhouettes obtained from background subtraction.\nIn the area of using level set surface representations for fitting surfaces to 3D data, this paper\nis related to the work in [9] where surfaces are fitted to points using a distance potential. In\n[10] apparent contours were incorporated for level set based surface fitting. Surfaces can\nalso be estimated using 2D dense data, as in [11] based on photo-consistency. Shape priors\nfor level sets have previously been applied to segmentation, e.g. [12].\n\n1.2 Contribution of the Paper\n\nThe main contribution of this paper is the approach itself - the combination of a state-of-the-\nart surface reconstruction technique and the learned statistical model of surface features.\nThis merging results in a robust and efficient method for surface reconstruction without\nany need for manual intervention. There is no need for an abundant number of images, in\nfact, a single image is sufficient.\nAnother key contribution is the introduction of a multi-view feature model which is ca-\npable of representing both 2D and 3D data in a consistent manner. As the model is fully\nprobabilistic, the missing data problem can be handled in a natural way. By only incor-\nporating distinct surface features, compared to a full morphable model, we not only gain\ncomputational efficiency, but also robustness to specularities and other illumination effects.\nThis point is a also valid when compared to other surface reconstruction methods based on\nimage-correlation [11]. In the case of face modelling, this is known to cause problems due\nto the complexity of the BRDF for human skin. The main contribution within the field of\nlevel sets is in the incorporation of an a priori 3D model used for surface regularization.\n\n2 Part I: A Learned Shape Model\n\nIn this section, we develop a statistical model which is needed in order to be able to au-\ntomatically extract and compute 3D surface features from the input images. The output\n\n\f\nfrom the model will serve as input to the level set reconstruction algorithm in order to get a\ncomplete surface model. In addition, we will use an a priori 3D surface model as a (weak)\nregularizer.\n\n2.1 The Feature Model\n\nSuppose we have a number of elements in a d-dimensional vector t, for example, a collec-\ntion of 3D point coordinates. Suppose t can be related to some latent vector u of dimension\nq where the relationship is linear:\n\n                                              t = W u + ,                                         (1)\n\nwhere W is a matrix of size d  q and  is d-vector allowing for non-zero mean. How-\never, our measurements take place in the images, which is a non-linear function of the 3D\nfeatures according to the perspective camera model. Denote the projection function with\nf : Rd  Re, projecting all 3D features to 2D image features, for one or several images1.\nAlso, we need to change coordinate system of the 3D features to suit the actual projec-\ntion function. Denote this mapping by T : Rd  Rd. Thus, f(T (t)) will project all\nnormalized 3D data to all images.\nFinally, a noise model needs to be specified. We assume that the image measurements\nare independent and normally distributed, likewise, the latent variables are assumed to be\nGaussian with unit variance u  N(0, I). Thus, in summary:\n\n                            t2D = f (T (t)) + = f (T (W u + )) + ,                                (2)\n\nwhere  N(0, 2I) for some scalar .\nBefore the model can be used, its parameters need to be estimated from training data.\nGiven that it is a probabilistic model, this is best done with maximum likelihood (ML).\nSuppose we are given n examples {t2D,i}n , the ML estimate for W and  is obtained\n                                                   i=1\nby minimizing:\n\n                      n       1 ||(t\n                             2         2D,i - f (Ti(W ui + ))||2 + ||ui||2    ,                  (3)\n                     i=1\n\nover all unknowns. The standard deviation  is estimated a priori from the data. Once the\nmodel parameters W and  have been learned from examples, they are kept fix. In practice,\nto minimize (3) we alternatively optimize over (W, ) and the latent variables {ui}n using\n                                                                                            i\ngradient descent. Initial estimates can be obtained by intersecting 3D structure from each\nset of images and then applying standard PPCA algorithms for the linear part [4]. The\nnormalization Ti() is chosen such that each normalized 3D sample has zero mean and unit\nvariance.\nA 3D point which is visible in m > 1 images will be represented in the vector t with its 3D\ncoordinates (X, Y, Z). For points visible in only one image, m = 1, no depth information\nis available, and such points are represented similarly to apparent contour points. A curve\nwill be represented in the model by a number of points along the curve. In the training\nof the model, it is important to parameterize each 3D curve such that each point on the\ncurve approximately corresponds to the same point on the corresponding curve in the other\nexamples. As for curves, we sample the apparent contours (in the images) using arc-length\nparametrization. However, there is no 3D information available for the apparent contours\nas they are view-dependent. A simple way is to treat contours points as 3D points with a\nconstant, approximate (but crude) depth estimate.\n\n   1In the experiments, f() will model the projection of three calibrated perspective cameras.\n\n\f\n2.2 The Grey-Level Model\n\nThe missing component in the model is the relationship between 2D image features and\nthe underlying grey-level (or color) values at these pixels. Again, we adopt a linear model\n(PPCA). Using the same notation as in (1), but now with the subscript gl for grey-level, the\nmodel can be written\n                                 tgl = Wglugl + gl + gl ,\nwhere tgl is a vector containing the grey-level values of all the 2D image features and\n gl is Gaussian noise in the measurements.      In the training phase, each data sample of\ngrey-levels is normalized by subtracting the mean and scaling to unit variance. The ML-\nestimate of Wgl and gl is computed with the EM-algorithm [4]. The complete statistical\ntwo-layer model with one feature model and one grey-level model is very similar to the\nconcept of ASM [2]. In principle, the same techniques as used for ASM can be applied to\nautomatically compute the latent variables of the system, i.e. u and ugl.\n\n2.3 The 3D Model\n\nThe two-layer feature model produces only a sparse set of features in 3D space. Even if\nthese cues are characteristic for a particular sample (or individual), it is often not enough\nin order to infer a complete surface model, in particular, in regions where the features\nare sparse. Therefore, we introduce a 3D surface model consisting of the complete mean\nsurface serving as domain-specific regularizer. The mean surface is obtained from laser\nscans with the technique described in [13].\n\n3 Part II: Surface Reconstruction\n\n3.1 Level Set Formulation\n\nLet x be a point in the open set   R3. The time dependent surface (t) is represented\nimplicitly as the zero level set of a function (x, t) :   R+  R as\n\n                                 (t) = {x ; (x, t) = 0} ,                              (4)\n\nwhere  is defined such that (x, t) < 0 inside  and (x, t) > 0 outside. Using the\ndefinition above gives the outward unit normal n and the mean curvature  as\n\n                                                              \n                             n =             and  =                .                    (5)\n                                    | |                     | |\n\nOne important, frequently used example is the signed distance function, where the require-\nment | (x)| = 1 is imposed.\nThe zero set of (x, t) represents (t) at all times t. This means that (x(t), t)  0 for a\npoint on the curve x(t)  (t). Differentiating with respect to t gives\n\n                          t + v      = 0  t + vn| | = 0 ,                          (6)\n\nwhere v = dx(t)/dt and vn is the velocity normal to the surface. This PDE is solved\nin order to move the surface according to some derived velocity v. For a more thorough\ntreatment of level set surfaces, see [1].\n\n3.2 Surface Fitting to Points\n\nIn [9] surfaces are fitted to a point set S using the level set method. An initial surface is\ndeformed in the gradient direction of an energy functional which involves elastic energy\n\n\f\nand potential energy. The energy is expressed using a distance potential as the surface\nintegral\n                                   EP () =              d(x) d ,                         (7)\n                                                    \nwhere  is the surface, d surface area and d(x) = dist(x, S) is the distance from the point\nx to S. The gradient descent of (7) is, cf. [9],\n                             t = ( d(x)  n + d(x)) | | ,                               (8)\nwhere n is the surface normal and  the mean curvature.\nThis motion is known to interfere with surface regularization, since all surface points are\nattracted to the 3D features. Therefore we cut the influence of the potential by setting\nd(x) = min(d(x), dmax).\n\n3.3 Surface Fitting to Apparent Contours\n\nLet  be an apparent contour in an image parameterized as (s) : I  R  R2. The\nback-projected cone, written in homogeneous coordinates,\n                                   C(s) = c + P +(s) ,                                   (9)\n(where c is the camera center and P + the pseudo-inverse of the 3  4 camera matrix of\na perspective projection) should graze the surface at the location of the contour generator.\nIt is undesirable to attract the surface to the entire back-projected cone of the apparent\ncontour. The cone should only touch the surface along a curve - the so called contour\ngenerator.\nWe propose to solve this in the following manner. For each point on the curve m = (s),\nlet x denote the point closest to  (If there are several, then choose the one with smallest\ndepth). The function  is kept to be a signed distance function, i.e. | | = 1. This means\nthat the point x is easily found by checking the values of  along the line of sight given\nby (9). The set of these points, S , will be a (possibly discontinuous) space curve (, s).\nThis set is then added to the distance potential as\n                                 d(x) = min(d(x), d (x)) ,                                (10)\nwhere d (x) = dist(x, S ) is updated at appropriate intervals as the surface evolves.\n\n3.4 Adding a 3D Shape Prior\n\nSince the data, i.e. the points, curves and contours, are sparse it is customary to use a prior\nfor the regions where there is no information. Instead of the common choice of minimal\nsurface type models, we propose to use a learned shape model, as described in Section 2.3.\nBy first aligning the mean shape to the data, the deviation can be expressed similar to (7)\nas\n                              EP rior() =          dP rior(x) d ,                       (11)\n                                               \nwhere dPrior(x) is the distance potential of the aligned mean shape.\n\n3.5 The Combined Motion PDE\n\nAdding all the components above gives a functional ETot = EP +EPrior, where   R+\ndetermines the weight of the prior shape. Combining the components (8), (10) and (11)\nabove leads to a PDE for the motion of the surface as\n              t = [( d(x) +  dP rior(x))  n + (d(x) + dP rior(x))]| | .             (12)\nThis PDE is solved iteratively until a steady state is reached, which yields a (local) mini-\nmum of ETot.\n\n\f\nFigure 1: Extracted image features for one image triple and the reconstructed 3D surface.\n\n4 Experiments\n\n4.1 The Shape Model\n\nAll images were taken by a stereo setup with three (synchronized) cameras. The setup\nwas pre-calibrated for both extrinsic and intrinsic camera parameters. Example images are\ngiven in Figures 1 and 2. In total a database of 28 image triplets were collected of faces\nfrom different persons (23 males and 5 females of ages ranging from 7 to 65). 25 of these\nwere used for training and 3 for testing the two-layer feature model. The 3D mean surface\nwas computed from a database of 24 persons (different from the 28 persons above) using a\nlaser scanner as described in [13].\nThe 28 triplets in the training and test set were manually labelled - 36 points, 8 curves and\n5 apparent contours were extracted for each person, see Figure 1. The two-layer model has\nq = 12 elements in the latent variable u for the geometrical part and qgl = 15 elements\nin ugl for the grey-level model. These numbers were found empirically in order to capture\nmost of the variations in the data. The model is able to adopt quite well to the test set. In\nfact, even for one (frontal) input image of the test set, the model predicts the two profile\nviews remarkably well. As the pose over all faces is (essentially) constant for both test and\ntraining images, the normalizing coordinate mapping T is restricted to scale and translation.\n\n4.2 Surface Reconstruction\n\nOnce 3D data has been obtained by fitting the two-layer shape model to the input images,\nsurfaces are fitted to this data by solving the PDE (12). In the standard level set formulation,\nthe surface must be closed in the computational domain in order to make  continuous. We\nuse the technique in [14] for initializing and evolving open implicit surfaces.\nSurfaces were reconstructed for a number of persons, a selection is shown in Fig-\nures 1 and 2, where the zero set is visualized and triangulated using the marching cubes\nalgorithm . The value of  was 0.3 and dmax was set from the maximum distance of the\nfeature points to the initial surface (4-5 voxels). For the reconstructions in Figure 2, the\nmean and median distances (measured in voxel width) of the feature points to the recon-\nstructed surface have been computed. See Figure 3 for a typical histogram and a table of\nthe results. Most points have sub-voxel distance to the surface, i.e. the deviation is of the\nsame order as the surface resolution. This shows that we are not fitting a surface to the\nmean shape but that it really fits the feature data. The reader is encouraged to zoom in on\nthe reconstructions in Figure 2 to verify the quality.\n\n5 Conclusions and Future Work\n\nIn this paper, a framework for reconstructing surface geometry based on learned shape\nmodels and level sets has been presented. The approach relies solely on images as input\nand the output of the system is a geometric 3D surface consistent with the input images. A\nnew regularization approach is introduced in the level set framework, where the distance to\nan a priori 3D model is used instead of the common mean curvature regularization.\n\n\f\nFigure 2: Input images and the reconstructed surface for three persons, two in the training\ndata and one (bottom) in the test data. For each person: One of the input images, triangu-\nlated surfaces and surfaces with texture. Note that the profiles above are different from the\nones in the input.\n\n\f\n          Histogram for distance to the surface for all 3D points\n\n            70\n            60                                                               mean    median\n            50                                                       Person 1 0.978 0.788\n            40\n            30                                                       Person 2 0.805 0.601\n            20                                                       Person 3 0.861 0.843\n            10\n              00          1         2          3\n                         Distance in voxel width\n\nFigure 3: Histogram of the deviations of the feature points from the surface for the second\nperson in Figure 2. The table displays mean and median deviations for all three persons.\n\nOur current work focuses on incorporating a robust photo-consistency measure in the en-\nergy potential (7) to enable more detailed geometry. Also, the total number of faces is\nquite small in the database and we will collect and label more images. Currently, only\nimages taken with the tri-stereo setup have been used with heads facing the middle cam-\nera (cf. Figure 1). Once the statistical model has been learned, it can be utilized for other\n(nearby) poses as well, but to what extent is yet to be explored.\n\nReferences\n [1] J.A. Sethian. Level Set Methods and Fast Marching Methods Evolving Interfaces in Com-\n     putational Geometry, Fluid Mechanics, Computer Vision, and Materials Science. Cambridge\n     University Press, 1999.\n [2] T. F. Cootes and Taylor C. J. Active shape model search using local grey-level models: A\n     quantatitative evaluation. In British Machine Vision Conf., pages 639648, 1993.\n [3] T.F. Cootes, G.V. Wheeler, K.N. Walker, and C.J. Taylor. View-based active appearance models.\n     Image and Vision Computing, 20(9-10):657664, 2002.\n [4] M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Phil. Trans. Royal\n     Soc. London B, 61(3):611622, 1999.\n [5] V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, pages\n     187194, 1999.\n [6] S. Romdhani and T. Vetter. Efficient, robust and accurate fitting of a 3d morphable model. In\n     Int. Conf. Computer Vision, pages 5966, Nice, France, 2003.\n [7] P. Fua. Regularized bundle-adjustment to model heads from image sequences without calibra-\n     tion data. Int. J. Comput. Vision, 38(2):153171, 2000.\n [8] B. Moghaddam, J. Lee, H. Pfister, and R. Machiraju. Model-based 3d face capture with shape-\n     from-silhouettes. In IEEE International Workshop on Analysis and Modeling of Faces and\n     Gestures (AMFG), pages 2027, 2003.\n [9] H.K. Zhao, S. Osher, B. Merriman, and M. Kang. Implicit and non-parametric shape recon-\n     struction from unorganized points using a variational level set method. In Computer Vision and\n     Image Understanding, pages 295319, 2000.\n[10] J.E. Solem and F. Kahl. Surface reconstruction from the projection of points, curves and con-\n     tours. In 2nd Int. Symposium on 3D Data Processing, Visualization and Transmission, Thessa-\n     loniki, Greece, 2004.\n[11] O. Faugeras and R. Keriven. Variational principles, surface evolution, PDEs, level set methods,\n     and the stereo problem. IEEE Transactions on Image Processing, 7(3):336344, 1998.\n[12] M. Rousson and N. Paragios. Shape priors for level set representations. In Proc. European\n     Conf. on Computer Vision, volume 2351 of Lecture Notes in Computer Science. Springer, 2002.\n[13] K. Skoglund. Three-dimensional face modelling and analysis. Master's thesis, Informatics and\n     Mathematical Modelling, Technical University of Denmark, DTU, 2003.\n[14] J.E. Solem and A. Heyden. Reconstructing open surfaces from unorganized data points. In\n     International Conference on Computer Vision and Pattern Recognition, Washington DC, 2004.\n\n\f\n", "award": [], "sourceid": 2654, "authors": [{"given_name": "Jan", "family_name": "Solem", "institution": null}, {"given_name": "Fredrik", "family_name": "Kahl", "institution": null}]}