{"title": "Differentiable Cloth Simulation for Inverse Problems", "book": "Advances in Neural Information Processing Systems", "page_first": 772, "page_last": 781, "abstract": "We propose a differentiable cloth simulator that can be embedded as a layer in deep neural networks.  This approach provides an effective, robust framework for modeling cloth dynamics, self-collisions, and contacts. Due to the high dimensionality of the dynamical system in modeling cloth, traditional gradient computation for collision response can become impractical. To address this problem, we propose to compute the gradient directly using QR decomposition of a much smaller matrix. Experimental results indicate that our method can speed up backpropagation by two orders of magnitude. We demonstrate the presented approach on a number of inverse problems, including parameter estimation and motion control for cloth.", "full_text": "Differentiable Cloth Simulation for Inverse Problems\n\nJunbang Liang\nMing C. Lin\nUniversity of Maryland, College Park\n\nVladlen Koltun\n\nIntel Labs\n\nAbstract\n\nWe propose a differentiable cloth simulator that can be embedded as a layer in deep\nneural networks. This approach provides an effective, robust framework for model-\ning cloth dynamics, self-collisions, and contacts. Due to the high dimensionality\nof the dynamical system in modeling cloth, traditional gradient computation for\ncollision response can become impractical. To address this problem, we propose to\ncompute the gradient directly using QR decomposition of a much smaller matrix.\nExperimental results indicate that our method can speed up backpropagation by\ntwo orders of magnitude. We demonstrate the presented approach on a number of\ninverse problems, including parameter estimation and motion control for cloth.\n\n1\n\nIntroduction\n\nDifferentiable physics simulation is a powerful family of techniques that applies gradient-based\nmethods to learning and control of physical systems [7; 8; 29; 13; 25]. It can enable optimization for\ncontrol, and can also be integrated into neural network frameworks for performing complex tasks.\nOur work focuses on cloth simulation, which relates to applications in robotics, computer vision, and\ncomputer graphics [6; 19; 3; 31; 23; 16; 11]. Our goal is to enable differentiable cloth simulation,\nwhich can provide a uni\ufb01ed approach to a variety of inverse problems that involve cloth.\nDifferentiable cloth simulation is challenging due to a number of factors, which include the high\ndimensionality of cloth (as compared for example to rigid bodies [7]) and the need to handle contacts\nand collision. For example, a simple 16\u21e516 grid-based cloth mesh has 289 vertices, 867 variables, and\n512 faces when triangulated. Typical resolutions for garments would be at least many thousands, if\nnot millions, of vertices and faces. Previous work that tackled differentiable simulation with collisions\nset up a static linear solver to account for all constraints [7]. In our simple example with cloth, the\nnumber of pairwise constraints would be at least 289\u21e5512 = 140K for vertex-face collisions alone,\nwhich renders existing methods impractical even for this simple system. Even if a dynamic solver\nis applied upon collision, solving a dense linear system with such high dimensionality makes the\ngradient computation infeasible.\nIn this paper, we propose a differentiable cloth simulation algorithm that overcomes the above\ndif\ufb01culties. First, we use dynamic collision detection since the actual collision pairs are very\nsparse. The collision response is solved by quadratic optimization, for which we can use implicit\ndifferentiation to compute the gradient. We directly solve the equations derived from implicit\ndifferentiation by using the QR decomposition of the constraint matrix, which is much smaller than\nthe original linear system and is often of low rank. This approach reduces the gradient computation\nto a linear system of a small upper triangular matrix (the R component of the decomposition), which\nenables fast simulation and backpropagation.\nOur experiments indicate that the presented method makes differentiable cloth simulation practical.\nUsing our method, the largest size of the linear system is 10x-20x smaller than the original solver\nin the backpropagation of the collision response, and the solver is 60x-130x faster. We demonstrate\nthe potential of differentiable cloth simulation in a number of application scenarios, such as physical\nparameter estimation and motion control of cloth. With only a few samples, the differentiable\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fsimulator can optimize its input variables to \ufb01t the data, thereby inferring physical parameters from\nobservations and reaching desired control goals.\n\n2 Related Work\n\nDifferentiable physics. With recent advances in deep learning, there has been increasing interest in\ndifferentiable physics simulation, which can be combined with other learning methods to provide\nphysically consistent predictions. Belbute-Peres et al. [7] and Degrave et al. [8] proposed rigid body\nsimulators using a static formulation of the linear complementarity problem (LCP) [5; 4]. Toussaint\net al. [29] developed a robot reasoning system that can achieve user-de\ufb01ned tasks and is based\non differentiable primitives. Hu et al. [13] implemented a differentiable simulator for soft robots\nbased on the Material Point Method (MPM). They store the object data at every simulation step so\nthat the gradient can be computed out of the box. Schenck and Fox [25] embedded particle-based\n\ufb02uid dynamics into convolutional neural networks, with precomputed signed distance functions for\ncollision handling. They solved or avoided collisions by assuming special object shapes, transferring\nto an Eulerian grid, or solving the corresponding collision constraint equation.\nNone of these methods can be applied to cloth simulation. First, cloth is a 2D surface in a 3D world;\nthus methods that use an Eulerian grid to compute material density, such as MPM [13], are not\napplicable. Second, the collision constraints in cloth simulation are more dynamic and complex\ngiven the high number of degrees of freedom; thus constructing a static dense LCP for the entire\nsystem [7; 8] or constructing the overall state transition graph [29] is inef\ufb01cient and usually impossible\nfor cloth of common resolution, since contact can happen for every edge-edge or vertex-face pair.\nLastly, the shape of cloth changes constantly so self-collision cannot be handled by precomputed\nsigned distance functions [25].\nIn contrast, our method uses dynamic collision detection and computes the gradients of the collision\nresponse by performing implicit differentiation on the quadratic optimization used for computing\nthe response. We utilize the low dimensionality and rank of the constraint matrix in the quadratic\noptimization and minimize the computation needed for the gradient propagation by giving an explicit\nsolution to the linear system using QR decomposition of the constraint matrix.\nDeep learning and physics. Supervised deep networks have been used to approximate physical\ndynamics. Mrowca et al. [21] and Li et al. [17] learned interaction networks to model particle systems.\nIngraham et al. [14] trained a model to predict protein structures from sequences using a learnable\nsimulator; the simulator predicts the deformation energy as an approximation to the physical process.\nDeep networks have also been used to support the simulation of \ufb02uid dynamics [28; 15; 20]. Our\nmethod differs from many works that use deep networks to approximate physical systems in that\nwe backpropagate through the true physical simulation. Thus our method conforms to physical law\nregardless of the scale of the problem. It can also naturally accept physical parameters as input, which\nenables learning from data.\nDeep learning and cloth. Coupling cloth simulation with deep learning has become a popular way\nto solve problems such as detail re\ufb01nement, garment retargeting, and material estimation. Yang et\nal. [31] proposed a recurrent model to estimate physical cloth parameters from video. L\u00e4hner et\nal. [16] trained a GAN to generate wrinkles on a coarse garment mesh which can then be automatically\nregistered to a human body using PCA. Gundogdu et al. [11] trained a graph convolutional framework\nto generate drapes and wrinkles given a roughly registered mesh. Santesteban et al. [24] developed\nan end-to-end retargeting network using a parametric human body model with displacements to\nrepresent the cloth.\nThese applications may bene\ufb01t from our method. For garment retargeting problems, the relationship\nbetween body pose and vertex displacement is made explicit via the computed gradient, which can\nthen be applied in network regularization for better performance. For parameter estimation, the\ndifferentiable simulation provides an optimization-based solution rather than a learning-based one.\nInstead of learning statistics from a large amount of data, we can directly apply gradient-based\noptimization via the simulator, which does not require any training data.\n\n2\n\n\f3 Differentiable Cloth Simulation\n\nIn this section, we introduce the main algorithms for the gradient computation. In general, we\nfollow the computation \ufb02ow of the common approach to cloth simulation: discretization using the\n\ufb01nite element method [9], integration using implicit Euler [2], and collision response on impact\nzones [22; 12]. We use implicit differentiation in the linear solve and the optimization in order to\ncompute the gradient with respect to the input parameters. The discontinuity introduced by the\ncollision response is negligible because the discontinuous states constitute a zero-measure set. During\nthe backpropagation in the optimization, the gradient values can be directly computed after QR\ndecomposition of the constraint matrix.\n\n3.1 Overview\nWe begin by de\ufb01ning the problem formally and providing common notation. A triangular mesh\nM = {V,E,F} consists of sets of vertex states, edges, and faces, where the state of the vertices\nincludes both position x and velocity v. Given a cloth mesh Mt together with obstacle meshes Mobs\nat step t, a cloth simulator can compute the mesh state Mt+1 at the next step t + 1 based on the\ncomputed internal and external forces and the collision response. A simple simulation pipeline is\nshown in Algorithm 1, where M is the mass matrix, f is the force, and a is the acceleration. For more\ndetailed description of cloth simulation, please refer to Appendix B. All gradients except the linear\nsolve (Line 4 in Algorithm 1) and the collision response (Line 7) can be computed using automatic\ndifferentiation in PyTorch [26].\n\nt\n\nAlgorithm 1 Cloth simulation\n1: v0 0\n2: for t = 1 to n do\n3: M, f compute_forces(x, v)\nat M1f\n4:\n5:\nvt vt1 + att\n6:\nxt xt1 + vtt\nxt xt+ collision_response(xt, vt, xobs\n7:\n8:\nvt (xt  xt1)/t\n9: end for\n\nt\n\n, vobs\n\nt\n\n)\n\n3.2 Derivatives of the Physics Solve\nIn modern simulation algorithms, implicit Euler is often used for stable integration results. Thus the\nmass matrix M used in Algorithm 1 often includes the Jacobian of the forces (see Appendix B for\nthe exact formulation). We denote it below as \u02c6M in order to mark the difference. A linear solve\nwill be needed to compute the acceleration since it is time consuming to compute \u02c6M1. We use\nimplicit differentiation to compute the gradients of the linear solve. Given an equation \u02c6Ma = f with\na solution z and the propagated gradient @L@a|a=z, where L is the task-speci\ufb01c loss function, we can\nuse the implicit differentiation form\n(1)\n\nto derive the gradient as\n\n(2)\n\n(3)\n\nwhere da is obtained from the linear system\n\n@L\n@ \u02c6M\n\n= d>a ,\n\n\u02c6M@a = @f  @ \u02c6Ma\n= daz> @L\n\n@f\n\n\u02c6M>da =\n\n>\n\n.\n\n@L\n@a\n\nThe proof is as follows. We take @L@f as an example here, the derivation of @L\npendix A.1:\n\n@ \u02c6M is shown in Ap-\n\n@L\n@f\n\n=\n\n@L\n@a \u00b7\n\n@a\n@f\n\n= d>a\n\n\u02c6M \u00b7 \u02c6M\u2020I = d>a .\n\n(4)\n\n3\n\n\fThe \ufb01rst equality is given by the chain rule, the second is given by Equations 1 and 3, and \u02c6M\u2020 is the\npseudoinverse of matrix \u02c6M.\n\n3.3 Dynamic Collision Detection and Response\nAs mentioned in Sec. 1, a static collision solver is not suitable for cloth because the total number\nof possible collision pairs is very high: quadratic in the number of faces. A common approach in\ncloth simulation is to dynamically detect collision on the \ufb02y and compute the response. We use a\nbounding volume hierarchy for collision detection [27], and non-rigid impact zones [12] to compute\nthe collision response.\nSpeci\ufb01cally, we solve a cubic equation to detect the collision time t of each vertex-face or edge-edge\npair that is suf\ufb01ciently close to contact:\n\n(x1 + v1t) \u00b7 (x2 + v2t)\u21e5(x3 + v3t) = 0,\n\n(5)\nwhere xk and vk (k = 1, 2, 3) are the relative position and velocity to the \ufb01rst vertex. A solution\nthat lies in [0, 1] means that a collision is possible before the next simulation step. After making\nsure that the pair indeed intersects at time t, we set up one constraint for this collision, forcing the\nsigned distance of this collision pair at time t to be no less than the thickness of the cloth . The\nsigned distance of the vertex-face or edge-edge pair is linear to the vertex position x. The set of all\nconstraints then makes up a quadratic optimization problem as discussed later in Sec. 3.4.\nFor backpropagation, we need to compute the derivatives of the solution t since it is related to the\nparameters of the constraints. We use implicit differentiation here to simplify the process. Generally,\ngiven a cubic equation ax3 + bx2 + cx + d = 0, its implicit differentiation is of the following form:\n(6)\n\n(3ax2 + 2bx + c)@x = @ax3 + @bx2 + @cx + @d.\n\nTherefore we have\n\n(7)\n\n@x\n@b\n\n@x\n@c\n\n\u21e5 @x\n\n@a\n\n@x\n\n@d\u21e4 =\n\n1\n\n3ax2 + 2bx + c\u21e5x3 x2 x 1\u21e4 .\n\n3.4 Derivatives of the Collision Response\nA general approach to integrating collision constraints into physics simulation has been proposed\nby Belbute-Peres et al. [7]. However, as mentioned in Sections 1 and 2, constructing a static LCP is\noften impractical in cloth simulation because of high dimensionality. Collisions that actually happen\nin each step are very sparse compared to the complete set. Therefore, we use a dynamic approach\nthat incorporates collision detection and response.\nCollision handling in our implementation is based on impact zone optimization [22]. It \ufb01nds all\ncolliding instances using continuous collision detection (Sec. 3.3) and sets up the constraints for\nall collisions. In order to introduce minimum change to the original mesh state, we develop a QP\nproblem to solve for the constraints. Since the signed distance function is linear in x, the optimization\ntakes a quadratic form:\n\n1\n2\n\nz\n\n(z  x)>W(z  x)\n\nminimize\nsubject to Gz + h \uf8ff 0\n\n(8)\n(9)\nwhere W is a constant diagonal weight matrix related to the mass of each vertex, and G and h are\nconstraint parameters (see Appendix B for more details). We further denote the number of variables\nand constraints by n and m, i.e. x 2 Rn, h 2 Rm, and G 2 Rm\u21e5n. Note that this optimization is a\nfunction with inputs x, G, and h, and output z. Our goal here is to derive @L@x , @L@G, and @L@h given @L@z ,\nwhere L refers to the loss function.\nWhen computing the gradient using implicit differentiation [1], the dimensionality of the linear\nsystem (Equation 13) can be too high. Our key observation here is that n >> m > rank(G), since\none contact often involves 4 vertices (thus 12 variables) and some contacts may be linearly dependent\n(e.g. multiple adjacent collision pairs). OptNet [1] solves a linear equation of size m + n, which is\nmore than necessary. We introduce a simpler and more ef\ufb01cient algorithm below to minimize the size\nof the linear equation.\n\n4\n\n\f3.4.1 QR Decomposition\nTo make things simpler, we assume that G is of full rank in this section. At global minimum z\u21e4 and\n\u21e4 of the Lagrangian, the following holds for stationarity and complementary slackness conditions:\n\nWz\u21e4  Wx + G>\u21e4 = 0\nD(\u21e4)(Gz\u21e4 + h) = 0,\n\nwith their implicit differentiation as\n\n\uf8ff W\nD(\u21e4)G D(Gz\u21e4 + h)\uf8ff@z\n\n@ =\uf8ff M@x  @G>\u21e4\n\nD(\u21e4)(@Gz\u21e4 + @h) ,\n\nG>\n\n(10)\n(11)\n\n(12)\n\nwhere D() transforms a vector to a diagonal matrix. Using similar derivation to Sec. 3.2, solving the\nequation\n\n\uf8ffW G>D(\u21e4)\nG D(Gz\u21e4 + h)\uf8ffdz\n\nd =\uf8ff @L@z\n0 \n\n>\n\ncan provide the desired gradient:\n\n@L\n@x\n@L\n@G\n@L\n@h\n\n= dT\n\nz W\n\n= D(\u21e4)dz\u21e4>  \u21e4d>z\n= dT\n\n D(\u21e4).\n\n(13)\n\n(14)\n\n(15)\n\n(16)\n\n(See Appendix A.2 for the derivation.) However, as mentioned before, directly solving Equation 13\nmay be computationally expensive in our case. We show that by performing a QR decomposition, the\nsolution can be derived without solving a large system.\nTo further reduce computation, we assume that no con-\nstraint is \u2018over-satis\ufb01ed\u2019, i.e. Gz\u21e4 + h = 0. We will\nremove these assumptions later in Sec. 3.4.2. We compute\nthe QR decomposition of pW1\n\nG>:\n\npW1\n\nG> = QR.\n\n(17)\n\nThe solution of Equation 13 can be expressed as\n\ndz = pW1\n(I  QQ>)pW1 @L\n@z\nd = D(\u21e4)1R1Q>pW1 @L\n>\n@z\n\n>\n\n(18)\n\n,\n\nFigure 1: Impact of perturbation. A\nsmall perturbation of the target position\nwill cause the \ufb01nal result to move along\nthe constraint surface.\n\n(19)\nwhere pW1 is the inverse of the square root of a diago-\nnal matrix. The result above can be veri\ufb01ed by substitution\nin Equation 13.\nThe intuition behind Equation 18 is as follows. When perturbing the original point x in an opti-\nmization, the resulting displacement of z will be moving along the surface of Gx + h = 0, which\nwill become perpendicular to the normal when the perturbation is small. (Fig. 1 illustrates this idea\nin two dimensions.) This is where the term I  QQ> comes from. Note that pW1\nG> is an\nn\u21e5m matrix, where n >> m and the QR decomposition will only take O(nm2) time, compared to\nO((n + m)3) in the original dense linear solve. After that we will need to solve a linear equation in\nEquation 19, but it is more ef\ufb01cient than solving Equation 13 since it is only of size m, and R is an\nupper-triangular matrix. In our collision response case, where n \uf8ff 12m, our method can provide up\nto 183x acceleration in theory. The speed-up in our experiments (Sec. 4) ranges from 60x to 130x for\nlarge linear systems.\n\n5\n\n\f3.4.2 Low-rank Constraints\nThe algorithm above cannot be directly applied when G is low-rank, or when some constraint is not\nat boundary. This will cause R or D(\u21e4) to be singular. We now show that the singularity can be\navoided via small modi\ufb01cations to the algorithm.\nFirst, if k = 0 for the kth constraint then dk doesn\u2019t matter. This is because the \ufb01nal result contains\nonly components of D(\u21e4)d but not d alone, as shown in Equations 15 and 16. Intuitively, if the\nconstraint is over-satis\ufb01ed, then perturbing the parameters of that constraint will not have impact on\nz. Based on this observation, we can remove the constraints in G when their corresponding  is 0.\nNext, if G is of rank k, where k < m, then we can rewrite Equation 17 as\n\npW1\n\nG> = Q1[R1 R2],\n\n(20)\n\nwhere Q1 2 Rn\u21e5k, R1 2 Rk\u21e5k, and R2 2 Rk\u21e5(mk). Getting rid of R2 (i.e. removing those\nconstraints from the beginning) does not affect the optimization result, but may change  so that\nthe computed gradients are incorrect. Therefore, we need to transfer the Lagrange multipliers to the\nlinearly independent terms \ufb01rst:\n\nwhere 1 and 2 are the Lagrange multipliers corresponding to the constraints on R1 and R2.\n\n1 1 + R1\n\n1 R22,\n\n(21)\n\n4 Experiments\n\nWe conduct three experiments to showcase the power of differentiable cloth simulation. First, we\nuse an ablation study to quantify the performance gained by using our method to compute the\ngradient. Next, we use the computed gradient to optimize the physical parameters of cloth. Lastly,\nwe demonstrate the ability to control cloth motion.\n\n4.1 Ablation Study\n\nAs mentioned in Sec. 3.4.1, our method for computing the gradients of the optimization can achieve a\nspeed-up of up to 183x in theory. We conduct an ablation study to verify this estimate in practice. In\norder to clearly measure the timing difference, we design a scenario with many collisions. We put\na piece of cloth into an upside-down square pyramid, so that the cloth is forced to fold, come into\nfrequent contact with the pyramid, and collide with itself, as shown in Fig. 2.\nWe measure the running time of backpropagation in each\nquadratic optimization and also the running time of the\nphysics solve as a reference. With all other variables \ufb01xed,\nwe compare to the baseline method where the gradients are\ncomputed by directly solving Equation 13. Timings are\nlisted in Tab. 1. In this experiment, the backpropagation\nof the physics solve takes from 0.007s to 0.5s, which,\ntogether with the timings of the baseline, implies that the\ncollision handling step is the critical bottleneck when there\nare many collisions in the scene. The results in Tab. 1 show\nthat our proposed method can signi\ufb01cantly decrease the\nmatrix size required for computation and thus the actual\nrunning time, resolving the bottleneck in backpropagation.\nThe experimental results also match well with the theory\nin Sec. 3.4. Each collision involves a vertex-face or edge-edge pair, which both have 4 vertices and\n12 variables. Therefore, the original matrix size (n + m = 13m) should be about 13 times bigger\nthan in our method (m). In our experiment, the ratio of the matrix size is indeed close to 13. Possible\nreasons for the ratio not being exactly 13 include (a) multiple collision pairs that share the same\nvertex, making n smaller, and (b) the constraint matrix can be of low rank, as described in Sec. 3.4.2,\nmaking the effective m smaller in practice.\n\nFigure 2: Example frame from the abla-\ntion study. A piece of cloth is crumpled\ninside a square pyramid, so as to gener-\nate a large number of collisions.\n\n6\n\n\fMesh\n\nresolution\n\n16x16\n32x32\n64x64\n\nBaseline\n\nMatrix size\n599 \u00b1 76\n1326 \u00b1 23\n2024 \u00b1 274\n\nRuntime (s)\n0.33 \u00b1 0.13\n1.2 \u00b1 0.10\n4.6 \u00b1 0.33\n\nOurs\n\nMatrix size\n66 \u00b1 26\n97 \u00b1 24\n242 \u00b1 47\n\nRuntime (s)\n0.013 \u00b1 0.0019\n0.011 \u00b1 0.0023\n0.072 \u00b1 0.011\n\nSpeedup\n\nMatrix size Runtime\n\n8.9\n13\n8.3\n\n25\n112\n64\n\nTable 1: Statistics of the backward propagation with and without our method for various mesh\nresolutions. We report the average values in each cell with the corresponding standard deviations. By\nusing our method, the runtime of gradient computation is reduced by up to two orders of magnitude.\n\n4.2 Material Estimation\nIn this experiment, our aim is to learn the material parameters of cloth from observation. The scene\nfeatures a piece of cloth hanging under gravity and subjected to a constant wind force, as shown\nin Fig. 3. We use the material model from Wang et al. [30]. It consists of three parts: density d,\nstretching stiffness S, and bending stiffness B. The stretching stiffness quanti\ufb01es how large the\nreaction force will be when the cloth is stretched out; the bending stiffness models how easily the\ncloth can be bent and folded.\nWe used the real-world dataset from Wang et al. [30], which consists of 10 different cloth materials.\nThere are in total 50 frames of simulated data. The \ufb01rst 25 frames are taken as input and all 50 frames\nare used to measure accuracy. This is a case-by-case optimization problem. Our goal is to \ufb01t the\nobserved data in each sequence as well as possible, with no \u201ctraining set\u201d used for training.\nIn our optimization setup, we use SGD with learning rate ranging\nfrom 0.01 to 0.1 and momentum from 0.9 to 0.99, depending on the\nconvergence speed. The initial guess is the set of average values\nacross all materials. We de\ufb01ne the loss as the average MSE across all\nframes. In order to speed up optimization, we gradually increase the\nnumber of frames used. Speci\ufb01cally, we \ufb01rst optimize the parameters\nusing only 1 simulated frame. We proceed to the second frame after\nthe loss decreases to a certain threshold. This optimization scheme\ncan help obtain a relatively good guess before additional frames are\ninvolved.\nAs a simple baseline, we measure the total external force and divide\nFigure 3: Example frame from\nit by the observed acceleration to compute the density. For the\nthe material estimation scene\nstretching stiffness, we simplify the model to an isotropic one and\nfor cloth blowing in the wind.\nrecord the maximum deformation magnitude along the vertical axis.\nSince the effect of the bending stiffness is too subtle to observe, we directly use the averaged value as\nour prior. We also compare our method with the L-BFGS optimization by Wang et al. [30] using\n\ufb01nite difference. We used the PyTorch L-BFGS implementation and set the learning rate ranging\nfrom 0.1 to 0.2 depending on the convergence speed.\nFor the performance measurement, we use the Frobenius norm normalized by the target as the metric\nfor the material parameters:\n\nE(P) = kP  P0kF\nkP0kF\n\n,\n\n(22)\n\nwhere P and P0 are the estimated and the target physics parameters, which stand for either density d,\nstretching stiffness S, or bending stiffness B. In order to show the \ufb01nal visual effect, we also measure\nthe average distance of the vertices between the estimated one and the target normalized by the size\nof the cloth as another metric:\n\nE(X) =\n\n1\n\nnT L X1\uf8ffi\uf8ffT,1\uf8ffj\uf8ffn\n\nkXi,j  Yi,jk2,\n\n(23)\n\nwhere L is the size of the cloth, and X and Y are T \u21e5n\u21e53 matrices denoting the n simulated vertex\npositions across T frames using the estimated parameter and the target, respectively.\nTab. 2 shows the estimation result. We achieve a much smaller error in most measurements in\ncomparison to the baselines. The reason the stiffness matrices do not have low error is that (a) a large\n\n7\n\n\fpart of them describes the nonlinear stress behavior that needs a large deformation of the cloth and\nis not commonly observed in our environment, (b) different stiffness values can sometimes provide\nsimilar results, and (c) the bending force for common cloth materials is too small compared to gravity\nand the wind forces to make an impact. The table shows that the linear part of the stiffness matrix\nis optimized well. With the computed gradient using our model, one can effectively optimize the\nunknown parameters that dominate the cloth movement to \ufb01t the observed data. We show in the\nsupplementary video that the estimated parameters yield very similar qualitative behavior to the\noriginal observation.\nCompared with regular simulators, our simulator is designed to be embedded in deep networks. When\ngradients are needed, our simulator shows signi\ufb01cant improvement over \ufb01nite-difference methods, as\nshown in Tab. 2. Regular simulators need to run one simulation for each input variable to compute the\ngradient, while our method only needs to run once for all gradients to be computed. Therefore, the\nmore input variables there are during learning, the greater the performance gain that can be achieved\nby our method over \ufb01nite-difference methods.\n\nMethod\n\nRuntime\n\n(sec/step/iter)\n\nNon-ln streching\nstiffness error (%)\n\nLn streching\n\nstiffness error (%)\n\nBending stiffness\n\nDensity\nerror (%)\n68 \u00b1 46\n4.2 \u00b1 5.6\n1.8 \u00b1 2.0\n\n-\n\n2.89 \u00b1 0.02\n2.03 \u00b1 0.06\n\nBaseline\nL-BFGS [30]\nOurs\nTable 2: Results on the material parameter estimation task. Lower is better. \u2018Ln\u2019 stands for \u2018linear\u2019.\nValues of the material parameters are the Frobenius norms of the difference normalized by the\nFrobenius norm of the target. Values of the simulated result are the average pairwise vertex distance\nnormalized by the size of the cloth. Our gradient-based method yields much smaller error than the\nbaselines.\n\n160 \u00b1 119\n72 \u00b1 90\n45 \u00b1 41\n\n74 \u00b1 23\n64 \u00b1 34\n57 \u00b1 29\n\nerror (%)\n70 \u00b1 42\n70 \u00b1 43\n77 \u00b1 36\n\nSimulation\nerror (%)\n12 \u00b1 3.0\n4.9 \u00b1 3.3\n1.6 \u00b1 1.4\n\n4.3 Motion Control\n\nFigure 4: Example frame from the mo-\ntion control experiment: dropping cloth\ninto a basket.\n\nWe further demonstrate the power of our differentiable\nsimulator by optimizing control parameters. The task is\nto drop a piece of cloth into a basket, as shown in Fig. 4.\nThe cloth is originally placed on a table that is away from\nthe basket. The system then applies external forces to the\ncorners of the cloth to lift it and drop it into the basket.\nThe external force is applied for 3 seconds and can be\nchanged during this period. The basket is a box with an\nopen top. A planar obstacle is placed between the cloth\nand the basket to increase the dif\ufb01culty of the task.\nWe de\ufb01ne the loss here as the squared distance between\nthe center of mass of the cloth and the bottom of the basket.\nTo demonstrate the ability to embed the simulator into neural networks, we also couple our simulator\nwith a two-layer fully-connected (FC) network that takes the mesh states as input and outputs the\ncontrol forces. Our methods here are compared to two baselines. One of the baselines is a simple\nmethod that computes the momentum needed at every time step. The entire cloth is treated as a\npoint mass and an external force is computed at each time step to control the point mass towards the\ngoal. Obstacles are simply neglected in this method. The other baseline is the PPO algorithm, as\nimplemented in Ray RLlib [18]. The reward function is de\ufb01ned as the negative of the distance of the\ncenter of mass of the cloth to the bottom of the basket. Please refer to the Appendix for additional\ndetails.\nTab. 3 shows the performance of the different meth-\nods and their sample complexity. The error shown in\nthe table is the distance de\ufb01ned above normalized by\nthe size of the cloth. Our method achieves the best\nperformance with a much smaller number of simula-\ntion steps. The bottom of the basket in our setting has\nthe same size as the cloth, so a normalized error of\nless than 50%, as our methods achieve, implies that\nthe cloth is successfully dropped into the basket.\n\nTable 3: Motion control results. The table\nreports the smallest distance to the target posi-\ntion, normalized by the size of the cloth, and\nthe number of samples used during training.\n\nMethod\nPoint mass\nPPO [18]\nOurs\nOurs+FC\n\nError (%)\n111\n432\n17\n39\n\nSamples\n\u2013\n10,000\n53\n108\n\n8\n\n\f5 Conclusion\nWe presented a differentiable cloth simulator that can compute the analytical gradient of the simulation\nfunction with respect to the input parameters. We used dynamic collision handling and explicitly\nderived its gradient. Implicit differentiation is used in computing gradients of the linear solver and\ncollision response. Experiments have demonstrated that our method accelerates backpropagation by\nup to two orders of magnitude.\nWe have demonstrated the potential of differentiable cloth simulation in two application scenarios:\nmaterial estimation and motion control. By making use of the gradients from the physically-aware\nsimulation, our method can optimize the unknown parameters faster and more accurately than\ngradient-free baselines. Using differentiable simulation, we can learn the intrinsic properties of cloth\nfrom observation.\nOne limitation of our existing implementation is that the current simulation architecture is not\noptimized for large-scale vectorized operations, which introduces some overhead. This can be\naddressed by a specialized, optimized simulation system based solely on tensor operations.\n\nReferences\n[1] Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as a layer in neural\n\nnetworks. In International Conference on Machine Learning (ICML), 2017.\n\n[2] David Baraff and Andrew Witkin. Large steps in cloth simulation. In SIGGRAPH, 1998.\n\n[3] Katherine L. Bouman, Bei Xiao, Peter Battaglia, and William T. Freeman. Estimating the\nmaterial properties of fabric from video. In International Conference on Computer Vision\n(ICCV), 2013.\n\n[4] Michael Bradley Cline. Rigid Body Simulation with Contact and Constraints. PhD thesis,\n\nUniversity of British Columbia, 2002.\n\n[5] Richard W Cottle. Linear Complementarity Problem. Springer, 2009.\n\n[6] Marco Cusumano-Towner, Arjun Singh, Stephen Miller, James F. O\u2019Brien, and Pieter Abbeel.\nBringing clothing into desired con\ufb01gurations with limited perception. In International Confer-\nence on Robotics and Automation (ICRA), 2011.\n\n[7] Filipe de Avila Belbute-Peres, Kevin A. Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico\nKolter. End-to-end differentiable physics for learning and control. In Advances in Neural\nInformation Processing Systems, 2018.\n\n[8] Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis wyffels. A differentiable physics\n\nengine for deep learning in robotics. Frontiers in Neurorobotics, 13, 2019.\n\n[9] Olaf Etzmu\u00df, Michael Keckeisen, and Wolfgang Stra\u00dfer. A fast \ufb01nite element solution for cloth\n\nmodelling. In Paci\ufb01c Conference on Computer Graphics and Applications, 2003.\n\n[10] Eitan Grinspun, Anil N Hirani, Mathieu Desbrun, and Peter Schr\u00f6der. Discrete shells. In\n\nSymposium on Computer Animation, 2003.\n\n[11] Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann,\nand Pascal Fua. GarNet: A two-stream network for fast and accurate 3D cloth draping. In\nInternational Conference on Computer Vision (ICCV), 2019.\n\n[12] David Harmon, Etienne Vouga, Rasmus Tamstorf, and Eitan Grinspun. Robust treatment of\n\nsimultaneous collisions. ACM Trans. Graph., 27(3), 2008.\n\n[13] Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B. Tenenbaum, William T. Freeman,\nJiajun Wu, Daniela Rus, and Wojciech Matusik. ChainQueen: A real-time differentiable\nphysical simulator for soft robotics. In International Conference on Robotics and Automation\n(ICRA), 2019.\n\n9\n\n\f[14] John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure\nIn International Conference on Learning Representations\n\nwith a differentiable simulator.\n(ICLR), 2019.\n\n[15] J Nathan Kutz. Deep learning in \ufb02uid dynamics. Journal of Fluid Mechanics, 814, 2017.\n[16] Zorah L\u00e4hner, Daniel Cremers, and Tony Tung. DeepWrinkles: Accurate and realistic clothing\n\nmodeling. In European Conference on Computer Vision (ECCV), 2018.\n\n[17] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning\nparticle dynamics for manipulating rigid bodies, deformable objects, and \ufb02uids. In International\nConference on Learning Representations (ICLR), 2019.\n\n[18] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E.\nGonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement\nlearning. In International Conference on Machine Learning (ICML), 2018.\n\n[19] Stephen Miller, Jur van den Berg, Mario Fritz, Trevor Darrell, Kenneth Y. Goldberg, and Pieter\n\nAbbeel. A geometric approach to robotic laundry folding. I. J. Robotics Res., 31(2), 2012.\n\n[20] Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep\ndynamical modeling and control of unsteady \ufb02uid \ufb02ows. In Advances in Neural Information\nProcessing Systems, 2018.\n\n[21] Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Fei-Fei Li, Josh Tenenbaum, and\nDaniel L. Yamins. Flexible neural representation for physics prediction. In Advances in Neural\nInformation Processing Systems, 2018.\n\n[22] Rahul Narain, Armin Samii, and James F. O\u2019Brien. Adaptive anisotropic remeshing for cloth\n\nsimulation. ACM Trans. Graph., 31(6), 2012.\n\n[23] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black. ClothCap: Seamless 4D\n\nclothing capture and retargeting. ACM Trans. Graph., 36(4), 2017.\n\n[24] Igor Santesteban, Miguel A. Otaduy, and Dan Casas. Learning-based animation of clothing for\n\nvirtual try-on. In Eurographics, 2019.\n\n[25] Connor Schenck and Dieter Fox. SPNets: Differentiable \ufb02uid dynamics for deep neural\n\nnetworks. In Conference on Robot Learning (CoRL), 2018.\n\n[26] Benoit Steiner, Zachary DeVito, Soumith Chintala, Sam Gross, Adam Paszke, Francisco Massa,\nAdam Lerer, Gregory Chanan, Zeming Lin, Edward Yang, et al. PyTorch: An imperative style,\nhigh-performance deep learning library. In Advances in Neural Information Processing Systems,\n2019.\n\n[27] Min Tang, Dinesh Manocha, and Ruofeng Tong. Fast continuous collision detection using\ndeforming non-penetration \ufb01lters. In Symposium on Interactive 3D Graphics and Games, 2010.\n[28] Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating\nEulerian \ufb02uid simulation with convolutional networks. In International Conference on Machine\nLearning (ICML), 2017.\n\n[29] Marc Toussaint, Kelsey Allen, Kevin Smith, and Joshua Tenenbaum. Differentiable physics and\nstable modes for tool-use and manipulation planning. In Robotics: Science and Systems (RSS),\n2018.\n\n[30] Huamin Wang, James F. O\u2019Brien, and Ravi Ramamoorthi. Data-driven elastic models for cloth:\n\nModeling and measurement. ACM Trans. Graph., 30(4), 2011.\n\n[31] Shan Yang, Junbang Liang, and Ming C. Lin. Learning-based cloth material recovery from\n\nvideo. In International Conference on Computer Vision (ICCV), 2017.\n\n10\n\n\f", "award": [], "sourceid": 394, "authors": [{"given_name": "Junbang", "family_name": "Liang", "institution": "University of Maryland, College Park"}, {"given_name": "Ming", "family_name": "Lin", "institution": "University of Maryland - College Park"}, {"given_name": "Vladlen", "family_name": "Koltun", "institution": "Intel Labs"}]}