Dynamic garment reconstruction from monocular video is an important yet challenging task due to the complex dynamics and unconstrained nature of the garments. Recent advancements in neural rendering have enabled high-quality geometric reconstruction with image/video supervision. However, implicit representation methods that use volume rendering often provide smooth geometry and fail to model high-frequency details. While template reconstruction methods model explicit geometry, they use vertex displacement for deformation which results in artifacts. Addressing these limitations, we propose NGD, a Neural Gradient-based Deformation method to reconstruct dynamically evolving textured garments from monocular videos. Additionally, we propose a novel adaptive remeshing strategy for modeling dynamically evolving surfaces like wrinkles and pleats of the skirt, leading to high-quality reconstruction. Finally, we learn dynamic texture maps to capture per-frame lighting and shadow effects. We provide extensive qualitative and quantitative evaluations to demonstrate significant improvements over existing SOTA methods and provide high-quality garment reconstructions.
Unlike previous approaches, our method aggregates features across multiple timestamps, enabling the reconstruction of plausible geometric details in all views, including the back view, which remains occluded from the camera.
The method consists of geometry and appearance reconstruction modules. For geometry, it introduces a new deformation parameterization over a base garment mesh that separates global shape from pose-specific surface deformations. To enhance detail, the approach includes a gradient-based remeshing strategy that increases mesh resolution in high-curvature regions, allowing accurate modeling of features like wrinkles and folds.
The appearance reconstruction module learns both a frame-invariant base texture map and a frame-dependent dynamic texture map to capture the visual details of the garments.
@article{park2021nerfies,
author = {Park, Keunhong and Sinha, Utkarsh and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Seitz, Steven M. and Martin-Brualla, Ricardo},
title = {Nerfies: Deformable Neural Radiance Fields},
journal = {ICCV},
year = {2021},
}