Mesh-to-4D: Animating Textured Meshes using Video Diffusion Priors

H. Ata Atasoy, Yujin Chen
Course Project
Code
Mesh-to-4D method overview

Mesh-to-4D is a framework for animating arbitrary triangle meshes using motion priors from video diffusion models. Our approach operates in two stages:

In the Static Stage, we optimize Surface-Aligned Gaussian Splats (SuGaR) from multi-view renders to anchor high-fidelity appearance to the mesh geometry. This stage ensures that the visual details remain consistently tethered to the underlying mesh surface during subsequent deformations.

In the Dynamic Stage, we extract an underlying Skeleton from a pre-trained Neural Skinning Model and calculate Joint-Vertex Bindings based on geodesic distance to facilitate structured deformation. Guided by a Video Diffusion Model (VDM), we train a deformation MLP to predict per-vertex displacements. These movements are regulated via Adaptive Hybrid Skinning (LBS + DQS) and optimized under photometric, SDS, and As-Rigid-As-Possible (ARAP) losses to ensure the mesh evolves realistically while preserving its topological integrity.

Results

Limitations and Future Work

While Mesh-to-4D demonstrates promising results in animating arbitrary meshes, there are several limitations and opportunities for future enhancement:

  • Topological Priors: The framework currently assumes that the articulation model can extract a geometrically sound skeleton. For meshes with complex or non-manifold topologies, skeletal extraction may fail.
  • Robust Appearance Modeling: While SuGaR provides a strong geometric foundation, standard RGB and Silhouette losses are sensitive to the inherent noise and "texture flickering" found in current video diffusion models. Integrating robust deep features, such as those from DINOv2/v3, could provide more semantically consistent supervision.
  • Multi-View Motion Consistency: Our current approach relies on single-view supervision from camera-conditioned generation, which can be ambiguous for motion occurring along the optical axis. Integrating generative models like GCD could enable the synthesis of consistent novel views for the generated video. This multi-view supervision would allow the model to observe and supervise the scene from multiple angles, leading to a more robust 3D understanding of complex movements.
  • Monocular Geometric Priors: A straightforward enhancement is the incorporation of monocular depth or normal estimations (e.g., from Marigold or Depth Anything). While these priors provide useful local geometric cues, they were not included in our final pipeline as they were insufficient to resolve failure cases involving rapid, drastic movements, particularly in regions with uniform appearance.
  • Explicit Motion Tracking: Currently, the model relies on implicit visual supervision. Our experiments suggest that integrating dense pixel trackers like Co-Tracker provides a powerful supervision signal; however, these trackers remain susceptible to failure during high-velocity or dramatic occlusions.

In summary, the framework's robustness could be significantly enhanced by replacing raw RGB losses with deep feature supervision and incorporating explicit motion tracking. Furthermore, while multi-view video generation offers a path toward resolving 3D ambiguities, it introduces the complex challenge of maintaining temporal and cross-view consistency between synthesized viewpoints.

Related Work

This project builds upon and relates to several key works in the field:

DreamMesh4D: An approach for video-to-4D generation of 3D meshes. Our method builds heavily on their ideas.

Animating the Uncaptured: A novel approach for text-driven animations of humanoid meshes.

Anima-X: A feed-forward 3D animation framework that bridges the motion priors of video diffusion models with the controllable structure of skeleton-based animation.

AnimateAnyMesh: feed-forward framework for text-driven universal mesh animation.

Puppeteer: Given an input mesh, Puppeteer first transforms it into an animation-ready model through automatic rigging, and subsequently animates it under video guidance.

SuGaR (Surface-Aligned Gaussian Splatting): The foundation for our high-fidelity appearance modeling.