ViDS: Video Diffusion Shader using 3D Face Tracking

Wenbo Ji¹, Davide Davoli², Zhe Chen³, Liam Schoneveld³, Matthias Nießner¹, Jiapeng Tang^1,†

¹Technical University of Munich ²Toyota Motor Europe NV/SA ³Woven by Toyota

Video Diffusion Shader creates expressive and identity-preserving portrait animation from a single image.

ViDS animates sculptural, stylized, and photorealistic source portraits across diverse expressions and head poses using dense 3D face-tracking normal maps — Given a single source portrait (left), ViDS transfers diverse expressions and head poses from a driving sequence using dense 3D face-tracking normal maps (insets), producing identity-preserving animations across sculptural, stylized, and photorealistic subjects.

Abstract

We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips.

Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices.

Method Overview

ViDS conditions generation on a source image, a sequence of Pixel3DMM normal maps, and a text prompt generated from the driving video by a vision-language model. The reference identity shape is estimated once and kept fixed while pose and expression parameters are transferred from the driving sequence. The resulting normal maps provide dense, pixel-aligned, and lighting-agnostic geometric guidance.

The source image, control sequence, and noise latents are VAE-encoded and combined through a unified conditioning layer before entering the DiT backbone. For long videos, overlapping sliding windows reuse the same reference embedding, fix a latent prefix from the preceding window, and blend overlapping decoded frames to improve identity and temporal consistency.

Qualitative Results

We compare ViDS with Follow-Your-Emoji, X-Portrait, HunyuanPortrait, and Wan-Animate across self-reenactment, cross-reenactment, and zero-shot in-the-wild generalization. ViDS handles challenging expressions and large head-pose changes while preserving source identity and appearance. On the VFHQ test set, ViDS achieves the best self-reenactment SSIM, LPIPS, PSNR, CSIM, AKD, and FID, together with the best cross-reenactment CSIM and APD.

Qualitative comparison of ViDS against four portrait-animation baselines across self-reenactment, cross-reenactment, and in-the-wild generalization — **Qualitative comparisons.** From top to bottom: self-reenactment, cross-reenactment, and in-the-wild generalization. ViDS provides stronger identity preservation, visual quality, and expression and pose transfer under difficult motion.

Generation Beyond 3DMM Regions

Although the 3DMM supplies structural guidance only for the face, the diffusion shader synthesizes non-parametric details such as teeth, tongue, and hair. Text prompts complement the geometric condition when the requested motion extends beyond the tracked mesh.

ViDS examples showing an extreme open-mouth expression and a large head-pose change — **Extreme expression and pose.** ViDS generates faithful inner-mouth details and preserves appearance under large rotations.

ViDS example showing hair motion generated with complementary text guidance — **Hair motion.** Text guidance adds dynamics outside the rigid 3DMM region.

Quantitative Results

On the VFHQ test set, ViDS leads most reconstruction, identity, motion, and perceptual metrics. In self-reenactment, it achieves the best SSIM, LPIPS, PSNR, CSIM, AKD, and FID. In cross-reenactment, it obtains the best CSIM and APD while remaining competitive in perceptual quality.

Quantitative comparisons on the VFHQ test set
Method	Self-Reenactment									Cross-Reenactment
Method	SSIM ↑	LPIPS ↓	PSNR ↑	CSIM ↑	AED ↓	APD ↓	AKD ↓	FID ↓	FVD ↓	CSIM ↑	AED ↓	APD ↓	IQA ↑
Follow-Your-Emoji	0.691	0.209	20.25	0.833	0.157	0.036	4.008	22.55	0.016	0.366	0.305	0.053	59.34
X-Portrait	0.683	0.227	19.46	0.853	0.118	0.031	3.134	24.10	0.031	0.311	0.285	0.079	58.97
HunyuanPortrait	0.622	0.309	17.54	0.673	0.113	0.043	17.47	26.20	0.018	0.252	0.279	0.080	56.74
Wan-Animate	0.713	0.187	21.13	0.876	0.118	0.017	2.775	24.01	0.013	0.227	0.286	0.053	64.51
Ours	0.722	0.178	21.19	0.879	0.121	0.022	2.752	21.29	0.020	0.372	0.298	0.049	62.85

Best results are bold; second-best results are underlined. Rankings use unrounded values. Swipe horizontally to view all metrics.

Comparisons Versus Baselines

Self Reenactment (VFHQ Test Set)

Reference Image Target Image Follow-Your-Emoji X-Portrait HunyuanPortrait Wan-Animate Ours

Cross Reenactment (VFHQ Test Set)

Reference Image Target Image Follow-Your-Emoji X-Portrait HunyuanPortrait Wan-Animate Ours

In-the-wild Generalization (Human-like Synthetic Characters)

Reference Image Target Image Follow-Your-Emoji X-Portrait HunyuanPortrait Wan-Animate Ours

In-the-wild Generalization (Stylized Synthetic Characters)

Reference Image Target Image Follow-Your-Emoji X-Portrait HunyuanPortrait Wan-Animate Ours

Ablation Studies

Types of Conditioning Signals

We compare MediaPipe landmarks, SHeaP normals, and three Pixel3DMM renderings. Landmarks are too sparse for precise control, while UV maps and gray meshes omit fine local surface detail. Pixel3DMM normal maps provide the strongest overall balance of identity preservation and expression transfer.

Ablation comparing MediaPipe landmarks, SHeaP normals, Pixel3DMM gray meshes, UV maps, and normal maps — **Ablation on conditioning signals.** Dense Pixel3DMM normal maps retain local geometry and give more accurate expression and pose control.

Reference / Target MediaPipe Landmark SHeaP Normal Pixel3DMM Mesh Pixel3DMM UV Pixel3DMM Normal

Reference Shape Parameters

Keeping the reference subject's 3DMM shape parameters is important for identity preservation. Using the driving subject's shape produces inaccurate proportions and contours; swapping in the reference identity restores the intended facial structure.

Reference shape parameter ablation comparing results without and with identity-shape swapping — **Ablation on reference shape parameters.** Reference-identity shape parameters recover the correct proportions, contours, and facial structure.