TL;DR: HeteroAvatar combines 3D Gaussian and mesh rendering to reconstruct photorealistic, animatable, and geometrically accurate Gaussian head avatars from monocular videos of dynamic heads.
Head avatar reconstruction is a crucial research topic with wide-ranging applications. For this purpose, utilizing the promising technique of 3D Gaussian splatting significantly improves rendering quality. However, this method encounters challenges, such as imprecise geometry and noisy surfaces. To address these issues, we propose HeteroAvatar, a novel method for generating a Gaussian head avatar with correct geometry from a monocular video of a dynamic head. To achieve this, we introduce the hetero rendering method, which concurrently employs both 3D Gaussian rendering and mesh rendering techniques during training. This approach enables HeteroAvatar to accurately capture the head geometry with smooth surfaces while maintaining photorealistic rendering quality. Additionally, our approach is capable of generating a mesh-based head avatar, which ensures compatibility with a wide range of traditional graphics applications. Experimental results demonstrate that HeteroAvatar outperforms state-of-the-art methods in terms of reconstructed geometry quality and photorealistic rendering with fewer 3D Gaussians.
Given a monocular video, a head tracker algorithm is employed to extract face-related parameters (\(\boldsymbol{\beta}\), \(\boldsymbol{\psi}\), \(\boldsymbol{\theta}\)) along with a FLAME mesh model. We then utilize a geometry-guided 3D Gaussian initialization, where a 3D Gaussian is associated with each vertex of the mesh. This process yields both an initial Gaussian model \(\boldsymbol{\mathcal{G}_0}\) and a corresponding mesh model \(\boldsymbol{\mathcal{M}_0}\). Learnable features \(\boldsymbol{L_i}\) are subsequently assigned to each vertex. The training pipeline utilizes canonical and deformation modules to refine the shape and learn Gaussian parameters, followed by the use of both mesh rendering and Gaussian rendering for image generation. Mesh rendering is essential for accurately capturing the geometry, as it ensures the alignment of 3D Gaussian centers with the actual head surface. Meanwhile, 3D Gaussian rendering contributes significantly to improving the visual quality. In the lower part of the figure, the red and blue blobs represent 3D Gaussians attached to base and subdivided vertices, respectively. The green blobs represent free Gaussians for better image quality.
The Gaussian positions of head avatars reconstructed across multiple subjects demonstrate that HeteroAvatar consistently aligns the Gaussians with the underlying head geometry, producing geometrically accurate and smooth surfaces across diverse identities.
Qualitative comparisons across multiple subjects demonstrate that HeteroAvatar achieves visual quality comparable to or better than existing state-of-the-art methods, while preserving accurate head geometry and fine appearance details.
Additional video comparisons across diverse subjects demonstrate that HeteroAvatar faithfully reproduces ground-truth appearance, facial expressions, head motion, and fine facial details with consistently high photorealistic quality.
Novel-view rendering results across multiple subjects demonstrate that HeteroAvatar synthesizes photorealistic and consistent head appearances from previously unseen viewpoints while preserving facial details and accurate head geometry.
Cross-identity reenactment experiments across multiple subjects demonstrate that HeteroAvatar enables reconstructed head avatars to be animated using the head poses and facial expressions of different individuals while preserving the target identity and maintaining realistic visual quality.
HeteroAvatar additionally generates mesh-based head avatars through the mesh-rendering process, accurately capturing the underlying head geometry and producing smooth surface representations.
The generated mesh-based head avatars are compatible with existing commercial graphics applications, enabling their direct use in various applications using only the mesh model and its corresponding texture map.
If you find this work useful in your research, please consider citing:
@article{seker2025heteroavatar,
title = {HeteroAvatar: Generation of Gaussian Head Avatars With Correct Geometry Using Hetero Rendering},
author = {Seker, Ahmet Cagatay and Kang, Jun Seok and Ahn, Sang Chul},
journal = {IEEE Access},
volume = {13},
pages = {33768--33782},
year = {2025},
publisher = {IEEE}
}