RAE-NeRF: Residual-Based Audio-Video Encoder with Denoising in Talking Head Synchronization

1. Introduction

With the rapid advancement of artificial intelligence, digital avatars have shown great potential in various application scenarios such as intelligent assistants, customer service, virtual reality, and film production. To meet the requirements of specific tasks, constructing highly realistic and personalized digital human faces has become a growing research focus.

Among these technologies, speech-driven talking head synthesis has emerged as a core component, aiming to generate synchronized lip movements and facial expressions based on speech input, thereby enabling natural and lifelike dynamic face generation. This technology plays a particularly important role in fields such as digital assistants [1], virtual reality [2], and visual effects in film production [3], where higher demands are placed on visual realism and spatiotemporal coherence.

Although deep generative models have achieved significant progress in recent years [1,4,5,6,7,8,9], existing methods still fall short of satisfying the increasing demand for high-quality and highly interactive digital avatars. Traditional approaches based on Generative Adversarial Networks (GAN) [10,11,12,13] perform well in modeling lip movements, but often suffer from identity inconsistency, leading to visual artifacts such as unstable tooth size and fluctuating lip thickness during synthesis. Recently, NeRF-based methods have gained attention due to their ability to preserve facial details and maintain identity consistency. However, NeRF still faces several challenges in speech-driven talking head synthesis tasks, such as lip-sync mismatches, difficulty in controlling facial expressions, and unstable head poses-all of which negatively affect the naturalness and immersive quality of the generated videos.

Although many recent speech-driven talking head synthesis methods have made significant strides in improving synthesis quality and inference speed, several key challenges remain unresolved. For example, SyncTalk [14], ER-NeRF [15], and RAD-NeRF [16] incorporate Instant-NGP [17] technology to accelerate the generation of dynamic 3D avatars and enhance real-time performance. However, these methods still suffer from limitations in modeling precision and rendering fidelity. First, audio inputs are often affected by background noise, resulting in inaccurate feature extraction and subsequently degrading lip-sync accuracy and the naturalness of facial movements. Second, it remains challenging to fully model high-level correlations between audio and visual modalities during feature alignment, which is crucial for producing detailed and realistic dynamic expressions.

To address the aforementioned challenges, in this paper, we propose an enhanced speech-driven facial synthesis framework that focuses on improving audio-visual alignment accuracy and facial detail fidelity. Unlike previous approaches that directly feed raw audio into the encoder, our framework introduces a speech denoising module prior to the audio-visual encoding process. This effectively suppresses background noise interference, improving the clarity and stability of the audio signal at the source. By enhancing the expressiveness of speech features, our method allows for more accurate control of lip movements and facial dynamics, thereby significantly improving the naturalness of the synthesized avatars.

In addition, we design a Residual-based Audio-Visual Encoder to deeply extract and fuse the spatiotemporal correlations between audio and visual modalities. By incorporating residual connections, the encoder strengthens feature learning capacity and alleviates the gradient vanishing problem in high-dimensional spaces. This design improves the model’s accuracy in capturing complex facial dynamics and enhances the stability and consistency of audio-visual alignment. Notably, it delivers more natural motion restoration in detailed regions such as lips, cheeks, and jaw.

Experimental results demonstrate that our proposed method significantly outperforms existing models on several benchmark datasets, achieving noticeable improvements in lip-sync precision, facial expression naturalness, and overall visual fidelity. These advancements provide a solid foundation for building high-quality, photorealistic digital avatars and offer new insights for practical applications of speech-driven virtual humans.

In summary, the main contributions of this paper are as follows:

• Speech Denoising Module: We introduce a noise suppression component before the audio encoding stage to purify the input signal, reducing the impact of background noise and enhancing the usability of audio features.
• Residual-based Audio-Visual Encoder: We propose a novel encoder architecture that leverages residual connections to improve cross-modal feature fusion and alignment, thereby enhancing modeling accuracy and visual output quality.
• Extensive Experimental Validation: We conduct comprehensive quantitative and qualitative experiments across multiple datasets to evaluate the proposed method in terms of lip-sync accuracy, facial detail restoration, and visual realism. The results consistently demonstrate superior synthesis quality and robustness compared to existing approaches.

The remainder of this paper is organized as follows. Section II summarize related research achievement in the field of audio-driven talking head synthesis. Then in Section III, we present the details of our design, including the architecture and the neural networks. Subsequently, we provide experimental results to validate our design. Finally, Section V concludes the paper.

2. Related Works

Researchers have made substantial efforts to improve the performance of audio-driven talking head synthesis, introducing various denoising techniques to enhance signal-to-noise ratio (SNR) and improve signal quality. The main achievements in this field are summarized as follows.

2.1. Audio-Driven Talking Head Synthesis

In recent years, audio-driven talking head synthesis based on GANs has attracted significant attention [4,7,18,19,20,21,22,23,24]. While these methods have achieved notable progress in generating coherent video sequences, they still face challenges in maintaining identity consistency of the synthesized subjects. A typical class of approaches focuses on synthesizing the lip region [5,10,11,12,13,25], modeling lip movements to produce synchronized lip animations and enhance the expressiveness of talking heads. For instance, Wav2Lip [5] introduces a lip-sync expert module to improve the accuracy of lip movements. However, due to its reliance on multiple reference frames for reconstruction, it demonstrates limited capability in preserving identity consistency.

In contrast, works such as [4,6,8,26] attempt full-face synthesis. While these methods improve the overall facial dynamics, they often struggle to maintain synchronization between facial expressions and head poses. To address the limitations of GAN-based methods in terms of generation speed and spatial consistency, recent studies have turned toward NeRF-based approaches.

With the rapid development of NeRF techniques [27,28,29,30,31,32], their ability to efficiently model 3D structures has shown promising potential in talking head synthesis. However, traditional NeRF renderers are computationally intensive, slow to render, and require large memory, making them unsuitable for real-time applications.

To mitigate these issues, SSP-NeRF [33] introduces a semantically guided sampling mechanism that effectively captures the varying impact of audio signals on different facial regions, thereby enhancing the modeling of local facial dynamics. RAD-NeRF [16], built upon the Instant-NGP framework [17], further improves rendering efficiency and visual quality. Nevertheless, it still relies on a complex audio processing pipeline, which increases the training burden and computational cost. Moreover, some multi-stage strategies [34,35] pretrain audio-visual alignment modules and utilize NeRF-based renderers to generate higher-quality images.

While these methods enhance expression fidelity, the additional training stages often introduce synchronization errors, limiting their effectiveness in tasks that demand high temporal and spatial consistency.

2.2. Speech Denoising Techniques

Speech Enhancement (SE) aims to improve the quality of speech signals degraded by complex acoustic environments, making device-captured audio clearer and more intelligible. By suppressing background noise and highlighting relevant speech components, SE provides more reliable audio inputs for downstream tasks such as speech communication and automatic speech recognition (ASR), thereby enhancing overall performance and human-computer interaction experiences.

Current mainstream SE methods can be broadly categorized into two types: time-domain models and time-frequency (TF) domain models. Time-domain approaches directly model the signal at the waveform level by encoding noisy waveforms into latent representations, which are then processed using architectures such as Transformers and reconstructed into clean speech [36,37]. In contrast, TF-domain methods first extract Short-Time Fourier Transform (STFT) features from the noisy signal, use deep neural networks to predict the frequency domain representation of clean speech, and finally apply the inverse STFT (ISTFT) to reconstruct the waveform [38,39].

To address the quality degradation under low SNR conditions, recent studies have incorporated complex-valued modeling in the TF domain and employed both explicit and implicit strategies for optimizing magnitude and phase estimation of STFT [40], achieving significant improvements in speech restoration. Unlike ASR models that predominantly focus on temporal modeling (e.g., processing hidden features $X\in {\mathbb{R}}^{B\times T\times C}$) [41], SE and Speech Separation (SS) [42,43] tasks typically retain both temporal and frequency dimensions. In processing feature tensors $X\in {\mathbb{R}}^{B\times T\times F\times C}$, they often employ dual-path architectures to model along both T and F axes in parallel.

Although SE and SS models usually have fewer parameters (typically in the range of millions) compared to ASR models [44], their dual-path mechanisms introduce notable computational costs. In recent years, the development of efficient ASR encoders such as Efficient Conformer [45], Squeezeformer [46], and Zipformer [47] has significantly reduced the computational burden by introducing hierarchical downsampling mechanisms. For example, Squeezeformer applies a U-Net structure to downsample the temporal dimension of intermediate representations to 12.5 Hz, while Zipformer further introduces multi-level, non-uniform downsampling strategies to improve efficiency. Inspired by these advances, ZipEnhancer [48] extends temporal downsampling techniques to SE models and combines them with frequency-domain downsampling, achieving a favorable trade-off between speech quality and computational efficiency. This development paves the way for lightweight and high-performance SE solutions.

3. RAE-NeRF Architecture Design

In this section, we present the proposed RAE-NeRF framework in detail, with an overview of the architecture illustrated in Figure 1. First, we describe the core component of the ZipEnhancer model [48] used in the speech denoising module-namely, the DualPath Zipformer Blocks. Then, we elaborate on the structure of our Residual-based Audio-Visual Encoder. Finally, we introduce the tri-plane hash representation employed to model spatial geometric information.

3.1. DualPathZipformer Blocks

3.1.1. Down-UPSampleStacks

Unlike the single-path downsampling modules commonly used in automatic speech recognition (ASR), we adopt a dual-path structure for time-frequency domain modeling, which simultaneously performs upsampling and downsampling in both the time and frequency dimensions. As shown in Figure 2, the Down-UPSampleStack consists of paired DownSample and UpSample modules to achieve symmetric compression and expansion along either the temporal or frequency axes.

Specifically, we denote the down-up modules in the temporal dimension as T_DownSample and T_UpSample, and their counterparts in the frequency dimension as F_DownSample and F_UpSample. The sampling factor is represented by r. Our sampling strategy follows the simple method used in Zipformer: the DownSample module applies r learnable scalar weights (normalized by softmax) to aggregate every r frames, whereas the UpSample module repeats each frame r times. Through this process, the original input feature map $Y_l^{\prime }\in {\mathbb{R}}^{C\times T\times F}$, where C is the number of channels, T is the temporal length, and F is the frequency dimension-is downsampled into $Y_l^{\prime *}\in {\mathbb{R}}^{C\times {\displaystyle \frac{T}{r}}\times {\displaystyle \frac{F}{r}}}$ at layer l. The downsampled features are then passed to a DualPath ZipformerBlock for efficient modeling in both time and frequency domains at a reduced frame rate.

In addition, a Bypass module, analogous to residual connections, is included to preserve unsampled information and enhance the model’s representational capacity. It is worth noting that the dashed DownSample components shown in Figure 2 are optional and are omitted when $r=1$.

3.1.2. ZipformerBlock

In the Dual-Path structure, both the F_ZipformerBlock and T_ZipformerBlock share the same model architecture. For the compressed feature $Y_l^{\prime *}$ within a mini-batch, shaped as $B\times {\displaystyle \frac{T}{r}}\times {\displaystyle \frac{F^{\prime }}{r}}\times C$, where B is the batch size, the data is first reshaped to ${\displaystyle \frac{B\times T}{r}}\times {\displaystyle \frac{F^{\prime }}{r}}\times C$ and fed into the F_ZipformerBlock to capture frequency-domain dependencies. Then, it is reshaped to ${\displaystyle \frac{B\times F^{\prime }}{r}}\times {\displaystyle \frac{T}{r}}\times C$ and passed through the T_ZipformerBlock to model temporal correlations. The final output is the updated feature representation $Y_{l+1}^{\prime *}$.

3.1.3. Bypass

The Bypass module introduces a learnable residual connection that flexibly integrates the input x and the output y of a wrapped intermediate module. The output is defined as:

$$O=(1-w)\odot x+w\odot y$$

(1)

where w denotes a learnable channel-wise weight, and ⊙ indicates element-wise multiplication along the channel dimension. A smaller value of w indicates a higher tendency to bypass the intermediate module, thereby preserving more of the original input signal.

3.2. Residual-Based Audio-Visual Encoder

To effectively extract deep semantic information from input audio features, we design a residual-connected audio encoder module, termed the Residual-based Audio-visual Encoder. This module consists of multiple Fully Connected Residual Blocks (FCResBlocks), which help maintain the network’s expressive capacity while improving training stability.

3.2.1. Network Architecture

As illustrated in Figure 4, the input to the Residual-based Audio-Visual Encoder is a 512-dimensional audio feature vector. After passing through three FCResBlocks, it is mapped to a low-dimensional audio representation of size $d_{\mathrm{aud}}$ (set to 64 in our implementation). The transformation process is listed as follows:

• First FCResBlock: Input dimension = 512, Output dimension = 256;
• Second FCResBlock: Input dimension = 256, Output dimension = 128;
• Third FCResBlock: Input dimension = 128, Output dimension = $d_{\mathrm{aud}}$.

Each FCResBlock contains two fully connected (Linear) layers, with a LeakyReLU activation in between. A residual connection is employed to preserve low-level information from the input. The detailed formulation is provided in Equation 4.

3.2.2. Definition of FCResBlock

Given an input feature $\mathbf{x}\in {\mathbb{R}}^{d_{\mathrm{in}}}$, the output $\mathbf{y}$ of an FCResBlock is defined as:

$$\begin{array}{cc}\hfill {\mathbf{h}}_1& \hfill =\mathrm{LeakyReLU}({\mathbf{W}}_1\mathbf{x}+{\mathbf{b}}_1),\hfill \\ \hfill {\mathbf{h}}_2& \hfill ={\mathbf{W}}_2{\mathbf{h}}_1+{\mathbf{b}}_2,\hfill \\ \hfill \mathbf{r}& \hfill =\left\{\begin{array}{cc}\hfill \mathbf{x},\hfill & \hfill \mathrm{if}{d}_{\mathrm{in}}=d_{\mathrm{out}}\hfill \\ \hfill {\mathbf{W}}_{\mathrm{res}}\mathbf{x}+{\mathbf{b}}_{\mathrm{res}},\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right.,\hfill \\ \hfill \mathbf{y}& \hfill =\mathrm{LeakyReLU}({\mathbf{h}}_2+\mathbf{r}).\hfill \end{array}$$

(2)

Here, ${\mathbf{W}}_1$, ${\mathbf{W}}_2$, and ${\mathbf{W}}_{\mathrm{res}}$ are learnable weight matrices, and ${\mathbf{b}}_1$, ${\mathbf{b}}_2$, and ${\mathbf{b}}_{\mathrm{res}}$ are their corresponding biases. This structure ensures that essential information from the input is retained during the transformation process, helping to mitigate the vanishing gradient problem.

Figure 3. The overall architecture of Residual-based Audio-visual Encoder.

Figure 3. The overall architecture of Residual-based Audio-visual Encoder.

3.2.3. Down-UPSampleStacks

Unlike the single-path downsampling modules commonly used in automatic speech recognition (ASR), we adopt a dual-path structure for time-frequency domain modeling, which simultaneously performs upsampling and downsampling in both the time and frequency dimensions. As shown in Figure 2, the Down-UPSampleStack consists of paired DownSample and UpSample modules to achieve symmetric compression and expansion along either the temporal or frequency axes.

Specifically, we denote the down-up modules in the temporal dimension as T_DownSample and T_UpSample, and their counterparts in the frequency dimension as F_DownSample and F_UpSample. The sampling factor is represented by r. Our sampling strategy follows the simple method used in Zipformer: the DownSample module applies r learnable scalar weights (normalized by softmax) to aggregate every r frames, whereas the UpSample module repeats each frame r times. Through this process, the original input feature map $Y_l^{\prime }\in {\mathbb{R}}^{C\times T\times F}$, where C is the number of channels, T is the temporal length, and F is the frequency dimension-is downsampled into $Y_l^{\prime *}\in {\mathbb{R}}^{C\times {\displaystyle \frac{T}{r}}\times {\displaystyle \frac{F}{r}}}$ at layer l. The downsampled features are then passed to a DualPath ZipformerBlock for efficient modeling in both time and frequency domains at a reduced frame rate.

In addition, a Bypass module, analogous to residual connections, is included to preserve unsampled information and enhance the model’s representational capacity. It is worth noting that the dashed DownSample components shown in Figure 2 are optional and are omitted when $r=1$.

3.2.4. ZipformerBlock

In the Dual-Path structure, both the F_ZipformerBlock and T_ZipformerBlock share the same model architecture. For the compressed feature $Y_l^{\prime *}$ within a mini-batch, shaped as $B\times {\displaystyle \frac{T}{r}}\times {\displaystyle \frac{F^{\prime }}{r}}\times C$, where B is the batch size, the data is first reshaped to ${\displaystyle \frac{B\times T}{r}}\times {\displaystyle \frac{F^{\prime }}{r}}\times C$ and fed into the F_ZipformerBlock to capture frequency-domain dependencies. Then, it is reshaped to ${\displaystyle \frac{B\times F^{\prime }}{r}}\times {\displaystyle \frac{T}{r}}\times C$ and passed through the T_ZipformerBlock to model temporal correlations. The final output is the updated feature representation $Y_{l+1}^{\prime *}$.

3.2.5. Bypass

The Bypass module introduces a learnable residual connection that flexibly integrates the input x and the output y of a wrapped intermediate module. The output is defined as:

$$O=(1-w)\odot x+w\odot y$$

(3)

where w denotes a learnable channel-wise weight, and ⊙ indicates element-wise multiplication along the channel dimension. A smaller value of w indicates a higher tendency to bypass the intermediate module, thereby preserving more of the original input signal.

3.3. Residual-Based Audio-Visual Encoder

To effectively extract deep semantic information from input audio features, we design a residual-connected audio encoder module, termed the Residual-based Audio-visual Encoder. This module consists of multiple Fully Connected Residual Blocks (FCResBlocks), which help maintain the network’s expressive capacity while improving training stability.

3.3.1. Network Architecture

As illustrated in Figure 4, the input to the Residual-based Audio-Visual Encoder is a 512-dimensional audio feature vector. After passing through three FCResBlocks, it is mapped to a low-dimensional audio representation of size $d_{\mathrm{aud}}$ (set to 64 in our implementation). The transformation process is as follows:

• First FCResBlock: Input dimension = 512, Output dimension = 256;
• Second FCResBlock: Input dimension = 256, Output dimension = 128;
• Third FCResBlock: Input dimension = 128, Output dimension = $d_{\mathrm{aud}}$.

Each FCResBlock contains two fully connected (Linear) layers, with a LeakyReLU activation in between. A residual connection is employed to preserve low-level information from the input. The detailed formulation is provided in Equation 4.

3.3.2. Definition of FCResBlock

Given an input feature $\mathbf{x}\in {\mathbb{R}}^{d_{\mathrm{in}}}$, the output $\mathbf{y}$ of an FCResBlock is defined as:

$$\begin{array}{cc}\hfill {\mathbf{h}}_1& \hfill =\mathrm{LeakyReLU}({\mathbf{W}}_1\mathbf{x}+{\mathbf{b}}_1),\hfill \\ \hfill {\mathbf{h}}_2& \hfill ={\mathbf{W}}_2{\mathbf{h}}_1+{\mathbf{b}}_2,\hfill \\ \hfill \mathbf{r}& \hfill =\left\{\begin{array}{cc}\hfill \mathbf{x},\hfill & \hfill \mathrm{if}{d}_{\mathrm{in}}=d_{\mathrm{out}}\hfill \\ \hfill {\mathbf{W}}_{\mathrm{res}}\mathbf{x}+{\mathbf{b}}_{\mathrm{res}},\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right.,\hfill \\ \hfill \mathbf{y}& \hfill =\mathrm{LeakyReLU}({\mathbf{h}}_2+\mathbf{r}).\hfill \end{array}$$

(4)

Here, ${\mathbf{W}}_1$, ${\mathbf{W}}_2$, and ${\mathbf{W}}_{\mathrm{res}}$ are learnable weight matrices, and ${\mathbf{b}}_1$, ${\mathbf{b}}_2$, and ${\mathbf{b}}_{\mathrm{res}}$ are their corresponding biases. This structure ensures that essential information from the input is retained during the transformation process, helping mitigate the vanishing gradient problem.

Figure 4. The overall architecture of Residual-based Audio-visual Encoder.

Figure 4. The overall architecture of Residual-based Audio-visual Encoder.

3.3.3. Overall Module Workflow

The complete structure of the Residual-based Audio-Visual Encoder is shown in Figure 4. Suppose the input is a 3D tensor $\mathbf{X}\in {\mathbb{R}}^{B\times T\times 512}$, where B is the batch size and T is the time window size (defaulting to 16). The model encodes each frame independently along the temporal dimension, resulting in an audio representation $\mathbf{Z}\in {\mathbb{R}}^{T\times d_{\mathrm{aud}}}$, which is used for subsequent cross-modal alignment or task-specific modeling.

3.4. Tri-Plane Hash Representation

To represent a static 3D scene, NeRF [27] leverages multi-view images and their corresponding camera poses to construct an implicit function $\mathcal{F}$ for scene modeling. The function is defined as $\mathcal{F}:(x,y,z)\to (c,\sigma )$, where $(x,y,z)$ denotes a point in 3D space and $(\theta ,\varphi )$ indicates the viewing direction. The output consists of $c=(r,g,b)$, representing the radiance color at that location in the given direction, and $\sigma$, representing the volumetric density at that location.

The color value $\widehat{C}\left(r\right)$ of each pixel in the final rendered image is obtained by integrating the volumetric information sampled along a ray $r\left(t\right)=o+td$ originating from the camera position o in the direction d. The integration is computed as follows:

$$\widehat{C}\left(r\right)={\int }_{t_n}^{t_f}\sigma \left(r\left(t\right)\right)·c(r\left(t\right),d)·T\left(t\right)dt$$

(5)

Here, $t_n$ and $t_f$ represent the near and far bounds of sampling along the ray, respectively. $T\left(t\right)$ is the accumulated transmittance, which measures the probability that the ray is not occluded from the origin up to time t. It is defined as:

$$T\left(t\right)=exp\left(-{\int }_{t_n}^t\sigma \left(r\left(s\right)\right)ds\right)$$

(6)

To alleviate hash collisions and enhance feature representation capabilities, we introduce three orthogonal 2D hash grids [15]. Specifically, a 3D spatial coordinate $x=(x,y,z)\in {\mathbb{R}}^{\mathrm{XYZ}}$ is encoded using three multi-resolution 2D hash encoders from different projection directions, following the design in [17]. Each encoder projects the 3D point onto a 2D plane and maps the projected coordinates $(a,b)$ to a feature vector, as defined by:

$${\mathcal{H}}^{AB}:(a,b)\to {\mathrm{f}}_{ab}^{AB}$$

(7)

Here, the output feature ${\mathrm{f}}_{ab}^{AB}\in {\mathbb{R}}^{LD}$ encodes geometric information on the projection plane ${\mathbb{R}}^{AB}$, where L denotes the number of resolution levels and D is the feature dimensionality at each resolution level. We denote the multi-resolution hash encoder on plane ${\mathbb{R}}^{AB}$ as ${\mathcal{H}}^{AB}$.

The final geometric feature vector ${\mathrm{f}}_{\mathrm{x}}\in {\mathbb{R}}^{3\times LD}$ is obtained by concatenating the hash features from three orthogonal planes XY, YZ, and XZ:

$${\mathrm{f}}_{\mathrm{x}}={\mathcal{H}}^{XY}(x,y)\oplus {\mathcal{H}}^{YZ}(y,z)\oplus {\mathcal{H}}^{XZ}(x,z)$$

(8)

where ⊕ denotes the concatenation operation.

The resulting feature ${\mathrm{f}}_{\mathrm{x}}$, together with the viewing direction d, the lip feature $f_l$, and the expression feature $f_e$, is fed into an MLP decoder. The overall implicit function based on the tri-plane hash representation is defined as:

$${\mathcal{F}}^{\mathcal{H}}:(\mathrm{x},\mathrm{d},f_l,f_e;{\mathcal{H}}^3)\to (c,\sigma )$$

(9)

where ${\mathcal{H}}^3$ denotes the combination of the three hash encoders ${\mathcal{H}}^{XY}$, ${\mathcal{H}}^{YZ}$, and ${\mathcal{H}}^{XZ}$ as specified in Equation 7.

4. Experiments

4.1. Experimental Settings

The dataset used in our experiments consists of two parts: one comprises publicly available video sources, including datasets from [33,49,50]; the other consists of internally recorded videos, covering speech content in English, Chinese, and Korean. On average, each video contains approximately 6160 frames. The internally recorded videos have a frame rate of 30 frames per second (FPS), while the publicly available datasets operate at 25 FPS. In terms of resolution, internal videos are $540\times 540$, videos from the AD-NeRF dataset [49] are $450\times 450$, and other public videos are $512\times 512$. All videos feature centrally aligned human portraits.

For both qualitative and quantitative performance comparisons, we selected two NeRF-based methods as baselines: ER-NeRF [15] and SyncTalk [14], as well as a GAN-based method, Wav2Lip [5].

The training of our model is divided into two stages. In the coarse stage, the portrait head model undergoes 100,000 training iterations. This is followed by a fine stage with an additional 25,000 iterations to further improve synthesis quality. We employ 2D hash encoders with a resolution level of $L=14$ and a feature dimension $F=1$. In each iteration, ${256}^2$ rays are sampled for optimization. We use the AdamW optimizer [51], with a learning rate of 0.01 for the hash encoders and 0.001 for all other modules. All training processes are conducted on an NVIDIA GeForce RTX 3090 GPU, with the total training time being approximately 2 hours.

4.2. Denoising Performance Analysis

To evaluate the denoising effectiveness of the proposed ZipEnhancer network, we conducted a quantitative analysis using audio clips extracted from videos of six speakers in the publicly available dataset provided by [34].

We employed five widely adopted objective evaluation metrics to assess speech quality and intelligibility: Segmental Signal-to-Noise Ratio (SNRseg), Weighted Spectral Slope (WSS), Short-Time Objective Intelligibility (STOI), Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) [52], and Mel-Cepstral Distance (MCD) [53].

As illustrated in the Figure 5 and Figure 6 , we conducted a comprehensive quantitative evaluation of the ZipEnhancer module using five objective metrics. For the SI-SDR metric, the denoised speech achieved a score of 30.35 dB, representing a substantial improvement over the noisy input’s score of -9.05 dB. This significant gain indicates that the denoised signal is much closer to the original clean speech in terms of spectral power distribution, with interference effectively suppressed.

In terms of SNRseg, the segmental signal-to-noise ratio increased from -1.97 dB for the noisy input to 24.53 dB after denoising, demonstrating that the energy of speech segments relative to noise was greatly enhanced. Regarding the STOI metric, the denoised speech achieved a score of 0.9998, compared to 0.9982 for the noisy signal, indicating improved short-time intelligibility. This suggests that the denoised speech is more comprehensible and thus more suitable for downstream applications such as speech recognition and communication.

For the MCD metric, the denoised speech yielded a score of 0.178, significantly lower than the 5.275 measured for the noisy input. This reflects a major reduction in spectral distortion and a notable enhancement in perceived audio quality. Lastly, the WSS score dropped from 3.48 (noisy input) to 0.57 (denoised), indicating that the spectral tilt of the denoised signal more closely resembles that of the clean reference, thereby improving the naturalness and realism of the speech.

Taken together, the results across all five evaluation metrics consistently demonstrate that the ZipEnhancer network exhibits strong denoising performance in both speech quality and intelligibility. These improvements provide a robust foundation for subsequent talking-head synthesis tasks.

4.3. Quantitative Evaluation

Full Reference Quality Assessment To comprehensively evaluate the quality of the generated images, we employ several full-reference image quality metrics, including Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS) [54], Multi-Scale Structural Similarity (MS-SSIM), and Frechet Inception Distance (FID) [55]. These metrics measure different aspects of image clarity, structural fidelity, and perceptual similarity.

No Reference Quality Assessment Given that high PSNR images may still exhibit perceptual inconsistencies in texture details [56], we further incorporate two no-reference quality assessment methods for more precise and perceptually aligned evaluations: the Natural Image Quality Evaluator (NIQE) [57] and the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [58]. These metrics help assess the perceptual realism of generated images without relying on reference ground truths.

Synchronization Assessment To evaluate facial motion accuracy and synchronization, we adopt the Landmark Distance (LMD) metric, which measures the spatial discrepancy between predicted and ground-truth facial landmarks. A lower LMD score indicates better temporal alignment and motion fidelity.

Evaluation Results The quantitative evaluation results for head reconstruction are summarized in Table 1 and Table 2. We compare our method with existing GAN-based and NeRF-based approaches. It can be observed that our method achieves superior performance across all image quality metrics. Moreover, in terms of facial motion synchronization, our approach outperforms most competing methods, demonstrating its effectiveness in both visual quality and temporal consistency.

In the full reference quality evaluation (in Table 1), RAE-NeRF achieves a PSNR value of 42.642164, which is slightly higher than that of SyncTalk (42.492073) and significantly surpasses those of Wav2Lip and ER-NeRF. This indicates that RAE-NeRF generates images with lower luminance error and higher clarity. Regarding structural similarity, the SSIM score of RAE-NeRF reaches 0.999274, which is close to SyncTalk’s 0.999254 and notably higher than those of the other two methods. This suggests that RAE-NeRF effectively preserves structural details and produces images highly consistent with the original. In terms of perceptual quality, RAE-NeRF achieves the lowest LPIPS score (0.003332), indicating that its outputs are perceptually closest to the ground truth and visually more realistic. Furthermore, the FID score of RAE-NeRF is 1.401045, also the lowest among the four methods, demonstrating that the distribution of its generated images closely matches that of real images.

In the no reference quality and synchronization evaluation (in Table 2), RAE-NeRF shows superior performance as well. For lip motion synchronization, the LMD score of RAE-NeRF is 1.090, outperforming Wav2Lip (2.0289) and ER-NeRF (1.9174), and even better than SyncTalk (1.9961), indicating more accurate alignment between facial motion and speech. In terms of no-reference image quality metrics, RAE-NeRF achieves a BRISQUE score of 49.8398, lower than those of Wav2Lip and ER-NeRF, and comparable to SyncTalk (49.9054), reflecting a good balance of naturalness and perceptual quality. Its NIQE score is 7.3829, slightly lower than those of Wav2Lip (7.4609) and ER-NeRF (7.4267), and nearly identical to SyncTalk (7.3830), further confirming the superior perceptual quality of RAE-NeRF under no-reference conditions.

Taken together, the results from Table 1 and Table 2 demonstrate that our RAE-NeRF model consistently outperforms existing methods in both visual quality and dynamic accuracy. It not only ensures high image fidelity, structural consistency, and perceptual realism, but also achieves more precise synchronization of facial motion with speech, offering a robust and high-quality solution for head reconstruction tasks.

4.4. Qualitative Evaluation

Compared with existing methods such as Wav2Lip, ER-NeRF, and SyncTalk, the proposed RAE-NeRF demonstrates significant advantages across multiple dimensions, as illustrated in Figure 7. Overall, the generated videos achieve superior alignment with reference frames, more accurate lip-sync, and more natural facial expressions. Specifically, the lip movements generated by our method closely mirror the dynamic articulation patterns of real human speech. The eye gaze and facial expressions are also highly consistent with those in the original video, resulting in a strong sense of realism and naturalness. In contrast, Wav2Lip exhibits noticeable lip-sync errors in certain frames; ER-NeRF occasionally generates mismatched expressions or gaze directions; and SyncTalk suffers from incorrect mouth shapes and blinking artifacts in some frames, negatively impacting synchronization and visual quality.

In terms of eye movement, our method accurately captures and reproduces natural eyeball motion and maintains a realistic blinking frequency, effectively avoiding the “dead-eye” effect. This enhances the liveliness and emotional expressiveness of the generated faces. In comparison, SyncTalk shows considerable discrepancies in the eye region across several frames, while ER-NeRF suffers from slight misalignment or uneven blinking, leading to a more artificial appearance. By precisely modeling the dynamics from reference videos, our method significantly improves realism and immersion in conversational scenarios.

Regarding facial detail preservation, our method produces clearer facial features and retains more comprehensive details, including skin textures, consistent lighting, and facial shadows, resulting in visual outputs highly similar to the reference video. Wav2Lip, on the other hand, generates blurry results in some frames, while ER-NeRF and SyncTalk show limitations in rendering fine-grained textures.

In terms of facial expression generation, our method enables precise control over gaze direction, eyebrow movement, and subtle facial variations synchronized with speech rhythm, further enhancing the realism of synthesized videos. By contrast, SyncTalk shows weaker coordination between eye and mouth movements in some frames, and ER-NeRF produces relatively static facial expressions, lacking dynamic variation.

Moreover, our method excels in head pose consistency and stability, naturally reproducing variations in head orientation, tilt, and movement as observed in the reference video. ER-NeRF exhibits visible jitter in some frames, while SyncTalk, though generally stable, occasionally generates slightly rigid facial details.

Figure 8. Lip motion comparison results.

Figure 8. Lip motion comparison results.

Finally, in terms of lip motion, our method achieves higher accuracy and naturalness in modeling lip opening, shape variation, and movement trajectory. The generated results align closely with the reference video frames, effectively reflecting the dynamic articulatory patterns of human speech, thereby enhancing the realism and immersive experience of the synthesized talking-head videos.

4.5. Audio-video Encoder

We extracted the Audio-video Encoder used in SyncTalk (AudioNet_ave) and our proposed Residual-based Audio-visual Encoder (AudioNet_FCResBlock), and trained them on simulated noisy audio data to compare their convergence behavior in terms of training loss. As shown in the Figure 9, both models exhibit significant loss fluctuations during the initial training phase (approximately the first 40 epochs), but the overall trend indicates convergence. As training progresses, the loss of AudioNet_FCResBlock decreases more rapidly and becomes noticeably lower than that of AudioNet_ave after around 80 epochs, demonstrating its superior robustness and feature extraction capability. Furthermore, AudioNet_FCResBlock shows a more stable convergence trajectory throughout training, and achieves a significantly lower final loss value. These experimental results indicate that the proposed residual structure contributes to improved model fitting and generalization when dealing with noise-contaminated speech data, validating the effectiveness of our design in the talking head synthesis task.

5. Conclusion

This paper proposes an enhanced speech-driven facial synthesis framework, RAE-NeRF, aiming to address the instability, imprecise expression control, and suboptimal 3D reconstruction quality often encountered in existing methods under noisy speech conditions. The RAE-NeRF framework consists of three key modules: (1) the ZipEnhancer module, which effectively enhances speech clarity and provides robust audio features for subsequent processing; (2) a residual-based audio-visual encoder that introduces a residual structure to efficiently fuse audio and visual features, thereby improving expression-driving accuracy; and (3) a tri-plane hash encoder that enables high-quality 3D facial modeling and rendering while maintaining computational efficiency. Experiments on multiple datasets demonstrate that RAE-NeRF outperforms current mainstream methods in terms of realism, facial synchronization, and noise robustness. Notably, even under poor speech quality conditions, the proposed framework maintains stable and natural facial synthesis performance, showcasing strong robustness and generalization ability. Future work will explore the extension of RAE-NeRF to multi-language and diverse emotional states, and incorporate emotion recognition mechanisms to enhance the naturalness and intelligence of virtual human interactions.