Enhanced Multi-Scale Attention-Driven 3D Human Reconstruction from Single Image

1. Introduction

In the fields of computer vision and graphics, reconstructing 3D human models from a single image is both a challenging and significant task [1,2]. Compared to multi-view reconstruction, single-image reconstruction offers advantages such as ease of data acquisition, lower costs, and wide applicability. These benefits make single-image reconstruction particularly promising for various applications, providing rich possibilities for augmented reality, virtual try-on, human-computer interaction, and animation production [3,4]. For instance, in augmented reality, accurate 3D human models enable more immersive experiences by seamlessly integrating virtual objects with the real world [5]. Virtual try-on applications benefit from single-image reconstruction by allowing users to visualize how clothes fit without physically trying them on, enhancing online shopping experiences [6]. In human-computer interaction, realistic 3D models improve gesture recognition and enhance interactive applications [7,8], while in animation production, single-image reconstruction simplifies the creation of lifelike characters, reducing the need for extensive motion capture setups [9,10].

Recent methods leverage implicit functions to generate the final human mesh [11,12], thus avoiding explicit mesh partitioning and more naturally capturing shapes and details. Implicit functions offer greater flexibility by producing outputs at arbitrary resolutions and densities [13]. Another key advantage is their ability to significantly reduce memory consumption while maintaining the accuracy and quality of the models. As a result, implicit functions are gaining increasing attention from researchers in the field of 3D human reconstruction. The flexibility of implicit functions allows for smooth handling of complex geometries and fine details, which is particularly beneficial for applications requiring high precision, such as medical imaging and detailed character modeling in video games.

Despite the development of robust representation functions, learning the detailed textures of the non-visible side from a single image remains a major challenge. Existing methods often employ generative adversarial networks to predict normal maps for the non-visible side, thereby providing rich details and clothing the bare parameterized models with intricate garments [11,14]. However, these methods typically capture only local features, neglecting global and multi-scale information. This limitation becomes evident in complex scenes, particularly when dealing with intricate human details and structures [15]. Additionally, generating normal maps requires accurate modeling of different parts of the input image. Traditional CNN networks, constrained by fixed convolution kernel sizes, struggle to capture spatial relationships across various scales and sizes, resulting in insufficient modeling capacity for spatial relationships between different scale features.

Inspired by the powerful attention mechanism[16], we introduce an enhanced multi-scale attention module, starting from predicting the normal map of clothed human bodies. This mechanism aids the network in learning more discriminative feature representations at different scales, thereby enhancing the network’s modeling capability for complex scenes. Moreover, it allows the network to adaptively adjust weights based on feature information from different positions, facilitating better capture of spatial relationships between different positions in the image. This is beneficial for improving the accuracy of normal prediction and enhancing robustness to human body poses.

Furthermore, traditional CNN networks often rely on simple convolution and pooling operations to fuse features across different scales, limiting effective information fusion and interaction. This limitation hampers the network’s ability to fully utilize the correlation between different scale features when learning complex scene characteristics. To address this, we enhance the model’s capability to capture details and global structures in complex scenes by leveraging features at different scales and employing adaptive weights. This enhancement leads to improved accuracy and robustness in normal prediction.

To address issues of discontinuity and noise in the normal maps generated during reconstruction, we developed a hybrid loss function. This function reduces differences between adjacent pixels to suppress noise and discontinuities, resulting in more coherent and smooth normal maps. Our contributions are as follows:

• We propose a novel network model, Enhanced Multi-Scale Attention-Driven 3D Human Reconstruction from Single Image (EMAR), for robustly reconstructing 3D human meshes from a single image.
• To effectively capture and integrate global and local features, we introduce an enhanced multi-scale attention module that helps the network learn more discriminative feature representations at different scales, thereby improving its ability to model complex scenes.
• For effective multi-scale information fusion and interaction, we design a novel feature fusion module based on the enhanced multi-scale attention module, improving the accuracy and robustness of normal prediction.
• We introduce a hybrid loss function to supervise network training and ensure fast convergence, enhancing the network’s robustness in handling complex structures and resulting in high-fidelity 3D human models.

2. Related Works

Single image 3D human reconstruction. With the progression of deep learning, novel methodologies for reconstructing 3D human bodies from single images have made significant strides in precision and efficiency. [8] leverages supervised learning to train convolutional neural networks for direct volumetric prediction of the human body from a single image. [17] improve model robustness against complex real-world scenes using synthetic data and neural mesh rendering techniques, ensuring accurate 3D human posture and shape reconstruction. Lastly, [18] includes scene location information within complete frames to better comprehend the spatial position and posture of the human body, thereby enhancing the accuracy and robustness of single-view human body reconstruction. [19] combines conditional implicit functions and efficient voxel networks to accurately reconstruct 3D human models from a single image in complex scenes. [20] introduces an efficient network architecture that significantly enhances computational speed while maintaining detail fidelity, making it suitable for real-time applications. [21] proposes an explicit point-based 3D human reconstruction method that uses a novel network to generate high-resolution 3D point clouds from single-view RGB images, improving precision by accurately capturing surface details. [22] presents a new implicit representation method (IUVD) that decomposes the human body into multiple parts for separate processing, thereby enhancing reconstruction accuracy and efficiency. [23] proposes a tri-directional implicit function for high-fidelity 3D character reconstruction. By modeling from multiple directions, it improves detail level and accuracy in the reconstruction process. [24] introduces an implicit reconstruction method based on self-attention mechanisms and Signed Distance Functions (SDF). The self-attention mechanism enhances the model’s ability to capture complex geometric details, while SDF provides high-precision surface representation. This combination is particularly effective in reconstructing clothed human models, significantly improving reconstruction accuracy.

Attention mechanism. Attention mechanisms enhance the focus of neural networks on specific regions. To address partial occlusion issues, [25] employs motion continuity attention to capture temporal coherence, integrating features from temporally adjacent representations more effectively through a hierarchical attention feature mechanism. [26] uses a self-attention mechanism based on non-adjacent vertices of the body’s triangular mesh topology to minimize memory overhead and accelerate inference speed. [27] proposed a learnable sampling module for human mesh recovery to reduce inherent depth ambiguity, aggregating global information through joint adaptive markers and leveraging non-local information in the input image. [28] is a mesh transformer for end-to-end human mesh recovery, where the encoder captures interactions between vertices and joints, and the decoder outputs 3D joint coordinates and mesh structure. [29] introduced an end-to-end Transformer-based method for multi-person human mesh recovery in videos, extracting global features with a spatio-temporal encoder and predicting human posture and mesh using spatio-temporal pose and shape decoders. [30] utilizes the cross-attention mechanism in the transformer architecture to effectively decouple side-view features in the 2D-to-3D feature mapping process, significantly improving the accuracy and robustness of 3D models, especially in cases where SMPL-X estimation is imperfect. [31] applies the shifted window attention mechanism to better capture spatial relationships and details, enhancing single-view 3D reconstruction and demonstrating significant performance improvements. [32] employs attention mechanisms to enhance the generalization capability of Neural Radiance Fields, allowing high-quality 3D human models to be generated from sparse inputs, including single images.

3. Methodology

In this section, we present the details of our proposed EMAR. Section 3.1 describes the network overview of our EMAR. Section 3.2 explains the EMSA module. Section 3.3 shows the feature fusion module. Section 3.4 introduces our constructed hybrid loss function.

3.1. Network Overview

Our proposed EMAR, as shown in Figure 1, takes an RGB image ${\mathrm{I}}_{\mathrm{in}}$ as input and obtains the SMPL(-X) mesh M through human pose estimation, which can be replaced as needed. Using Pytorch3D rendering, the normalized maps ${\mathrm{N}}_{(\mathrm{V}/\mathrm{IN})}$ of the visible and invisible sides of the parameterized mesh are obtained. These two normal maps, along with the input image, are concatenated as the input to the clothing-normal prediction network. The network utilizes an enhanced multiscale attention module to extract multiscale features and a feature fusion module to merge features of different scales, thus predicting more accurate front and back clothing-normal maps ${\mathrm{N}}_{\mathrm{CB}(\mathrm{V}/\mathrm{N})}$. Finally, combining SDF features extracted from M predicts the final 3D human reconstruction result. Below, we will focus on describing our innovative part.

3.2. Enhanced Multi-Scale Attention Module

In the following sections, we will provide a detailed description of the Enhanced Multi-Scale Attention Module (EMSA). As illustrated in Figure 2, the input features are denoted as $X\in {\mathbb{R}}^{(c\times h\times w)}$, where $\mathrm{c}$ refers to the number of channels, and $\mathrm{h}$ and $\mathrm{w}$ represent the spatial dimensions of the input features. To enhance the learning and representation capabilities of features, we first divide the features into $\mathrm{g}$ groups, with each group having $\mathrm{c}/\mathrm{g}$ channels. In this paper, $\mathrm{g}$ is set to 3.

We map the input features of each group into four different branches. The second and third branches undergo horizontal and vertical pooling, respectively, extracting global information ${\mathrm{x}}_{\mathrm{h}}$ and ${\mathrm{x}}_{\mathrm{w}}$ along the corresponding directions. These features are then concatenated and processed through a $1\times 1$ convolution to further fuse the global information in the height and width directions, forming a new feature map. Subsequently, the fused feature map is split along the height and width directions to obtain new ${\mathrm{nx}}_{\mathrm{h}}$ and ${\mathrm{nx}}_{\mathrm{w}}$. To constrain the resulting values within the range [0, 1], ${\mathrm{nx}}_{\mathrm{h}}$ and ${\mathrm{nx}}_{\mathrm{w}}$ are transformed non-linearly through sigmoid activation functions. They are then combined with the original feature map of the first branch to inject global information along the height and width directions for feature enhancement. Normalization is performed to ensure that the mean feature value within each group is 0 and the variance is 1, further stabilizing the feature distribution. The process is outlined as follows:

$${\mathrm{f}}_1=\mathrm{GN}(\sigma \left({\mathrm{nx}}_{\mathrm{h}}\right)\times \sigma \left({\mathrm{nx}}_{\mathrm{w}}\right)\times {\mathrm{group}}_{\mathrm{x}})$$

Where ${\mathrm{f}}_1$ represents the output feature map, $\mathrm{GN}$ represents the group normalization function, $\sigma$ represents the sigmoid function, and ${\mathrm{group}}_{\mathrm{x}}$ represents the grouped data. After applying the sigmoid function to ${\mathrm{nx}}_{\mathrm{h}}$ and ${\mathrm{nx}}_{\mathrm{w}}$, element-wise multiplication is performed with ${\mathrm{group}}_{\mathrm{x}}$, and the resulting feature map is then processed through the group normalization function. Where ${\mathrm{nx}}_{\mathrm{h}}$ and ${\mathrm{nx}}_{\mathrm{w}}$ are obtained through the following functions:

$$\begin{array}{c}({\mathrm{nx}}_{\mathrm{h}},{\mathrm{nx}}_{\mathrm{w}})=\mathrm{split}(conv1\times 1(Xavgpool\left({\mathrm{group}}_{\mathrm{x}}\right)\\ +Yavgpool\left({\mathrm{group}}_{\mathrm{x}}\right)\left)\right)\end{array}$$

(1)

Where split denotes the splitting function, which splits the feature map after convolution into ${\mathrm{nx}}_{\mathrm{h}}$ and ${\mathrm{nx}}_{\mathrm{w}}$,$\mathrm{Conv}1\times 1$ represents a convolutional neural network with a $1\times 1$ kernel size. $\mathrm{Xavgpool}$ and $\mathrm{Yavgpool}$ respectively denote average pooling along the horizontal and vertical directions.

For the fourth branch, we directly apply two $3\times 3$ convolutions to increase the receptive field and enhance feature representation. Subsequently, the ReLU activation function is utilized to enhance the non-linear representation capability of features. Then, another $3\times 3$ convolution is performed to further extract and enhance features. Finally, the result of the fourth branch is combined with the results of the first three branches, which have undergone group normalization, and inputted into the feature fusion module (Sec. 3.3) for integration, ultimately yielding the final feature map. The data processing procedure for the fourth branch is outlined as follows:

$${\mathrm{f}}_2=\mathrm{conv}3\times 3\left(\mathrm{ReLU}(\mathrm{conv}3\times 3(\mathrm{conv}3\times 3))\right)$$

(2)

Here, ${\mathrm{f}}_2$ represents the result of processing in the fourth branch, ReLU denotes the ReLU activation function, and conv$3\times 3$ represents a convolutional neural network with a $3\times 3$ kernel size. The subsequent results from these two processes are fed into the feature fusion module, resulting in the final outcome:

$$f_{out}=ff(\mathrm{f}1,\mathrm{f}2)$$

(3)

Where $f_{out}$ represents the final output result, ff denotes the feature fusion module, and further details will be elaborated in Section 3C.

3.3. Feature Fusion Module

In 3D human reconstruction tasks, models need to handle feature information from different scales and resolutions. In complex scenes, features from different scales may contain diverse crucial information. However, conventional Convolutional Neural Networks (CNNs) often rely on simple convolution and pooling operations to fuse these feature representations. This approach fails to adequately capture the complex relationships and interactions among features at different scales, leading to insufficient modeling capabilities for complex scenes. To address these limitations, we propose a Feature Fusion Module (FFM) aimed at enhancing the model’s integration capability of features at different scales. This module achieves better capture of details and global structures in complex scenes through fusion of multi-scale features and adaptive weight adjustments.

Figure 3 illustrates our proposed Feature Fusion Module, which efficiently integrates multi-scale features through a series of refined operations, thereby enhancing the model’s adaptability in handling complex scenes. Specifically, this module receives feature information from two input feature maps (Input1 and Input2) and performs a series of processing operations on these two feature maps. Firstly, adaptive average pooling (AvgPool) is applied to each input feature map separately, compressing the global information of the feature maps into 1x1 feature maps to extract global features. Then, the pooled feature maps are reshaped to match subsequent matrix operations.

$$\mathrm{Reshape}(\mathrm{AvgPool}\left(\mathrm{Input}\right)={\displaystyle \frac{1}{H\times W}}\sum _{i=1}^H\sum _{j=1}^W\mathrm{Input}(i,j))$$

(4)

Then, the reshaped feature maps are subjected to the Softmax function to generate normalized weight matrices. These weight matrices reflect the importance of the feature maps at a global scale. Subsequently, through matrix multiplication (Matmul), these weight matrices are multiplied by the corresponding feature matrices to compute the fused weights. This step ensures that information from the feature maps at a local scale is preserved, and by weighting the fusion of global and local features, the expressive power of the features is enhanced.

$$f=\mathrm{Softmax}\left(\mathrm{AvgPool}\right(\mathrm{Input}\left)\right)$$

(5)

$$\begin{array}{c}\mathrm{Weights}=\mathrm{Matmul}(\left(f_1\right),\mathrm{Reshape}\left(\mathrm{Input}2\right))\\ +\mathrm{Matmul}(\left(f_2\right),\mathrm{Reshape}\left(\mathrm{Input}1\right))\end{array}$$

(6)

Here, $\mathrm{Input}$ denotes two inputs, ${\mathrm{Input}}_1$ and ${\mathrm{Input}}_2$, where ${\mathrm{f}}_1$ represents the processed value of ${\mathrm{Input}}_1$ and ${\mathrm{f}}_2$ represents the processed value of ${\mathrm{Input}}_2$.

After generating the fusion weights, the initial feature maps are combined with the fused weights through element-wise multiplication. This step enables the feature maps to adaptively adjust the weights of different features, thereby better integrating multi-scale feature information. Finally, after passing through the Sigmoid activation function, the final output feature map (Output) is generated. This feature fusion module effectively enhances the model’s ability to capture details and global structures, thereby improving its adaptability in complex scenes and enhancing the accuracy and robustness of normal prediction.

$$\mathrm{Output}=\sigma \left(\mathrm{sum}({\mathrm{Weights}}_1,{\mathrm{Weights}}_2)+{\mathrm{group}}_x\right)$$

(7)

Through this multi-step feature fusion process, the module dynamically adjusts and integrates feature information from different scales, fully exploiting the complementarity between global and local features, thereby significantly enhancing the model’s performance in handling complex scenes. This approach not only enhances the expressive power of features but also effectively addresses the issue of information loss in traditional convolutional neural networks when dealing with multi-scale features, greatly boosting the model’s performance.

3.4. Hybrid Loss Function

In normal map prediction, high-frequency noise and irregular edges may affect the quality of the results. To address this issue, we propose a mixed loss function consisting of three parts: L1 loss, VGG perceptual loss, and Total Variation regularization. The L1 loss calculates the absolute error between the predicted normal and the target normal, aiming to improve prediction accuracy, which can be computed by the following formula:

$$L_1={\parallel \mathrm{pred}-\mathrm{target}\parallel }_1$$

(8)

The VGG perceptual loss measures the difference between the predicted and target normals in the feature space using a pre-trained VGG network, enhancing visual quality and perceptual similarity, which can be calculated by the following formula:

$$L_{VGG}={\parallel \varphi \left(\mathrm{pred}\right)-\varphi \left(\mathrm{target}\right)\parallel }_2$$

(9)

Here, $\varphi$ denotes the feature extraction function of the VGG network. Total Variation regularization is employed to reduce high-frequency noise in the normal map, maintaining smooth edges, which can be calculated by the following formula:

$$\begin{array}{c}{L}_{TV}=\sum _{(i,j)}({({\mathrm{pred}}_{i+1,j}-{\mathrm{pred}}_{i,j})}^2\\ +{({\mathrm{pred}}_{i,j+1}-{\mathrm{pred}}_{i,j})}^2)\end{array}$$

(10)

Here, pred represents the predicted normal map, i and j are the pixel coordinates of the image, and ${\mathrm{pred}}_{i+1,j}$ and ${\mathrm{pred}}_{i,j+1}$ represent the pixel values of the image at positions $i+1,j$ and $i,j+1$, respectively. Additionally, in TV regularization, there is usually a regularization weight $\lambda$ to control the influence of the regularization term on the total loss, which typically ranges from positive real numbers, commonly between 0.01 and 10. In this paper, $\lambda =0.1$. Next, we calculate the losses separately for the visible and invisible sides of the normal map:

$$L_{\mathrm{total}\text{-}\mathrm{V}}=5.0\times L_{\mathrm{L1}\text{-}\mathrm{V}}+L_{\mathrm{VGG}\text{-}\mathrm{V}}+{\lambda }_{\mathrm{reg}}\times L_{\mathrm{TV}\text{-}\mathrm{V}}$$

(11)

$$L_{\mathrm{total}\text{-}\mathrm{IN}}=5.0\times L_{\mathrm{L1}\text{-}\mathrm{IN}}+L_{\mathrm{VGG}\text{-}\mathrm{IN}}+{\lambda }_{\mathrm{reg}}\times L_{\mathrm{TV}\text{-}\mathrm{IN}}$$

(12)

Finally, the total losses of the frontal and back normals are combined to form the final total loss:

$$\begin{array}{c}{L}_{\mathrm{final}}=[L_{\mathrm{total}\text{-}\mathrm{V}},L_{\mathrm{total}\text{-}\mathrm{IN}}]\end{array}$$

(13)

4. Experiments

In this section, we validate the feasibility of our algorithm through extensive experiments. Section 4.1 introduces the datasets and experimental settings used. Section 4.2 presents the three evaluation metrics used in the experiments. Section 4.3 outlines the comparison results with state-of-the-art methods. Section 4.4 provides ablation studies and related discussions, while Section 4.5 presents failure cases and analysis.

4.1. Datasets and Implementation

The Thuman2.0 [33] and Cape [34] datasets are widely used in the computer vision domain, each with its own unique characteristics. Thuman2.0 is a dataset for human pose estimation, providing rich samples of human poses where different poses have varying effects on the morphology of different body parts, which is crucial for reconstructing 3D human models. The Cape dataset, on the other hand, provides images and models of humans wearing different clothing. Clothing occlusion and shape significantly impact the appearance of the human body, necessitating the consideration of these factors in the training data. Therefore, we chose these two datasets.

We evaluated our approach on the CAPE dataset, which consists of 150 models, to test the generalization of EMAR without any module trained on the Cape dataset. We divided the Cape dataset into "CAPE-Hard" and "CAPE-Easy" categories based on the presence of "difficult poses" and "non-difficult poses", with "CAPE-Hard" containing 100 models and "CAPE-Easy" containing 50 models, to better analyze the generalization ability on complex poses.

The proposed method was implemented on a desktop computer with an Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz, NVIDIA RTX A6000 GPU, and 128GB of memory. The EMAR model, as proposed in this paper, supports end-to-end training and utilizes the Adam optimizer with an initial learning rate of 1e-4, a batch size of 4, and trained for 20 epochs.

Considering the characteristics of point clouds, we opted to use three specific loss functions: point-to-surface distance (p2s), Chamfer Distance, and Normal Difference. These loss functions are particularly well-suited for our task as they directly measure the geometric accuracy and surface smoothness of the 3D models. The p2s loss ensures that the generated point cloud closely aligns with the target surface, Chamfer Distance minimizes the overall discrepancy between two point clouds, and Normal Difference helps maintain consistent and accurate surface normals, thereby enhancing the fidelity of the reconstructed 3D geometry.

4.2. Comparison Experiments

Qualitative Experiments. As shown in Figure 4, the qualitative experimental results demonstrate that our method outperforms PIFu, PaMIR, and 2K2K. This improvement is attributed to our incorporation of the Enhanced Multi-Scale Attention Module, which enables the model to focus on more discriminative features at different scales. In contrast, PIFu and PaMIR are less effective in capturing details and handling complex features. The 2K2K method struggles with accurately estimating depth information due to the limited resolution of the input images, resulting in an unclear representation of the human form. Compared to ICON, our method shows significant improvements in detail, as illustrated in Figure 5. This is because we predict more accurate normal maps, providing the model with more reliable surface details.

Quantitative Experiments. We present the quantitative results in Table 1, where we tested all methods under two pose difficulty levels, "CAPE-Easy" and "CAPE-Hard." The metrics on the rightmost side under CAPE are the averaged results of "CAPE-Easy" and "CAPE-Hard" for the same row. It is evident from the table that our method outperforms the others.

4.3. Ablation Experiments

In this section, we compare and discuss the roles of the proposed modules, including the EMSA module and the Feature Fusion module.

To evaluate the effects of each component, we conduct ablation experiments using ICON as the baseline model. We implement the Feature Fusion module through channel concatenation, which is the most common method. Subsequently, we separately add the EMSA module and the Feature Fusion module to complete the experimental settings. Table 2 presents the numerical results of different combinations of these modules. With the addition of various key components, the performance of the network gradually improves, with our model achieving the best performance. Following is the baseline model with the EMSA module, while the baseline model performs the worst among all indicators. This further confirms the necessity and effectiveness of our proposed approach.

4.4. Failure Cases and Analysis

As shown above, our proposed EMAR can effectively reconstruct the 3D human mesh from a single image. However, our model still faces some challenging issues. As shown in Figure 6, when the body undergoes self-intersection, the predicted human mesh by the network often exhibits various faults. In Figure 6-A, the target person’s hands are clasped together, but the predicted human mesh shows the hands crossing fingers. In Figure 6-B, the target person is standing with one leg bent and the other leg stretched forward, while trying to reach forward to grab the heel with both hands, but the predicted human mesh is in a squatting position. This is due to inaccurate human pose estimation, which affects the final reconstruction results. Precise pose estimation will be the direction of our next work.

5. Conclusion

This paper presents an enhanced single-image 3D human body reconstruction method driven by multi-scale attention. To address the issue of inability to utilize feature correlations across different scales, we propose an Enhanced Multi-Scale Attention (EMSA) module, which helps the network learn more distinctive feature representations at various scales, thereby improving robustness to various human poses. To tackle the problem of ineffective integration of information at different scales, leading to insufficient modeling capabilities for complex scenes, we design a Feature Fusion Module (FFM) to enhance the model’s integration capability of features at different scales. Additionally, we introduce a loss function more suitable for normal map prediction, enabling the network to generate smoother normal maps. Finally, we conduct comparative experiments and ablation studies, and the results demonstrate that our method surpasses most state-of-the-art approaches.