Lightweight Context-aware Feature Transformer Network for Human Pose Estimation

1. Introduction

The goal of human pose estimation is to predict keypoints of human anatomy in the images. It has extensive applications in the field of computer vision for instance human action recognition [1,2,3,4], human pose tracking [5,6,7,8,9], 3D human pose estimation [10,11,12,13] and so on.

CNNs have obtained praiseworthy accomplishments in human pose estimation [14,15,16,17,18]during a recent period of time. However, the convolution’s receptive field is confined, which makes the CNNs unable to capture the dependence of remote interaction information. Recently, different methods [19,20,21,22,23] have been presented to remedy the shortcomings of the convolution limitation problem. A typical solution is to expand the receptive field to learn the global dependency information, for example by increasing the network depth [24,25,26]. However, deepening of the network will lead to a sharp increase in the number of parameters. Recently, Transformer [27] with self-attention has become a novel choice for a variety of visual tasks [28,29,30] for its capturing interactions between any pairwise positions. For human pose estimation, we expect to leverage global dependencies captured by self-attention to provide contextual clues for occluded keypoints. Because the body keypoints themselves have certain connections, as shown in Figure 1, global dependencies are able to improve the ability to locate difficult keypoints depending on easily detecting keypoints, thereby enhancing the performance of the overall network. There have been some recent works in CNN [31,32] directly model global dependencies with self-attention instead of convolution. For instance, CoT [32] encodes contextual information into the self-attention module, increasing representation ability of features. CoT can substitute $3\times 3$ convolutions in ResNet [33], while owning fewer parameters.

In order to fully leverage the advantages of CNNs and self-attention mechanisms, some researchers have combined [34,35]them to extract features. However, there are still some drawbacks for multi-scale networks [36,37,38,39]. Each subnetwork of multi-scale neural networks has a different resolution in order to exchange information between multiple resolution representations in feature fusion. High resolution features with more attention to detail information can precisely locate the position information of the keypoints. Low resolution features with a larger receptive field can capture global information about the human pose. In feature fusion, the accuracy of keypoints detection will be enhanced if our model can fully exploit the benefits of high and low resolution. However, some existing methods [40,41] ignore the differences between features at different resolutions, resulting in undesired fusion of noise features. To bridge the differences between features at different resolutions, an effective approach is to utilize the attention mechanism. Because attention can make the network to stress or restrain information through learning, so that the network can better grasp the information we need to pay attention to. Recently, some scholars have conducted relevant research [42,43,44]. For example, CBAM [45]considers channel and spatial relationships and generates spatial attention maps finally. Therefore, we also expect our model to have the ability to learn information in both channel and spatial orientations.

Based on the above studies, in this article, we put forth a Lightweight Context-aware Feature Transformer Network (CaFTNet) built upon HRNet, to improve network efficacy by enhancing the localization accuracy of occluded keypoints. Firstly, to strengthen the semantic features of contextual information, we design a Transformerneck structure. Transformerneck directly replaces the $3\times 3$ convolution in bottleneck with Contextual Transformer (CoT) block, while reducing the complexity of the network. Then inspired by the CBAM, to further refine the features of the fusion output, we design an Attention Feature Aggregation Module(AFAM). Due to the diversity of human poses, CBAM is still insufficient for spatial processing as it only employs a $7\times 7$ convolution filter for feature fusion, while spatial attention is decided by the value of each pixel, not the region of $7\times 7$. So, we propose an ARM to activate the obtained features. Therefore, our method further reinforces the feature fusion of multi-scale networks and ameliorates the output features. On the COCO [46], our model achieves better result than other methods with the CNN as the backbone. What’s more, notably, the model reduces the number of parameter by $72.9$%. On the MPII [47], our method takes advantage of 60.7% of the number of parameters, acquiring semblable results to other methods with the CNN as the backbone.

2. Related Work

2.1. Human Pose Estimation

Currently, CNNs have gained tremendous success in the field of human pose estimation. Hourglass [36] belongs to an hourglass type of network structure, which can perceive more global information. CPN [48] has two stages: GlobalNet and RefineNet, which can alleviate the detection problem of hard keypoints. Simple baseline [24] adds some transpose convolutional layers to restore the resolution. It indicates importance of high resolution feature maps. HRNet [14] proposes a network with high-resolution representations through the whole process, which repeats multi-scale fusions to improve the representation power of feature maps. Accordingly, HRNet achieves impressive results in multiple benchmark datasets. However, HRNet still falls within the category of CNNs, facing the issue of limited receptive fields. Therefore, global information needs to be ameliorated.

2.2. Attention enhanced Convolution

Convolution is dependent on a well-sized convolution kernel to gather information, which leads to the inability of the CNNs to establish global dependencies. The existing multiple approaches to image attention mechanisms suggest that they can compensate for the problem of confining of the convolution receptive field. Therefore, many scholars have explored the apply of attention to improve the capability of CNNs. SENet [43] models the interactions between the channels by using global mean pooling and two fully-connected layers. On the basis of SENet, ECANet [44] is came up with. Local cross-channel interaction strategy without decreasing the dimension is designed, which further improves the performance. CBAM [45] calculates attention maps on the channel and spatial directions to better learn useful information in the feature maps.

Recently, with the rise of the self-attention of Transfomer, the interest is aroused due to the powerful global dependence modeling ability of self-attention. Some works [49,50,51,52] have shown that self-attention modules are proposed as individual blocks, which can wholly substitute for the convolutions in HRNet. Although self-attention can effectively capture interactions between any paired positions. However, the pairwise query-key relationships are learned individually on isolated query-key pairs, without taking into account the abundant contextual information between them during the learning process.This seriously restricts the self-attention learning ability of two-dimensional feature maps for visual representation learning. Most recently, [31] displaces $3\times 3$ convolutions with self-attention in the final stage of the network. [32] replaces $3\times 3$ convolution in each bottleneck by leveraging the CoT block, which can take full advantage of the context of the query-key to model global dependencies.

2.3. HRNet

HRNet [14] utilizes a stem to fastly downsample the input feature. As shown in Figure 2, the HRNet network can be segmented into four stages. The first stage mainly has a high-resolution subnetwork. And from the second stage, a low-resolution subnetwork is added to each stage. The resolution of the new subnetwork is half of the lowest resolution of the previous stage. Each stage will interact with information through multi-resolution blocks.

HRNet has achieved remarkable success as a feature extractor. The problem of the limited receptive field inherent to the convolution operation needs to be improved. HRNet is unable to establish long-term dependencies, resulting in incorrect estimation of some human poses. For this reason, this paper proposes a Lightweight Context-aware Feature Transformer Network(CaFTNet). CaFTNet firstly capitalizes on the CoT block to enhance the expressiveness of the features. Then in feature fusion, the CaFTNet exploits AFAM to enhance the representative power of the output feature maps. And our final results are also better.

3. Methods

In this section, we put forward a CaFTNet to better perform the feature extraction. Figure 3 depicts the framework of our presented model. To begin with, we briefly review framework of our CaFTNet. Then, we introduce Transformerneck and AFAM in detail.

3.1. Context-aware Feature TransformerNetwork(CaFTNet)

The aim of this paper is to enhance the representational ability of the feature maps and lighten network model in pose estimation. The overall architecture of CaFTNet is revealed in Figure 3 . The CaFTNet puts HRNet as the backbone and enhances it with the presented Transformerneck and Attention Feature Aggregation Module(AFAM).

First, our proposed Transformerneck is used to extract preliminary input features with contextual information. It is represented by a blue dashed line box. Transformerneck is to replace the 3 × 3 convolution with CoT while keeping the bottleneck framework unchanged. Secondly, this input features continue to encode the feature information through the Mid-HRNet. Then, we place the AFAM on the head of the neural networks to further refine the enriched contextual features. AFAM is represented by a green dashed line box. AFAM successively deduces attention maps along channel and spatial directions. Adaptive refinement module(ARM) is exploited to activate the obtained attention maps. The input undergoes adaptive feature refinement through multiplication with the activated attention maps. Next, output of the AFAM goes through Transformer Encoder Layer to encode position representation of keypoints. Finally, a head is attached to Transformer Encoder output to predict keypoints heatmaps.

3.2. Transfomerneck

For a middle input $X\in R^{H\times W\times C}$ , an output H is first obtained through a $1\times 1$ convolution and a nonlinear activation layer. The output H is sent into the CoT (As shown in the green rectangle enclosed in Figure 4). H is represented as:

$$H=ReLU\left(BN\left(Con{v}_{1\times 1}\left(X\right)\right)\right).$$

(1)

H will then be defined the K, Q and V along three different paths. The K first produces contextualized $K_1$ through $3\times 3$ convolutions. The formula of $K_1$ is described as follows:

$$K_1=Con{v}_{3\times 3}\left(K\right).$$

(2)

Then, $K_1$ and Q are conducted the operation of concatenation. And the result of the operation generates the attention map M by two series of $1\times 1$ convolutions. The formula of M is calculated :

$$M=Con{v}_{1\times 1}\left(ReLU\left(Con{v}_{1\times 1}\left(Concat\left(K_1,Q\right)\right)\right)\right).$$

(3)

Next, V first passes through a $1\times 1$ convolution to obtain $V_1$, the feature map $K_2$ can be computed as follows:

$${\mathit{V}}_{\mathit{1}}=Con{v}_{1\times 1}\left(V\right),$$

(4)

$$K_2=F\left(V_1\otimes M\right),$$

(5)

where $F(\otimes )$ denotes the matrix multiplication operation. The final output Z of CoT is thus calculated as the fusion of $K_1$ and $K_2$ . Z continues to produce T through a nonlinear activation layer and a $1\times 1$ convolution layer. The T and a shortcut connection are added element-wise to produce Y with context relations. Finally, Y is sent to the next module via the Relu activation function.

3.3. Attention Feature Aggregation Module(AFAM)

To begin with, we consider feature map $F\in R^{H\times W\times C}$ as input in Figure 5. F through a cs module, generating the spatial attention map $F_s$ that we need. This process can be described as two steps in Figure 6. The first step, $F_c$ can be described as:

$$F_c=Sigmoid\left(MLP\left(AvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)\otimes F.$$

(6)

The second step, the $F_c$ was fed to the spatial attention model to gain the $F_s$. The $F_s$ is adopted as:

$$F_s=Sigmoid\left(Con{v}_{7\times 7}\left(\left[AvgPool\left(F_c\right);MaxPool\left(F_c\right)\right]\right)\right)\otimes F_c.$$

(7)

Next, $F_s$ is reshaped to feature sequences $F_q$, $F_k$ and $F_v$ in order to model the spatial context relations of the corresponding features. Detailed description of this process is as follows: (1)$F_s$ obtains the spatial context feature $F_v$ through a $1\times 1$ convolution and a sigmoid layer in the last row. $F_v$ is represented as:

$$\begin{array}{c}\hfill F_v=Sigmoid\left(Con{v}_{1\times 1}\left(F_s\right)\right).\end{array}$$

(8)

(2) $F_s$ rearranges the spatially related context features together respectively through two $1\times 1$ convolutions and a non-linear activation layer to obtain $F_q$ and $F_k$. $F_q$ and $F_k$ are represented as:

$$\begin{array}{c}\hfill F_q=Co{n}_{1\times 1}\left(ReLU\left(Con{v}_{1\times 1}\left(F_s\right)\right)\right),\end{array}$$

(9)

$$\begin{array}{c}\hfill F_k=Co{n}_{1\times 1}\left(ReLU\left(Con{v}_{1\times 1}\left(F_s\right)\right)\right).\end{array}$$

(10)

(3) The $F_q$ and $F_k$ are multiplied element-wise to obtain an attention map with contextual relationships, which is subsequently applied to feature for recalibrating the output feature $F_1$. $F_1$ is represented as:

$$\begin{array}{c}\hfill F_1=F_v\otimes Sigmoid\left(F_q\otimes F_k\right).\end{array}$$

(11)

Finally, $F_1$ and F are added element-wise to achieve $F^{\prime }$. The $F^{\prime }$ is adopted as:

$$F^{\prime }=F_1\oplus F.$$

(12)

4. Experiments

This section may be divided by subheadings. It should provide a concise and precise description of the experimental results, their interpretation as well as the experimental conclusions that can be drawn.

4.1. Model Variants

Based on HRNet, we present Lightweight Context-aware Feature Transformer Network. In our structure, there are two different depths of CNNs to extract the input features. Detailed setup information are presented in Table 1. The network utilized by CaFTNet-R is ResNet. The backbone network utilized by CaFTNet-H4 is the HRNet-W48. From Table 2, we can find that the model reaches best result when the network employs CaFTNet-H4.

4.2. Technical details

Our model takes advantage of the top-down [53,54,55] approach. The experimental environment configuration is shown as follows: Two RTX 2080s are deployed. Python Version is 3.7. Framework is PyTorch 1.10.0. The network model is optimized utilizing the Adam [56] optimizer during training, with an initial learning rate of 0.001 and 0.00001 at 220 rounds. The network is trained 230 rounds with a batchsize of 16 for each GPU. Because the size of the pictures in the dataset is different, the pictures are modified by image pre-processing. Here, images are cropped to 256 × 192 on the COCO and 256 × 256 on the MPII.

4.3. Results on COCO

4.3.1. Dataset and Evaluation Metrics

The COCO [46] has more than 200,000 images and 250,000 instances, each containing up to 17 human keypoints. The network model is trained on the train2017 dataset, and the network model is verified and tested on val2017 (including 5,000 images) and test-dev2017 (including 20,000 images). Our model is measured by Object Keypoint Similarity ($\mathit{OKS}$) on the COCO dataset. $\mathit{OKS}$ defines the similarity between different human keypoints, ${\mathit{AP}}^{\mathit{50}}$ indicates the accuracy of the keypoints at OKS = 0.5, and ${\mathit{AP}}^{\mathit{75}}$ is the accuracy of the keypoints at $\mathit{OKS}$ = 0.75. The $\mathit{mAP}$ is defined as the mean accuracy value of the predicted keypoints at the 10 thresholds of $\mathit{OKS}$ = 0.50, 0.55 ... 0.90, 0.95. ${\mathit{AP}}^{\mathit{M}}$ is utilized to describe the accuracy of the detection of medium size keypoints, and ${\mathit{AP}}^{\mathit{L}}$ represents the accuracy of large size keypoints. The formula of OKS is described as:

$$OKS=\frac{{\sum }_{i}exp\left(-d_i^2/2s^2{k}_i^2\right)\delta \left(v_i>0\right)}{{\sum }_i\delta \left(v_i>0\right)},$$

(13)

where $d_i$ is the Euclidean distance between the i-th predicted keypoint coordinate and the corresponding groundtruth, $v_i$ is the visibility flag of the keypoint, s is the object scale, and $k_i$ is a keypoint-specific constant.

4.3.2. Quantitative Results

The models are compared for their performance on the COCO val2017, and the results are shown in Table 3. Where the number of parameters and GFLOPs are calculated from the human pose estimation network model. The experimental results show that the CaFTNet model gains good performance with less number of parameters and GFLOPs. CaFTNet-H4 acquires 76.2 AP scores with an input size of 256×192, better than other models with the same input size. In contrast to the TransPose-R-A4 [57], CaFTNet-R has an 8.3% drop in the number of parameters, but increasing AP scores by 1.1. In contrast to the ResNet-152 [33], our CaFTNet-R obtains better performance, utilizing only 7.2% of the model parameters. With the complex network model of HRNet-W48 [14] contrast, CaFTNet acquires a good AP score with much lower complexity. Table 4 exhibitions the results of our approach and the other approaches on the COCO test-dev2017. Our CaFTNet-H4 reaches 75.5 AP. Due to the effective perceptual context semantic information, CaFTNet achieves a good balance between accuracy and complexity.

4.3.3. Qualitative Comparisons

Different keypoints rely on different regions. We can find that for the keypoints of the head like the nose, the eyes, etc. Their positioning depends mainly on the interdependencies between the keypoints, and it is worth noting that the prediction of the wrist or knees depend on the favorable cues around them. For instance, the prediction of the right knee depends on the left knee and the right lower limb. A closer look shows that our network has the ability to derive useful information from its relevant parts for keypoints to predict targets. In this way, we can understand why the model can predict the occluded keypoints(e.g. the occluded right knee in Figure 7 (a) ).

Visualization. Contrast the results for more visualizations are revealed on COCO in Figure . The source image is displayed at the top of the picture, and the middle row displays the HRNet results, and our results are displayed in the last row. The objects (enclosed in red circles) that were not detected by HRNet in the first and second columns of images are likely due to occlusion by other objects. HRNet may have treated the undetected objects as background during the detection process. In comparison, the proposed AFAM in this paper can weight the features during information fusion output, allowing for better prediction of occluded objects. Images of in the third column, our approach can accurately detect occluded keypoints. This is because our model introduce CoT, which allows for better capture of contextual information, providing beneficial cues for detecting occluded keypoints. As a result, our approach achieves superior results.

4.4. Results on MPII

4.4.1. Dataset and Evaluation metric

MPII[47] is a single-person pose estimation dataset that captures the whole-body pose of people in real scenes, including 28,821 training images, 11,701 test images, a benchmark dataset in single-person pose estimation. The division of the training and validation sets are 22,246 and 2958 images, respectively. The standard evaluation index of MPII is $PCKh$ (head-normalized percentage of correct keypoint), using the head segment length as the normalization reference. $PCKh$ is described as:

$$PCK{h}_i=\frac{{\sum }_p\delta \left(\frac{d_{pi}}{L_p^{head}}\le 0.5\right)}{{\sum }_{p}1},$$

(14)

$$PCK{h}_{mean}=\frac{{\sum }_p{\sum }_i\delta \left(\frac{d_{pi}}{L_p^{head}}\le 0.5\right)}{{\sum }_p{\sum }_{i}1},$$

(15)

where i represents the i-th keypoint, p represents the p-th pedestrian, $d_{pi}$ is the Euclidean distance between the p-th individual’s i-th predicted keypoint coordinate and the corresponding groundtruth, $\delta \left(·\right)$ represents the indicator function, ${\mathit{L}}_{\mathit{p}}^{\mathit{head}}$ indicates the p-th head segment length. We report $\mathit{PCKh}@\mathit{0}.\mathit{5}\left(\alpha =0.5\right)$ score for fair comparison with other methods.

4.4.2. Quantitative Results

Table 7 presents the contrast results of the different approaches on the MPII, which can find that the approaches in this paper exceed the HRNet. In more detail, we can find from the Table 7 that although our final results are only higher than 0.1 of the baseline method. Especially for ankles detection, our method is 0.3 higher than the TokenPose-L/D24. And each test result of our method outperforms the baseline method. And thus proves that our method achieves better performance on this dataset. More importantly, our method uses only 61 percent of the number of baseline method parameters. Our results are better compared to SimpleBaseline-Res152, and number of our parameter decreases by 74.8%.

4.4.3. Qualitative Comparisons

We reveal some contrasting results on the MPII dataset in Figure 9. The source image is displayed at the top of the picture, and the middle row displays the HRNet results, and our results are displayed in the last row. From the visualization results of the third line, our method can correctly detect the keypoints of the occlusion not detected by HRNet. Mainly because our model can better capture the contextual information and provide favorable clues for blocking keypoints. Thus, our approach achieves better results.

4.5. Ablation experiments

The ablation experiments are chosen for training validation on the COCO dataset, considering the role of Transformerneck and AFAM in the network model.

4.5.1. Transformerneck

In this paper, two sets of ablation experiments are devised to verify the effect of employing bottleneck or Transformerneck separately based on our different network models. When implementing our network model with CaFTNet-H, we replace the bottleneck structure with our proposed Transformerneck, while keeping the other structures unchanged. The structures using Transformerneck reached 75.7 AP in Table 5. We also report the results of replacing bottleneck with Transformerneck when employing CaFTNet-R. Results with Transformerneck yield 0.6 more values than results employing bottleneck. The results illuminate the fundamentality of excavating the contextual information for decoding the subsequent features.

4.5.2. Attention Feature Aggregation Module(AFAM)

We investigate the effects of different attention on the experimental results, for example (i)SENet; (ii)ECANet; (iii)CBAM; (iv)AFAM. Due to their different operation of the feature map, the influence on the results of the experiment is also different. SENet [43] models the interactions between the channels by using global mean pooling and two fully-connected layers. On the basis of SENet, [44] proposes ECANet. Local cross-channel interaction strategy without decreasing the dimension is designed, which further improves the performance. [45] puts forward CBAM to focus more on spatial attention maps. AFAM compensates for the lack of spatial processing of CBAM, and it keeps the network focused at our more desired features. Table 6 presents the contrasting results from our different settings. Although the difference between them is small, we can become conscious of our proposed AFAM at 0.5 higher than using SE. The results expose that more spatial information is needed when solving the feature fusion problems.

Table 6. Contrasting results for the COCO under different attentions.

Model	Baseline	SE	ECA	CBAM	AFAM	AP
CaFTNet-R	✓					72.6
CaFTNet-R	✓	✓				72.7
CaFTNet-R	✓		✓			72.8
CaFTNet-R	✓			✓		73.0
CaFTNet-R	✓				✓	73.2

Table 6. Contrasting results for the COCO under different attentions.

Model	Baseline	SE	ECA	CBAM	AFAM	AP
CaFTNet-R	✓					72.6
CaFTNet-R	✓	✓				72.7
CaFTNet-R	✓		✓			72.8
CaFTNet-R	✓			✓		73.0
CaFTNet-R	✓				✓	73.2

Table 7. Results on the MPII validation set (PCKh@0.5).

Model	Hea	Sho	Elb	Wri	Hip	Kne	Ank	Mean	Params
SimpleBaseline-Res50[24]	96.4	95.3	89.0	83.2	88.4	84.0	79.6	88.5	34.0M
SimpleBaseline-Res101[24]	96.9	95.9	89.5	84.4	88.4	84.5	80.7	89.1	53.0M
SimpleBaseline-Res152[24]	97.0	95.9	90.0	85.0	89.2	85.3	81.3	89.6	68.6M
HRNet-W32[14]	97.1	95.9	90.3	86.4	89.1	87.1	83.3	90.3	28.5M
TokenPose-L/D24[58]	97.1	95.9	90.4	86.0	89.3	87.1	82.5	90.2	28.1M
CaFTNet-H4	97.2	96.1	90.5	86.5	89.3	86.9	82.8	90.4	17.3M

Table 7. Results on the MPII validation set (PCKh@0.5).

Model	Hea	Sho	Elb	Wri	Hip	Kne	Ank	Mean	Params
SimpleBaseline-Res50[24]	96.4	95.3	89.0	83.2	88.4	84.0	79.6	88.5	34.0M
SimpleBaseline-Res101[24]	96.9	95.9	89.5	84.4	88.4	84.5	80.7	89.1	53.0M
SimpleBaseline-Res152[24]	97.0	95.9	90.0	85.0	89.2	85.3	81.3	89.6	68.6M
HRNet-W32[14]	97.1	95.9	90.3	86.4	89.1	87.1	83.3	90.3	28.5M
TokenPose-L/D24[58]	97.1	95.9	90.4	86.0	89.3	87.1	82.5	90.2	28.1M
CaFTNet-H4	97.2	96.1	90.5	86.5	89.3	86.9	82.8	90.4	17.3M

5. Conclusions

In this article, we put forth the Lightweight Context-aware Feature Transformer Network (CaFTNet) for enhancing the efficacy of human pose estimation models. Since CNNs cannot capture long-range dependencies between global regions, we devise Transformerneck. Furthermore, to bolster the representation power of the fusion output feature maps, we design Attention Feature Aggregation Module(AFAM). Extensive experiments carried out on the COCO and MPII corroborate the availability of the proposed approach.

6. Patents