Human Pose Estimation for Yoga using VGG-19 and COCO Dataset

Introduction

• Skeleton-based Models: These models represent the human body as a collection of joints connected by bones. The coordinates of the joints are tracked over time to understand the movement and posture.
• Contour-based Models: These models focus on the outer contour of the body, capturing the silhouette to infer pose and movement.
• Volume-based Models: These models create a volumetric representation of the body, capturing the full 3D structure, which is useful for more detailed analysis.

• 2D Pose Estimation: This technique involves estimating key points in the joints of the human body in the 2D space with respect to the image or video. It serves as a foundation for more advanced computer vision tasks like 3D human pose estimation, motion prediction, and human parsing.
• 3D Pose Estimation: This technique involves estimating the actual spatial positioning of the body in the 3D space, introducing the z-dimension. It provides a more comprehensive understanding of the body’s posture and movement [3].

Literature Review

A.

Early Models and Approaches

• Pictorial Structures: The concept of pictorial structures was introduced by Fischler and Elschlager in the 1970s. This approach represented objects using a collection of parts and their spatial relationships [4]. Felzenszwalb and Huttenlocher later made this method practical and tractable using the distance transform trick, which significantly improved its efficiency and accuracy [5].
• Datasets: Earlier models used smaller datasets like Parse and Buffy for evaluation. However, these datasets were not suitable for training complex models due to their limited size and variability. The introduction of larger datasets, such as the Leeds Sports Pose (LSP) dataset containing 10,000 images, marked a significant milestone in the development of HPE models [6].

B.

Advancements with Larger Datasets

• COCO Dataset: The COCO (Common Objects in Context) dataset is a large-scale object detection, segmentation, and captioning dataset that has become a standard benchmark for HPE. It provides a diverse set of images with annotated key points, making it ideal for training and evaluating HPE models [7].
• MPII Human Pose Dataset: The MPII dataset is another extensive dataset that includes around 25,000 images with annotated body joints. It covers a wide range of human activities and poses, providing a robust benchmark for HPE algorithms [8].

C.

Key Libraries and Frameworks

• • OpenPose: Developed by Zhe Cao and his team in 2019, OpenPose is a real-time multi-person key point detection library capable of detecting 135 key points. It uses a bottom-up approach, which is efficient for handling multiple persons in an image. OpenPose is trained on COCO and MPII datasets and has become one of the most popular tools in HPE [9].

    

    Figure 1. Overall pipeline of Real time 2D pose estimation.

    Figure 1. Overall pipeline of Real time 2D pose estimation.

    
  • DeepCut: Presented by Leonid Pishchulin in 2016, DeepCut uses a bottom-up approach with Integral Linear Programming to model detected key points and form a skeleton representation. It addresses the challenge of multi-person pose estimation by simultaneously detecting and associating body parts [10].
  • AlphaPose: Developed in 2016, AlphaPose uses a topdown approach for human pose estimation. It detects human bodies first and then localizes key points within the detected regions. AlphaPose supports various operating systems and is known for its high accuracy and robustness [11].

D.

Convolutional Neural Networks (CNNs) in HPE

• • VGG-19: VGG-19 is a convolutional network that is 19 layers deep and was trained on the ImageNet database. It can classify more than 1,000 objects and is known for its performance in image recognition tasks. In this project, we use the first 10 layers of VGG-19 for feature extraction, which provides the basis for detecting key points in human poses [12].
  • High-Resolution Net (HRNet): Introduced by Jingdong Wang, HRNet maintains high-resolution representations through the entire network. It has been used for semantic segmentation, object detection, and HPE, providing high accuracy and detailed pose estimations [13].

C.

Model Training

I. Methodology

A.

Data collection and Preprocessing

The backbone of our Human Pose Estimation (HPE) model is the COCO (Common Objects in Context) dataset, which is a large-scale dataset containing over 2 million images with annotated key points. The diversity and extensive scale of this dataset make it ideal for training robust HPE models. The initial step involved filtering and annotating images using COCO’s annotation files, which include detailed information about each image’s size, bounding boxes, segmentation, and key point locations [14]. From this dataset, we selected approximately 65,000 images with crucial points necessary for our training purposes

To ensure consistency and improve the model's performance, we normalized the images. The normalization process involved scaling the pixel values using the formula $({\displaystyle \frac{x}{256}}-0.5)$ , where x represents the pixel values. This transformation standardized the pixel values to fall within the range of −0.5 to 0.5, making the data more suitable for training the neural network. Furthermore, we converted each key point into a 32 × 32 × 17 matrix, which represents the probability function for the key points [15]. These matrices were essential for generating heatmaps that the model would use to learn the spatial distribution of key points.

B.

Model Architecture

The model architecture is built upon the VGG-19 convolutional neural network, specifically utilizing the first 10 layers for feature extraction [16]. VGG-19 is well-regarded for its performance in image recognition tasks due to its deep architecture and the use of small, 3 × 3 convolution filters, which effectively capture intricate details in the images. The output from these layers provided a robust set of features that served as the foundation for detecting key points and part affinity fields in the subsequent stages [16].

The extracted features were processed through a series of CNN layers, organized into stages. The first stage consisted of five convolutional layers designed to further refine the features extracted by VGG-19. The first three layers used 3 × 3 × 128 filters, the fourth layer used 3 × 3 × 512 filters, and the fifth layer employed 1 × 1 × 17 filters [17]. Each convolutional layer was followed by a ReLU activation function, introducing non-linearity and enabling the model to learn complex patterns [18].

In stages 2 through 6, the architecture branched into two separate paths: one path was responsible for generating heatmaps, while the other generated Part Affinity Fields (PAFs). These branches contained layers with 7 × 7 × 128 kernels, and the final layers had 1 × 1 × 128 and 1 × 1 × 34 kernels. The heatmaps represented the probability of key points in a two-dimensional space, while the PAFs depicted the location and orientation of limbs, forming pairs in the image domain [19].

C.

Model Training

The training and validation of the model were carried out using a split from the COCO dataset, where 50,000 images (90%) were used for training and 5,000 images (10%) for validation. This division ensured that the model had ample data to learn from while also providing a separate set of images to evaluate its performance. The training process spanned multiple epochs, during which the model’s parameters were optimized to minimize the loss functions for both branches (heatmaps and PAFs).

The loss function for the heatmaps was defined as

$$L_{\mathrm{heatmap}}={\displaystyle \sum }_j{\displaystyle \sum }_p\parallel S_j(p)-S_j^{\ast }(p){\parallel }_2^2$$

Where Sj(p) represents the predicted heatmap for the point j at the position p, S*j(p) represents the ground truth heatmap [20]. Similarly, the loss function of PAFs was defined as:

$$L_{\mathrm{PAF}}={\displaystyle \sum }_c{\displaystyle \sum }_p\parallel L_c(p)-L_c^{\ast }(p){\parallel }_2^2$$

Where Lc(p) represents the predicted PAF for Limb c at position p, and L*c(p) represents ground truth PAF [20].

D.

Data Flow

The data flow for the Human Pose Estimation project starts with collecting the COCO dataset, which is partitioned into training, validation, and testing subsets. Data IO processes ensure proper loading and saving of data during preprocessing, training, and evaluation. Samples are drawn for training and validation to refine the model.

Model selection involves choosing the architecture and hyperparameters, initializing the VGG-19 network for feature extraction, and defining loss functions for heatmaps and Part Affinity Fields (PAFs). The Adam optimizer is used to adjust model parameters, minimizing the loss functions.

The model undergoes fitting, with layers configured for extracting features and generating heatmaps and PAFs. Hyperparameters are fine-tuned to enhance performance. Model inference applies the trained model to validation samples, followed by evaluation using metrics like accuracy, precision, recall, F1 score, and Mean Squared Error (MSE).

Experimental results provide insights into model performance, with comparisons to benchmark its effectiveness. The validated model is then prepared for deployment in a mobile application, allowing real-time pose estimation using a mobile device’s front camera. This comprehensive data flow ensures the model is accurately trained and capable of effective human pose estimation.

Figure 2. Data Flow diagram.

Figure 2. Data Flow diagram.

E.

Deployment in Mobile Application

To make the HPE model accessible and user-friendly, we deployed it as a mobile application using Google’s Flutter framework. Flutter is an open-source framework that allows developers to create natively compiled applications for mobile, web, and desktop from a single codebase. This choice ensured that our application could run efficiently on various devices.

We used TensorFlow Lite to deploy the trained model on mobile devices. TensorFlow Lite is a lightweight version of TensorFlow designed specifically for mobile and embedded devices, providing efficient performance and low latency. The application was equipped with features for real-time pose estimation, using the front camera of a mobile device to capture and analyze poses.

The user interface of the application was designed to be intuitive, allowing users to select different exercises or yoga poses and receive instant feedback on their performance. The feedback mechanism used visual indicators, such as green for correct poses and red for incorrect poses, to help users adjust their posture in real-time.

The model’s performance was evaluated using standard metrics such as precision, recall, F1-score, and Mean Squared Error (MSE). Precision measured the accuracy of key point detection, recall assessed the model’s ability to detect all relevant key points, and the F1-score provided a balanced measure of accuracy by combining precision and recall. MSE evaluated the difference between the predicted and actual key point locations, giving a quantitative measure of the model’s accuracy.

Result and Analysis

A.

Accuracy and Loss Analysis

Figure 3. Epoch vs Training Accuracy.

Figure 3. Epoch vs Training Accuracy.

Figure 4. Epoch vs Validation Accuracy.

Figure 4. Epoch vs Validation Accuracy.

Figure 5. Epoch vs Training and Validation Accuracy.

Figure 5. Epoch vs Training and Validation Accuracy.

Figure 6. Epoch vs Training Loss.

Figure 6. Epoch vs Training Loss.

B.

Output

Figure 7. Heatmap of all keypoints.

Figure 7. Heatmap of all keypoints.

Figure 8. Heatmaps of Part Affinity Field.

Figure 8. Heatmaps of Part Affinity Field.

Figure 9. Part Affinity Field.

Figure 9. Part Affinity Field.

Figure 10. Key points represented by different colors.

Figure 10. Key points represented by different colors.

Figure 11. Joining keypoints and Final output.

Figure 11. Joining keypoints and Final output.

C. 

Output from Mobile Application

Figure 12. Output of squat in mobile app.

Figure 12. Output of squat in mobile app.

Figure 13. Output of arm raise in mobile app.

Figure 13. Output of arm raise in mobile app.

Figure 14. UI of Mobile app.

Figure 14. UI of Mobile app.

Conclusion

References

1. N. Barla, ”V7Labs,” [Online]. Available:https://www.v7labs.com/blog/human-pose-estimation-guide.
2. P. Ganesh, ”Towards Data Science,” 15 March 2019.[Online]. Available:https://towardsdatascience.com/human-pose-estimationsimplified-6cfd88542ab3.
3. M. A. Fischler and R. A. Elschlager, ”The Representation and Matching of Pictorial Structures,” IEEE Transactions on Computers, vol. C-22, no. 1, pp. 67-92, 1973. [CrossRef]
4. P. F. Felzenszwalb and D. P. Huttenlocher, ”Pictorial structures for object recognition,” International Journal of Computer Vision, vol. 61, no. 1,pp. 55-79, 2005. [CrossRef]
5. L. Johnson and C. Everingham, ”Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation,” in Proceedings of the British Machine Vision Conference, 2010, pp. 1-11. [CrossRef]
6. Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, “Electron spectroscopy studies on magneto-optical media and plastic substrate interface,” IEEE Transl. J. Magn. Japan, vol. 2, pp. 740–741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982]. [CrossRef]
7. T.-Y. Lin et al., ”Microsoft COCO: Common Objects in Context,” in European Conference on Computer Vision (ECCV), 2014, pp. 740-755. [CrossRef]
8. M. Andriluka et al., ”2D Human Pose Estimation: New Benchmark and State of the Art Analysis,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 3686-3693. [CrossRef]
9. Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, ”OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 172-186, 2019. [CrossRef]
10. L. Pishchulin et al., ”DeepCut: Joint Subset Partition and Labeling for Multi-Person Pose Estimation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4929-4937. [CrossRef]
11. H. Fang et al., ”RMPE: Regional Multi-person Pose Estimation,” in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2334-2343. [CrossRef]
12. K. Simonyan and A. Zisserman, ”Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv preprint arXiv:1409.1556, 2014.
13. J. Wang et al., ”Deep High-Resolution Representation Learning for Visual Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 10, pp. 3349-3364, 2020. [CrossRef]
14. T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dolla´r, ”Microsoft COCO: Common Objects in Context,” in European Conference on Computer Vision (ECCV), Zurich, Switzerland, 2014, pp. 740-755.
15. K. Simonyan and A. Zisserman, ”Very Deep Convolutional Networks for Large-Scale Image Recognition,” in International Conference on Learning Representations (ICLR), San Diego, USA, 2015.
16. Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, ”Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, USA, 2017, pp. 7291-7299.
17. D. P. Kingma and J. Ba, ”Adam: A Method for Stochastic Optimization,” in International Conference on Learning Representations (ICLR), San Diego, USA, 2015.
18. X. Peng and K. Saenko, ”Synthetic to Real Adaptation with Generative Correlation Alignment Networks,” in IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, USA, 2018, pp. 1982-1991.
19. A. Newell, K. Yang, and J. Deng, ”Stacked Hourglass Networks for Human Pose Estimation,” in European Conference on Computer Vision (ECCV), Amsterdam, Netherlands, 2016, pp. 483-499.
20. S. Johnson and M. Everingham, ”Learning Effective Human Pose Estimation from Inaccurate Annotation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, USA, 2011, pp. 14651472.
21. D. Shrestha and D. Valles, “Evolving Autonomous Navigation: A NEAT Approach for Firefighting Rover Operations in Dynamic Environments,” 24th Annual IEEE International Conference on Electro Information Technology (EIT2024). [In Press].