Mediapipe Pose AI Motion Capture Software for Upper Limb Exercise and Rehabilitative Therapy

1. Introduction

Stroke afflicts an increasing proportion of middle age and early elderly people both in advanced countries and emerging economies.[1] As stroke differentially affects both sides of the body, there is typical misaligned muscle power and dynamics on either the left or right upper limb, that contributes to dysfunction in ability to perform many daily activities.[2,3]

This project seeks to use modern AI-based motion capture by Mediapipe Pose to dynamically capture upper limb movements, catalogue and analyse joint angle evolution, and calculate angular velocity changes to aid in detecting early signs of mild stroke that remains difficult to detect clinically. Chief diagnostic tools for detecting mild stroke using this AI motion capture approach is postulated to be misaligned joint angle evolution, and differential joint angular velocity changes during three repeats of a standard upper shoulder flexion extension exercise. Three repeats are the maximum possible here because of the tremendous amount of computational power needed to execute laptop’s webcam-based AI motion capture by Mediapipe Pose.

Given that Mediapipe Pose motion capture software allows the full range of upper limb motions (both physiotherapist designed as well as mis-tuned) to be captured, it serves as a useful video-based record for physiotherapist to analyse, in detail, the type of pathomechanical gait exhibited, and possibly propose a suitable orthotic treatment for more severe cases. Beyond rehabilitative therapy, the AI motion capture software presented here could also serve as biomechanics gait diagnostic tool for upper limb freedom of movement to help gauge early signs of musculoskeletal disorders or early onset muscle dystrophy.

To help in documenting the motion capture exercise, joint angle changes and joint angular velocity evolution is documented at the per frame level, and the aggregate data is presented in two combinatorial plots (joint angle changes, and angular velocity changes) as well as two Excel files of the corresponding content. Together, these statistics serve as useful feedback for both the patient and physiotherapist; the latter for tracking treatment progress.

Presented here is demonstrative data for a single upper limb shoulder flexion extension rehabilitative exercise designed to analyse and check the overall functional performance of the shoulder-elbow joint movement system, which is responsible for many load-bearing as well as finer motor skills tasks needed for an adult to remain fully functional and independent in daily life. Interested readers are invited to further explore the utility of the software which is deposited on figshare at https://figshare.com/articles/software/_b_Mediapipe_Pose_AI_motion_capture_software_for_upper_limb_exercise_and_rehabilitative_therapy_b_/30681290

2. Implementation Method

At the outset, this AI motion capture software platform is built using the Python programming environment (version 3.11) with Mediapipe Pose providing body landmarks detection and tracking and OpenCV-Python providing computer vision support. Other Python packages used include numpy, matplotlib, random, time, and math.

Computational engine for this software comes from a combination of numeric CPU, graphics GPU, and neural NPU that, altogether, delivers a close to real-time upper limb and hands landmarks detection and tracking, and fast visual reconstruction on the laptop screen. This computational engine also helps realise a suitably fast motion tracking and exercise environment, suitable for capturing a wide range of upper limb rehabilitative exercises for a variety of patient groups and even healthy adults keen on an exercise gaming experience. Hardware platform used for this project is a HP laptop equipped with Intel Core Ultra 5 processor (Model: 125H) with Intel Arc Graphics support, 16 GB of RAM and 512 GB of solid state drive memory.

3. Results and Discussion of Demonstrative Use of Motion Capture Software

Figure 1 shows representative data from a shoulder-elbow flexion extension exercise that is captured by the AI motion capture software developed here. In Figure 1a,c, it can be seen that the flexion and extension cycle can be captured accurately by the software. Given the overlap of the curves for both the right and left shoulder, it is clear that there is no left-right asymmetric behaviour of the musculoskeletal system, indicating no stroke disease effects. On the corresponding angular velocity plots (Figure 1b,d), there are sharp peaks associated with rapid shoulder flexion and extension events, which suggests strong muscle power. Again, there is no left-right asymmetric musculoskeletal problems detected.