An Explainable Dance Pose Evaluation System Using Velocity-Adaptive Filtering and Dynamic Alignment

1. Introduction

The widespread availability of video streaming platforms has made digital performing arts education broadly accessible. Aspiring dancers, athletes, and movement enthusiasts now routinely rely on pre-recorded online tutorials to acquire complex physical motor skills. However, autonomous physical practice in the absence of an in-person instructor introduces substantial pedagogical hurdles. First-time and intermediate learners frequently struggle with spatial body awareness, experiencing a gap between what they believe their limbs are doing and their true orientation in space. Without corrective feedback, learners risk internalizing incorrect movement habits, ingraining flawed muscle memory, and exposing joints to unnecessary strain or injury.

While expert in-person instruction remains the benchmark for physical pedagogy, it is often limited by high financial cost, rigid schedules, and geographic constraints. Practice mirrors provide brief visual feedback, but they cannot offer quantitative diagnostic guidance; furthermore, turning one’s head to inspect a mirror during dynamic rotations or spins compromises body alignment. Consequently, there is an urgent need for automated, accessible movement evaluation systems capable of analyzing ordinary video recordings of students, aligning them against canonical teacher demonstrations, and generating clear diagnostic corrections.

Let ${\mathcal{V}}_{\mathrm{ref}}={\left\{I_{\mathrm{ref}}^{\left(t\right)}\right\}}_{t=1}^{T_{\mathrm{ref}}}$ denote an instructor’s reference demonstration captured at frame rate $f_{\mathrm{ref}}$, and let ${\mathcal{V}}_{\mathrm{lrn}}={\left\{I_{\mathrm{lrn}}^{\left(t^{\prime }\right)}\right\}}_{t^{\prime }=1}^{T_{\mathrm{lrn}}}$ denote an uncalibrated query video of a student replicating the sequence. An automated evaluation mapping must overcome four primary operational challenges:

1.

Body Proportion Variations: The teacher and student have differing heights, limb segments, and body mass distributions.

2.

Camera Variations: Lens zoom, sensor placement, and camera-to-performer distances vary arbitrarily between recordings.

3.

Temporal Phrasing: Human performers naturally introduce expressive micro-timing, hesitations, or slight tempo shifts ($50--250$ ms). Rigid frame-by-frame comparison penalizes small timing differences as technical posture flaws.

4.

High-Velocity Motion Artifacts: Rapid arm whips, spins, and floor transitions cause transient keypoint motion blur and self-occlusions.

Early motion tracking systems relied on wearable physical sensors or active depth cameras such as the Microsoft Kinect [9]. Although accurate, their hardware requirements restricted use outside laboratory environments. Deep learning architectures such as OpenPose [2] and MediaPipe BlazePose [1] have since enabled markerless tracking of 33 body landmarks directly from standard consumer video streams [10]. In the dance domain, Choi et al. [4] compared 2D joint coordinates frame by frame; however, rigid temporal matching proved fragile during rapid movements. Johari and Mukherjee [5] developed ReAL-T, demonstrating that automated joint-angle trajectory matching correlates well with expert ratings, yet their system did not incorporate adaptive filtering for high-speed dance steps or partition continuous sequences into discrete poses.

To bridge these gaps, our hypothesis is that an explainable kinematic evaluation pipeline—combining body normalization, velocity-adaptive filtering, bounded Dynamic Time Warping with temporal tolerance, and kinematic feature segmentation—can accurately evaluate dance movement from standard consumer cameras without penalizing natural expressive timing.

2. Materials and Methods

2.1. Research Design and Architecture

The framework operates as a modular, five-stage pipeline designed to evaluate student videos against reference performances (Figure 1). Video recordings are first processed to extract 2D skeletal coordinates, normalized for physical dimensions, smoothed using speed-sensitive filters, aligned dynamically in time, segmented into choreographic poses, and translated into natural-language diagnostic feedback.

2.2. Body-Scale Normalization and Missing Landmark Infilling

Frame images are processed using MediaPipe BlazePose to obtain 33 anatomical landmarks ${\mathbf{u}}_k=(u_x,u_y)\in {[0,1]}^2$ and detection confidence scores $c_k\in [0,1]$. Because performers occupy different areas of the camera frame, coordinates are centered at the pelvis midpoint ${\mathbf{p}}_{\mathrm{hip}}$ and scaled by torso length $L_{\mathrm{torso}}$:

$${\mathbf{p}}_{\mathrm{hip}}=\frac{1}{2}({\mathbf{u}}_{23}+{\mathbf{u}}_{24}),{\mathbf{p}}_{\mathrm{sh}}=\frac{1}{2}({\mathbf{u}}_{11}+{\mathbf{u}}_{12})$$

(1)

$$L_{\mathrm{torso}}={\parallel {\mathbf{p}}_{\mathrm{sh}}-{\mathbf{p}}_{\mathrm{hip}}\parallel }_2$$

(2)

$${\mathbf{p}}_k={\displaystyle \frac{{\mathbf{u}}_k-{\mathbf{p}}_{\mathrm{hip}}}{\max (L_{\mathrm{torso}},{10}^{-6})}}$$

(3)

This normalization ensures scale and position invariance across differing camera distances. When joint occlusions cause confidence to drop below 0.35, missing coordinates are infilled via bidirectional linear interpolation across confident neighboring frames.

2.3. Velocity-Adaptive Kinematic Kalman Filter

To suppress tracking jitter without falling behind rapid movements, a 1D constant-velocity Kalman filter [6] is implemented for each coordinate axis of every joint:

$${\mathbf{x}}_t=\left[\begin{array}{c}{x}_t\\ v_t\end{array}\right],\mathbf{F}=\left[\begin{array}{cc}1& \Delta t\\ 0& 1\end{array}\right]$$

(4)

Standard constant-velocity filters use a fixed process noise covariance $\mathbf{Q}$, which introduces an $85--450$ ms phase delay on sharp ballistic movements. We modulate ${\mathbf{Q}}_t$ continuously as a function of instantaneous landmark velocity $|{\widehat{v}}_{t-1}|$:

$${\mathbf{Q}}_t=\left[\begin{array}{cc}{\displaystyle \frac{\Delta t^3}{3}}& {\displaystyle \frac{\Delta t^2}{2}}\\ {\displaystyle \frac{\Delta t^2}{2}}& \Delta t\end{array}\right]q_0\left(1+\beta |{\widehat{v}}_{t-1}|\right)$$

(5)

where $q_0=0.005$ is baseline process noise, measurement variance is $R=0.012$, and velocity sensitivity is $\beta =12.0$. Rapid limb transitions expand ${\mathbf{Q}}_t$, increasing the Kalman gain to track quick trajectory changes without delay. During held postures ($|\widehat{v}|\approx 0$), ${\mathbf{Q}}_t\to q_0$, smoothing out tracking noise. Additionally, the measurement gate relaxes during fast movement:

$${\tau }_t=\max \left(0.15,0.35-0.5{\overline{v}}_t\right)$$

(6)

where ${\overline{v}}_t$ is the mean landmark velocity across the body.

2.4. Dynamic Time Warping with Temporal Leeway

Performances are aligned temporally via Dynamic Time Warping [8] constrained within a Sakoe-Chiba band ($\pm 25\%$ duration). The composite frame matching error $E_{\mathrm{total}}\in [0,1]$ is:

$$E_{\mathrm{total}}=0.55E_{\mathrm{pos}}+0.35E_{\mathrm{angle}}+0.10P_{\mathrm{conf}}$$

(7)

where $E_{\mathrm{pos}}$ is the confidence-weighted Euclidean distance across 16 primary dance joints ${\mathcal{K}}_{\mathrm{comp}}$, $E_{\mathrm{angle}}$ tracks angular error across eight major joints (elbows, shoulders, hips, knees), and $P_{\mathrm{conf}}=1.0-\min (c_{\mathrm{ref}},c_{\mathrm{lrn}})$ accounts for tracking confidence. The error maps to a standard performance score $\mathcal{S}\in [0,100]$:

$$\mathcal{S}=\max \left(0,\min \left(100,100\times \left(1-{\displaystyle \frac{E_{\mathrm{total}}}{0.50}}\right)\right)\right)$$

(8)

where $E_{\mathrm{total}}\ge 0.50$ clamps the score to 0.

To distinguish posture errors from minor phrasing offsets, aligned frames are evaluated across a local neighborhood window of $\pm 0.25$ s ($W_f$ frames) with an offset penalty $\lambda =1.5$ points per frame:

$${\mathcal{S}}^{*}(i,j)=\underset{|{\delta }_r|,|{\delta }_l|\le W_f}{\max }\left[\mathcal{S}\left({\mathbf{x}}_{\mathrm{ref}}^{(i+{\delta }_r)},{\mathbf{x}}_{\mathrm{lrn}}^{(j+{\delta }_l)}\right)-\lambda \left(\right|{\delta }_r|+|{\delta }_l\left|\right)\right]$$

(9)

2.5. Feature-Divergence Segmentation and Explainable Feedback

Continuous choreography is partitioned into discrete poses using a 40-dimensional kinematic vector ${\mathbf{f}}_t$ containing 16 joint coordinates (32 values) and 8 normalized joint angles. The vector is smoothed using a Kalman filter ($q_{\mathrm{seg}}=0.001,R_{\mathrm{seg}}=0.02$). A segment boundary is established when feature divergence exceeds ${\delta }_{\mathrm{pose}}=0.055$ for at least 0.15 s:

$${\displaystyle \frac{1}{40}}{\parallel {\mathbf{f}}_t-{\overline{\mathbf{f}}}_{\mathrm{curr}}\parallel }_1\ge {\delta }_{\mathrm{pose}}$$

(10)

For each detected pose, a deterministic rule engine inspects mean angular errors ($\left|\Delta \theta \right|\ge {10}^{\circ }$) and spatial offsets ($\parallel \Delta \mathbf{p}\parallel {}_2\ge 0.08$), generating actionable corrections (e.g., “Left knee is too bent by 37.2 degrees” or “Right foot is higher and more to the right”).

2.6. Benchmark Dataset and Evaluation Protocol

The system was evaluated on 15 benchmark pairs (30 video recordings, 10,550 frames) combining sequences from the public AIST Dance Video Database [7,11] and structured movement exercises. The dataset covers eight distinct movement styles (Ballet Jazz, Street Jazz, Popping, House, Breakdance Toprock, Contemporary, Ballroom Waltz, and Yoga) captured from frontal ($0^{\circ }$) and profile (${90}^{\circ }$) views. Full-body framing audits confirm that hands and feet remain in view across $88.3\%--100.0\%$ of frames. Sequence 8 (Yoga Tree Pose) incorporates a deliberate balance loss to evaluate fault detection sensitivity.

To establish ground-truth validation, human perceptual scores were recorded via manual review of all 15 routines prior to running the pipeline, evaluating joint placement, extension, and timing on a 0–100 scale. In addition, three professional dance instructors independently rated six routines using a dedicated double-blind review interface.

3. Results and Discussion

3.1. Benchmark Evaluation Across Dance Genres

Table 1 presents the performance evaluation across all 15 benchmark sequences alongside the human perceptual ground-truth scores. Across all 15 routines, the automated scores closely matched human perceptual assessments, exhibiting a mean absolute error of only 1.7 points.

Frontal routines achieved an average score of $86.79$ for regular choreography (or $77.93$ when including the deliberate balance-loss test case). Styles with clear, sharp poses such as Street Jazz scored high at $91.7$ (human evaluation: $93.0$), and Popping scored $89.4$ (human evaluation: $88.0$). Faster styles such as Breakdance Toprock scored lower at $78.9$ (human evaluation: $77.0$) due to foot placement variations during rapid footwork. In sequence 8, where the student lost balance and placed both feet down, the system score dropped to $7.1$, matching the human failure score of $5.0$.

Side-view (${90}^{\circ }$ profile) routines yielded an average score of $83.83$ (human perceptual average: $83.50$). Street Jazz scored $87.4$ from the side (compared to $91.7$ from the front), and Popping scored $82.8$ from the side (compared to $89.4$ from the front). These findings demonstrate that planar torso normalization functions reliably across viewpoints, provided limbs are not fully occluded.

3.2. Tempo Analysis: Slow Versus Fast Routines

To examine the effect of movement tempo on pose segmentation and scoring, we compared a slow contemporary routine (sequence 6, 939 frames) with a fast urban choreography routine (sequence 7, 766 frames) as shown in Table 2.

The slow performance segmented into 6 sustained poses (averaging 2.6 s each), with scores ranging from $86.95$ to $95.65$ (all Excellent). Minor errors were confined to small wrist height differences ($<0.08$ normalized units), with only one joint angle deviation above ${10}^{\circ }$ (Pose 5, left elbow bent by $15.6^{\circ }$).

In contrast, the fast routine segmented into 34 brief poses (averaging $0.37$ s each), with scores spanning $59.39$ to $87.14$ (11 Excellent, 17 Good, 6 Fair). Table 3 provides sample feedback generated during the fast routine. The lowest-scoring segment (Pose 34, score $59.4$) correctly flagged an arm whip where the student’s elbow remained bent by $83.9^{\circ }$ instead of extending fully.

3.3. Ablation Study and Baseline Comparison

To measure the contribution of each algorithmic module, we benchmarked the system against four baseline configurations on the 766-frame Fast Choreography sequence (Table 4).

Frame-by-frame evaluation without temporal alignment (Baseline 1) broke down entirely (score $22.4$, $88.2\%$ false penalties) due to natural timing drift. Static Kalman filtering (Baseline 2) smoothed over sharp peaks, introducing an 85 ms lag that lowered scores during fast transitions. Using joint angles alone (Baseline 3) failed to capture stance width and floor placements ($34.1\%$ false penalties). Removing the temporal tolerance window (Baseline 4) penalized minor musical timing variations as posture flaws ($18.4\%$ false penalties). The proposed complete system kept tracking latency under 15 ms, achieving an average score of $79.6$ with only $3.2\%$ false penalties.

3.4. Failure Modes and User Validation

The framework encountered four primary edge cases:

1.

Severe Technique Breakdown: In sequence 8, the student lost balance completely ($E_{\mathrm{total}}\ge 0.50$ across $94\%$ of frames). The score dropped to $7.1$, demonstrating that the metric correctly flags gross motor errors.

2.

Vertical Foreshortening on Footwork: Chest-height cameras compress depth along the floor, occasionally misinterpreting forward/backward foot placement as height changes.

3.

Sagittal Self-Occlusion: In ${90}^{\circ }$ profile views during complete spins, the near side of the body momentarily hides far-side limbs, lowering confidence for 4–8 frames.

4.

Fast Motion Blur: Rapid forearm snaps ($>{600}^{\circ }$/s) produced single-frame motion blur at 30 FPS capture, shifting wrist localization by up to 42 pixels. Recording at 60 or 120 FPS mitigated this error.

During independent validation with three professional dance instructors across six routines, automated scores correlated strongly with expert ratings (Pearson $r=0.88$, Spearman $\rho =0.86$, Mean Absolute Error $=4.8$ points). The instructors noted that partitioning continuous choreography into distinct poses mirrors how teachers critique movement in studio environments.

4. Discussion

Current limitations include single-dancer tracking constraints, monocular depth ambiguity during out-of-plane body rotations, and reliance on movement kinematics rather than musical audio beats for alignment. Future extensions will incorporate audio beat tracking into Dynamic Time Warping, implement self-supervised camera view normalization, and optimize the processing pipeline for real-time mobile browser execution using WebGPU.

5. Conclusion

We developed an explainable computer vision framework that evaluates dance movement from standard consumer video. By integrating body normalization, velocity-adaptive Kalman filtering, bounded Dynamic Time Warping with temporal tolerance, and feature segmentation, the system maintains tracking delay under 15 ms and evaluates posture fairly without penalizing natural expressive timing. The system partitions continuous movement into reviewable poses and generates clear, actionable corrections that correlate closely with professional dance instructor evaluations ($r=0.88$, MAE $=4.8$ points).

Not applicable.

To facilitate independent verification and future research, the implementation of the proposed evaluation pipeline, adaptive filters, and alignment algorithms is publicly available at:

https://github.com/Rahuldesai0/pose-evaluation-system

The repository contains the complete source code files required to run the pose estimation, filtering, and scoring modules described in this paper.