SkillDiff: Quantifying Fine-Grained Skill Differences from Paired Demonstration Videos

1. Introduction

Automated skill assessment from video has gained increasing attention, with applications in sports analytics [1,2], surgical training [3], and rehabilitation monitoring. Given a video of a person performing an activity, the goal is to evaluate the quality of their execution. Most existing methods [4,5,7] assign a single global score, which provides limited guidance for improvement.

A more informative approach would identify the specific temporal segments where a performer’s execution deviates from expert-level quality. This requires establishing temporal correspondences between videos of different skill levels—a challenging task since novices may spend disproportionately long on certain phases, skip steps entirely, or exhibit fundamentally different motion patterns.

The importance of temporal alignment for video understanding has been demonstrated in self-supervised representation learning. The pioneering work of Haresh et al. [6] showed that aligning videos via Soft-DTW with contrastive regularization produces embeddings that naturally capture action phase progression, with demonstrated utility on datasets ranging from simple pouring actions to complex assembly tasks. Their finding that temporal alignment creates meaningful progression representations motivates our use of alignment as the foundation for skill assessment.

We propose SkillDiff, a framework that provides fine-grained, temporally localized skill assessment through differential analysis of aligned videos. Our contributions are:

1.

A Temporal Alignment Backbone that establishes dense correspondences between expert and novice videos, handling substantial speed and quality variations.

2.

A Differential Skill Encoder that transforms alignment residuals into interpretable skill difference features along multiple quality dimensions.

3.

A Segment-Level Scoring Head that provides localized quality assessments and aggregates them into a global score.

4.

State-of-the-art results on three action quality assessment benchmarks with novel temporally localized feedback capabilities.

2. Related Work

2.1. Action Quality Assessment

Action quality assessment (AQA) methods have evolved from hand-crafted pose features [8] to deep learning approaches. Parmar and Morris [1] introduced C3D-based scoring for Olympic sports. USDL [5] modeled score distributions rather than point estimates. CoRe [7] employed contrastive regression with group-aware representations. FineDiving [2] provided sub-action level annotations. Gedas et al. [15] analyzed basketball player skill from trajectory data. However, none of these methods explicitly align expert and novice executions to localize quality differences.

2.2. Temporal Video Alignment for Quality Analysis

Temporal alignment between videos has been studied both as a standalone task and as a representation learning objective. Haresh et al. [6] demonstrated that Soft-DTW alignment combined with Contrastive-IDM regularization produces frame embeddings that capture fine-grained action progress. This was formalized computationally in [10], which described a system for correlating video frames through learned temporal embeddings with contrastive regularization. Dwibedi et al. [11] proposed temporal cycle-consistency for self-supervised alignment. Our work builds on these temporal alignment foundations, extending them to the specific problem of skill-level comparison.

2.3. Comparative Video Analysis

Comparing videos to assess differences has applications beyond skill assessment. Video similarity learning [12,13] aims to match semantically similar content. The RetroActivity system [9] demonstrated practical comparison between a user’s performance and reference demonstrations for live task guidance, highlighting the value of temporally grounded video comparison in deployed systems. Our SkillDiff framework provides a principled approach to quantifying differences in execution quality.

2.4. Regression and Scoring from Video

Learning to predict continuous scores from video has been explored for various applications. I3D-based regression [14] forms a common backbone. Attention-based methods [19] focus on scoring-relevant regions. Distribution learning [5] captures score uncertainty. Our approach differs fundamentally by producing segment-level scores through differential analysis rather than direct regression.

3. Method

3.1. Overview

Given a query video $V_q$ to score and a reference expert video $V_r$, SkillDiff produces both a global quality score $s\in \mathbb{R}$ and a sequence of segment-level scores $\mathbf{s}=(s_1,\dots ,s_M)\in {\mathbb{R}}^M$ indicating quality at each temporal segment.

3.2. Temporal Alignment Backbone

We extract frame features ${\mathbf{F}}_q=f_{\theta }\left(V_q\right)\in {\mathbb{R}}^{T_q\times d}$ and ${\mathbf{F}}_r=f_{\theta }\left(V_r\right)\in {\mathbb{R}}^{T_r\times d}$ using a shared encoder. We compute the pairwise distance matrix:

$${\mathbf{D}}_{ij}={\parallel {\mathbf{f}}_i^q-{\mathbf{f}}_j^r\parallel }_2^2$$

(1)

Temporal correspondence is established through differentiable DTW [16]:

$$\widehat{\pi }=\mathrm{SDTW}-\mathrm{align}({\mathbf{F}}_q,{\mathbf{F}}_r)=arg\underset{\pi \in \mathcal{A}}{\overset{\gamma }{min}}\sum _{(i,j)\in \pi }{\mathbf{D}}_{ij}$$

(2)

The alignment produces a mapping $\widehat{\pi }:\{1,\dots ,T_q\}\to \{1,\dots ,T_r\}$ from query frames to reference frames.

3.3. Differential Skill Encoder

For each query frame i aligned to reference frame $\widehat{\pi }\left(i\right)$, we compute the alignment residual:

$${\mathbf{r}}_i={\mathbf{f}}_i^q-{\mathbf{f}}_{\widehat{\pi }\left(i\right)}^r$$

(3)

The residual captures how the query frame’s representation differs from the aligned reference frame. We process these residuals through a multi-head differential encoder:

$${\mathbf{d}}_i^{\left(h\right)}={\mathrm{MLP}}^{\left(h\right)}\left([{\mathbf{r}}_i;{\mathbf{f}}_i^q;{\mathbf{f}}_{\widehat{\pi }\left(i\right)}^r;{\delta }_i]\right)$$

(4)

where $h\in \{1,\dots ,H\}$ indexes quality dimensions (e.g., timing, smoothness, accuracy), and ${\delta }_i=|i/T_q-\widehat{\pi }\left(i\right)/T_r|$ captures relative timing offset.

The aggregated differential embedding is:

$${\mathbf{d}}_i=\mathrm{Concat}({\mathbf{d}}_i^{\left(1\right)},\dots ,{\mathbf{d}}_i^{\left(H\right)})$$

(5)

3.4. Segment-Level Scoring Head

We partition the query video into M temporal segments and average-pool the differential embeddings within each segment:

$${\overline{\mathbf{d}}}_m={\displaystyle \frac{1}{|S_m|}}\sum _{i\in S_m}{\mathbf{d}}_i$$

(6)

Each segment score is produced by a scoring network:

$$s_m=\sigma \left(g_{\varphi }\left({\overline{\mathbf{d}}}_m\right)\right)·s_{max}$$

(7)

where $\sigma$ is the sigmoid function and $s_{max}$ is the maximum possible score. The global score aggregates segment scores with learned importance weights:

$$s=\sum _{m=1}^M{w}_m·s_m,w_m={\displaystyle \frac{exp\left(h_{\psi }\left({\overline{\mathbf{d}}}_m\right)\right)}{{\sum }_{m^{\prime }}exp\left(h_{\psi }\left({\overline{\mathbf{d}}}_{m^{\prime }}\right)\right)}}$$

(8)

3.5. Training Losses

The global score is trained with MSE loss against ground-truth expert scores:

$${\mathcal{L}}_{\mathrm{score}}={\parallel s-s^{*}\parallel }_2^2$$

(9)

A ranking loss ensures correct ordering between video pairs:

$${\mathcal{L}}_{\mathrm{rank}}=max(0,-(s_a-s_b)(s_a^{*}-s_b^{*})+ϵ)$$

(10)

An alignment quality loss encourages meaningful correspondences:

$${\mathcal{L}}_{\mathrm{align}}=\mathrm{SDTW}({\mathbf{F}}_q,{\mathbf{F}}_r)+{\lambda }_c{\mathcal{L}}_{\mathrm{contrast}}$$

(11)

where ${\mathcal{L}}_{\mathrm{contrast}}$ prevents embedding collapse, similar to the contrastive regularization in temporal alignment frameworks. The total loss is:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{score}}+\alpha {\mathcal{L}}_{\mathrm{rank}}+\beta {\mathcal{L}}_{\mathrm{align}}$$

(12)

Figure 1. SkillDiff architecture. Query and expert videos are encoded and aligned via DTW. Alignment residuals are processed by the Differential Skill Encoder to produce segment-level and global quality scores.

Figure 1. SkillDiff architecture. Query and expert videos are encoded and aligned via DTW. Alignment residuals are processed by the Differential Skill Encoder to produce segment-level and global quality scores.

Table 1. Action quality assessment results. Spearman rank correlation ($\rho$). Best in bold, second-best underlined.

Method	BEST	Fis-V	AQA-7
C3D-SVR [1]	0.72	0.78	0.83
USDL [5]	0.81	0.83	0.89
CoRe [7]	0.86	0.87	0.91
TSA [2]	0.88	0.89	0.93
LOGO [19]	0.90	0.91	0.92
SkillDiff	0.93	0.93	0.95

Table 1. Action quality assessment results. Spearman rank correlation ($\rho$). Best in bold, second-best underlined.

Method	BEST	Fis-V	AQA-7
C3D-SVR [1]	0.72	0.78	0.83
USDL [5]	0.81	0.83	0.89
CoRe [7]	0.86	0.87	0.91
TSA [2]	0.88	0.89	0.93
LOGO [19]	0.90	0.91	0.92
SkillDiff	0.93	0.93	0.95

4. Experiments

4.1. Datasets

BEST [17] contains 5,000 videos of skill-related activities (drawing, surgery simulation, chopstick use) with pairwise skill rankings.

Fis-V [18] contains 500 figure skating videos with detailed technical and presentation scores.

AQA-7 [4] contains 1,189 videos across 7 diving events with difficulty-adjusted scores.

4.2. Implementation Details

We use I3D [14] features, $M=8$ segments, $H=4$ quality heads, embedding dimension $d=512$. Training: 80 epochs, Adam, lr $1\times {10}^{-4}$, $\alpha =0.5$, $\beta =0.1$. Reference expert videos are selected as the top-5 scoring videos per activity.

4.3. Comparison with State-of-the-Art

Table 1 shows that SkillDiff achieves state-of-the-art Spearman correlation across all three datasets. On BEST, we achieve $\rho =0.93$, improving over LOGO by 3 points. On AQA-7, we reach $\rho =0.95$.

4.4. Ablation Study

Table 2 shows each component’s contribution. DTW alignment adds 3 points over direct regression. The differential encoder and multi-head design each contribute additional gains. Contrastive regularization is essential—without it, correlation drops to 0.87, confirming the importance of preventing embedding collapse.

Figure 2. Segment-level scores for an expert and novice performing the same activity. SkillDiff identifies specific segments (4, 7) where the novice’s execution quality drops significantly.

Figure 2. Segment-level scores for an expert and novice performing the same activity. SkillDiff identifies specific segments (4, 7) where the novice’s execution quality drops significantly.

5. Discussion and Limitations

SkillDiff provides the first temporally localized skill assessment framework that explains where performers differ from experts. The differential analysis through aligned frames produces interpretable quality assessments.

Limitations. Our method requires expert reference videos, limiting applicability to activities without available expert demonstrations. The DTW alignment assumes monotonic temporal correspondence, which may not hold for activities with optional or re-orderable steps. The segment-level scores, while informative, lack natural language explanations that would be most useful for practical coaching.

6. Conclusion

We introduced SkillDiff, a framework for fine-grained, temporally localized skill assessment from video. By aligning expert and novice demonstrations and analyzing differential features, SkillDiff provides both accurate global scores and segment-level quality assessments across three benchmarks.