Submitted:
12 November 2026
Posted:
13 November 2026
You are already at the latest version
Abstract
Pipa playing techniques involve string contact, hand movement, and acoustic characteristics that cannot be fully described by note information or a single modality. We present a framework for automatic annotation of pipa playing techniques that combines audiovisual information with expert rules. Synchronized recordings and basic fingering annotations establish temporal and spatial links among audiovisual segments, string and fret positions, and note events. Acoustic and visual modules are then organized according to the movement characteristics of individual techniques. The acoustic branch uses envelope spectra to characterize plucking rhythms and augments an Audio Spectrogram Transformer with patch mixing and focal loss. The visual branch identifies techniques from the displacement, direction, and oscillation of hand keypoints. Expert rules describing technique combinations and temporal relationships subsequently check the initial labels in context; correction benefits are assessed by counting both corrected errors and newly introduced errors. Experiments on a pipa performance dataset yield overall accuracies of 83.2% for right-hand techniques and 64.6% for left-hand techniques over all annotated segments. Because the left-hand categories depend on movement that the video must resolve, left-hand accuracy is additionally reported on the evaluable subset defined by the input conditions of the visual pathway: 0.837 of left-hand annotations meet these conditions, and accuracy on this subset is 0.755. Expert rules produce positive net correction gains for both hands. These results support organizing audiovisual evidence around performance actions and incorporating performance knowledge into annotation, providing a methodological basis for fine-grained pipa datasets and subsequent performance analysis.
Keywords:
pipa performance
; automatic playing-technique annotation
; multimodal analysis
; audio spectrogram transformer
; expert system
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.