Speech-Based Parkinson's Detection Using Pre-Trained Self-Supervised Automatic Speech Recognition (ASR) Models and Supervised Contrastive Learning

1. Introduction

Parkinson's disease (PD), a progressive neurodegenerative disorder, affects millions of people worldwide, causing significant motor, cognitive, and speech impairments [1,2]. With the global prevalence of PD projected to increase twofold by 2040, accurate and timely diagnosis, as well as monitoring of disease progression, have become crucial for optimizing treatment outcomes and improving quality of life for patients [3,4]. Traditional methods of assessing Parkinson's severity, such as the Unified PD Rating Scale (UPDRS), rely on subjective evaluations by clinicians, which can be time-consuming, costly, and prone to inter-rater variability [5,6].

Speech analysis has emerged as a promising biomarker for PD (PD) diagnosis and monitoring in recent years. Individuals with PD often exhibit distinct speech patterns, including tremors, rigidity, and hesitation, which can be indicative of the disease's progression [3]. These vocal changes are frequent among the earliest observable symptoms of the disease, making them particularly valuable for early detection and ongoing assessment [7].

Recent advancements in signal processing, machine learning (ML), and deep learning (DL) have shown significant interest in the automated detection and classification of Parkinson's disease severity through speech analysis [8]. Typically, these detection approaches employ a two-stage pipeline: feature extraction followed by classification. The development of these systems relies on supervised learning techniques, utilizing speech samples collected from both healthy individuals and those diagnosed with PD. The binary classification (Parkinson's vs. non-Parkinson's) is based on clinical diagnoses provided by neurologists and movement disorder specialists. Researchers have explored various models and feature extraction techniques to identify and highlight the distinctive characteristics of PD-affected speech [2,9,10,11].

Building on these advancements, recent studies have increasingly focused on two main approaches: feature selection for shallow classifiers and deep learning architectures. The first approach involves hand-crafting acoustic features that capture speech abnormalities associated with PD and using conventional machine learning algorithms, including support vector machine (SVM) classifiers, random forest models, and k-nearest neighbor (kNN) techniques [11,12,13,14,15]. The models were trained using diverse speech characteristics including acoustic features (jitter, shimmer, and harmonic-to-noise ratios) alongside complexity measurements to identify changes in voice production that occur in Parkinson's Disease patients [9,10,11,16,17]. While this approach has shown promising results, the second approach, which leverages deep learning to automatically learn features directly from speech data, has demonstrated superior performance. Various neural network architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and transformer-based models, have achieved state-of-the-art results on PD detection from speech [16,18,19,20,21,22,23]. These architectures can process different types of input representations - from raw audio waveforms that preserve the complete temporal information, to spectrograms that capture the time-frequency characteristics of speech, or even a combination of both. Overall, deep learning models excel at automatically learning relevant speech patterns, eliminating the need for manual feature engineering by experts. By leveraging transfer learning, these models can build upon knowledge gained from related speech tasks to improve their accuracy in detecting Parkinson's Disease [16,24].

However, in various fields, acquiring labeled data poses significant challenges. Consequently, there has been substantial progress in pre-training deep learning models with unlabeled data. Recently, a promising approach utilizing automatic speech recognition (ASR) models combined with self-supervised learning has emerged and gained widespread attention. This approach is helpful in research areas where datasets are often limited in size [25]. The process involves training models on large speech datasets in an unsupervised manner, allowing them to learn speech representations from tasks-audio that can be applied to specific downstream task. Pre-trained models like Wav2Vec 2.0 [26] and HuBERT [27] have demonstrated effectiveness in various speech technology applications, including voice pathology detection [25,28,29,30]. Studies have demonstrated that ASR models, originally designed for speech-to-text tasks, have the potential to be repurposed for other disease related detection by analyzing speech patterns and identifying acoustic biomarkers associated with the disease. These models have shown promise in tasks relevant to PD, such as detecting subtle speech changes and classifying severity levels of dysarthria, a common speech impairment in PD patients [25]. This study found that HuBERT features outperformed traditional acoustic features in detecting and classifying dysarthric speech, with absolute accuracy improvements of up to 2.86% and 10.46% in detection and severity level classification of tasks, respectively [25]. A recent study demonstrated that Wav2Vec features can effectively classify PD patients, with a minimum AUROC of 0.77 and a maximum AUROC of 0.98 in cross-database experiments [31]. In another study, Wav2Vec 2.0 was employed to classify raw speech recordings for PD detection using a dataset collected via a midrange Android smartphone, achieving a notable accuracy of up to 97.92% in distinguishing PD from non-PD individuals. However, the authors highlighted difficulties in making precise multi-class predictions across different stages of the disease [32]. Despite the promising results of recent studies, the application of ASR models in PD diagnosis and monitoring remains a relatively underexplored area of research.

In this study, we first explore the potential of popular ASR models, specifically Wav2Vec 2.0 and HuBERT, for the detection of PD from speech. We propose that these models, having learned rich speech representations from large datasets, are capable of effectively capturing the distinctive acoustic characteristics associated with PD, leading to accurate detection. We also aim to find out the explainability of these models by employing gradient-based feature importance methods like Grad-CAM [33], visualizing the speech signal regions that contribute most significantly to the model's decision.

Furthermore, we propose the integration of a supervised contrastive learning [34] approach in conjunction with a classification head within the training objectives, a strategy that has not yet been explored in this context of ASR models. The supervised contrastive approach is expected to help in grouping similar speech representations more closely, while distancing those that are dissimilar, which can improve the model’s ability to discern PD-specific features. In addition to leveraging advanced ASR models, we established baseline comparisons using two classical machine learning approaches: an extreme gradient boosting (XGBoost) [35] model and a multilayer perceptron (MLP). These models were trained on traditional acoustic features: mel-frequency cepstral coefficients (MFCCs) and the extended Geneva minimalistic acoustic parameter set (eGeMAPS) [36].

After a thorough assessment, which included fine-tuning and careful examination of the effects of supervised contrastive learning, it is indicated that this method has the capability to enhance overall performance.

2. Materials and Methods

This speech classification system, depicted in Figure 1, analyzes 5-second audio clips to identify individuals with PD. The process begins with a Transformer Encoder that converts raw audio into a meaningful feature vector. This vector is then processed through two parallel pathways: one task head that classifies the audio as "Healthy" or "Parkinson," and a projection network aimed at enhancing model robustness via supervised contrastive learning.

For the base model, only the classifier is utilized. To investigate the effects of supervised contrastive learning, both the classifier and the projection network are employed, with gradient flow from the classification head blocked. The architecture employs a unified training approach where the classifier, encoder, and projection networks are trained simultaneously. By blocking the backward flow of gradients from the classifier to the encoder, this design eliminates the traditional requirement for separate training phases.

2.1. Dataset

This study used the recently updated NeuroVoz voice and speech database, as originally described in [37,38], which includes recordings from 112 adult native speakers of Castilian Spanish. The participants are divided into two groups: 58 healthy controls (HC) and 54 individuals diagnosed with PD. The recordings were collected by the Bioengineering and Optoelectronics Group at the Universidad Politécnica de Madrid, in collaboration with the Otorhinolaryngology and Neurology Departments of Gregorio Marañón Hospital in Madrid, Spain. The database features a variety of speech tasks, such as sustained vowel phonation, diadochokinetic (DDK) tests, recitation of predetermined sentences, and spontaneous speech describing an image. Further details about the database can be found in reference [37,38].

In this study, we used a subset of the complete dataset, consisting of 16 repeated utterances totaling 1,695 audio samples. Of these, 828 were from individuals with PD, and 867 were from HC. This selection was made because we aimed to utilize an ASR model that had been pretrained on comparable large-scale dataset. The histogram in Figure S1 compares audio recording durations between two study groups: healthy controls (HC) and individuals diagnosed with PD. Further details about the speech tasks are provided in supplementary Tables S1 and S2. To our knowledge, while one study examined small subsets of this dataset using DDK and vowels [16], no comprehensive analysis of the complete subset has been conducted to date.

2.2. Data Preprocessing

To prepare the audio data for analysis by deep learning models, several preprocessing steps were implemented. The first step standardized all recordings to a 16000 kHz sampling rate. Next, silent segments at the beginning and end of each recording were eliminated. The dataset contained varied recording durations, as illustrated in Figure S1. To facilitate uniform batch processing during model training, each recording was adjusted to a standard 5-second length. This was achieved by adding padding to shorter segments and cutting off excess audio from longer recordings. Moreover, in order to enhance model generalization given the limited training data size, we expanded the dataset using audio augmentation techniques. Specifically, we applied low-pass filtering through the torch audio augmentation library [39] s part of the preprocessing pipeline. This augmentation was restricted to the training set only, while validation and test sets remained unmodified to maintain their real-world characteristics.

Moreover, in order to reduce overfitting given the initially small training dataset, the limited data set was expanded through the application of simple low-pass filter audio augmentation using the torch audio augmentation library prior to executing put into model. This data expansion aims to improve generalizability. Data augmentation was not used for the validation and test sets so that these sets would resemble real-world data.

2.3. Deep Learning Models

Our research evaluates the effectiveness of two leading self-supervised speech models - Wav2Vec 2.0 [26] and HuBERT [27]- in processing speech patterns for specialized applications. Both models were pretrained on the extensive LibriSpeech corpus, a well-regarded dataset in the speech processing community.

Wav2Vec 2.0 and HuBERT are self-supervised models for learning speech representations directly from raw audio, leveraging multi-layer CNN encoders and 12-layer transformer-based context networks (Figure 2). Wav2Vec 2.0 processes raw audio into 768-dimensional contextual embeddings through convolutional layers with a stride of 20 ms and a receptive field of 25 ms, followed by masked transformer-based context modeling and a contrastive learning objective to distinguish true quantized representations from distractors, as described by Baevski et al. [26]. In contrast, HuBERT employs a k-means clustering step to generate discrete target units from MFCC features, learning to predict embeddings for masked segments using cross-entropy loss, with clustering refined iteratively using deeper model layers [27]. Both models output 768-dimensional feature vectors from their final transformer layers, which we use for analysis. For PD detection, these feature vectors are passed through a simple one-layer fully connected classifier (Figure 2c), which reduces them to 64 dimensions via a linear transformation, applies ReLU activation, dropout for regularization, and a final linear transformation to produce predictions. The same classification architecture, including a 32-dimensional hidden layer in the projection network, is used across all models.

2.4. Supervised Contrastive Learning (SupCon)

Contrastive learning is a powerful self-supervised learning technique that has gained significant attention in recent years, particularly in the fields of computer vision and natural language processing [34,40,41]. The core idea behind contrastive learning is to learn representations by comparing similar and dissimilar samples in a dataset. This approach enables models to capture meaningful features and relationships within the data without relying on explicit labels. The method is frequently used in self-supervised learning, where the task is to learn meaningful features from unlabeled data [34]. In supervised settings, where labels are available, a given sample may have multiple positive samples (i.e., multiple samples that belong to the same class). The standard contrastive loss does not account for this and can only handle one positive per anchor sample, making it unsuitable for supervised learning tasks where there are several samples from the same class [34,42].

Contrastive loss for supervised learning can be generalized to incorporate multiple positive samples. This is done by summing over all positives for a given anchor sample, denoted as $\mathrm{P}\left(\mathrm{i}\right)$. As shown in Eq. 1, for instance, the contrastive loss for each sample $i$ is computed by considering all positive samples $p\in P\left(i\right)$, where $P\left(i\right)$ represents the set of positive samples for the anchor $i$. The supervised contrastive loss then evaluates the distance between the anchor and all positive samples relative to a set of negatives (samples that belong to different classes). The function inside the log term measures the similarity between the anchor $z_i$ and a positive sample $z_p$, normalized by the sum of exponentials of similarities between the anchor and all other samples $z_a\in A\left(i\right)$, where $A\left(i\right)$ is the set of all samples except the anchor. This is scaled by a temperature parameter $\mathsf{\tau }$to control the sharpness of the distribution. All our experiments used a temperature of τ = 0.07. Lower temperature benefits training more than higher ones, but extremely low temperatures are harder to train due to numerical instability [34,43,44]. This formulation enables the incorporation of multiple positive samples, making it particularly well-suited for supervised learning environments [42]. The contrastive loss is defined as follows:

$${\mathcal{L}}_{contrasive}={\sum }_{i\in I}{\displaystyle \frac{-1}{\left|P\left(i\right)\right|}}{\sum }_{p\in P\left(i\right)}\log {\displaystyle \frac{\exp \left(\frac{\mathrm{sim}\left({\mathit{z}}_{\mathit{i}},{\mathit{z}}_{\mathit{p}}\right)}{\mathsf{\tau }}\right)}{{\sum }_{a\in A\left(i\right)}\exp \left(\frac{\mathrm{sim}\left({\mathit{z}}_{\mathit{i}},{\mathit{z}}_{\mathit{a}}\right)}{\mathsf{\tau }}\right)}}$$

(1)

where:

$z\in R^{N\times d}$ represents the normalized projections of the input samples into an embedding space, where N is the number of samples and $d$ is the dimensionality of the embedding.

i ∈ I refers to each sample in the batch,

P(i) is the set of positive samples for the anchor i,

A(i) is the set of all other samples excluding the anchor,

$\tau$ is a temperature parameter to scale the similarities, typically set at $\mathsf{\tau }=0.07$.

$\mathrm{sim}\left({\mathit{z}}_{\mathit{i}},{\mathit{z}}_{\mathit{p}}\right)$denotes the cosine similarity between the projections ${\mathit{z}}_i$ and ${\mathit{z}}_{\mathit{p}}$.

In datasets with numerous easy-to-contrast pairs, contrastive loss gradients can become minimal, hindering informative updates [42,45]. To mitigate this, we employ hard pair scaling with a factor of 1.5, identifying the most challenging negative examples through cosine similarity analysis. Noting that, while supervised contrastive loss aims to distinguish pairs across classes, we complement this with a softmax-based cross-entropy loss [34], blocking its gradient to prevent encoder updates during training.

2.5. Baseline Features and Model Used for Comparison

A baseline approach was established using traditional acoustic feature extraction. Two primary feature sets were selected: MFCCs and eGeMAPS, known for their ability to distinguish pathological from healthy speech [20,25,46,47]. MFCCs were computed using Hamming-windowed frames of 25 ms with a 5 ms shift, producing a 39-dimensional vector per utterance, including cepstral coefficients and their derivatives, via the Librosa library [48]. eGeMAPS (extended Geneva minimalistic acoustic parameter set) [36], designed to capture key vocal characteristics, comprised 88 features, including prosodic, excitation, vocal tract, and spectral descriptors, extracted using the openSMILE toolkit [25,49].

The extracted features were classified using an XGBoost model [35] (details in supplementary materials) and a multilayer perceptron (MLP) with a predefined architecture. These classifiers predicted speech categories and served as a baseline for comparison with proposed ASR-based deep learning methods.

2.6. Experimental Setup and Training Parameters

We utilized several Python libraries in our implementation, including PyTorch [39] for developing deep learning models, Pandas [50] and NumPy [51] for data manipulation and analysis, and Matplotlib [52] along with Scikit-learn [53] for visualization and certain analytical tasks.

As detailed in Table 1, key training hyperparameters used during model optimization included learning rate, batch size, and number of epochs. The models were trained using an Adaptive Moment Estimation optimizer with Weight Decay (AdamW), an optimization algorithm with cross-entropy loss to measure prediction error [54]. A learning rate of 2e-5 was set initially and adjusted over time per a scheduler. We implemented the experiments using a system comprising an Intel Core i7 13700K CPU @ 5.2 GHz, with 128GB of RAM and GPU NVIDIA RTX 4090 24GB.

2.7. Evaluation Metrics

This study employed five primary evaluation criteria: accuracy, F1-score, specificity, sensitivity, and precision. The equations for these metrics are presented in Eqs. 2-6.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FN+\mathrm{ }FP}}$$

(2)

$$\begin{array}{c}Sensitivity={\displaystyle \frac{TP}{TP+FN}}\\ \end{array}$$

(3)

$$Specificity={\displaystyle \frac{TN}{TN+FP}}$$

(4)

$$\begin{array}{c}Precison={\displaystyle \frac{TP}{TP+FP}}\\ \end{array}$$

(5)

$$\begin{array}{c}F1={\displaystyle \frac{2\times Recall\times Precison}{Precison+Recall}}\\ \end{array}$$

(6)

where TP, TN, FP, and FN represent the counts of true positives, true negatives, false positives, and false negatives, respectively. Sensitivity and specificity are important statistical metrics that indicate the proportion of correctly identified positive and negative cases.

2.8. The Grad-CAM Technique for Visual Explanations

Grad-CAM (Gradient-weighted Class Activation Mapping) [33] can be adapted for ASR transformer models to visualize prediction relevance in audio data. By overlaying Grad-CAM heatmaps on audio waveforms, researchers can identify which time segments contain critical features for model inference [55]. The technique involves computing gradients of the target output with respect to attention weights in the final transformer layer, allowing visualization of the most contributory audio signal parts for tasks like Parkinson's disease classification.

3. Results and Discussion

This section presents a comprehensive analysis of the results and assesses the performance of the models examined in this study.

3.1. Classification Performance

To assess the models' performance in a fair and comprehensive manner, we divided the dataset into two parts: 80% for training and 20% for testing, ensuring no speaker overlap between them. The test set served as the final evaluation benchmark. Using the training set, a stratified speaker-independent 5-fold cross-validation approach was implemented for all experiments. The data was divided into 5 folds, ensuring no overlap of patients between folds to prevent data leakage. The model was trained on 4 folds (80% of training data) and assessed on the remaining fold (20%), with this process repeated 5 times so each fold served as the evaluation set once. Early stopping was employed during training to prevent overfitting by saving the best-performing model based on validation performance. Finally, we evaluated the best model's generalization capability on the previously held-out test set. This method provided a rigorous assessment of model performance on unseen data. The performance of each model on the test set, including precision, recall, F1 score, and accuracy, is detailed in Table 2. The graph representation of the corresponding performance (F1 score) for each model is shown in Figure 3.

Across all models, training with Supervised Contrastive consistently yields better F1 scores, Precision, and Sensitivity compared to Cross Entropy alone. Specifically, Wav2Vec2.0 with Supervised Contrastive achieves the highest F1 score of 90.0%, along with improved performance on other metrics, indicating that supervised contrastive training benefits the model's ability to accurately identify and classify relevant patterns.

The bar graph visually represents the F1 scores of these models, showcasing Wav2Vec2.0-base and HuBERT with Supervised Contrastive as the top-performing configurations, with HuBERT's performance closely matching that of Wav2Vec2.0. The baseline models (XGBoost and MLP) trained on acoustic features achieved F1-scores between 73.33% and 74.52%, and AUC scores between 0.80 and 0.81. These results were significantly lower than those of our ASR-based models, demonstrating the superior performance of fine-tuned ASR models in PD detection. Among the models, MLP (MFCCs features) and XGBoost (MFCCs features) exhibit the lowest F1 scores. Of these, XGBoost and MLP training with egeMAPS features performs slightly better, but both are outperformed by the other models.

The error bars demonstrate the variability across runs, with the highest-performing models exhibiting lower variability. These results emphasize the effectiveness of Supervised Contrastive training across both ASR models, while fine-tuning for Spanish (Wav2Vec2.0-FT[es]) shows a competitive yet slightly lower performance, suggesting some limitations in the transferability or adaptation of the model across languages.

The proposed models also were assessed via cumulative confusion matrices and receiver operating characteristic (ROC) curves across a 5-fold cross-validation. The cumulative confusion matrices provide a comprehensive view of model performance by summarizing results across all folds. The ROC curves illustrate the tradeoff between the true positive rate and the false positive rate, offering insights into the diagnostic power of the models. The area under the ROC curve (AUC) serves as an indicator of model performance, with higher AUC values reflecting superior classification ability. The cumulative confusion matrices in Figure 4 evaluate the diagnostic performance of three models trained with cross entropy and SupCon across five cross-validation folds.

In the context of diagnosing PD, recall (sensitivity) is particularly critical because Parkinson's is a progressive neurodegenerative disorder where early detection is essential for managing symptoms and potentially slowing disease progression. High recall ensures that the model captures as many true Parkinson cases as possible, minimizing false negatives that could delay essential treatment. Among the models, Wav2Vec 2.0 demonstrates the most balanced performance, achieving high sensitivity (92.6%) while maintaining competitive accuracy across both classes. HuBERT follows closely, but shows slightly lower sensitivity for Parkinson cases (91.3%), whereas the Wav2Vec 2.0-FT[es] model, although effective, has a lower recall for Healthy cases (73.8%) and performs best in identifying Parkinson cases (87.7%). This analysis shows that while all three models are good at differentiating between Healthy and Parkinson classes, Wav2Vec 2.0’s superior recall makes it particularly suitable for early diagnosis of PD, where maximizing sensitivity is essential to improving patient outcomes through timely treatment. Additionally, the cumulative confusion matrices for the baseline models are provided in Supplementary Figure S3.

Figure 5 presents the ROC curves comparing the performance of different model variants and training approaches. Overall, the baseline models (XGBoost and MLP) demonstrated lower performance (AUC < 0.81) compared to the ASR-based models. The SupCon methods demonstrated superior performance, with HuBERT SupCon and Wav2Vec SupCon achieving the highest AUC scores of 0.93 and 0.92, respectively. These results significantly outperformed the cross-entropy based approaches, which showed comparatively lower performance with AUC values ranging from 0.84 to 0.89. Specifically, Wav2Vec2-FT[es] CE showed the lowest performance with an AUC of 0.84, while Wav2Vec2 CE and HuBERT CE performed moderately better with AUC scores of 0.89 and 0.88, respectively. The steeper curves and higher true positive rates at lower false positive rates for the SupCon models indicate their enhanced ability to discriminate between classes compared to their CE-based counterparts.

To more effectively assess the model's performance across different utterances in the dataset, we analyzed the accuracy for each individual sentence type. Figure S4 presents a comparative analysis of the detection accuracy across various speech samples. The analysis includes both Wav2Vec 2.0 and HuBERT models, along with their variants, using Cross Entropy and Supervised Contrastive loss functions. The dataset comprises multiple sentence types, identified by specific IDs such as ABLANDADA, PIDIO, BURRO, and others, each representing distinct speech samples from the dataset.

Results showed that Wav2Vec 2.0 shows superior performance, especially when paired with Supervised Contrastive Loss, achieving accuracies between 0.87-0.94, compared to 0.83-0.89 with Cross Entropy Loss. The largest gain is observed for the " ABLANDADA " sentence, where accuracy reaches 0.94, showcasing this combination of model and loss function as particularly effective. HuBERT model performs slightly lower but still effectively, with accuracies of 0.84-0.91 using Supervised Contrastive Loss and 0.83-0.89 with Cross Entropy. Overall, Supervised Contrastive Loss consistently outperforms Cross Entropy across both models, with Wav2Vec 2.0 emerging as the most effective model for PD detection from this speech dataset.

3.2. Grad Cam Feature Visualization

Figure 6 shows Grad-CAM feature map visualizations for 10 sample audio clips taken from our dataset (ABLANDADA sentence). To ensure a meaningful comparison between Grad-CAM and t-SNE visualizations, we exclusively used models from the final fold. Grad-CAM analysis reveals which regions of the input contribute most significantly to the model's predictions. For this visualization, we focused on the output projections from the final self-attention layer. This layer encapsulates the highest-level features processed by the model. It directly influences the model's output by projecting the attention mechanism's output back into the embedding space, making it a good candidate for identifying the input regions that contribute most significantly to the final decision.

Both models highlight regions within the waveforms, but with differing intensity and spread, which suggests that they may be emphasizing slightly different acoustic features for classification. Regarding Healthy class, The wav2vec model's Grad-CAM heatmaps generally exhibit more consistent and expansive high-activation zones across the waveform compared to HuBERT, potentially indicating a broader approach in feature extraction. Conversely, HuBERT shows narrower, more localized regions of attention, suggesting it might be focusing on more specific, fine-grained features of the audio signal. When analyzing Parkinson's speech samples, both models demonstrate a consistent focus on the central regions of the audio waveforms. In most cases their focus is quite similar to each other. A notable observation is that the HuBERT model appears to strongly activate voiced segments in Parkinson's speech while paying attention to small unvoiced sections in healthy speech samples. This variation in activation regions could reflect differences in how each model captures the temporal structure of speech relevant to distinguishing healthy from Parkinsonian speech patterns.

Overall, the application of Grad-CAM can provide clinicians with user-friendly visualizations, highlighting the audio regions that influence a model's detection of PD. This capability allows clinicians to align the model’s decisions with their medical knowledge, thereby increasing confidence in the model's reliability, and identifying areas for improvement if the heatmap shows activation in irrelevant audio sections.

3.3. Ablation Study

In this section, we present the results of several ablation experiments to evaluate the impact of different components in our proposed integrated SupCon training. We specifically examine how factors such as projection size, temperature settings, freezing the encoder, utilizing features from various layers, and the impact of not scaling hard negatives influence performance.

Table 3 shows the ablation results of Wav2Vec 2.0 and HuBERT with SupCon across different projection head sizes. For Wav2Vec 2.0, the best accuracy (89.43%) and F1 score (90.00%) are achieved with a projection head size of 32. However, larger head sizes result in a slight decline in performance, with accuracy dropping to 87.43% for a head size of 256. Similarly, for HuBERT, the highest performance is observed with a projection head size of 32, achieving an accuracy of 88.06% and F1 score of 88.99%. Increasing the projection head size also leads to a marginal decrease in performance for HuBERT, with accuracy falling to 87.60% for a head size of 256.

Table 4 presents the performance metrics of three models: Wav2Vec 2.0 + CE, HuBERT+ CE, and Wav2Vec 2.0-FT[es] + CE, with and without fine-tuning. Fine-tuning consistently improved the performance across all models. Notably, HuBERT+ CE with fine-tuning achieved the highest accuracy (85.49%) and sensitivity (91.31%), indicating robust recognition capabilities. In contrast, Wav2Vec 2.0-FT[es] + CE, despite showing significant improvements post fine-tuning, had the lowest scores without fine-tuning, suggesting its dependency on additional training for optimal performance. The results underscore the importance of fine-tuning in enhancing model effectiveness, particularly for achieving higher specificity and sensitivity.

Wav2Vec 2.0 and HuBERT, both coupled with SupCon, were evaluated for classification performance at different temperatures (0.03, 0.07, and 0.10) as detailed in Table 5. Wav2Vec 2.0 achieved its highest accuracy (89.43% ± 2.31%) and F1-score (90.00% ± 2.35%) at a temperature of 0.07. HuBERT with SupCon showed comparable performance at 0.03 and 0.07 but a significant drop at 0.10, despite higher sensitivity. The optimal temperature appears model-specific, with 0.07 favoring Wav2Vec and HuBERT.

We also conducted an ablation study on feature extractor layers (Table S4) and the impact of scaling hard negatives with different projection sizes in SupCon (Table S5). For Cross Entropy criteria, both Wav2Vec 2.0 and HuBERT showed improved performance with deeper layers, peaking at Layer 12. Wav2Vec 2.0 achieved 85.89% accuracy (±4.27) and 86.86% F1-score (±3.42), while HuBERT reached 85.49% accuracy (±2.41) and 86.27% F1-score (±2.26). Sensitivity peaked at Layer 12 for both models, though specificity showed higher variability. For SupCon criteria, scaling hard negatives consistently improved metrics, with Wav2Vec 2.0 outperforming HuBERT, particularly in accuracy, F1-score, and precision. A projection size of 32 outperformed 64 for both models, with metrics showing stable variations (2-4%). Detailed results are provided in Tables S4 and S5 in the supplementary materials.

4. Conclusions

In summary, this study highlights the potential of leveraging advanced ASR models for the detection and monitoring of PD through speech analysis Utilizing the NeuroVoz voice and speech database. In these experiments, classification systems were developed using two popular pre-trained ASR models, wav2vec 2.0 and HuBERT, for feature extraction, alongside traditional baseline features and the base models (XGBoost and MLP). The ASR-based models were trained using two approaches: straightforward cross-entropy and supervised contrastive learning, aimed at accurately predicting the output labels. Results showed that features from pre-trained models outperformed the baseline features, suggesting that unsupervised training on diverse healthy speech data provides a strong foundation for pathological voice detection.

Our approach yielded promising results, with Wav2Vec 2.0 achieving a F1-score of up to 90.00% in distinguishing PD from healthy individuals. Furthermore, the application of SupCon enhanced the model's performance, improving the ability to discern subtle speech changes linked to PD. In this context, the contrastive learning strategy improved Wav2Vec 2.0's performance from 0.89 and 86.86% to 0.92 and 90.00% for the metrics the AUROC and F1-Score. The integration of gradient-based feature importance methods, such as Grad-CAM, allowed us to visualize and understand the specific regions of the speech signal that contributed most to the model's decisions, adding a layer of interpretability.

In conclusion, these results underscore the efficacy of ASR models along with contrastive learning in non-invasive PD diagnosis and monitoring, paving the way for their potential application in clinical settings. Despite the challenges posed by the limited availability of medical datasets, future work will focus on extending this method to cross-linguistic PD prediction, thereby contributing to the development of more robust and universally applicable diagnostic tools.