ECG-SSL: A Systematic Benchmark for Self-Supervised Learning Strategies on Multi-Center Electrocardiogram Data

1. Introduction

Cardiovascular diseases (CVDs) remain the leading cause of mortality worldwide, accounting for an estimated 17.9 million deaths annually [1]. The twelve-lead electrocardiogram (ECG) is the most widely deployed non-invasive cardiac diagnostic tool, recording the electrical activity of the heart from twelve standardized vantage points. Despite its ubiquity, ECG interpretation remains subject to significant inter-observer variability, with disagreement rates of 12–33% even among board-certified cardiologists [2,3]. This variability has motivated substantial interest in automated ECG interpretation through deep learning.

1.1. Supervised Learning and Its Limitations

Early deep learning approaches for ECG classification adopted fully supervised training, achieving remarkable performance when large annotated datasets are available. Hannun et al. [4] demonstrated cardiologist-level performance on rhythm classification using a 34-layer convolutional neural network trained on 91,232 annotated recordings. Ribeiro et al. [5] extended this to 12-lead ECG with a residual network trained on 2.3 million recordings from the Brazilian telehealth network. However, supervised approaches face a fundamental bottleneck: they require large volumes of expert-annotated data that are expensive, time-consuming, and difficult to scale across institutions with heterogeneous recording equipment, patient populations, and annotation practices.

1.2. Self-Supervised Learning for ECG

Self-supervised learning (SSL) offers a compelling alternative by learning representations from unlabeled data through pretext tasks [6,7,8]. The core idea is to train a model to solve a task that does not require human annotations—such as predicting the relationship between augmented views of the same recording (contrastive learning), maintaining consistency between a student and a slowly-updating teacher network (feature-consistency learning), or reconstructing masked portions of the input signal (generative learning). The resulting representations can then be fine-tuned for diverse downstream tasks with minimal labeled data.

In the ECG domain, several SSL approaches have been explored. Contrastive methods include 3KG [9], which applied contrastive learning to 12-lead ECG, and CLOCS [10], which leveraged temporal and spatial invariances specific to ECG. Multi-modal approaches such as ECG-Chat [11] and MERL [12] combine ECG signals with paired clinical text from MIMIC-IV-ECG. Masked signal reconstruction has been explored by MAEFE [13] and others.

1.3. Gaps in the Current Landscape

Here we introduce OpenECG, an open benchmark that addresses these gaps by unifying 1,233,337 twelve-lead ECG recordings from 483,837 patients across nine centers spanning four countries (Table 7). We perform the first systematic comparison of three representative SSL paradigms—SimCLR [6], BYOL [7], and MAE [8]—under identical data conditions. Through six complementary evaluation protocols, we provide actionable evidence that:

1.

Feature-consistency (BYOL) and generative (MAE) SSL approaches are substantially more data-efficient, label-efficient, and augmentation-robust than contrastive learning (SimCLR) for multi-center ECG data.

2.

Dataset diversity contributes more to cross-institutional generalization than dataset size alone, with high-variability datasets providing disproportionate benefits.

3.

SSL pre-trained representations transfer more robustly across institutions than supervised baselines.

4.

Lead masking and temporal masking are the most critical augmentation components for ECG-SSL.

Three critical gaps persist. First, existing studies typically evaluate a single SSL method on one or two datasets under method-specific experimental conditions, making fair cross-method comparison impossible. Second, the interaction between SSL strategy, pre-training data diversity, downstream label availability, and augmentation design is poorly characterized. Third, no open benchmark provides standardized data splits, preprocessing pipelines, and evaluation protocols for reproducible comparison.

1.4. Contributions

Figure 1. Overview of OpenECG. Top: data collection from nine public 12-lead ECG datasets worldwide, self-supervised pre-training pipeline, and cross-institutional generalizability testing. Bottom: the three evaluated SSL paradigms—(A) SimCLR (contrastive learning with positive and negative pairs), (B) BYOL (feature-consistency learning with positive pairs only, using a momentum-updated target network), and (C) MAE (masked signal modeling via reconstruction of masked ECG segments).

Figure 1. Overview of OpenECG. Top: data collection from nine public 12-lead ECG datasets worldwide, self-supervised pre-training pipeline, and cross-institutional generalizability testing. Bottom: the three evaluated SSL paradigms—(A) SimCLR (contrastive learning with positive and negative pairs), (B) BYOL (feature-consistency learning with positive pairs only, using a momentum-updated target network), and (C) MAE (masked signal modeling via reconstruction of masked ECG segments).

2. Results

2.1. Cross-Institutional Generalization

To assess how well SSL-trained models generalize to unseen institutions, we employed a leave-one-dataset-out (LODO) protocol: for each of six labeled datasets, we pre-trained on all remaining data and evaluated on the held-out dataset as an independent test set. This protocol simulates the clinically realistic scenario of deploying a pre-trained model at a new institution without local training data.

Table 1 and Figure 2 present the LODO results. All three SSL methods substantially outperformed random initialization (consistently 3–8% lower AUROC across datasets). Performance was broadly comparable on well-standardized datasets (Chapman, PTB-XL; $P>0.1$), but the performance gap widened substantially on high-variability datasets. On Georgia, BYOL achieved an F1-score of 0.262 compared to only 0.123 for SimCLR—a 113% relative improvement. On INCART, MAE outperformed SimCLR by 1.4% in AUROC (0.712 vs. 0.698).

2.2. Impact of Dataset Inclusion in Pre-Training

To disentangle the contribution of each dataset to cross-institutional generalization, we compared model performance with and without the target dataset included in pre-training (Table 2, Figure 3).

The inclusion benefit varied systematically with dataset variability. INCART and CPSC showed the largest improvements when included: BYOL gained +4.2% AUROC on INCART and +3.8% on CPSC. Well-standardized datasets showed minimal benefit (Chapman ≤1.1%, PTB-XL ≤1.5%). The method-by-dataset interaction was statistically significant (two-way ANOVA: $F_{10,72}=2.84$, $P=0.012$). BYOL showed the largest average inclusion benefit (+2.8%), followed by MAE (+2.2%) and SimCLR (+1.9%).

2.3. Pre-Training Data Scaling Behavior

We systematically varied the pre-training data volume from 1% to 100% in logarithmic increments (Figure 4). Three distinct scaling regimes emerged:

Rapid learning phase (1–30% data). All methods showed steep performance gains (8–15 AUROC percentage points). MAE exhibited the fastest initial learning rate on larger datasets.

Diminishing returns phase (30–70% data). BYOL and MAE reached >95% of their maximum AUROC by approximately 60–70% of total data.

Saturation phase (70–100% data). BYOL and MAE showed marginal improvements (<0.5% AUROC), while SimCLR continued to improve up to 80–90% of data before plateauing—reflecting its reliance on diverse negative pairs.

2.4. Label Efficiency Under Few-Shot Regimes

We evaluated fine-tuning performance using 1%, 5%, 10%, 50%, and 100% of available diagnostic labels (Figure 5, Table 3).

At 10% labeled data, BYOL achieved an average of 94% of its full-data performance, compared to 88% for SimCLR and only 78% for random initialization. This means BYOL requires roughly 10× less labeled data than training from scratch to achieve 90% of the best possible performance. The label-efficiency gap was largest on Georgia (BYOL at 10%: 0.510 AUROC; random at 10%: 0.418; $\Delta =+9.2\%$). At 50% labels the gap narrowed to <2%.

2.5. Representation Quality: Linear Probing

To characterize the quality of learned representations independent of fine-tuning capacity, we trained a single linear classification layer on frozen pre-trained features (Table 4, Figure 6).

BYOL and MAE produced consistently higher linear probing performance than SimCLR (mean AUROC: 0.779 and 0.773 vs. 0.750). The differences were statistically significant (BYOL vs. SimCLR: $t=6.44$, $P=0.001$; MAE vs. SimCLR: $t=3.28$, $P=0.022$), while the difference between BYOL and MAE was not significant ($t=1.22$, $P=0.277$). SimCLR showed the largest gap between fine-tuned and linear-probe performance (mean $\Delta$: 7.5%), while BYOL’s gap was smallest (3.9%).

2.6. Cross-Domain Transfer Robustness

For each ordered pair of labeled datasets $(A,B)$ with $A\ne B$, we fine-tuned the pre-trained model on dataset A and evaluated on dataset B (Figure 7). BYOL and MAE achieved higher mean off-diagonal transfer AUROC (0.749 and 0.750) than SimCLR (0.740). The strongest cross-domain transfer occurred between CPSC and Chapman (mean AUROC 0.89); the largest domain gap was observed when Georgia served as the source (mean target AUROC: 0.68).

2.7. Augmentation Strategy Ablation

Starting from the full augmentation pipeline (temporal masking, lead masking, Gaussian noise, amplitude scaling, baseline wander), we removed each component individually (Figure 8, Table 5).

Lead masking is the most critical augmentation. Removing it produced the largest performance drop: −2.5% for SimCLR, −1.9% for BYOL, and −0.8% for MAE. Lead masking forces the model to learn cross-lead dependencies encoding critical diagnostic information.

Temporal masking ranks second. Removing it caused drops of −1.8% (SimCLR), −1.2% (BYOL), and −0.6% (MAE).

SimCLR is most sensitive to augmentation; MAE is most robust. Without all augmentations, SimCLR dropped 6.2% vs. only 2.2% for MAE—a 3× sensitivity difference. Gaussian noise ($\Delta <1\%$) and baseline wander ($\Delta <0.6\%$) had minimal impact on all methods.

2.8. Representation Space Visualization

t-SNE visualization on a held-out Chapman subset comprising six cardiac conditions (Figure 9) showed that BYOL representations had the tightest within-class clustering and clearest between-class separation, consistent with its highest linear probing performance. MAE produced intermediate clustering quality, while SimCLR representations were the most diffuse.

2.9. Comparison with Prior SSL-Based ECG Models

Table 6 benchmarks OpenECG against published SSL-based ECG approaches. On CPSC, our MAE achieved F1/AUROC of 74.5/93.2, competitive with MERL (72.8/92.6) and approaching ECG-Chat (80.1/95.7). On Chapman, SimCLR achieved 52.3/95.1, substantially outperforming 3KG (43.2/88.3). A critical distinction is evaluation breadth: while most prior models report results on 1–3 datasets, OpenECG evaluates on six independent datasets across four countries.

2.10. Multi-Dimensional Method Comparison

Figure 10 synthesizes the three SSL methods across six evaluation dimensions: mean AUROC, mean F1, linear probing quality, data efficiency, cross-institutional robustness, and low-data performance. BYOL and MAE dominate SimCLR on four of six dimensions (data efficiency, linear probing, cross-institutional robustness, low-data performance), while SimCLR maintains a slight edge in mean AUROC.

3. Discussion

This study establishes OpenECG as the first open, standardized benchmark for systematic comparison of SSL strategies on multi-center ECG data.

3.1. Feature-Consistency and Generative SSL Outperform Contrastive Learning

BYOL and MAE consistently surpassed SimCLR across all evaluation axes. The mechanistic explanation centers on the role of negative pairs: SimCLR’s contrastive objective maximizes agreement between augmented views of the same recording while pushing apart views from different recordings. In highly heterogeneous multi-center data, two recordings from different institutions may differ more in acquisition characteristics than in cardiac pathology, causing the model to learn institutional signatures rather than cardiac features. BYOL avoids this problem by using only positive pairs; MAE sidesteps it by framing the pretext task as signal reconstruction. Our findings parallel recent observations in computer vision, where BYOL and DINO [14] have shown advantages over SimCLR on diverse datasets [15], and extend them to the ECG domain.

3.2. Dataset Diversity Supersedes Volume

INCART, with only 75 recordings but highly distinctive characteristics (257 Hz, 30-minute duration), provided a +4.2% inclusion benefit for BYOL—comparable to or larger than the benefit from much larger but more standardized datasets. This finding challenges the prevailing “more data is always better” paradigm and suggests that strategic curation emphasizing diversity in recording conditions, sampling rates, durations, and patient demographics may be more resource-efficient than accumulating homogeneous data from a single institution.

3.3. SSL Dramatically Reduces Annotation Requirements

BYOL achieves 94% of its full-data performance with only 10% of labeled data (vs. 78% for random initialization), enabling an order-of-magnitude reduction in annotation burden. These results have direct practical implications: in clinical settings where cardiologist annotation time is the primary bottleneck, SSL pre-training can dramatically reduce the annotation budget without substantially compromising performance.

3.4. Augmentation Design Is Critical and Method-Dependent

Lead masking is the most critical augmentation ($\Delta$ up to −2.5%), as it forces the model to learn cross-lead dependencies encoding critical spatial information about cardiac electrical activity propagation. The 3× sensitivity difference between SimCLR and MAE provides practical guidance: when optimal augmentation strategies are unknown, MAE-style generative approaches offer a safer starting point because they are less dependent on augmentation design.

3.5. Limitations

Several limitations should be acknowledged. First, we evaluate three SSL methods but do not cover all recent variants (e.g., VICReg, Barlow Twins, DINO, data2vec). Second, the benchmark is limited to 12-lead hospital ECG; wearable single-lead data warrants separate benchmarking. Third, SimCLR and BYOL use ResNet-50 while MAE uses ViT-Base, confounding the SSL objective and backbone architecture. Fourth, our evaluation focuses on classification of 24 conditions; risk stratification tasks require external outcome data. Fifth, we do not evaluate in prospective clinical settings. Finally, demographic information is inconsistently available, limiting fairness analysis.

4. Materials and Methods

4.1. Study Design and Data Sources

OpenECG integrates nine publicly available twelve-lead ECG datasets encompassing 1,233,337 recordings from 483,837 patients across four countries (Table 7). Two large-scale datasets—MIMIC-IV-ECG [16] (800,035 recordings) and CODE-15 [17] (341,292 recordings)—served as unlabeled pre-training data. Six labeled datasets from the PhysioNet Challenge 2020 [18] (CPSC, Georgia, PTB, PTB-XL, St. Petersburg INCART) and Chapman-Shaoxing [19] provided supervised evaluation data covering 24 cardiac conditions defined by SNOMED CT codes [20].

Table 7. Key characteristics of ECG datasets in OpenECG. “Labeled” datasets provide diagnostic labels for evaluation; “Unlabeled” datasets provide pre-training data without labels.

Dataset	#Patients	#Recordings	Duration	Hz	Country	Role
MIMIC-IV	161,352	800,035	10 s	500	USA	Unlabeled
CODE-15	233,770	341,292	7–10 s	400	Brazil	Unlabeled
CPSC	14,053	14,053	6–60 s	500	China	Labeled
Georgia	10,344	10,344	10 s	500	USA	Labeled
PTB	249	549	var.	1000	Germany	Labeled
PTB-XL	18,885	21,837	10 s	500	Germany	Labeled
St. Pet.	32	75	30 min	257	Russia	Labeled
Chapman	45,152	45,152	10 s	500	China	Labeled
Total	483,837	1,233,337

Table 7. Key characteristics of ECG datasets in OpenECG. “Labeled” datasets provide diagnostic labels for evaluation; “Unlabeled” datasets provide pre-training data without labels.

Dataset	#Patients	#Recordings	Duration	Hz	Country	Role
MIMIC-IV	161,352	800,035	10 s	500	USA	Unlabeled
CODE-15	233,770	341,292	7–10 s	400	Brazil	Unlabeled
CPSC	14,053	14,053	6–60 s	500	China	Labeled
Georgia	10,344	10,344	10 s	500	USA	Labeled
PTB	249	549	var.	1000	Germany	Labeled
PTB-XL	18,885	21,837	10 s	500	Germany	Labeled
St. Pet.	32	75	30 min	257	Russia	Labeled
Chapman	45,152	45,152	10 s	500	China	Labeled
Total	483,837	1,233,337

4.2. Data Preprocessing and Standardization

All ECG recordings were standardized to a uniform format of $12\times 1,000$ time points through linear interpolation resampling. For recordings shorter than the target length, zero-padding was applied at the end. For recordings longer than 10 seconds, a random 10-second segment was extracted during training, while the first 10 seconds were used during evaluation. Each lead was independently normalized to zero mean and unit variance. Recordings with flat-line segments (amplitude variance $<{10}^{-6}$) on any lead were excluded. Data splits were performed strictly at the patient level to prevent data leakage.

4.3. Self-Supervised Pre-Training Methods

We evaluated three representative SSL paradigms:

SimCLR [6] (Contrastive learning): Two augmented views of each ECG recording are passed through a shared ResNet-50 encoder [21] and projection MLP. The NT-Xent loss maximizes agreement between views of the same recording while pushing apart views from different recordings. We used temperature $\tau =0.07$ and batch size 256.

BYOL [7] (Feature-consistency learning): An online network (encoder + projector + predictor) learns to predict the output of a target network applied to a different augmented view. The target network is updated via exponential moving average ($\tau =0.996$) of the online network. No negative pairs required. ResNet-50 backbone.

MAE [8] (Masked signal modeling): The input ECG is divided into patches, 75% of which are randomly masked. A Vision Transformer (ViT-Base [22]) encoder processes only the visible patches, and a lightweight decoder reconstructs the masked patches using MSE loss.

All models were pre-trained for 200 epochs using the AdamW optimizer with cosine learning rate scheduling (initial LR: $1\times {10}^{-3}$, minimum: $1\times {10}^{-6}$, weight decay: 0.05, warmup: 10 epochs).

4.4. Data Augmentation Strategies

We employed five ECG-specific augmentation strategies:

1.

Temporal masking: A contiguous block of 100 time points (10% of the signal) is set to zero at a random position.

2.

Lead masking: 1–3 leads are randomly selected and set to zero.

3.

Gaussian noise: Additive noise with $\sigma =0.05$ applied to all leads.

4.

Amplitude scaling: Random multiplicative factor sampled uniformly from $[0.8,1.2]$.

5.

Baseline wander: Sinusoidal drift with random frequency (0.1–0.5 Hz) and amplitude (0–0.2 mV).

4.5. Evaluation Protocols

Leave-one-dataset-out (LODO). For each of six labeled datasets, the model was pre-trained on all remaining data and the held-out dataset served as an independent test set.

Pre-training data scaling. Pre-training data volume was varied from 1% to 100% (1%, 5%, 10%, 20%, 30%, 50%, 70%, 80%, 90%, 100%).

Few-shot label efficiency. Fine-tuning with 1%, 5%, 10%, 50%, and 100% of available labeled data. A randomly initialized baseline was trained under the same conditions.

Linear probing. A single linear classification layer trained on frozen pre-trained representations.

Cross-domain transfer. For each ordered pair of labeled datasets $(A,B)$, the pre-trained model was fine-tuned on A and evaluated on B.

Augmentation ablation. Each augmentation component was individually removed, plus a control with all augmentations removed.

4.6. Evaluation Metrics and Statistical Analysis

Primary metrics include macro-averaged AUROC and F1-score for multi-label classification of 24 cardiac conditions. All reported metrics include 95% confidence intervals computed via 1,000-iteration bootstrap resampling with patient-level stratification. Between-method comparisons were assessed using paired bootstrap tests with Bonferroni correction. The interaction between SSL method and dataset characteristics was tested using two-way ANOVA. All statistical analyses were performed in Python 3.10 using scikit-learn, scipy, and statsmodels.

4.7. Computational Resources

All experiments were conducted on a computing cluster with NVIDIA A100 GPUs (80 GB VRAM). Pre-training each SSL method required approximately 48 hours on 4×A100 GPUs. The total computational budget was approximately 2,400 GPU-hours.

5. Conclusions

OpenECG provides the first systematic, head-to-head comparison of SSL strategies for multi-center ECG analysis. Our principal findings are:

1.

BYOL and MAE outperform SimCLR across all evaluation axes, with advantages most pronounced on high-variability datasets and in low-label regimes.

2.

Dataset diversity supersedes volume: heterogeneous curation yields better generalization than large homogeneous collections.

3.

SSL reduces annotation requirements by ∼10×: BYOL achieves 94% of full-data performance with only 10% of labels.

4.

Lead masking and temporal masking are the most critical augmentation components; contrastive methods are substantially more augmentation-sensitive than generative methods.

These results offer actionable, evidence-based guidance for SSL strategy selection and establish an open, reproducible benchmark to accelerate progress in AI-driven cardiovascular diagnostics.