Comparative Analysis of CNN and Transformer Models for Multi-Class Diabetic Retinopathy Grading Using Fundus Images

1. Introduction

Retinal diseases are responsible for a substantial proportion of vision impairment and blindness worldwide, posing an increasing challenge to healthcare systems [1]. These disorders, including diabetic retinopathy (DR), pathological myopia (PM), age-related macular degeneration (AMD), and glaucoma (GLA), impose a substantial burden on healthcare systems and significantly affect quality of life, particularly among aging and working-age populations [2]. Early detection and regular monitoring of retinal abnormalities play a crucial role in preventing permanent vision loss.

Among these conditions, diabetic retinopathy (DR), a microvascular complication of diabetes, is one of the most prevalent and clinically significant causes of blindness [2,3]. The global burden of DR continues to increase, with the number of affected individuals projected to rise from approximately 126.6 million in 2010 to 191.0 million by 2030. In particular, cases of vision-threatening diabetic retinopathy (VTDR) are expected to grow from 37.3 million to 56.3 million if timely intervention is not achieved [4]. Despite the availability of effective screening programs and treatment strategies, DR remains a leading cause of vision loss among working-age populations [5]. Early detection and accurate grading of disease severity are essential for timely intervention and effective treatment planning. However, manual examination of retinal fundus images is time-consuming, requires expert ophthalmologists, and is subject to inter-observer variability.

The rapid evolution of artificial intelligence (AI) has accelerated the adoption of deep learning (DL) for automated medical image analysis. This technology has significantly improved the accuracy of disease detection across various image modalities [5,6,7,8], including various retinal diseases [9,10]. Transfer learning (TL) and pretrained models have further improved performance in data-limited clinical settings [11]. In particular, convolutional neural networks (CNNs) have demonstrated strong performance in retinal disease classification tasks due to their ability to learn hierarchical spatial representations from fundus images [12]. Despite their success, CNN-based models primarily focus on local feature extraction and may struggle to capture global contextual relationships, which can be critical for distinguishing subtle differences in diabetic retinopathy severity levels. More recently, transformer-based models have emerged as an alternative paradigm in computer vision, offering the capability to model global contextual relationships through self-attention mechanisms [13]. These models have shown competitive performance in various image classification tasks.

While both CNN-based and transformer-based models have demonstrated promising performance, their relative effectiveness for fine-grained multi-class classification tasks, such as diabetic retinopathy grading, remains an active research topic. As deep learning architectures continue to evolve, consistent comparative evaluations conducted under identical experimental settings for understanding their relative strengths, limitations, and practical applicability in automated retinal screening systems. Moreover, relatively fewer studies investigate the trade-off between classification performance and computational efficiency, which is an important consideration for deploying deep learning models in practical retinal screening systems.

To address these gaps, this study presents a comprehensive comparative analysis of six state-of-the-art pretrained deep learning architectures for five-class diabetic retinopathy grading using retinal fundus images. Three representative CNN-based models, namely ResNet50, EfficientNet-B0, and MobileNetV2, are compared with three transformer-based models, including Vision Transformer (ViT), Swin-Tiny Transformer, and Swin Transformer. These models are trained and evaluated under identical preprocessing, augmentation, transfer learning, training protocols, and evaluation criteria to ensure a fair comparison and to provide a consistent experimental protocol for comparative evaluation. In addition to prediction performance, this study analyzes the computational efficiency of each architecture in terms of model complexity and inference speed, providing practical insights into selecting suitable DL models for automated diabetic retinopathy screening.

The key contributions of this study can be summarized as follows:

• A unified experimental framework for the comparative evaluation of representative CNN-based and transformer-based architectures for five-class diabetic retinopathy grading.
• A systematic evaluation of six pretrained deep learning models, including ResNet50, EfficientNet-B0, MobileNetV2, Vision Transformer, Swin-Tiny Transformer, and Swin Transformer, using transfer learning.
• Investigation of multi-class diabetic retinopathy grading under an imbalanced dataset using a unified training strategy based on weighted loss and balanced sampling.
• A comprehensive comparative analysis consisting of overall and per-class classification performance, QWK, learning and convergence behavior, computational efficiency, and performance stability across five independent runs.

2. Related Work

Deep learning has become the dominant approach for automated diabetic retinopathy grading, leading to the development of numerous CNN-based and transformer-based architectures. Existing studies have explored various strategies to improve classification accuracy, ranging from designing more effective convolutional backbones to adopting self-attention mechanisms for global feature modeling [14,15,16]. This section reviews recent research in three categories: CNN-based approaches, transformer-based approaches, and comparative studies that motivate the proposed work.

2.1. CNN-Based Approaches for Diabetic Retinopathy Classification

CNN-based methods remain the foundation for automated DR grading, largely due to their strong performance on fundus imagery and their compatibility with transfer learning. Early work focused on adapting established backbones such as VGG and ResNet, then progressively shifted toward more parameter-efficient and task-specific designs, including EfficientNet and its variants through transfer learning.

One line of research refines classical VGG-style networks [17]. The VGG-NiN architecture augments VGG16 with spatial pyramid pooling and network-in-network blocks to better capture scale-varying lesions while reducing parameters, enabling robust multiclass grading at varying image resolutions compared with vanilla VGG and traditional pipelines [17]. Building on the advantages of pretraining, Saeed et al. introduce an adaptive fine-tuned CNN that reinitializes and selectively. fine-tunes early convolutional layers on DR data, replaces fully connected layers with a PCA-based feature stage, and employs gradient-boosted classifiers, which together improve generalization on EyePACS and Messidor for multiclass diagnosis [18].

As CNN architectures continued to mature, research shifted toward designing more specialized networks for diabetic retinopathy analysis. A pretrained NASNet-based framework, referred to as DR-NASNet, incorporated Ben Graham preprocessing, CLAHE-based contrast enhancement, data augmentation, and dense blocks to improve feature extraction while maintaining a relatively compact model with lower computational complexity [19].

Similarly, a CNN with residual blocks (DRCNNRB) combined image preprocessing with geometric data augmentation, including rotation, flipping, shearing, zooming, and rescaling, to alleviate class imbalance and improve classification robustness. The proposed framework demonstrated superior diagnostic performance and computational efficiency compared with several existing CNN-based approaches [20]. More recently, lightweight CNN architectures have received increasing attention for practical DR screening. An enhanced MobileNet framework introduced architectural modifications, including Swish activation functions, depthwise separable convolutions, and improved dropout regularization, achieving competitive classification performance while outperforming several benchmark models, including ResNet101, DenseNet121, EfficientNet-B7, and hybrid CNN-based approaches, with substantially lower computational complexity. These findings highlight the potential of lightweight CNN models for resource-constrained and edge-based healthcare applications [21].

As the availability of retinal image datasets increased, researchers began systematically comparing different CNN architectures for diabetic retinopathy grading. One study. [22] systematically evaluate VGG, ResNet, and EfficientNet variants and show that EfficientNet-B3 yields superior performance for five-class DR grading, reflected in a quadratic weighted kappa of 0.85 that better accounts for ordinal misclassification than raw accuracy. Similarly, another study [23] investigated the performance of multiple CNN architectures, including VGG16, ResNet50, and ResNet101, for multiclass DR classification using a large clinical fundus image dataset. Their framework incorporated image preprocessing, data augmentation, and regularization to address data imbalance and improve model generalization. Among the evaluated models, ResNet101 consistently outperformed ResNet50 and VGG16, demonstrating the effectiveness of deeper residual networks for retinal image classification. Their findings further demonstrated the advantage of residual learning over conventional CNN architectures, motivating the inclusion of ResNet-based models in subsequent comparative studies.

Beyond improving standalone CNN architectures, recent studies have explored hybrid deep learning frameworks that integrate CNNs with complementary learning techniques to further enhance diabetic retinopathy classification. One such framework combined retinal image preprocessing, lesion segmentation, transfer learning using SqueezeNet, feature dimensionality reduction, and a He-Weighted Bi-directional Long Short-Term Memory (HWBLSTM) classifier to exploit both spatial and sequential feature representations. The proposed approach achieved competitive performance on the APTOS and MESSIDOR datasets while maintaining relatively low computational overhead, demonstrating the potential of combining CNN-based feature extraction with advanced learning modules for DR grading [24].

Overall, CNN-based approaches have evolved from conventional transfer learning using standard architectures toward specialized network designs, lightweight architectures, comparative evaluations of modern backbones, and hybrid learning frameworks. Despite these advances, most CNN models rely primarily on convolutional operations to capture local spatial information, which may limit their ability to model long-range contextual relationships in retinal fundus images. This limitation has motivated the growing adoption of transformer-based architectures that employ self-attention mechanisms to jointly capture local and global image representations.

2.2. Transformer-Based Approaches for Retinal Image Analysis

Transformer-based architectures have recently emerged as a promising alternative to convolutional neural networks for retinal image analysis, particularly for diabetic retinopathy classification [13,25]. Unlike CNNs, which mainly learn local spatial representations through convolutional operations, transformers employ self-attention to model relationships between distant image regions and learn global contextual representations. These characteristics make transformer models particularly suitable for diabetic retinopathy grading, where disease severity is often determined by multiple lesions distributed throughout the retinal fundus.

Early studies demonstrated the feasibility of applying pure Vision Transformers (ViTs) to diabetic retinopathy classification. A patch-based ViT framework showed that transformer models could achieve competitive performance for five-class DR grading while outperforming several conventional CNN-based approaches, highlighting the effectiveness of global attention mechanisms for retinal image analysis [26]. As transformer models continued to evolve, research increasingly focused on overcoming their dependence on large, annotated datasets and improving feature representation. Self-supervised learning strategies were introduced to pretrain Vision Transformers using unlabeled retinal images before fine-tuning for diabetic retinopathy grading, substantially improving label efficiency while maintaining competitive classification performance [27]. Similarly, optimized Vision Transformer frameworks specifically designed for retinal disease analysis, called OcuViT, show that carefully fine-tuned ViT-Base models, combined with streamlined preprocessing, can achieve state-of-the-art performance on both binary and five-class DR grading (APTOS) as well as AMD grading, outperforming earlier CNN- and ViT-based systems in accuracy and robustness [28].

Recent studies have also explored architectural improvements to better exploit the high resolution and heterogeneity of retinal fundus images. Transformer with Multiple Instance Learning (TMIL) introduced high-resolution fundus images segmentation into 224×224 patches. After that, a patch aggregation strategy was applied using a dedicated Global Instance Computing Block. This design allows reuse of pre-trained ViT weights without image down-scaling, improving accuracy, minimizing the computational cost, and reducing inference time by 62% compared with naïvely feeding high-resolution images into ViT, enabling more efficient analysis of retinal images without excessive image down-sampling [29]

More recently, research has focused on developing advanced transformer architectures that improve robustness, computational efficiency, and feature representation for retinal image analysis. Swin-DRNet [30] introduced a hierarchical Swin Transformer framework that combines contrast-adaptive preprocessing, class-aware representation learning, and focal-loss optimization to improve diabetic retinopathy screening under heterogeneous imaging conditions and severe class imbalance. Experimental evaluation on the APTOS 2019, IDRiD, and Messidor2 datasets demonstrated consistent performance across different datasets, highlighting the strong generalization capability of hierarchical transformers for real-world DR screening. Similarly, an Attention Dual Transformer with Adaptive Temporal Convolution (ADT-ATC) [31] incorporated dual spatial transformer modules, adaptive temporal convolution, and hierarchical cross-attention to model fine retinal lesions and global contextual information jointly. The proposed framework achieved superior classification performance compared with conventional deep learning approaches, demonstrating the potential of advanced attention mechanisms for diabetic retinopathy detection.

While pure ViTs and their variants improve global context modeling, several studies argue that the most effective DR grading systems combine transformers with CNNs to exploit both local lesion sensitivity and long-range relational reasoning. Hybrid models thus emerge as a natural next step after CNN and pure ViT pipelines. Therefore, recent research has investigated hybrid CNN–Transformer frameworks that combine the complementary strengths of convolutional feature extraction and self-attention mechanisms. Representative approaches integrate residual or densely connected CNN backbones with transformer modules to simultaneously capture local retinal lesions and long-range contextual relationships, leading to improved multiclass DR grading performance compared with standalone CNN or transformer models. Examples of these frameworks include: CTNet (combines residual CNN with transformer) [32], and D-TNet (couples DenseNet121 with a Transformer architecture) [33].

Overall, transformer-based approaches have progressed from conventional vision transformers toward improved optimization strategies, self-supervised learning, hierarchical attention mechanisms, and hybrid CNN–Transformer frameworks. These developments have substantially advanced automated diabetic retinopathy grading and demonstrated the effectiveness of attention-based models for retinal image analysis.

2.3. Research Gap and Motivation

The reviewed literature demonstrates substantial progress in automated diabetic retinopathy grading. CNN-based approaches have evolved from conventional transfer learning frameworks toward specialized, lightweight architectures, while transformer-based methods have introduced self-attention mechanisms, hierarchical feature learning, and hybrid CNN–transformer frameworks to further improve classification performance. These advances have significantly enhanced deep learning’s capabilities for retinal image analysis.

Despite these developments, some studies have compared CNN- and transformer-based models for diabetic retinopathy classification. However, differences in the selected architectures, preprocessing pipelines, transfer learning strategies, training protocols, and evaluation criteria make direct comparison across studies challenging. Moreover, most existing studies primarily emphasize predictive performance, with limited comprehensive evaluation of classification performance, class-specific behavior, performance stability, learning behavior, and computational efficiency within a common experimental framework. Therefore, a consistent and controlled benchmark using representative standard and lightweight CNN and transformer architectures remains valuable for understanding the trade-offs between predictive performance and computational cost. Motivated by those gaps, this study presents a unified comparative evaluation of representative pretrained CNN and transformer architectures for five-class diabetic retinopathy grading using consistent preprocessing, training, and evaluation settings.

3. Materials and Methods

3.1. The Proposed Framework Overview

In this study, diabetic retinopathy (DR) grading is formulated as a supervised multi-class classification task. The objective is to automatically classify retinal fundus images into five DR severity levels, ranging from no apparent DR to advanced disease stages. The proposed methodological framework, illustrated in Figure 1, consists of seven main steps:

• Data Acquisition: Retinal fundus images representing the five DR severity classes are collected from publicly available datasets.
• Data Splitting into training, validation, and testing subsets for model development and evaluation.
• Data Preprocessing: Standardization of input images to ensure compatibility across different DL architectures.
• Data Augmentation and Class Imbalance Handling: apply data augmentation and training strategies to improve model generalization and address class distribution imbalance.
• Transfer Learning using Pretrained DL Models: Adaptation of CNN- and transformer-based pretrained models for multi-class DR grading.
• Classification: Prediction of DR severity grades using the learned representations and classification layers.
• Model Evaluation and Analysis: Evaluation and comparison of model performance by the use of standard evaluation metrics and computational analysis.

The following subsections describe each stage of the proposed workflow in detail.

3.2. Dataset Acquisition and Statistics

In this study, the APTOS 2019 blindness detection dataset was utilized for multi-class diabetic retinopathy (DR) grading [34]. The dataset is publicly available through the Kaggle repository and was originally released as part of the Asia Pacific Tele-Ophthalmology Society (APTOS) challenge. It consists of 3,662 RGB retinal fundus images that were clinically examined and annotated by trained ophthalmologists according to the International Clinical Diabetic Retinopathy Disease Severity Scale (ICDRSS) [35]. Each fundus image is assigned to one of five DR severity grades: no diabetic retinopathy (No DR), mild DR, moderate DR, severe DR, and proliferative DR.

The publicly available dataset includes predefined training, validation, and testing subsets, which were directly adopted in this study without performing any additional data splitting to ensure reproducibility. The predefined subsets preserve a similar class distribution across the training (approximately 80%), validation (10%), and test (10%) sets, resulting in an approximately stratified distribution across the five DR severity grades, as summarized in Table 1. The dataset download source is provided in the Data Availability Statement. As shown in Table 1, the dataset exhibits a clear class imbalance, with a substantially larger number of normal retinal images than advanced DR stages. This characteristic was considered during model training by employing class-balanced optimization strategies, as described in Section 4.

Representative retinal fundus images corresponding to the five diabetic retinopathy severity grades are presented in Figure 2. These examples illustrate the progressive visual characteristics of diabetic retinopathy, ranging from normal retinal appearance to proliferative disease.

3.3. Data Preprocessing

All retinal fundus images were processed using a unified preprocessing pipeline to ensure consistency across DL architectures and enable fair comparison between CNN- and transformer-based models. First, all images were converted into RGB format to maintain a consistent three-channel representation. The images were then resized to a fixed resolution of 224x224 pixels to meet the input requirements of the selected pretrained models.

Pixel intensity values were normalized to provide a standardized input distribution and improve training stability. Since the models were initialized with weights pretrained on ImageNet, the images were normalized using the corresponding ImageNet mean and standard deviation. Additionally, the categorical diabetic retinopathy severity labels were converted to numerical class labels corresponding to the five grading levels: No DR (0), Mild DR (1), Moderate DR (2), Severe DR (3), and Proliferative DR (4). The same preprocessing procedure was applied across all evaluated architectures to ensure that performance differences were primarily attributable to model characteristics rather than to variations in data preparation.

3.4. Data Augmentation and Class Imbalance Handling

To improve model generalization and reduce overfitting, data augmentation was applied only to the training set. The augmentation pipeline consisted of random horizontal flipping, small-angle rotation, slight geometric transformations, and mild brightness and contrast adjustments. No augmentation was applied to validation or testing images to ensure unbiased model evaluation. To reduce the effect of class imbalance among DR severity grades, a class-weighted loss function was employed during training, assigning higher penalties to underrepresented classes.

3.5. Selection of Pretrained Deep Learning Architectures

To investigate the performance differences between convolution-based and attention-based architectures, this study selected representative pretrained models from CNN and transformer families. The selected CNN models (ResNet50, EfficientNet-B0, and MobileNetV2) represent complementary convolutional design strategies, namely residual learning, compound model scaling, and lightweight network design. Similarly, the selected transformer models (ViT, Swin-Tiny Transformer, and Swin Transformer) represent complementary attention-based paradigms, including global self-attention and hierarchical window-based attention, while covering both lightweight and higher-capacity architectures. This selection enables a comprehensive comparison of classification performance and computational requirements for multi-class diabetic retinopathy grading.

3.5.1. CNN-Based Models

To evaluate the effectiveness of convolution-based architectures for diabetic retinopathy grading, three representative CNN models were selected: ResNet50, EfficientNet-B0, and MobileNetV2. These models represent different CNN design strategies, including deep residual learning, efficient model scaling, and lightweight architectures. CNN-based models are particularly effective at extracting local spatial patterns from retinal images, which are important for identifying DR-related abnormalities, including lesions, texture variations, and structural changes.

ResNet50:

ResNet50 [36] is a deep CNN architecture based on residual learning, in which shortcut connections are introduced to improve gradient flow and mitigate the degradation problem associated with training deeper networks. Through multiple convolutional layers, ResNet50 learns hierarchical visual representations ranging from low-level features, such as edges and textures, to more complex disease-related patterns. In this study, ResNet50 was selected as a standard CNN baseline to evaluate the capability of deep residual architectures for multi-class DR grading.

EfficientNet-B0:

EfficientNet-B0 [37] is based on compound scaling, which systematically balances network depth, width, and input resolution to improve performance while maintaining computational efficiency. This balanced scaling allows the model to learn representative visual features with fewer parameters compared with conventional CNN architectures. EfficientNet-B0 was included to investigate the relationship between classification performance and computational efficiency in retinal image analysis.

MobileNetV2:

MobileNetV2 [38] is a lightweight CNN architecture designed to reduce computational complexity through depth wise separable convolutions and inverted residual blocks. These mechanisms reduce the number of trainable parameters while preserving effective feature extraction. In this study, MobileNetV2 was evaluated to assess the potential of lightweight CNN architectures for efficient DR grading, particularly in scenarios with limited computational resources.

3.5.2. Transformer-Based Models

To compare convolution-based models with attention-based approaches, three transformer architectures were evaluated: Vision Transformer (ViT), Swin-Tiny Transformer, and Swin Transformer. Unlike CNNs, which primarily focus on local feature extraction, transformer-based models use self-attention mechanisms to capture relationships between different image regions. This capability may be beneficial for retinal image analysis, where disease severity can depend on abnormalities distributed across different areas of the fundus image.

Vision Transformer (ViT):

ViT [39] adapts transformer architectures for image classification by dividing an image into fixed-size patches and modeling relationships among these patches using self-attention mechanisms. This design enables the model to capture global contextual information across the entire retinal image. ViT was selected as the standard transformer baseline to evaluate the effectiveness of global attention-based representation learning for DR grading.

Swin-Tiny Transformer:

Swin-Tiny Transformer [40] is a lightweight hierarchical transformer architecture that introduces shifted window-based self-attention. By limiting attention computation within local windows while enabling information exchange between regions, it reduces computational complexity while maintaining contextual feature learning. Swin-Tiny was included to investigate the performance of efficient transformer architectures compared with lightweight CNN models.

Swin Transformer (Base variant):

Swin Transformer [40] extends the vision transformer concept by incorporating hierarchical feature extraction and shifted-window attention mechanisms, enabling the model to capture multi-scale representations. This architecture combines local feature modeling with broader contextual understanding, which can be valuable for analyzing retinal abnormalities appearing at different scales. In this study, Swin Transformer was selected as a stronger hierarchical transformer architecture for comparison with CNN-based models.

The selected architectures allow analysis of model behavior across different design families, including CNN versus transformer models, standard versus lightweight architectures, and accuracy–efficiency trade-offs.

3.6. Transfer Learning, Fine-Tuning, and Classification Strategy

To leverage knowledge learned from large-scale pretrained architectures, this study adopted a transfer learning approach for multi-class diabetic retinopathy grading from retinal fundus images. As described in Section 3.4, the selected CNN-based and transformer-based models were pretrained on large-scale image datasets and have demonstrated strong capabilities in extracting meaningful visual representations. Through transfer learning, these pretrained representations were adapted to the DR classification task, reducing the need for training models from scratch and improving the learning process when using a limited amount of medical imaging data.

To adapt each pretrained architecture for the proposed classification task, the original classification layer was replaced with a task-specific classification head consisting of five output neurons corresponding to the five DR severity grades: Class 0: No DR, Class 1: Mild DR, Class 2: Moderate DR, Class 3: Severe DR, and Class 4: Proliferative DR. The output of each model was passed through a SoftMax activation function to generate a probability distribution across the five classes, where the class with the highest probability score was selected as the final predicted DR grade.

During fine-tuning, all pretrained model parameters, including the backbone and the task-specific classification head, were optimized using the training data to adapt the learned feature representations to retinal fundus images. The validation set was used to monitor model performance and select the best-performing model checkpoint during training. The same fine-tuning strategy was applied across all evaluated architectures to provide a consistent experimental setting for comparing CNN-based and transformer-based models.

4. Experimental Setup and Evaluation Protocols

4.1. Implementation Environment

The proposed framework was implemented in Python using the PyTorch deep learning framework. Torchvision was employed for image preprocessing and data augmentation, while pretrained CNN and transformer models were loaded using the TIMM library. Model evaluation and visualization were performed using Scikit-learn and Matplotlib, respectively. Additional libraries, including Pillow and TQDM, were used for image handling and monitoring the training process. All experiments were conducted using Google Colab Pro+ with a Python 3 runtime and NVIDIA A100 GPU acceleration. To ensure consistency throughout the experiments, all models were trained and evaluated under the same hardware environment and software configuration. Furthermore, five independent experiments were performed using different random seeds, while the random seeds for Python, NumPy, and PyTorch were fixed within each run to ensure reproducibility.

4.2. Hyperparameter Configuration and Training Setup

All models were trained using the same experimental configuration to ensure a consistent comparison between the evaluated architectures. The models were initialized with ImageNet pretrained weights and fully fine-tuned using the training set, while the validation set was used to monitor model performance, guide early stopping, and select the best-performing checkpoint during training. Several hyperparameters were considered during model optimization, including the learning rate, batch size, optimizer, and number of training epochs. The AdamW optimizer was employed with an initial learning rate of 1×10⁻⁴, a weight decay of 1×10⁻⁴, and a batch size of 32 for all evaluated models. All models were trained for a maximum of 20 epochs. Early stopping with a patience of six epochs was employed based on the validation loss to reduce overfitting and prevent unnecessary training. The selected training configuration was then applied consistently across all evaluated architectures during model training and evaluation. To address the variability introduced by random initialization and stochastic training, each model was independently trained and evaluated five times using different random seeds. The same training configuration and data splitting were maintained across all runs to ensure a fair comparison. Classification results were aggregated across the five runs and reported as mean ± standard deviation (std), providing a more reliable assessment of model performance and stability than a single experimental run. The resulting run-level performance was also used for the statistical analysis of differences among the evaluated models.

To mitigate class imbalance, Weighted Cross-Entropy Loss together with a WeightedRandomSampler was adopted during training. The final hyperparameter configuration used in this study is summarized in Table 2.

4.3. Evaluation Metrics

To comprehensively evaluate the performance of the selected CNN-based and transformer-based models for multi-class diabetic retinopathy grading, two categories of evaluation criteria were considered: classification performance and computational efficiency.

For classification performance, all models were evaluated using accuracy, precision, recall, F1-score, and the area under the receiver operating characteristic curve (AUC-ROC). Since the APTOS dataset exhibits an imbalanced distribution across the five DR severity classes, both macro-average and weighted-average precision, recall, F1-score, and AUC were reported. For multiclass AUC, a one-vs-rest (OvR) strategy was employed, where each DR severity label was evaluated against the remaining labels. The resulting class-specific AUC values were then summarized using macro-average and weighted-average. Macro-average metrics assign equal importance to each class regardless of its size, providing a better assessment of the model’s performance on minority classes. In contrast, weighted-average metrics account for the number of samples in each class, providing an overall performance measure that reflects the class distribution. These complementary metrics offer a more comprehensive evaluation of model performance under class imbalance.

Since diabetic retinopathy grading is an ordinal classification problem, Quadratic Weighted Kappa (QWK) [41] was additionally employed to quantify the agreement between the predicted and true DR severity grades while accounting for the severity of misclassification. In addition, QWK is less affected by class imbalance than conventional classification metrics, making it particularly suitable for evaluating the imbalanced APTOS2019 dataset. Unlike standard classification metrics, QWK assigns larger penalties to predictions that differ by multiple severity grades than to those differing by only one grade. Consequently, it provides a more clinically meaningful evaluation of model performance for ordinal diabetic retinopathy grading.

All classification metrics were computed from the confusion matrix, which consists of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The definitions and mathematical formulations of these evaluation metrics are summarized in Table 3. For all classification metrics, values closer to one indicate better predictive performance, whereas lower loss values indicate better model convergence. The reported overall classification results represent the mean and standard deviation obtained across the five independent runs. To further analyze the learning behavior of each architecture, the weighted cross-entropy loss was employed as the optimization objective during training, while training and validation loss curves were monitored to assess model convergence. The model corresponding to the lowest validation loss was selected as the final model for testing.

In addition to prediction performance, the computational efficiency of each architecture was also investigated. The number of trainable parameters was reported as a measure of model complexity, while inference time was measured to assess the time required to generate predictions for unseen retinal fundus images. These complementary measures provide practical insight into the trade-off between predictive performance and computational cost, which is particularly important when selecting models for real-world diabetic retinopathy screening systems.

5. Experimental Results and Comparative Analysis

This section presents the experimental results obtained from the six pretrained deep learning models for five-class diabetic retinopathy grading. The performance of the CNN-based and transformer-based architectures is evaluated using the metrics described in Section 4.3 under the same experimental configuration to ensure a fair comparison. The analysis focuses on three aspects: overall classification performance, learning behavior during training, and computational efficiency. Finally, the strengths and limitations of each architecture are discussed to provide practical insights into selecting appropriate deep learning models for automated DR screening.

5.1. Overall Classification Performance

The overall classification performance of the evaluated CNN-based and transformer-based models on the independent test set using accuracy, precision, recall, F1-score, AUC, and QWK is reported in Table 4 and illustrated in Figure 3. To provide a comprehensive evaluation under the imbalanced class distribution of the APTOS 2019 dataset, both macro-average and weighted-average performance metrics are reported. Macro-average metrics assign equal importance to all diabetic retinopathy severity classes, regardless of sample size, making them more suitable for evaluating recognition of minority classes. In contrast, weighted-average metrics account for the class distribution by assigning greater importance to majority classes, thereby reflecting the overall predictive performance on the complete dataset.

Overall, the transformer-based models outperformed the CNN-based models across most evaluation metrics. Among the CNN architectures, EfficientNet-B0 achieved the strongest performance, with an accuracy with an accuracy of 0.791, QWK of 0.850, macro F1-score of 0.604, and weighted F1-score of 0.793. This indicates that EfficientNet-B0 was the most effective CNN baseline, outperforming both ResNet50 and MobileNetV2. ResNet50 showed the weakest performance, particularly in macro F1-score (0.479), suggesting limited ability to correctly classify minority DR severity grades.

Among the transformer-based models, Swin-Tiny achieved the highest accuracy (0.823), QWK (0.898), macro precision (0.667), macro F1-score (0.644), weighted recall (0.823), and weighted F1-score (0.821), demonstrating the strongest overall classification performance. The highest QWK value further indicates the strongest agreement between the predicted and true DR severity grades while accounting for the ordinal nature of the classification task. Swin Transformer achieved the highest macro recall (0.640), weighted precision (0.832), and weighted AUC (0.966), while obtaining a QWK (0.897) comparable to that of Swin-Tiny. ViT also demonstrated competitive performance, particularly in terms of weighted AUC (0.957) and weighted F1-score (0.809). The relatively small differences between ViT, Swin-Tiny, and Swin Transformer indicate that transformer-based architectures performed comparably, with Swin-based architectures showing the strongest overall balance across metrics. The small standard deviation values across the five separate runs indicate generally stable performance across different random seeds. Particularly, QWK showed low variability among the transformer-based models, with standard deviations ranging from 0.010 to 0.015. Although some variation was observed across individual classification metrics, the overall performance patterns were still consistent across the repeated runs.

A noticeable difference is observed between the macro-average and weighted-average metrics across all evaluated models. In general, the weighted-average values were consistently higher than their corresponding macro-average values. This behavior is expected due to the imbalanced distribution of the APTOS 2019 dataset, where the majority classes contribute more heavily to the weighted evaluation metrics. For example, ResNet50 achieved a weighted F1-score of 0.672, whereas its macro F1-score decreased to 0.479, indicating substantially weaker performance on the minority DR severity grades. Similar trends were observed for EfficientNet-B0 and MobileNetV2. In comparison, the transformer-based models exhibited smaller gaps between the macro and weighted metrics, suggesting more balanced classification performance across both majority and minority classes.

The AUC and QWK results further support these findings. All transformer-based models achieved weighted AUC values above 0.95, substantially higher than those of the CNN-based architectures. Similarly, the transformer models obtained the highest QWK values, indicating stronger agreement between the predicted and true DR severity grades while accounting for the ordinal relationship between disease stages. The consistently higher AUC and QWK values suggest that transformer models possess stronger discriminative power in separating DR severity grades across a wide range of decision thresholds. Furthermore, the relatively small gap between macro and weighted AUC values indicate that transformer-based architectures maintain reliable class discrimination even for minority classes, whereas CNN models exhibit greater sensitivity to class imbalance.

Finally, to determine whether the observed performance differences were statistically meaningful, statistical analysis was conducted across the five independent runs using QWK, macro F1-score, and weighted F1-score. The Friedman test [42] was first applied to assess overall differences among the six models, followed by pairwise Wilcoxon signed-rank tests with Holm correction [42] to account for multiple comparisons. The Friedman analysis identified significant overall differences among the evaluated architectures for all three metrics (Holm-adjusted p < 0.05), confirming that model architecture had a measurable effect on classification performance. Post-hoc analysis showed that the differences among individual leading models were not statistically significant after correction, indicating comparable performance among the strongest architectures despite differences in their mean values. Swin-Tiny nevertheless achieved the highest mean QWK, macro F1-score, and weighted F1-score across the five runs.

5.2. Per-Class Performance Analysis

To further examine the classification behavior of the best-performing CNN-based model (EfficientNet-B0) and transformer-based model (Swin-Tiny), a per-class analysis was conducted using precision, recall, and F1-score for each diabetic retinopathy (DR) severity grade. The per-class performance metrics reported in Table 5 correspond to the mean and standard deviation obtained across five independent runs. To provide a qualitative visualization of the classification behavior, the corresponding confusion matrices from a representative test run are presented in Figure 4.

Overall, Swin-Tiny demonstrated stronger per-class performance than EfficientNet-B0 across nearly all disease severity grades and evaluation metrics. The main exception was Mild DR recall, where EfficientNet-B0 achieved a higher value (0.553 vs. 0.506), while performance for the No DR class was largely comparable between the two models. Both models achieved excellent performance in identifying the majority class “No DR”, with F1-scores exceeding 0.97, indicating reliable discrimination of healthy retinal images. Performance was lower for the Mild and Moderate classes, reflecting the greater difficulty of distinguishing intermediate DR severity levels. For Mild DR, Swin-Tiny achieved higher precision (0.544) and F1-score (0.517), whereas EfficientNet-B0 achieved higher recall (0.553 vs. 0.506), indicating that it correctly identified a larger proportion of actual Mild DR cases and missed fewer cases at this severity level. For Moderate DR, Swin-Tiny showed a clearer advantage, achieving higher precision (0.710), recall (0.793), and F1-score (0.748) than EfficientNet-B0.

The Severe DR class remained the most challenging for both architectures. Swin-Tiny achieved slightly higher precision (0.368), recall (0.400), and F1-score (0.380) than EfficientNet-B0. The variability observed across the independent runs further indicates that performance estimates for this less represented class were less stable than those for the No DR class. Swin-Tiny also achieved higher mean performance for Proliferative DR, with precision of 0.740 compared with 0.647 for EfficientNet-B0, recall of 0.503 compared with 0.430, and F1-score of 0.598 compared with 0.515. However, the relatively large standard deviations for several Proliferative DR metrics indicate notable variation across the five runs. Overall, the standard deviations reported in Table 5 show greater variability for several of the less represented DR severity grades compared with the No DR class. This finding highlights the influence of class representation on the stability of per-class performance estimates and provides additional context for the observed differences between the two architectures.

The representative confusion matrices (see Figure 4) further illustrate the class-specific classification patterns. A substantial proportion of the classification errors involved neighboring DR severity grades, consistent with the gradual progression of diabetic retinopathy and the visual similarity between adjacent stages. Both models correctly identified the majority of No DR cases. Mild and Severe DR were frequently misclassified as Moderate, with Moderate also accounting for several misclassifications of Proliferative DR. Swin-Tiny correctly classified more Moderate (70 vs. 58) and Proliferative DR cases (19 vs. 14) than EfficientNet-B0 in the representative run, whereas EfficientNet-B0 correctly identified slightly more Mild cases (17 vs. 15). These patterns are generally consistent with the mean per-class results reported across the five independent runs in Table 5.

5.3. Learning Behavior and Convergence

To further analyze the training behavior of the evaluated models, the training and validation loss curves were examined throughout the training process. Figure 5 presents the learning curves from a representative experimental run (seed 123). As shown in Figure 5, all models exhibited a rapid decrease in training loss during the first few epochs, indicating successful optimization. The CNN-based models showed relatively smooth validation curves, whereas the transformer-based models exhibited larger fluctuations in validation loss after the initial training stage, indicating greater sensitivity to continued optimization. As training progressed, the gap between the training and validation losses gradually increased, particularly for the transformer-based models, suggesting increased overfitting during later epochs.

Among the CNN-based models, EfficientNet-B0 and MobileNetV2 showed a relatively rapid reduction in training loss, while ResNet50 exhibited a more gradual convergence pattern. The transformer-based models also converged rapidly, reaching very low training losses within the first few epochs. However, unlike the CNN-based models, their validation losses generally reached their minimum early and subsequently fluctuated or increased despite continued reductions in training loss. This pattern indicates that additional training increasingly optimized the training data without corresponding improvements in validation loss, indicating varying degrees of overfitting across transformer-based architectures.

To further characterize convergence behavior, Table 6 reports the minimum validation loss and its corresponding epoch for the representative run. The transformer-based models achieved their minimum validation losses much earlier than the CNN-based models. Specifically, ViT and Swin Transformer reached their minimum validation losses at epoch 3, while Swin-Tiny reached its minimum at epoch 5. In comparison, ResNet50, EfficientNet-B0, and MobileNetV2 reached their minimum validation losses at epochs 16, 9, and 16, respectively. These results indicate that transformer-based models learn discriminative features rapidly but begin to overfit earlier, while CNN-based models require more training epochs to reach their best generalization performance. The observed differences in convergence behavior also reflect the distinct characteristics of the evaluated CNN- and transformer-based architectures, which may respond in a different way to the common training configuration and reach their optimal validation performance at different stages of training. Early stopping was triggered for ViT, Swin-Tiny, and Swin Transformer at epochs 19, 16, and 19, respectively, whereas all CNN-based models completed the maximum 20 epochs.

5.4. Computational Efficiency Analysis

To compare the computational efficiency of the evaluated models, the number of trainable parameters and the inference latency per image were analyzed. Inference latency was measured using a batch size of one under the same hardware environment for all models, following warm-up iterations and repeated measurements. The latency results are reported as mean ± standard deviation across five independent runs. The results are summarized in Table 7.

Among the CNN-based models, MobileNetV2 exhibited the smallest model size, requiring only 2.23 million trainable parameters while achieving the lowest inference latency within the CNN family (7.023 ms/image). EfficientNet-B0 also demonstrated a compact model size, requiring 4.01 M parameters but exhibited a higher inference latency (9.835 ms/image) than both MobileNetV2 and ResNet50. ResNet50 despite requiring a substantially larger model size (23.52 M parameters), achieved an intermediate inference latency of (7.730 ms/image).

The transformer-based models required considerably more parameters than the CNN models. ViT and Swin Transformer contained approximately 86 M parameters, while Swin-Tiny reduced the model size to 27.52 M parameters. Interestingly, ViT achieved the lowest inference latency among all evaluated models despite having one of the largest model sizes. This observation suggests that inference latency is not solely determined by the number of trainable parameters but is also influenced by the underlying network architecture and its execution efficiency on the target hardware.

Overall, the results demonstrate that model size and inference latency are not directly proportional. While lightweight architectures generally reduce the number of trainable parameters, they do not necessarily achieve the lowest inference latency. These findings highlight the importance of considering predictive performance, model complexity, and inference efficiency when selecting models for automated diabetic retinopathy screening.

5.5. Comparison with Related Studies

To provide additional context for the obtained results, the best-performing model in this study was compared with selected studies that also addressed five-class DR grading using the APTOS 2019 dataset. The selected studies employed pretrained or hybrid DL approaches with additional strategies, including Siamese learning [43], patch-based feature extraction [44], and hybrid CNN architectures [45]. The reported accuracy, F1-score, and QWK values are summarized in Table 8, where available. However, several of these studies report a limited set of evaluation metrics, with accuracy frequently serving as the primary performance measure, while F1-score and QWK are not consistently reported. For an imbalanced multi-class dataset such as APTOS 2019, accuracy alone may not fully reflect performance across DR severity grades, particularly the less represented classes. Therefore, Table 8 presents accuracy, F1-score, and QWK for comparison where these measures are available.

As shown in Table 8, the performance of Swin-Tiny is competitive with the selected related studies, although some approaches achieved higher accuracy. These differences may be attributed to the use of additional task-specific strategies, including Siamese learning, patch-based processing, and hybrid architectures, as well as differences in the objectives and experimental designs of the studies. The comparison is also constrained by the limited availability of complementary evaluation metrics in some studies, making it difficult to assess performance beyond overall accuracy. In comparison, the present study considers multiple overall and class-sensitive metrics, including F1-score and QWK, together with per-class performance, computational efficiency, and performance stability across five independent runs. This broader evaluation provides additional insight into model behavior beyond a single aggregate performance measure.

5.6. Discussion and Limitations

The experimental results consistently demonstrate that transformer-based architectures outperform CNN-based models for five-class diabetic retinopathy grading. Among the evaluated architectures, Swin-Tiny achieved the highest accuracy, QWK, macro F1-score, and weighted F1-score, while Swin Transformer achieved the highest macro recall, weighted precision, and AUC. ViT also remained competitive across the evaluated metrics. Among the CNN models, EfficientNet-B0 provided the strongest overall performance. The per-class analysis further supported these findings, with Swin-Tiny achieving higher mean performance than EfficientNet-B0 across most DR severity grades and evaluation metrics. However, greater variability was observed for the less represented classes, particularly Severe DR, highlighting the greater difficulty of distinguishing these disease stages. These improvements can be attributed to the self-attention mechanism, which captures both local lesion characteristics and long-range contextual relationships within retinal fundus images.

The computational analysis further demonstrates that model size and inference latency are not directly proportional. MobileNetV2 had the smallest parameter count and the lowest inference latency among the CNN models, while ViT achieved the lowest overall inference latency despite its substantially larger model size. In contrast, Swin Transformer had the largest computational requirements, whereas Swin-Tiny provided a more favorable balance between predictive performance and model complexity, achieving the strongest overall classification performance with substantially fewer parameters than the larger Swin Transformer. These findings suggest that the choice of architecture should depend on the intended deployment scenario. When the primary objective is classification performance, transformer-based models provide a strong choice, with Swin-Tiny achieving the strongest overall performance in this study. However, when model size and computational resources are critical, lightweight CNN models such as MobileNetV2 provide an effective alternative with fewer parameters and competitive classification performance. The inference results further indicate that latency is architecture-dependent and does not necessarily increase with model size, as ViT achieved the lowest inference latency among the evaluated models.

Despite the comprehensive analysis we presented, this study has some limitations. First, the experiments were conducted using a single benchmark dataset, which limits conclusions regarding generalizability across different populations, imaging devices, and acquisition conditions. Second, the imbalanced distribution of APTOS 2019 resulted in relatively few test samples for some DR severity grades, increasing the variability of class-specific performance estimates. Several solutions can be explored for addressing class imbalance and improving the recognition of underrepresented DR severity grades as future direction. Finally, the study employed a common training configuration across architectures rather than architecture-specific hyperparameter optimization, as the primary objective was to provide a controlled architectural comparison. Finally, the study employed a common training configuration across architectures rather than architecture-specific hyperparameter optimization, as the primary objective was to provide a controlled architectural comparison. Future evaluation across additional retinal datasets and optimized architecture-specific configurations would further establish the generalizability of the observed findings.

6. Conclusion

This study presented a comparative analysis of six pretrained CNN-based and transformer-based models for five-class diabetic retinopathy grading using retinal fundus images. All models were evaluated under the same preprocessing, training, and evaluation settings to ensure a fair comparison. The results demonstrated that transformer-based architectures consistently outperformed CNN-based models in overall classification performance. Among the evaluated models, Swin-Tiny achieved the strongest overall classification performance, while EfficientNet-B0 provided the strongest CNN baseline. In addition, the study highlighted the relationship between predictive performance and computational efficiency, demonstrating that model size and inference latency are not necessarily proportional and that architecture selection should consider both predictive and computational requirements.

The comprehensive evaluation across overall and per-class metrics further demonstrated the challenges associated with distinguishing less represented DR severity grades, particularly Severe DR. Future work will extend the evaluation to additional retinal datasets to evaluate the generalizability of these findings and investigate more recent architectures and architecture-specific optimization. Data augmentation and advanced imbalance-aware learning strategies can also be explored as future direction to improve the recognition of underrepresented DR severity grades. Addressing these aspects can further improve and support the development of reliable automated diabetic retinopathy screening systems.