Dual-Stream Contrastive Latent Learning GAN for Brain Image Synthesis and Tumor Classification

1. Introduction

Brain tumors (BTs) remain a significant public health concern, ranking as the 10th leading cause of death in the United States only [1]. Research indicates that brain tumors are significantly impacting patients' lives through physical, cognitive, and psychological impairments [2]. BTs can be benign or malignant, with benign tumors being slow-growing and localized, while malignant tumors are highly aggressive with metastasis [3]. Brain tumors (BTs) encompass a diverse range of types with varying degrees of aggressiveness, including glioblastoma, the most aggressive form; meningioma, which is often benign; and pituitary adenomas [4,5].The World Health Organization (WHO) classifies brain tumors on a scale from Grade I to IV based on their degree of spread, biological behavior, and prognosis [6].Targeted efforts are essential to enhance early detection methods and deepen the understanding of brain tumor progression, as early diagnosis expands treatment options and improves survival rates [7].

Magnetic Resonance Imaging (MRI) is a highly effective imaging modality for detecting and characterizing various aspects of brain tumors. It offers superior soft tissue contrast while minimizing patient exposure to ionizing radiation [8]. However, BT diagnosis through MRI scans is highly time-intensive and heavily dependent on the radiologist’s expertise. Accurately labeling the scans without misclassification presents challenges in both time and precision. To address this issue, various computer-aided solutions have been developed to support automated decision-making systems [9,10]. Artificial Intelligence (AI) has emerged as a promising tool for the early detection of BTs and is being leveraged to enhance diagnosis by analyzing MRI scans. Various deep learning architectures have been documented in the literature for the detection and classification of BTs using MRI scans [11,12,13,14,15]. In [16], an automated brain tumor classification method is proposed using an enhanced deep learning approach with DenseNet121. Transfer learning is applied, and hyper-parameter tuning optimizes the CNN model. MRI images of three distinct brain tumor types were analyzed using the DenseNet169 model for feature extraction. The extracted features were then fed into three multi-class machine learning classifiers, Random Forest (RF), Support Vector Machine (SVM), and XGBoost to enhance performance [17]. Elsewhere [18], the effectiveness of deep transfer learning has been assessed using ResNet152, VGG19, DenseNet169, and MobileNetv3 models. Similarly, [19] presented the CapsNet model for multiclass classification of neuro- degenerative diseases using a modified DenseNet-169 framework coupled with the Enhanced DeepLab V3+ model. Further, a brief summary of seven deep learning models for BTs detection including VGG-16, VGG-19, ResNet50, Inception, ResNetV2, InceptionV3, Xception, and DenseNet201 and five traditional classifiers considering SVMs, RF, Decision Trees, AdaBoost, and Gradient Boosting are documented in the literature [20]. Although these advanced architectures have achieved promising results, they largely overlook the inherent limitations of imaging datasets, particularly significant class imbalances. The challenge of acquiring large, diverse datasets encompassing patients at various disease stages constrains the full potential of deep learning networks.

To overcome the challenges posed by limited medical imaging datasets, image augmentation techniques help in enriching data diversity, mitigating class imbalance, and improve model generalization. Adversarial learning frameworks have been employed to optimize the interaction between the generator and discriminator for creating high quality synthetic medical images. In this regard, an automatic data GAN was used to learn the available annotated MRI samples of BRATS15 Challenge dataset [21] and inception, resnetv2, inceptionv3, transfer learning, and BRAIN-TUMOR-net were published in [22] for MRI images of glioma tumor, meningioma and pituitary BTs. [23] reported progressive GANs (PGGANs), in which multistage generative training was to generate BT images that were challenging for conventional GANs and further detailed reviews are available at [24,25,26,27,28] concerning the standard used augmentation methods and fusion deep learning models including U- Net.

Transformers and auto-encoder architectures are imperative for MRI image augmentations due to their ability to overcome key limitations of deep learning CNN models including long-range dependencies, and global contextual awareness. A cross- transformer was published to include self-care model keys, queries, and values for the classification of BTs in MRI images and compared the results with InceptionResNetV2, InceptionV3, DenseNet121, Xception, ResNet50V2, VGG19, and EfficientNetB7 networks [29]. In [30], the possibilities of vision transformers (ViTs) in this research as a viable alternative to CNNs for brain magnetic resonance imaging was pitched by incorporating self-attention mechanisms to establish relationships between image patches for a comprehensive understanding. Likewise [31] highlighted the relevance of dual vision transformer model (DSUNET) in providing reliable and efficient differentiation between brain tumors and other brain regions by leveraging MRI BRATS 2020 dataset. Besides swin transformer was introduced into UNet++ network, in which local features of BTs were extracted by convolutional layer in UNet++ and global resolutions were captured via shift window operation of swin transformer [32]. Researchers deployed a shrinking linear time vision transformer (SL(t)-ViT) network for enhanced classification across multiple datasets of BTs [33]. Hybrid architectures combined with cross-fusion allows parallel systems to be merged between branches, resulting in reliable prediction of various tumors and similar studies are available in the literature [34,35]. In these researches, either a two-branch parallel model that integrates the transformer module (TM) with the self-attention mechanism was used to classify brain tumors in MR images or hybrid shifted windows multi-head modules were deployed. To address the challenge of explainability in adversarial transformer models, graph attention network (GAT)-based transformer schemes were published [36]. Diffusion models on the other side learn the actual data distribution through likelihood-based training, making them more interpretable as compared to adversarial networks. Therefore, researchers comprehensively evaluated four GANs (progressive GAN, StyleGAN 1–3) and a diffusion model using two segmentation networks, U-Net and a Swin transformer for the task of brain tumor segmentation [37]. To address the issue of inter-class and intra-class problems, a gated global-local attention (GGLA) mechanism was evolved. The gating mechanism within the GGLA dynamically balances the contributions of global and local information, enabling the model to adaptively focus on the most relevant features for accurate classification. Additionally, an enhanced super-resolution generative adversarial network (ESRGAN) was coupled to generate images that balance the MRI image data [38]. The cutting-edge techniques in multiple stages within deep image recognition generative adversarial network (DIR-GAN) reported robustness of brain tumor detection and classification in MRI scans [39]. Similar work was published [40] where HARA-GAN was proposed by integrating residual U-net with Hybrid attention and Relative average discriminator, to mitigate noise caused by low under sampling rates. The results indicated that HARA-GAN outperforms DAGAN, RefineGAN and RSCA-GAN methods based on error maps and quantitative evaluation metrics in terms of both image quality and consistencies on MRI brain dataset. To improvise on capturing long-range dependencies and spatial variations [41] reported residual attention U-shaped network (RAUNet) for brain tumor segmentation which leverages the robust feature extraction capabilities of U-Net and the global context awareness provided by transformers to improve segmentation accuracy and [42] introduced hybrid federated adversarial MRI enhancement (FAME) by integrating advanced GAN architectures such as multi-scale convolutions, attention mechanisms, and GNNs. Furthermore, self-generating few-shot brain tumor segmentation models like CDSG-SAM are also part of the literature [43] where dynamic fuzzy support mask decoder module DFSMD were used within the to enhance the classification accuracy of BTs.

Despite the cited advancements in deep learning for brain tumor classification, a critical research gap persists in the generation and diversity of synthetic MRI images. Existing studies largely focus on the aspect of classification using CNNs, transformers, and hybrid architectures leading to sub- optimal generalization in terms of high quality and diverse tumor representations. The resolve of this study is to propose a dual- stream generator architecture in GANs, which incorporate contrastive learning and multi-stream feature fusion for enhancing the diversity and realism of synthetic MRI images. The present research is focus on leveraging dual-stream generator frameworks to ensure a balanced image synthesis.

The dual-stream generator and three discriminators were designed within an adversarial framework to enhance competitive learning. Traditional GAN-based augmentation suffers from mode collapse due to separate generators, a limitation addressed by our dual-stream architecture. The integration of a Contrastive Latent Projection (CLP) module preserves semantic consistency in augmented images, while contrastive learning ensures diverse yet coherent feature representations. We propose a novel objective function for DSCLPGAN, optimized for discriminative feature learning. Performance evaluation using assessment metrics demonstrates superior diversity and generalization in the generated images.

The main contributions of the paper are summarized as under;

• The proposed dual-stream augmentation framework utilizes a single generator with dual perturbations to enhance realism and diversity by effectively capturing both local and global variations in medical images.
• A rigorous mathematical formulation is developed, incorporating a CLP module to preserve semantic integrity and enhance model generalization in image augmentation tasks.
• A three-discriminator architecture is introduced, operating in parallel to assess image quality, diversity, and frequency consistency. Additionally, D1 performs classification, eliminating the need for a separate brain tumor (BT) classifier network.

2. Materials & Methods

A dual-stream single-generator with CLP is an efficient, robust, and medically meaningful way to augment MRI images while ensuring high-quality and diversity in synthetic images.

2.1. Dual Stream Generator of Our Proposed Model (DSCLPGAN)

Instead of using two separate generators, a single generator handles both augmentation streams. A single generator with dual perturbations avoids image similarity issues generated by two generators by maintaining diversity within one network. In our model, one stream applies local variations and other handles the global variations. An encoder extracts the latent representation from the input MRI scan. The latent space is split into two parallel streams with one to emphasize localized variations and the second to emphasize global changes as illustrated in Figure 1.

As illustrated in Figure 1, the dual stream generator uses CNN/ transformer encoder for the extraction of global features and patch wise CNN for learning the local features within an image. The CLP module applies contrastive learning to project real and fake features into a contrastive latent space to ensure that the augmented images remain semantically close to the original. It applies contrastive learning using a similarity-based loss function as given in Eq. 1.

$$L_{CLP}=-log{\displaystyle \frac{exp\left(sim\right(Z_1,Z_2)/\tau }{\raisebox{1ex}{${\sum }_{j}exp\left(sim\right(Z_1,Z_j)$}\!\left/ \!\raisebox{-1ex}{$\tau $}\right.}}$$

(1)

where sim (.) is cosine similarity, $\tau$ is temperature parameter controlling how strongly differences are penalized.

The do while pseudocode representation of CLP with input MRI image (X), encoder and projection head with perturbations is given in Figure 2 along with the description of involved parameters.

The dual- stream decoder reconstructs the image using the global Reconstruction Path, local refinement path and skip connections as illustrated in Figure 1. The global path exploited ResNet/ transformer decoder and local path involves the Deconv and CNN blocks. The final output includes high quality MRI images with structure & texture accuracy.

2.2. Complete Architecture of Our Proposed Model (DSCLPGAN)

In this paper, dual stream generator and triple discriminators are used for adversarial training of diverse synthetic images as presented Figure 3. The first discriminator D₁ (global discriminator) is for the classification of BTs. The second discriminator D₂ (local discriminator) is designed to analyze small patches within an image including specific tumor regions. The D₃ converts images to frequency space for global consistency. A detailed schematic including the details of CNN and layers presented in Figure 3.

2.3. Mathematical Formulation

The dual stream generator (G) consists of a spatial stream G_S (z) to generate spatial features. The latent contrastive stream G_L (z) enforces latent space similarity via CLP. The generator learns to map latent noise to a realistic MRI image as given in Eq. 2.

$$x^{\text{'}}=G\left(\right)G_s\left(z\right)+G_l\left(z\right)$$

(2)

where $G_s\left(z\right)$ generates high resolution spatial details and $G_l\left(z\right)$ enforces feature consistency.

D₁ learns to classify both the real and generated MRI images into medical conditions given in Eqs. 3- 4.

p (y_r∣x_r)= D₁(x_r)

(3)

p (y’∣x’)= D₁(G(z))

(4)

where y_r is the ground truth class.

The classification loss is governed by Eq. 5.

$${\mathrm{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}=-{\mathrm{E}}_{~{\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{x}\right)}}{\sum }_{i=1}^c{y^i}_{r}log{D}_1^i\left(x_r\right)$$

(5)

where c represents the number of classes and ${y^i}_r$ is ground truth class indicator.

The local D₂ discriminator evaluates patches from real and fake MRI images and the adversarial loss is given by Eq. 6.

$${\mathrm{L}}_{\mathrm{D}2}={-\mathrm{E}}_{x_r~{\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{x}\right)}}\left[\log D_2\left(x_r\right)\right]+{\mathrm{E}}_{\mathrm{z}~{\mathrm{P}}_{\mathrm{z}\left(\mathrm{z}\right)}}\left[\mathrm{l}\mathrm{o}\mathrm{g}\right({1-D}_2\left(G\left(z\right)\right)\left)\right]$$

(6)

MRI scans are evaluated in the Fourier domain by the D₃ to ensure frequency consistency and the Fourier adversarial loss is computed using the expression in Eq. 7.

$${\mathrm{L}}_{\mathrm{D}3}={-\mathrm{E}}_{x_r~{\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{x}\right)}}\left[\log D_3\left(F(x_r\right)\right)]-{\mathrm{E}}_{\mathrm{z}~{\mathrm{P}}_{\mathrm{z}}}[\mathrm{l}\mathrm{o}\mathrm{g}\left({1-D}_3\right(F\left(G(z\right)\left)\right)\left)\right]$$

(7)

The loss function is the aggregate loss total of all three discriminators, CLP loss and the hyper-parameters balancing each loss term and is given by Eq. 8.

$$L_D={\lambda }_{cls}{L}_{D1}+{{\lambda }_{adv}L}_{D2}+{\lambda }_{freq}{L}_{D3}+{\lambda }_{clp}{L}_{clp}$$

(8)

The objective of training is to minimize the generator loss and to maximize the discriminators loss as indicated in Eq. 9.

$${{\mathrm{G}}^{\mathrm{*}}=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{G}}{{max}_{D}L}_G$$

(9)

where $L_G={\lambda }_{cls}{L}_{D1}+{{\lambda }_{adv}L}_{D2}+{\lambda }_{freq}{L}_{D3}+{\lambda }_{clp}{L}_{clp}$

The ultimate objective is to produce MRI images that appear realistic at the global level with accurate and realistic local details and further maintain semantic consistency with the real image.

3. Results & Analysis

This brain tumor dataset (Figure Share 2024) contains 3064 T1-weighted contrast- enhanced images with three types of brain tumor: meningioma (708 slices), glioma (1426 slices), and pituitary tumor (930 slices). The 5-fold cross-validation indices are also provided. Prior to augmentation and training phase, the MRI scans were subjected to a series of pre- processing stages for dataset standardization. Using our proposed DSCLPGAN architecture, the image data in each class is amplified by a factor 75 to increase the diversity and to validate the generalization ability of our proposed network. We test our GAN generated image data underlying objective to underscore the impact and extent of image augmentation in improvising diversity and generalization. All the models are implemented using Pytorch™. The four possible output labels are: 0- no tumor, 1- meningioma tumor, 2- glioma tumor, and 3- pituitary tumor. A latent dimension of 100 and batch size of 64 is selected for 50 epochs with learning rate of 0.0001. Adam optimizer is used for optimization with parameters β₁ vale of 0.5 and β₂ of 0.999. settings having The L₃ Regularization is applied to the weights of the network to penalize large weights. The training process is exhibited in Figure 4 in which the dual-stream generator is trained by simultaneously optimizing two streams: one generating realistic augmented images and the other enforcing latent space consistency. An adversarial loss helps the generator produce high-quality outputs, while a reconstruction loss maintains fidelity to the original data.

Few sample images generated by our model using the original MRI images are presented in Figure 5.

The Structural Similarity Index Measure (SSIM) loss shows a remarkable improvement from 0.65 to 0.99 over 20 epochs as indicated in Figure 6. This demonstrates the superior structural quality of our Dual-Stream Generator with CLP, hence ensuring that augmented MRI images remain highly realistic and clinically valuable. Such a high SSIM score reinforces the robustness of our method in generating high-quality medical images.

Similarly, the Fréchet Inception Distance (FID) in Figure 7 demonstrates a remarkable decline from 45 to 12 over 20 epochs. This signifies that our model progressively refines image quality, making synthetic data perceptually closer to real MRI scans. Such a low FID score highlights the effectiveness of our method in preserving critical medical features while ensuring diverse and realistic augmentation.

The Contrastive Loss exhibits a value 0.97 after 20 epochs as indicated in Figure 8. This reduction signifies improved feature consistency, ensuring that the generated images retain essential medical characteristics without losing diversity.

Augmentation Diversity Level refers to the degree of variation introduced in synthetic data to enhance model generalization while preserving essential structural features. At an augmentation diversity level of 5, our model achieves an outstanding classification accuracy of 99.76% as illustrated in Figure 9. , proving its unparalleled ability to generate diverse yet highly realistic MRI images. This exceptional performance is attributed to the CLP module in our model for ensuring meaningful variations.

Figure 10 suggests that the latent space distance shows an upward trend with the number of epochs which is reflecting the model’s ability to enhance feature distinction. The latent space distance ensures that the generated MRI images are not only diverse but also structurally coherent, preventing redundancy and mode collapse. Our model achieves PSNR of 34. 6 dB within 20 epochs as indicated in Figure 11, suggesting that our augmented images closely resemble the real MRI scans.

To highlight the effectiveness of the proposed DSCLPGAN, a comparative analysis is conducted against state-of-the-art GAN-based and transformer-based augmentation models and is illustrated in Table 1. The evaluation considers key performance metrics such as SSIM, PSNR and FID score. The models selected for comparison include BIGGAN [44], MAGE [45], TransGAN [46], SR TransGAN [47], CTGAN [48], StyleGANv2 [49], SFCGAN [50], VQ-GAN [51], and 3D Pix2Pix GAN [52]. An inter- comparison with such models having benchmark performance characteristics underscores the efficiency of DSCLPGAN in generating high quality and diverse medical images.

4. Conclusions

This study introduces a dual-stream GAN with CLP for MRI brain tumor image synthesis, ensuring structural fidelity and enhanced downstream performance. Experimental results demonstrate the model’s effectiveness across key metrics, including SSIM, FID, contrastive loss, and latent space consistency. The integration of triple discriminators refines image quality by evaluating realism from multiple perspectives while preserving latent space coherence. Notably, classification accuracy improves with increased augmentation diversity, highlighting the model’s superior generalization capabilities. These findings underscore its potential in data-limited medical imaging scenarios and its adaptability to broader clinical applications beyond MRI.