Dual Generators and Dynamically Fused Discriminators Adversarial Network to Create Synthetic Coronary Optical Coherence Tomography Images for Coronary Artery Disease Classification

1. Introduction

Coronary artery disease (CAD) is a serious health challenge that is affecting substantial individuals and imposes burdens on healthcare systems worldwide [1,2]. CAD is an artery narrowing and blockage, leading to severe complications like heart attacks and heart failure [3,4]. Collaborative efforts by researchers and professionals resulted in advancements in treating and managing this disease. Evolving CAD prevention and management procedures embraces modern AI driven architectures and cutting-edge diagnostics including the Optical Coherence Tomography (OCT) [5,6,7]. OCT provides high-resolution images of the coronary artery walls for detailed analysis of plaque composition, and identification of vulnerable plaques [8]. With OCT, different plaque components can be distinguished based on their optical properties, including fibrous tissue, lipid cores and calcium deposits [9]. Fibrous plaque appears as a homogeneous, signal-rich region on OCT whereas lipid-rich plaque indicate as a low-signal region on OCT. Fibrocalcific plaque presents a heterogeneous appearance with sharp borders in OCT scans. The OCT imaging datasets suffers from key challenges including size, diversity, accurate manual annotations and significant variability which hinders their full potential utilization in modern data driven architectures [10,11,12,13,14,15].

Artificial Intelligence has emerged as a potential candidate when it comes to detection and characterization of coronary atrial plaques and is therefore being exploited to support improve the diagnosis via analyzing OCT scans. Multifarious deep learning architectures were reported in the literature regarding the detection and classification of coronary plaques using OCT scans [16,17,18,19,20,21,22,23,24,25,26,27]. In [28], a fully automated, two-step deep learning approach for characterizing coronary calcified plaque in intravascular optical coherence tomography (IVOCT) images is reported where a 3D convolutional neural network (CNN) was used with a SegNet deep learning model. Similarly, an automated atherosclerotic plaque characterization method that used a hybrid learning approach and self attention based U- Net architecture were reported elsewhere [29,30] for the better classification performance of coronary atrial plaques than existing methods.

Vision Transformers (ViTs) have also gained traction for coronary plaque detection especially in cases where global feature relationships are essential. In this regard [31] explored the use of ViTs for coronary plaque detection where ViTs outperformed CNN-based models for large datasets. Instead of relying on lumen segmentation, the proposed method identifies the bifurcation image using a ViT-based classification model and then estimate bifurcation ostium points by a ViT-based landmark detection model. The proposed ViT-based model are 2.54% and 16.08% higher than that of traditional non-deep learning methods [32]. For better generalization characteristics, a transformer-based pyramid network called AFS-TPNet for robust, end-to-end segmentation of CCP from OCT images [33]. Researchers also used physics-informed deep network QOCT-Net to recover pixel-level optical attenuation coefficients directly from standard IVOCT B-scan images [34]. While these cutting-edge architectures delivered satisfactory results but from a clinical standpoint, acquiring large and diverse datasets with patients across different disease stages is stimulating, shadowing real potential of proposed deep learning architectures. To address this issue, data augmentation (DA) techniques were deployed to augment limited medical imaging training data that can be further fed to deep learning algorithms for better insights. Generative Adversarial Networks (GANs) have offered promising solutions by generating high-quality images that closely resemble real OCT scans, thereby aiding in dataset augmentation, image enhancement, and training deep learning classifiers. In [35], pseudo labeling, using model predictions as labels for unlabeled data, has been employed as a data augmentation technique. This method has demonstrated improvements in model performance by increasing the effective size of the training dataset. The StyleGAN2 and Cyclic GANs frameworks were used to generate high resolution synthetic patches for data augmentation for an improved data augmentation performance in the case of low data, across three different OCT datasets encompassing a range of scanning parameters [36,37]. To improvise generalization across different datasets, sparsity constrained GANs with baseline accuracy is available in the literature [38].

However, the recent works show that despite high structural similarity between the synthetic data and the real images, a considerable distortion is observed in frequency domain, therefore dual and triple discriminator architectures including Fourier acquisitive GAN (DDFA-GAN) to generate more realistic OCT images were proposed [39,40,41]. By applying multiple discriminators, the proposed models were jointly trained with the Fourier and spatial details of the images and results were are compared with popular GANs including deep convolutional GAN (DCGAN), Wasserstein GAN with gradient penalized (WGAN-GP), and least square GAN (LS-GAN). In [42] a multi-stage and multi-discriminatory generative adversarial network (MultiSDGAN) specifically for super-resolution and segmentation of OCT scans was proposed. This resulted in an improved performance by satisfying all the discriminators at multiple scales and including perceptual loss function. While the use of multiple discriminators in GANs attracted much attention but due to overloading of a single generator in the network, the desired generalization and diversity in the generated images is yet an open problem.

The resolve of this study was to perform data augmentation on an OCT dataset using a novel dual generator multiple fusion discriminator network for synthesizing high quality coronary images. The use of dual generators exploited in this paper helps to achieve better generalization and generate diverse of images of coronary arterial plaques. In our model, the two generators G1 and G2 receives the same conditional input y but they generate different variations of the OCT images as each generator acts as a regularizer for the other. In our model, one generator’s output can serve as a reference for improving the other and this mutual regularization enhances the ability of both generators to generalize better across different features and conditions. Additionally, we skipped the need of a separate classification architecture, rather uses a discriminator within our presented GAN architecture for classification of coronary atrial plaques into three classes. The objective functions of the generator and the five discriminators were set in competition against each other. We derive a novel objective function for DGDFGAN and optimize it through Whales algorithm during model training. This makes the model training more stable and improving the quality and diversity of the generated images. Using assessments, we illustrated how populating real images with created instances during the training phase increased our confidence in the reliable label prediction. Experimental results demonstrate that DGCGAN achieves optimal results in terms of the similarity and generalization characteristics.

The main contributions of the paper are summarized as under;

• A novel dual generator architecture using mutual regularization to achieve better generalization for OCT images of coronary arterial plaques. A discriminator (D1) within the network is used for classification alleviating the need for a separate classification architecture resulting in computational efficiency.
• A novel architecture of dynamic multi-discriminator fusion is designed to play an adversarial game against two generators. We introduce a dynamic fusion mechanism to adjust the weighting of discriminators based on the image specific conditions.
• Essentially, we incorporated adversarial loss, perceptual loss, L1 loss, cross entropy and diversity loss in our loss function for an enhanced realism in the generated images.

2. Methods and Materials

Generally, the cGAN takes two input data, namely the latent variable z and the conditional constraints into both generator G and discriminator D as illustrated in Figure 1. The G combines the initial noise input Z with label information to produce the generated output after reshaping to feature ma, transposed Convolutions, batch normalization and activation functions as presented in Figure 1. The final layer of the generator typically uses an activation function.

2.1. OCT Dataset and Its Pre-Processing

A dataset of 51 patients was created using a commercially available OCT system. The focus of the study was on vessels affected by stenosis, though for simplicity, cases involving serial stenosis, mixed plaques and bypass graft stenosis were excluded. Ethical approval for this research was granted by the Galway Clinical Research Ethics Committee (GCREC), and informed consent was obtained from all participants.

Three clinicians independently annotated the OCT images, but final labels were determined through consensus agreement. The classification task was structured by designating a specific label in contrast to the others. Prior to model input, pre- processing steps were applied to the raw OCT images. Vulnerable plaques meeting the predefined fibrous cap thickness criteria were excluded from the analysis. Plaque characterization was based on signal intensity relative to the lumen, classifying them into three classes namely; lipid plaques, calcified plaques and as “no plaque.”

2.2. Proposed Dual Generator Multi-Fusion GAN

In our configuration of the dual generators presented in Figure 2, each generator learns different plausible ways to generate an OCT image based on the same conditioning input y. The two generators G₁ and G₂ in Figure 2 learn to capture different aspects of the underlying OCT image structure. G₁ focus on capturing fine details of the OCT image, while G₂ emphasize on abstract features including overall tissue structure and patterns. With dual generators, the risk of mode collapse is also reduced because each generator learns a distinct aspect of the data. In case, if one generator fails to cover the diversity of the distribution, the other generator fills in missing modes which helps to ensure more realistic and varied synthetic OCT images. Likewise, the two generators learn different modes of the data distribution, they help the model to generalize better for unseen OCT data.

In the presented formulation, the role and parameters of each discriminator is illustrated in Figure 3. The ultimate aim of dynamic discriminators fusion is to improve quality, realism, and diversity of the generated OCT images. As presented in Figure 3, Discriminator 1 (D₁) is used for the classification of coronary atrial plaques into three predefined classes. designed to focus on a different characteristic of the OCT image. D₂ evaluates the temporal consistency of the generated MRI images and D₃ ensures that the generated image matches a particular condition. The role of D₄ is to ensure global realism of the generated images by taking into account spatial and temporal features. D₅ embeds perceptual loss which is computed as the distance between the feature maps of the real and generated images. This loss is combined with the adversarial loss to guide the discriminators on high level perceptual differences.

2.3. Mathematical Formulation

We use dual- generator cGANs and multi-discriminators for OCT synthetic image generation. The generator takes a latent vector z (random noise) and a label y, and produces a synthetic MRI image x.

x= G (z, y)

(1)

where z is the latent vector (random noise), y is the label and x is the generated MRI image.

A latent vector sampled from a Gaussian distribution to introduce randomness and variability into the generated images. The conditioning information in the form of one-hot vector representing the class label of the OCT image is concatenated with random noise z that forms the initial input to the generators. Each discriminator receives the same generated image and label but has a slightly different focus in its evaluation. D₁ is used for classification that to which class the generated image belongs using the expression given in Equation (2).

$$L_1=-E_{z~P_{data\left(z\right)}}{\sum }_{c=1}^3{y}_{c}log{D}_1\left(x\right)-E_{z~P_z}{\sum }_{c=1}^{3}log\left[\right(1-D_{1}G\left(z\right)\left)\right]$$

(2)

where P_data is the real data distribution, P_z is the distribution over the latent space and y_c is the one-hot encoding vector indicating the true class of image x.

D2 performs label-specific evaluation whether the generated image matches the expected distribution of the label using Equation (3).

$$L_2=-E_{z~P_{data\left(z\right)}}log{D}_2\left(x\right)-E_{z~P_z}log\left[\right(1-D_{2}G\left(z\right)\left)\right]$$

(3)

where D₂(x) is the probability that the image x is real.

Discriminator 3 evaluates perceptual quality, which compares the high-level features of the generated and real images. This can be modeled using a perceptual loss based on the difference in feature representations between real and generated images as given in Equation (4).

$${\mathrm{L}}_3={{\mathrm{E}}_{\mathrm{x}}}_{~{\mathrm{P}}_{\mathrm{data}}}\parallel \mathrm{f}\left(\mathrm{x}\right)-\mathrm{f}\left(\mathrm{G}\right(\mathrm{z}\left)\right){\parallel }_2^2$$

(4)

Discriminator 4 evaluates the quality of the image, which can be linked to the luminance or image gradient loss. It attempts to measure how sharp and high-quality the generated images using the L1 loss.

$$L_4=E_{z~P_{data}}{\Vert x-G\left(z\right)\Vert }_1-E_{z~P_z}log\left[\right(1-D_{4}G\left(z\right)\left)\right]$$

(5)

where x−G(z) denotes the L1 normalized between the real image x and the generated image G(z).

Discriminator 5 evaluates the consistency loss between the generated image to maintain style over different transformations of the input.

$$L_5=E_{z~P_z}{\Vert G^{-1}\left(G\right(z\left)\right)-z\Vert }_1$$

(6)

where G⁻¹(G(z)) represents the inverse transformation.

The total combined loss L_T can be written as:

$$L_T={\lambda }_1{L}_1+{\lambda }_2{L}_2+{\lambda }_3{L}_3+{\lambda }_4{L}_4+{\lambda }_5{L}_5$$

(7)

where λ1 controls the importance of the classification loss in discriminator 1, λ2 controls the importance of the realism/quality loss via discriminator 2, λ3 controls the importance of the perceptual loss in discriminator 3), λ4 controls the importance of the quality loss in discriminator 4 and λ5 controls the importance of the consistency loss via discriminator 5

$$L_T={{\lambda }_1(-E}_{z~P_{data\left(z\right)}}{\sum }_{c=1}^3{y}_{c}log{D}_1\left(x\right)-E_{z~P_z}log{\sum }_{c=1}^{3}log\left[\right(1-D_{1}G\left(z\right)\left)\right])+{\lambda }_2(-E_{z~P_{data\left(z\right)}}log{D}_2(x)-E_{z~P_z}log[(1-D_{2}G(z\left)\right)])+{{\lambda }_3(E_{z~P_{data}}\Vert x-G(z)\Vert }_2^2)+{\lambda }_4(E_{z~P_{data}}{\Vert x-G\left(z\right)\Vert }_1-E_{z~P_z}log[(1-D_{4}G(z\left)\right)])+{\lambda }_5(E_{z~P_z}{\Vert G^{-1}\left(G\right(z\left)\right)-z\Vert }_1$$

(8)

We use latent interpolation between the two generators to encourage diverse image generation. By interpolating between the latent space of G1 and G2, we force both generators to produce more diverse outputs, allowing them to explore a broader space of possibilities using Equation9.

z′=α⋅z₁+(1−α)⋅z₂

(9)

where z₁ and z₂ are two different latent codes, and α∈[0,1] is a blending factor that encourages diverse outputs. The generator now operates on the blended latent space as given in Equation (10).

G₁(z′∣y) and G₂(z′∣y)

(10)

Grey Wolf Optimization is used to explore the search space effectively for an optimal solution where each wolf represents a candidate solution in the optimization space. Each wolf represents a set of parameters of the neural network that is optimized by the GWO algorithm.

3. Results and Analysis

In this paper, the experiments are conducted according to the baseline architecture in Figure 3. Our original dataset contains 27 images (no plaque), 22 images (calcified) and 20 images of lipid plaques. After augmentation by a factor of 100 the dataset’s size was increased 6900 images with 2700 (no plaque), 22oo calcified images and 2000 lipid images. The samples generated by our model for each class is presented in Figure 4. Then, we applied cross validation (10-CV) to estimate the generalization performance of the compared segmentation models on the unseen data.

The DGDFGAN model is used for experimentation, and we tried different combinations where the multi-stage aspect or the multi-discriminatory aspect was treated incrementally. We analyzed the Structural Similarity Index Measure (SSIM) loss and the Fréchet Inception Distance (FID) score as an additional cost function. All of the experiments were run for 100 epochs for each fold. SSIM index is a metric that evaluates the degradation of images from a perceptual point of view and FID score underlines the diversity impact. We also included the L1 loss over 10 fold cross validation (10-CV). L1 loss measures the absolute difference between predicted and ground-truth values. It is used to enforce pixel-wise similarity in image generation tasks.

An inter-comparison our proposed model with the cutting edge Multi SDGAN model is presented in Table 1 in terms of SSIM, L1 and FID scores. It is evident that employing the dual generators and multi-discriminatory fusion module improves the performance in both similarity and diversity aspects. The dual generator helps in achieving an improved diversity in generated images as ach generator focus on different aspects of OCT image synthesis. This reduces mode collapse, where a single generator might otherwise get stuck producing similar images. Each generator interacts with different discriminators, ensuring that features are well-learned from multiple perspectives for an enhanced feature representation. By training on different discriminators, the generators learn robust representations that can generalize better to unseen OCT images. As illustrated in Table 2, the two generators share the load, leading to faster convergence resulting in lower FID (better realism) and higher SSIM (better structural similarity) when compared to other approaches including DDFAGAN, MHWGAN and MultiSDGAN.

We computed the SSIM loss which is a perceptual loss function to evaluate the similarity between the original and generated OCT images. It exploits contrast, and structure rather than just pixel-wise differences. The algorithm and related parameters description are presented in Figure 5.

The FID score is used to measure similarity between generated images and OCT real images by comparing their feature distributions. It exploits the mean and covariance of the feature embeddings. FID considers both realism and diversity for making it a better generative model. The used algorithm and the relevant variables are illustrated in Figure 6.

As illustrated in Table 2, an increase of 10. 81% in SSIM index observed as compared to leading SSIM measurements in case of Multi- SDGAN [42]. This is indicative of achieving better structural similarity between the original and generated OCT images. An improvement of 41. 66% in the FID score with baseline results of Multi- SDGAN confirms that our model significantly squeeze the difference between the feature distributions of real and generated images. L1 score reported in Table 2 is further suggestive of the fact that our model predictions are close to ground truth. The dual-generator setup mitigates the problem of biased learning, where a single generator struggle to capture all variations in the OCT dataset. By working in parallel, the generators foster a more comprehensive understanding of data distribution. Consequently, the multi-discriminator framework enhanced the FID scores by ensuring a richer adversarial signal to produce images with better realism and diversity.

4. Conclusions

In this work, we designed a dual generator architecture, entitled DGDFGAN which achieved the better diversity and generalization characteristics for generating synthetic OCT images. The two generators balance the load of multi- discriminators and helps in good generalization via mutual regularization. Further, we skipped the need of a separate classification model by using one discriminator within the DGDFGAN for classification, hence reducing computational effort. The multi-discriminatory modules are responsible for discriminating generators outputs using different loss functions. We computed SSIM loss, FID score and other losses for our model and results suggest that dual generator multi- discriminator set up leads to higher-quality image synthesis and diverse data augmentation which helps in improving diagnostic reliability of medical image datasets including OCT for clinical AI applications.