Generative Models for Medical Image Creation and Translation: A Systematic Review

1. Background

In recent years, deep learning has gained widespread prominence in medical image analysis [1,2,3,4,5,6,7]. Within the scope of this review, we focus on one of the most compelling applications of deep learning: generative AI in medical imaging-a dynamic and rapidly advancing field of research. The rapid advancement of deep learning and computer vision over the past few decades has had profound implications across a wide range of applications, with the field of medical image generation significantly benefiting from these developments [8,9,10,11,12,13,14].

In this review, we focus on the application of generative models to medical image creation and modality translation. As shown in Figure 1, image creation aims to generate new images based on potential conditional variables. In deep learning-based image generation, a large dataset of real images is typically used to train the model initially. Subsequently, random noise or conditional inputs are utilized to produce new images. This approach is primarily employed to address challenges in medical imaging, such as data scarcity, insufficient annotations, and severe class imbalances, which are common obstacles in training robust deep learning models [15,16].

Imaging modalities, including Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Positron Emission Tomography (PET), are commonly used in clinical workflow, each providing unique structural, functional, and metabolic information [17]. Image translation aims to map images from one or more modalities to another while preserving semantic and informational content. The primary goal of medical image translation is to optimize clinical workflows, particularly in situations where traditional imaging methods are impractical due to constraints related to time, labor, or cost [18].

In this context, several key questions guide our investigation: What are the latest advancements in generative models for medical image creation and cross-modality translation? How do different generative model architectures—such as generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models —perform in the context of medical imaging, and what are their respective strengths and limitations? Furthermore, how do advanced optimization strategies, including adversarial training, uncertainty modeling, and gradient perturbation, contribute to improving the fidelity, realism, and clinical utility of synthetic medical images? Finally, what are the primary evaluation metrics used to assess the quality, anatomical accuracy, and clinical applicability of these generated images, and how do these metrics align with the standards of real-world medical practice? By addressing these questions, this review aims to provide a comprehensive understanding of the current state of generative models in medical imaging, highlight emerging trends, and identify areas for future research that could further enhance the capabilities and clinical integration of these powerful technologies.

In this review, we categorize the relevant literature according to their respective applications and thoroughly examine their clinical implications. Furthermore, we explore recent trends and potential future directions in the field. To summarize, the primary contributions of our work are as follows:

• This review conducts a thorough review of three widely employed generative models: VAEs, GANs, and diffusion models (DMs). We outline algorithms within these generative models that have found extensive applications in the domain of medical image analysis and provide analyses thereof.
• This review categorizes the applications of generative models in medical image analysis into creation and translation. We present an extensive review of creation methods and classify their downstream applications into three distinct categories: classification, segmentation, and others. We classify translation methods based on the target modality.
• This review organizes previous studies into categories and offer practical implementation guidelines gleaned from the lessons learned in these works.

The architecture of this review is shown in Figure 2. In Section 2, we provided a comprehensive comparison with related works. In Section 3, we introduced the search methods for literature and analyzed the trend of the generated model’s publication. In Section 4, we introduced three most used generation models, VAEs, GANs, DMs, and their variants. In Section 5, we reviewed medical image creation and classified the literature according to different downstream tasks. In Section 6, we reviewed medical image translation and classified the literature according to different target modalities. In Section 7, we summarized the application of generative models in medical image creation and translation and provided implementation guidelines, as well as limitations and future research in this review.

2. Related Works

Numerous studies have reviewed the application of generative models in medical image analysis, reflecting the rapid development and growing interest in this field. Yi et al. [19] conducted an early review of the applications of GANs in medical image analysis, covering research up to October 2018. Similar to the work of Yi et al., Kazeminia et al. [20] extended this work by reviewing the applications of GANs in medical image analysis up to October 2019. Their review comprehensively categorized the use of GANs across various tasks, including medical image synthesis, segmentation, reconstruction, detection, denoising, registration, and classification.

Beyond GAN-specific reviews, Wang et al. [18] provided a broader perspective by examining deep learning-based methods for medical image translation, highlighting advancements in cross-modality image synthesis. Dayarathna et al. [17] conducted a comprehensive survey on deep learning-based medical image translation, covering research from 2018 to 2023. Their review focused on the generation of pseudo-CT, synthetic MR, and synthetic PET images, providing a detailed overview of synthetic contrasts in medical imaging. Additionally, they summarized the most frequently employed deep learning architectures for medical image synthesis, highlighting key methodologies and their applications in cross-modality image generation.

Additionally, several studies have focused specifically on the role of GANs in medical image augmentation. Chen et al. [21], Goceri et al. [15], and Kebaili et al. [16] conducted a comprehensive and systematic review and analysis of GAN based medical image augmentation work. Osuala et al. [22] reviewed the application of image synthesis and adversarial networks in the field of cancer imaging. Zhao et al. [23] summarized the application of GAN based on attention mechanism in tasks such as medical image segmentation, synthesis, and detection.

While these reviews provide valuable insights into the use of GANs and related techniques across diverse medical imaging tasks, there remains a significant gap in the literature. To date, no comprehensive review focuses exclusively on the application of deep learning-based generative models for medical image creation and cross-modality translation. Given the increasing complexity of modern generative architectures—such as diffusion models, VAEs, and transformer-based models—and their transformative potential in medical imaging, a dedicated review in this area is both timely and necessary. This work aims to address this gap by systematically analyzing recent advancements in deep learning-based generative models for medical image synthesis and translation, with a focus on their clinical relevance, methodological innovations, and future research directions.

3. Methodology

We conducted a rigorous and comprehensive literature search across multiple well-established academic databases, including Web of Science, Google Scholar, Elsevier, Springer, and IEEE Xplore, to ensure the inclusion of high-quality and diverse studies on generative models for medical image creation and translation. Our search strategy was meticulously designed to capture a broad spectrum of relevant research while maintaining precision and relevance. We employed a combination of targeted keywords and phrases, such as “generative models,” “medical image synthesis,” “GAN,” “diffusion models,” and “image translation,” using Boolean operators (e.g., AND, OR) to construct complex search queries that enhanced the sensitivity and specificity of the search. To maintain a contemporary focus, we restricted the search to peer-reviewed articles published between 2018 and 2023, thereby reflecting the latest advancements and emerging trends in the field. Preprint papers were deliberately excluded from our analysis due to the absence of rigorous peer-review processes, ensuring that only validated and credible research findings were considered.

The selection process was conducted in multiple stages to uphold methodological rigor and reduce potential bias. Initially, we performed a broad screening of article titles and abstracts to identify studies potentially meeting our inclusion criteria. This was followed by a comprehensive full-text review of shortlisted articles, which was independently performed by two authors. Studies were included if they explicitly applied generative models to medical image synthesis, provided detailed descriptions of model architectures and training methodologies, and quantitatively evaluated model performance using established metrics. We excluded articles that lacked methodological transparency, focused solely on theoretical aspects without empirical validation, or addressed non-medical applications. Any disagreements between the two reviewers were resolved through discussion, with unresolved discrepancies adjudicated by a third author to ensure consensus and objectivity. This multi-step screening approach minimized selection bias and enhanced the reliability and reproducibility of our study identification process.

In this review, we exclude the applications of generative models in medical image denoising, reconstruction, super-resolution, registration, etc. This review focuses on modalities primarily used for clinical diagnosis, such as CT, MRI, X-Ray, and PET. These modalities can non-invasively obtain images of entire organs or systems, aiding in clinical diagnosis and treatment monitoring. In this review, we exclude imaging modalities used for studying the microscopic structure of cells and tissues, such as histology and fluorescence microscopy. These modalities are common in pathology, cell biology, and molecular biology research, and are mainly used in laboratory settings to study features at the cellular and molecular levels. They rely on tissue sections and staining techniques, making them suitable for detailed observation at the cellular and tissue levels.

Through this meticulous and systematic screening, we identified a total of 232 articles that met all predefined inclusion criteria. These articles were incorporated into our review, providing a more extensive and structured analysis compared to prior surveys on the same topic [18,19,21,23,24]. Our systematic approach not only ensures comprehensive coverage of the literature but also facilitates a critical examination of the methodologies, optimization strategies, and clinical implications of generative models in medical imaging. To provide a clear visual representation of our search and selection process, Figure 3 presents a detailed flowchart outlining each stage, from the initial identification of studies to the final inclusion. This figure also illustrates the distribution of the selected articles across different model architectures and medical imaging applications, offering valuable insights into current research trends and gaps in the field.

4. Generative Models

Generative models are designed to learn the underlying distribution of a given dataset, in order to generate new data points that resemble the original dataset [25]. These models can generate new data samples that are like the training data, but not identical. Some popular generative models include VAE, GANs, and DMs.

4.1. Variational Autoencoder

VAEs [26] have already shown promise in generating complicated nature image [27,28,29] and medical image [30,31]. As show in Figure 4, the VAE model comprises an encoder network that transforms input data into a latent space representation and a decoder network that reconstructs new data samples from this latent space. Unlike conventional autoencoders, VAEs learn a probabilistic representation of the input data, enabling them to generate novel data samples that closely resemble the original input data [32]. VAEs can generate new, synthetic medical images that are similar to the original training data, which can be used to augment training datasets and improve the performance of machine learning models. However, the images generated by VAEs tend to be blurrier compared to those generated by other generative models like GANs. This is due to the inherent nature of the VAE’s probabilistic framework, which averages over many possible outputs.

In Figure 4, x represents the input image, which is fed into the encoder to obtain two sets of encodings, namely the mean encoding $\mu$ and the variance encoding $\sigma$. $ϵ$ represents the random noise encoding. By combining the original encoding with the noise encoding after weighted allocation, a new latent code $z$ is obtained, which is then sent to the decoder to reconstruct the original image.

4.2. Generative Adversarial Network

As show in Figure 5(a), GAN [33] consist of two neural networks that are trained in an adversarial manner: a generator that to generate ‘fake’ data samples that are indistinguishable from the real data, and a discriminator that learns to distinguish between the generated and real data samples [34]. The generator generates new data samples by transforming a low-dimensional input noise vector into a high-dimensional output space that resembles the original data. The discriminator is trained to differentiate between the generated data and the real data. These two networks are optimized through a minimax game framework, wherein the generator aims to create data that can deceive the discriminator, while the discriminator strives to correctly classify the generated data as fake [14].

As show in Figure 5, variants of GANs have been proposed to address some of the challenges of traditional GANs. For example, conditional GANs (cGANs) add an additional input layer to the generator and discriminator networks, allowing the generator to generate data that satisfies specific conditions, such as class labels or image attributes. Similarly, deep convolutional GANs (DCGANs) use convolutional layers to learn hierarchical features from image data, improving the quality of the generated images. For image translation, Pix2Pix [35] and CycleGAN [36] are the two most commonly used models, and currently most medical image translation models are modified based on these two models [37,38,39]. GANs and their variants have shown remarkable success in various applications. However, they can be challenging to train and require careful tuning to avoid issues such as mode collapse and instability.

4.3. Diffusion Model

Diffusion models have been applied to various fields of image generation [34]. As show in Figure 6, the diffusion model is a type of probability generation model that gradually adds noise to the data to break the structure of the data, and then learns a corresponding reverse process to denoise, thereby learning the distribution of the original data. The forward diffusion process incrementally adds noise to the input data, progressively increasing the noise level until the data is entirely transformed into pure Gaussian noise. This process systematically disrupts the underlying structure of the data distribution. The reverse diffusion process, often referred to as denoising, is then employed to reconstruct the original data structure from the perturbed distribution. This step effectively reverses the degradation introduced by the forward diffusion process. As a result, a highly flexible and tractable generative model is achieved, capable of accurately modeling complex data distributions starting from random noise [34].

Recently, diffusion models and their variants have been applied to medical image analysis, include medical image creation [40,41], translation [42], reconstruction [43,44], denoising [45], registration [46], classification [47], and segmentation [48,49].

5. Creation

Due to the inherent structural complexity and large parameter scale of deep learning models, a significant amount of labeled data is typically required for their effective training. The acquisition of labeled medical image data heavily depends on the subjective expertise and professional judgment of radiologists [50]. Additionally, it is susceptible to issues related to image quality, leading to significant challenges such as data scarcity, insufficient annotations, and pronounced class imbalances. These limitations significantly hinder the broader adoption of deep learning models and represent a critical obstacle in the development of deep learning-based medical diagnostic systems [15].

Medical image data augmentation serves as a technique employed to augment the quantity and diversity of available medical images for training machine learning models [16]. Traditional data augmentation techniques include methods such as image quality enhancement, adjustments to brightness or contrast, and geometric transformations like rotation, scaling, and deformation [15]. The ascendancy of deep learning-based generative models in the synthesis of data has garnered substantial attention. Within the domain of medical image analysis, the utilization of deep learning-based generative models for the generation of medical image data assumes paramount significance. This approach can simulate a substantial volume of challenging-to-obtain medical image data, effectively mitigating the adverse impact of data scarcity on the domain of medical image analysis [22].

In this section, we summarize the application of generative models in medical image creation. We review the literature based on downstream tasks, namely classification tasks, segmentation tasks, and other tasks. As shown in Figure 7(a), it is used for creating medical image data for classification tasks. Specifically, various classes of medical images are created from random noise, and then the created data is used to train a classification model. Figure 7(b) is used for creating medical image data for segmentation tasks. Medical images are created from segmentation masks, and then the created data and masks are used to train a segmentation model. Figure 7(c) is used for creating medical image data for other downstream tasks, such as regression, object detection, and survival prediction.

5.1. Metrics of Medical Image Creation

In order to verify the performance of the proposed medical image creation method, it is necessary to use metrics to evaluate the similarity between the synthesized image. Table 1 lists several commonly used image similarity evaluation metrics. ${\mathbb{P}}_r$ denotes the real image distribution, ${\mathbb{P}}_g$ denotes the generated image distribution. $p_{\mathcal{M}}\left(y|x\right)$ denotes the label distribution of $x$ as predicted by $\mathcal{M}$, and $p_{\mathcal{M}}\left(y\right)=\int p_{\mathcal{M}}\left(y|x\right)d{\mathbb{P}}_g$. $p_{\mathcal{M}}\left(y^{*}\right)=\int p_{\mathcal{M}}\left(y|x\right)d{\mathbb{P}}_r$ is the marginal label distribution for the samples from the real data distribution.$\mathsf{\Gamma }({\mathbb{P}}_r,{\mathbb{P}}_g)$ denotes the set of all joint distributions (i.e., probabilistic couplings) whose marginals are respectively ${\mathbb{P}}_r$ and ${\mathbb{P}}_g$, and $d(x_r,x_g)$ denotes the base distance between the two samples.

The Inception Score (IS) uses a pre-trained Inception-v3 model to compute the KL-divergence between the predicted class distributions of generated images and their overall diversity [51]. A higher IS indicates better image quality and diversity. The Mode Score (MS) adds a measure of similarity between the probability distributions of generated samples and real samples based on the IS. Kernel Maximum Mean Discrepancy (MMD) quantifies the difference between the probability distributions of generated samples and real samples using a fixed kernel function. Wasserstein Distance (WD) serves as a metric to evaluate the similarity between two distributions, where a smaller WD indicates greater similarity. Fréchet Inception Distance (FID) computes the Wasserstein-2 distance between the distributions of feature vectors derived from generated and real images, utilizing a pre-trained Inception-v3 model for feature extraction [51]. A lower FID indicates better image quality.

5.2. Classification

In recent years, there have been notable developments in medical image classification techniques, driven by advancements in deep learning algorithms [52]. However, several challenges and limitations remain. First, acquiring large-scale medical image datasets is often difficult due to privacy concerns, limited accessibility, and ethical constraints. Second, training medical image classification models necessitates the involvement of expert radiologists, pathologists, or clinicians to manually annotate the images with appropriate labels or categories. This annotation process is not only labor-intensive but also requires specialized expertise, creating significant barriers to effectively training deep learning models. Third, medical datasets frequently exhibit class imbalance, where certain disease categories are underrepresented compared to others, further complicating model training and evaluation [53]. Detecting rare diseases or conditions with limited training samples poses a challenge, as models tend to favor the majority classes during training.

In this section, we undertake a comprehensive review of the pertinent literature about medical image creation for classification. We compile essential information from the literature and present it in Table 2.

Table 2 provides a comprehensive overview of 27 literature sources, with most of them being based on GAN. Among these 27 sources, the highest number of publications is focused on chest, with 14 of them specifically targeting chest-related studies. The most common application is in the generation of X-ray and CT images. Additionally, most of the literature is based on 2D models, possibly due to limitations in GPU memory.

Pesteie et al. [30] introduced a variational generative model to learn the probability distribution of image data conditioned on latent variables and corresponding labels. The trained model is employed to generate new images for data augmentation. The efficacy of this approach is demonstrated through its application to ultrasound images (US) of the spine and brain MRI. This model resulted in a notable enhancement in the accuracy of the classification task.

Salehinejad et al. [54] proposed a DCGAN to create synthesized chest X-rays. They utilized both real and synthesized images to train a model for the detection of pathology across five classes of chest X-rays. A comparative analysis of DCNNs trained with a mixture of real and synthesized images revealed that the model outperformed its counterparts trained exclusively with real images.

Pan et al. [55] proposed an image synthesis framework based on diffusion model utilizing a Swin-transformer-based network. This model encompasses a forward Gaussian noise process and a reverse process employing the transformer-based diffusion model for denoising. COVID-19 classification models were trained using real images, synthetic images, and combinations of both.

Applying generative model-based data augmentation methods to medical image classification tasks offers several advantages. First, generative models can learn the underlying data distribution and produce synthetic samples that closely resemble real images. These generated samples can be used to augment the original dataset, enhancing the performance and robustness of classification models [56]. Second, medical image datasets often exhibit class imbalance, where certain disease categories are underrepresented. Synthesis algorithms can generate synthetic samples for these minority classes, helping to balance the class distribution and reduce bias toward majority classes during training. Third, synthesis algorithms can introduce intra-class variations by creating diverse samples with different appearances, shapes, or textures. This enables the model to learn more robust and discriminative features, improving its ability to generalize to unseen data. Furthermore, medical datasets are often limited due to privacy concerns, the rarity of certain conditions, or the high costs associated with data acquisition [57]. Synthesis algorithms allow researchers to artificially expand the dataset by generating additional samples, effectively increasing the amount of training data available for the classification task. In conclusion, by leveraging synthesis algorithms for data augmentation, medical image classification models can benefit from increased dataset size, improved class balance, intra-class diversity.

5.3. Segmentation

Developing a medical image segmentation model necessitates the expertise of radiologists or clinical professionals to manually annotate the images, thereby establishing ground truth data that serves as a reference for training and evaluating the segmentation model [58]. Manual annotation is a time-intensive, subjective process reliant on expert knowledge, rendering the task of constructing extensive and diverse datasets a formidable endeavor.

The generative models provide the images and masks required for training medical image segmentation models by converting masks into synthetic images. This approach significantly mitigates the demand for annotated data. In this section, we embark on an exhaustive review of the pertinent literature, which we present comprehensively in Table 3.

Table 3 offers a comprehensive overview of 26 literature sources, with the majority of them centering on GAN. Similar to the emphasis on data creation for classification tasks, a significant number of publications focus on chest and lung-related topics. The most prevalent applications involve the generation of X-ray, CT, and ultrasound images. As same as creation for classification, most of the literature is based on 2D models.

Guo et al. [59] introduced a confidence-guided synthesis of anatomic and molecular MR image networks (CG-SAMR) that enables the synthesis of data by leveraging lesion contour information into multi-modal MR images. Additionally, they extended the proposed architecture to support training with unpaired data. The synthesized data proves valuable for data augmentation, especially in the context of images containing pathological information related to gliomas.

Zhang et al. [60] presented an improved Dense GAN for data augmentation. They harnessed the power of Dense GAN to generate CT images, facilitating effective semi-supervised segmentation.

Amirrajab et al. [61] proposed a method for synthesizing cardiac MR images with plausible heart shapes and appearances to create labeled data. The approach dissects image synthesis into two tasks: label deformation and label-to-image translation. Label deformation is achieved through latent space interpolation within VAE model, while label-to-image translation is accomplished using a conditional GAN.

5.4. Other Tasks

In addition to classification and segmentation tasks, there are other tasks in the field of medical image analysis, such as regression, object detection, and survival prediction. There are currently many proposed data augmentation methods based on generative models for these tasks. We compile essential information from the literature and present it in Table 4. In addition, we also collected some literature without specified downstream tasks, and they are all listed in Table 4.

Han et al. [62] introduced a 3D Multi-Conditional GAN (MCGAN) to generate nodules on lung CT images to enhance sensitivity in object detection. The MCGAN incorporates two discriminators: the context discriminator and the nodule discriminator. The results demonstrate that 3D CNN-based detection achieves increased sensitivity for nodules of any size or attenuation at fixed False Positive rates, effectively addressing the scarcity of medical data by leveraging MCGAN-generated realistic nodules.

Kamli [63] proposed a Synthetic Medical Image Generator (SMIG) with the primary aim of generating synthetic MRI using GAN to provide anonymized data. Furthermore, to predict the growth of glioblastoma multiform tumors, the authors developed a tumor growth predictor. The authors emphasized the significance of employing synthetic data generated by SMIG. Despite the limited dataset size available from public dataset, the results demonstrate valuable accuracy in predicting tumor growth.

Li et al. [64] introduced DeepAnat, a method to synthesize high-quality T1 images from diffusion MRI and to perform brain segmentation on synthesized T1 images and assist co-registration using synthesized T1 images. This study underscores the advantages and practical feasibility of creating medical images to support various diffusion MRI data analyses and their utility in neuroscientific applications.

6. Translation

Medical image modality translation refers to the process of converting images from one or more modalities into different modalities [119], such as transforming from CT to MRI or from T1 and T2 to FLAIR. Medical image modality translation proves invaluable when medical imaging data is scarce or when patients cannot undergo specific imaging modalities due to medical or technical constraints. Modality translation empowers medical professionals and researchers to access more comprehensive information about a patient’s medical condition, enhancing the accuracy of diagnosis and treatment planning [18].

Conventional methods entail the utilization of models with predefined rules to effectuate the conversion of images from one modality to another. These models necessitate manual parameter adjustments to achieve optimal performance and are often tailored to specific applications, contingent upon the distinctive characteristics of the involved imaging modality [18]. Consequently, numerous intricate and application-specific techniques have been developed. However, these methods confront challenges when the two imaging modalities provide disparate information, rendering the establishment of an effective model a formidable undertaking.

In tandem with the advancement of deep learning, an increasing array of modality translation methods grounded in deep learning principles have emerged. Deep learning-based generative models, exemplified by GANs and diffusion models, have exhibited tremendous potential in the domain of medical image modality translation [20]. They excel by acquiring the capability to learn the mapping between different modalities and generating high-quality synthetic images.

In this section, we classify the collected modality translation literature according to the target modality. In Figure 8, the literature quantity is shown for translations between CT, MRI, CBCT, X-ray, PET, and ultrasound image. The number of studies of the six source modalities is in the order of MRI (63), CT (18), CBCT (15), PET (7), X-ray (3) and US (1). The number of studies of the six targeted modalities is in the order of CT (78), MRI (18), PET (6), X-ray (4), US (1). There are four kinds of translations worthy attention. The first is the translation from MRI to CT (59 studies), primarily focusing on dose calculation for MRI-guided radiation therapy. The second is the translation from CT to MRI (13 studies), primarily aiming for more accurate segmentation. The third is the translation from CBCT to CT with the main objective of 12, primarily serving the objectives of image denoising and dose calculation. The fourth is to translate PET to CT specifically for attenuation correction.

In addition, the translation between non-contrast images and contrast images has also been a research hotspot in recent years, and we will separately organize them in Section 6.7.

6.1. Metrics of Medical Image Translation

In order to verify the performance of the proposed modality translation method, it is necessary to use metrics to evaluate the similarity between the synthesized image and the real image. Table 5 lists several commonly used image similarity evaluation metrics.

Mean Absolute Error (MAE) provides a straightforward, easy-to-interpret measurement of error. It gives equal weight to all errors, regardless of their magnitude, making it less sensitive to outliers. Mean Squared Error (MSE) gives a more significant penalty to large errors compared to MAE, which can be desirable in some contexts. It is also widely used and mathematically convenient. Peak Signal-to-Noise Ratio (PSNR) is based on MSE and shares some of its limitations [120]. It does not always align with human perception, especially for complex images or artifacts like blockiness. Structural Similarity Index (SSIM) is designed to measure the similarity between two images in terms of luminance, contrast, and structure, which aligns better with human perception [121]. It is often considered more accurate than PSNR for evaluating image quality. In summary, MAE and MSE are simple and widely used metrics that are easy to compute but may not always align with human perception. PSNR is also widely used and easy to interpret but may not correlate well with perceptual quality. SSIM, on the other hand, is more aligned with human perception but can be more computationally expensive. Choosing the right metric depends on the specific requirements of the application and the aspects of image quality that are most important.

In the equations in Table 5, $x_i$ and $y_i$ are the pixel values of position $i$ in the image $x$ and $y$, respectively. MAX is the maximum possible pixel value. ${\mu }_x$ and ${\mu }_y$ are the mean value of image $x$ and $y$, respectively. ${\sigma }_x^2$ and ${\sigma }_y^2$ are the variance of image $x$ and $y$, respectively.${\sigma }_{xy}$ is covariance of image $x$ and $y$. ${\mathrm{C}}_1$ and ${\mathrm{C}}_2$ are constants.

6.2. Generating MRI

6.2.1. Multi-Contrast MRI Synthesis

MRI stands as a non-invasive medical imaging technique utilizing a potent magnetic field and radio waves to generate intricate images of internal organs and tissues within the human body [38]. Various MRI modalities, including T1-weighted (T1w), T2-weighted (T2w), Diffusion-Weighted Imaging (DWI), Magnetic Resonance Angiography (MRA), and Fluid-Attenuated Inversion Recovery (FLAIR), offer distinctive characteristics and applications. In tumor analysis, T1-weighted scans excel at differentiating gray and white matter in brain images, while T2-weighted images enhance the contrast between fluid and cortical tissue. FLAIR (Fluid-Attenuated Inversion Recovery) sequences are particularly effective in suppressing cerebrospinal fluid signals, improving lesion visibility. T1 contrast-enhanced (T1ce) images are valuable for delineating tumor regions in brain scans. Magnetic Resonance Angiography (MRA) is primarily used to evaluate vascular anatomy and detect abnormalities that may predispose to hemorrhages. Proton density (PD) images are widely utilized in radiology for inferring water content, aiding in lesion classification and multispectral segmentation. The integration of these multimodal MRI scans provides complementary information, with each modality offering unique insights into the body’s internal structures and functions. Together, they deliver a comprehensive assessment of the patient’s condition [38].

In some cases, it may be difficult to collect complete modalities for medical image analysis due to factors such as the cost of long-term examinations and uncooperative patients, particularly children and the elderly [122]. In such situations, synthesizing missing or damaged modalities using successfully acquired modalities can improve the availability of diagnosis-related images and enhance analysis tasks such as classification and segmentation. In recent years, with the development of deep learning based generative models, there has been an increasing amount of work on the translation between MRI modalities. Table 6 lists essential information of these works.

In Figure 9(a), based on the number of studies, the primary translations can be ranked as T1-to-T2 (17), T2-to-T1 (13), T1-to-FLAIR (10), T2-to-PD (7), T2-to-FLAIR (6), PD-to-T2 (6), FLAIR-to-T1 (4), FLAIR-to-T2 (4), T1-to-PD (2), PD-to-T1 (2), T2-to-DWI (2), and DWI-to-T2 (1). In Figure 9(b), according to the number of studies, the main translations can be ranked as (T1, FLAIR)-to-(T2, T1ce) (4), (T1, T1ce)-to-(T2, FLAIR) (4), (T1, T2)-to-(T1ce, FLAIR) (4), (T1ce, FLAIR)-to-(T1, T2) (4), (T2, FLAIR)-to-(T1, T1ce) (4), and (T2, T1ce)-to-(T1, FLAIR) (4). In Figure 9(c), the main translations are T1-to-(T2, T1ce, FLAIR) (4), T2-to-(T1, T1ce, FLAIR) (4), FLAIR-to-(T1, T2, T1ce) (4), and T1ce-to-(T1, T2, FLAIR) (4). In Figure 9(d), the primary translations are (T1, T1ce, FLAIR)-to-T2 (7), (T1, T2, FLAIR)-to-T1ce (7), (T1, T2, T1ce)-to-FLAIR (7), and (T2, T1ce, FLAIR)-to-T1 (7). The fundamental objective of MRI image translation across contrast mechanisms is to avoid the acquisition of actual scans and provide the unavailable MRI modality necessary for diagnosis and treatment.

In most cases, as the number of source modalities increases, the model’s performance tends to improve. On the same dataset, the performance of multi-to-single translation is superior to single-to-single translation. This is because more modalities can provide complementary information to each other, leading to a more realistic target modality. However, when the number of source modalities remains the same and the number of target modalities increases, no fixed trend in performance has been observed.

As show in Table 6, there are several widely used datasets in cross-modality MRI translation, such as IXI, BraTS, and ISLES. The IXI dataset comprises nearly 600 MRIs obtained from normal and healthy subjects. The MRI acquisition protocol for each subject includes a comprehensive set of sequences: T1, T2, PD, MRA, and DWI. These data have been collected across three different hospitals. The Brain Tumor Segmentation (BraTS) dataset is a widely recognized and frequently used collection of medical images specifically designed for brain tumor research, particularly in the field of medical image analysis and machine learning. The dataset includes multimodal brain MRI scans, typically comprising T1, T1ce, T2, and FLAIR images. The Ischemic Stroke Lesion Segmentation (ISLES) challenge is dedicated to evaluating infarct segmentation in both acute and sub-acute stroke cases, leveraging multimodal MRI data. The inaugural ISLES challenge, held in 2015, was divided into two sub-challenges: Sub-acute Stroke Lesion Segmentation (SISS) and Stroke Perfusion Estimation (SPES). SISS aimed to segment subacute stroke lesions using conventional post-stroke MRI sequences, including T1, T2, FLAIR, and DWI. The ISLES 2018 challenge focused on predicting infarct core delineation in DWI using CT perfusion data. The primary objective of the ISLES 2022 challenge is to segment stroke lesions from DWI, ADC, and FLAIR sequences, with a dataset comprising 400 cases.

Currently, most methods are based on GANs, and most of these methods utilize 2D network architecture, possibly due to memory constraints. Furthermore, algorithms that require paired data for training are more prevalent than those that can use unpaired data because paired images can provide better supervision, leading to improved model performance.

Salman et al. [38] proposed pGAN and cGAN for multi-contrast MRI synthesis, leveraging conditional GANs. The proposed approach preserves intermediate-to-high frequency details through an adversarial loss, providing enhanced synthesis performance using pixel-wise and perceptual losses for registered multi-contrast images and a cycle-consistency loss for unregistered images.

Zhou et al. [123] introduced a Hybrid-fusion Network (Hi-Net) for multi-modal MRI synthesis, which learns a mapping from multi-modal source images to target images. In Hi-Net, a modality-specific network is employed to learn representations for each individual modality, and a fusion network is utilized to learn the common latent representation of multi-modal data. Subsequently, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images.

Muzaffer et al. [42] proposed SynDiff, employing adversarial diffusion model for multi-contrast MRI translation. To capture a direct correlate of the image distribution, SynDiff utilizes a conditional diffusion process that progressively maps noise and source images onto the target image. For efficient and accurate image sampling during inference, large diffusion steps are taken with adversarial projections in the reverse diffusion direction.

6.2.2. Generating MRI from Other Modalities

In this section, we summarize the papers on the translation from non-MRI modalities to MRI. The number of papers of CT-to-MRI is the highest. Table 7 lists essential information of these works.

Wang et al. [124] introduced a bidirectional learning model, denoted as dual contrast CycleGAN (DC-CycleGAN), designed to synthesize MRI from CT. Specifically, a dual contrast loss is incorporated into the discriminators to indirectly establish constraints between real source and synthetic images. This is achieved by leveraging samples from the source domain as negative samples, enforcing the synthetic images to diverge significantly from the source domain. Additionally, cross-entropy and the structural similarity index (SSIM) are integrated into the DC-CycleGAN to consider both the luminance and structure of samples during image synthesis.

Lei et al. [125] proposed a method generating MRIs with superior soft-tissue contrast from CBCT images to aid CBCT segmentation. The entire segmentation process comprises three major steps. Firstly, CycleGAN is utilized to estimate a synthetic MRI (sMRI) from CBCT images. Secondly, a deep attention network is trained based on sMRI and its corresponding manual contours. Finally, segmented contours for a query patient are obtained by feeding the patient’s CBCT images into the trained sMRI estimation and segmentation model.

Bazangani et al. [126] proposed a separable convolution-based Elicit Generative Adversarial Network (E-GAN). The architecture can generate 3D T1 weighted MRI corresponding to FDG-PET

6.3. Generating CT

CT is a potent medical imaging technique that employs X-ray technology and computer processing to generate cross-sectional images of the human body. CT delivers highly detailed cross-sectional views of internal structures, allowing for precise examination and analysis of anatomical features, organs, and bones [177]. CT scanning plays a pivotal role in diagnosing a wide array of medical conditions, including traumatic injuries like fractures and internal hemorrhaging, as well as the detection and assessment of tumors, vascular disorders like aneurysms and blockages, lung diseases such as pneumonia and cancer, and neurological disorders like strokes, brain tumors, and related conditions [178,179].

However, it’s imperative to consider potential risks associated with CT scans due to their use of ionizing radiation, particularly when repeated imaging is necessary [180]. Furthermore, CT serves as the primary imaging modality for radiation therapy, as it provides essential electron density data for dose calculations. While MRI excels in visualizing soft tissues and tumors, it lacks the tissue attenuation information required for accurate dose calculations in radiation therapy. The utilization of generative models to translate MRI into CT images is pivotal in enabling MRI-only radiotherapy, which can yield cost savings, reduce patient radiation exposure, and eliminate registration errors associated with using two distinct imaging modalities [181].

Cone Beam Computed Tomography (CBCT) represents an advanced medical imaging technique widely applied in fields such as dentistry and maxillofacial radiology [182]. CBCT employs a cone-shaped X-ray beam and a specialized detector to produce high-resolution, three-dimensional images of specific regions of interest within the human body, primarily focusing on the craniofacial area. Notably, CBCT offers the advantage of lower radiation doses, enhancing patient safety, while still delivering exceptional image clarity for detailed visualization of anatomical structures like teeth, bones, and soft tissues. However, CBCT does have inherent limitations, including lower contrast for soft tissues and reduced spatial resolution compared to conventional CT. Additionally, CBCT is more susceptible to metal artifacts, potentially compromising image quality when scanning patients with dental restorations or implants. Therefore, the development of generative models to translate CBCT images into CT is of considerable significance [183].

In this section, we provide a comprehensive summary of research papers related to the translation from various imaging modalities, including CBCT, MRI, PET, and X-Ray, into CT. Table 8 lists essential information of these works.

Zhang et al. [183] decomposed CBCT-to-CT synthesis into artifact reduction and intensity correction. They proposed a Multimodal Unsupervised Representation Disentanglement (MURD) learning framework that disentangles content, style, and artifact representations from CBCT and CT images in the latent space. MURD can synthesize different forms of images by recombining disentangled representations. Additionally, they introduced a multipath consistency loss to enhance structural consistency in synthesis and a multidomain generator to improve synthesis performance.

Dong et al. [184] proposed a 3D CycleGAN framework to synthesize CT images from non-attenuation corrected PET (NAC PET). The method learns a transformation that minimizes the difference between sCT, generated from NAC PET, and true CT. It also learns an inverse transformation such that the cycle NAC PET image generated from the sCT is close to the true NAC PET image.

Zhou et al. [185] proposed a multimodality MRI synchronous construction based deep learning framework from a single T1-weight image for MRI-guided radiation therapy (MRIgRT) synthetic CT (sCT) image generation. The network is primarily based on GAN with sequential subtasks of intermediate synthetic MRI generation and joint sCT image generation from the single T1 MRI. It comprises a multitask generator and a multibranch discriminator, where the generator consists of a shared encoder and a split multibranch decoder.

6.4. Generating X-Ray Image

In this section, we provide a comprehensive summary of research papers related to the translation from various imaging modalities into X-ray. A summarized overview of these works is presented in Table 9, highlighting essential information for reference.

Yuen et al.[249] introduced a CT-based Chest X-ray (CXR) synthesis framework, named ct2cxr, for data augmentation in pneumonia classification. Leveraging GANs and a customized loss function tailored for model training, the approach aims to preserve target pathology and maintain high image fidelity. The results indicate that CXR images generated through style mixing enhance the performance of general pneumonia classification models. Evaluation on a Covid-19 dataset demonstrates similar improvements over baseline models.

Huang et al. [247] proposed a sigmoid-based intensity transform, utilizing the nonlinear optical properties of X-ray films, to enhance image contrast in synthetic cephalograms generated from 3D volumes. Super-resolution deep learning techniques are explored to improve image resolution. For low-dose purposes, Pix2pix is introduced for 2D cephalogram synthesis directly from two cone-beam projections. An efficient automatic landmark detection method for synthetic cephalograms is proposed, combining LeNet5 and ResNet50.

Shen et al. [250] proposed a strategy for obtaining X-ray projection images at novel view angles without the need for actual projection measurements. Specifically, a Deep Learning-based Geometry-Integrated Projection Synthesis (DL-GIPS) framework is proposed for generating novel-view X-ray projections. The deep learning model extracts geometry and texture features from a source-view projection, then performs geometry transformation on the extracted features to accommodate the change in view angle. In the final stage, the X-ray projection in the target view is synthesized from the transformed geometry and shared texture features via an image generator.

6.5. Generating PET Image

Positron Emission Tomography (PET) is a powerful medical imaging technique. It is based on the principle of detecting and visualizing the distribution of positron-emitting radionuclides within the body [253]. PET imaging has a wide range of clinical applications. PET is used to detect and stage various types of cancers by highlighting areas with increased metabolic activity. PET is valuable in studying brain function and diagnosing conditions such as Alzheimer’s disease, Parkinson’s disease, and epilepsy. PET can assess blood flow and myocardial viability, helping in the evaluation of heart conditions, including coronary artery disease and myocardial infarction. PET is used to identify sites of infection or inflammation in the body, which can aid in the diagnosis and monitoring of infectious diseases and inflammatory disorders. PET allowing scientists to study various physiological processes, develop new drugs, and better understand diseases at the molecular level. PET provides functional and metabolic information, complementing the structural information obtained from techniques like CT and MRI [273]. It can detect diseases at an early stage when structural changes may not yet be apparent. PET has high sensitivity and specificity, making it a valuable tool for accurate disease detection and treatment monitoring [255].

Of course, PET also has some limitations. PET involves exposure to ionizing radiation due to the use of radiopharmaceuticals. It requires specialized equipment and trained personnel. PET scans may be expensive compared to some other imaging modalities. So, there is currently some work dedicated to converting other commonly used medical image modalities, such as MRI and CT, into PET. In this section, we provide a comprehensive summary of research papers related to the translation from CT or MRI into PET image. A summarized overview of these works is presented in Table 10, highlighting essential information for reference.

Hu et al. [253] introduced a 3D end-to-end synthesis network named Bidirectional Mapping GAN (BMGAN) for brain MR-to-PET synthesis, effectively utilizing image contexts and latent vectors. The proposed bidirectional mapping mechanism is designed to embed the semantic information of PET images into the high-dimensional latent space. Furthermore, the architecture includes a 3D Dense-UNet generator and hybrid loss functions to enhance the visual quality of cross-modality synthetic images.

Ben-Cohen et al. [255] combined a fully convolutional network (FCN) with a conditional GAN to simulate PET data from input CT data. From a clinical perspective, such solutions may facilitate lesion detection and drug treatment evaluation in a CT-only environment, potentially reducing the need for more expensive and radioactive PET/CT scans.

6.6. Generating Ultrasound Image

Ultrasound imaging is a non-invasive medical imaging technique that uses high-frequency sound waves to create real-time, dynamic images of the internal structures of the human body [274]. These images, known as ultrasound scans or ultrasound images, are valuable in medical diagnosis, monitoring pregnancies, and guiding various medical procedures.

Ultrasound imaging does not involve radiation or invasive procedures. It provides dynamic, real-time images, making it suitable for observing movement and function. Ultrasound is safe for pregnant women, infants, and individuals with contraindications to other imaging methods. Ultrasound machines come in various sizes, including handheld devices, making them highly portable for use in different clinical settings.

Grimwood et al. [275] proposed the use of CycleGAN to create synthetic Endoscopic ultrasound (EUS) images from CT data, that can be used as a data augmentation strategy when EUS data is scarce.

6.7. Non-Contrast and Contrast-Enhanced Image

Non-contrast enhanced medical imaging entails the acquisition of images without the administration of contrast agents. This imaging modality relies on the inherent contrast of natural tissues to visualize anatomical structures and identify potential abnormalities. Non-contrast imaging is commonly employed for routine screenings, initial assessments, and follow-up examinations. It is considered a safer option for patients with contraindications or allergies to contrast agents. Nonetheless, there are scenarios where non-contrast imaging may be limited, and the use of contrast-enhanced imaging could offer additional diagnostic insights.

Contrast-enhanced medical imaging involves the introduction of contrast agents, typically through intravenous administration, to enhance the visualization of specific anatomical structures or physiological processes [276]. These contrast agents contain substances that augment the visibility of blood vessels, organs, tumors, or regions with altered perfusion. Contrast-enhanced imaging proves particularly valuable in situations where non-contrast imaging may not provide adequate diagnostic information. For instance, in contrast-enhanced CT scans, iodine-based contrast agents are intravenously injected to accentuate blood vessels, tumors, and regions with abnormal blood flow, thereby improving the detection and characterization of lesions, vascular anomalies, and tumors. In contrast-enhanced MRI, gadolinium-based contrast agents are commonly utilized to enhance the visualization of blood vessels, brain tumors, and areas with compromised blood-brain barrier integrity, making it indispensable in neuroimaging and the diagnosis of conditions such as multiple sclerosis [277].

Contrast-enhanced imaging plays a pivotal role in diagnosing and characterizing various medical conditions, including tumors, vascular irregularities, inflammation, and ischemia. It furnishes critical insights into the dynamic behavior of tissues, enhancing the specificity and sensitivity of imaging investigations. However, certain patients may not be eligible for contrast agent injections due to various factors. To address this challenge, generative models can be employed to translate non-contrast enhanced images into contrast-enhanced images [278].

In this section, we provide an overview of research papers focused on the translation between non-contrast enhanced images and contrast-enhanced images. Table 11 presents essential details from these studies, offering a valuable reference for further exploration of this topic.

Zhao et al. [258] introduced a Tripartite Generative Adversarial Network (Tripartite-GAN) for synthesizing contrast-enhanced MRI (CEMRI) to detect tumors without the need for contrast agent injection. The Tripartite-GAN comprises three interconnected networks—an attention-aware generator, a convolutional neural network-based discriminator, and a region-based convolutional neural network-based detector. This integrated framework facilitates the synthesis of CEMRI and tumor detection, with the generator aiding accurate tumor detection by synthesizing tumor-specific CEMRI and the detector enhancing the generator for precise CEMRI synthesis through back-propagation.

Chen et al. [265] proposed a deep learning-based approach for contrast-enhanced T1 synthesis in brain tumor patients. A 3D high-resolution FCN designed to maintain high-resolution information and aggregate multi-scale information in parallel is employed to map pre-contrast MRI sequences (T1, T2, and ADC) to CEMRI sequences. To address data imbalance between normal tissues and tumor regions, a local loss is introduced to enhance the contribution of tumor regions, resulting in improved tumor enhancement.

Ristea et al. [269] presented a novel approach for translating NCCT scans to CECT scans and vice versa. The approach, named CyTran (cycle-consistent generative adversarial convolutional transformers), is trainable on unpaired images due to the integration of a multi-level cycle consistency loss. In addition to the standard cycle-consistency loss at the image level, additional cycle-consistency losses between intermediate feature representations are proposed, enforcing cycle-consistency at multiple representation levels, and leading to superior results. To handle high-resolution images, a hybrid architecture based on convolutional and multi-head attention layers is designed.

7. Discussion

This review provides a comprehensive summary of prior research on the utilization of generative models in the domain of medical image analysis. Through a synthesis of relevant literature, we categorize the applications of generative models in medical image analysis into two main segments: creation and translation. Building upon the diverse application scenarios of generative models in medical image analysis, this paper organizes previous studies into categories and offers practical implementation guidelines gleaned from the lessons learned in these works.

In Section 4, we conduct a thorough review of three widely employed generative models: VAEs, GANs, and diffusion models. We outline algorithms within these generative models that have found extensive applications in the domain of medical image analysis and provide analyses thereof.

In Section 5, we present an extensive review of creation methods. Depending on the downstream tasks, we classify creation’s downstream applications into three distinct categories: classification, segmentation, and others. Among these, 27 studies focus on classification tasks, 26 studies on segmentation, and 11 studies on various other tasks. Our literature review consistently indicates that across various downstream tasks, data augmentation methods grounded in generative models consistently result in enhanced model performance, particularly when dealing with limited annotation resources.

Section 6 classifies translation methods based on the target modality. For the MRI modality, we identify 61 studies, with 42 studies primarily centered on inter-modal translation within MRI, particularly concentrated on brain images, while 19 studies encompass modalities such as CT, PET to MRI translation. Additionally, there are 77 studies for CT, 5 for X-ray, 6 for PET, and 1 for US, respectively. Furthermore, we conduct a separate analysis of 19 studies involving non-contrast enhanced and contrast-enhanced image translations.

Our comprehensive literature review underscores the notable advancements in GANs over recent years, with the majority of translation tasks predominantly relying on GAN-based methodologies. Furthermore, the introduction of DDPM has witnessed an increasing number of diffusion models being employed for translating medical images across different modalities. The remarkable image synthesis capabilities of diffusion models have significantly elevated the quality of synthesized images, albeit the inherent challenge of slow inference speed remains a critical concern [34].

Given the promising and rapidly evolving nature of medical image synthesis research, alongside the ongoing exploration of optimal image synthesis algorithms, researchers are encouraged to not only fine-tune strategies and pre-trained weights but also systematically investigate self-supervised learning techniques across various categories within their medical image datasets [39]. Additionally, testing newly developed strategies on multiple datasets, ideally encompassing diverse modalities and medical imaging domains, is recommended to foster a more comprehensive understanding of their potential and limitations.

7.1. Implementation Suggestion

Given the rapidly evolving nature and significant practical implications of medical image generation and translation, along with the increasing prominence of diffusion models, the pursuit of an optimal medical image generative model remains an ongoing challenge. In response, we have conducted a thorough survey and detailed comparative analysis of prior research. Our goal is to offer researchers a comprehensive set of implementation guidelines to support their exploration of methodologies in medical image generation and translation.

7.1.1. Unified Model or Specific Task Model?

The unified model refers to the use of the same network architecture to accomplish any medical image synthesis task [38,139,147]. Specific task model is designed to excel at a single task [160,161,162]. It is optimized specifically for that task, without considering other related tasks. Specific task models are often used when the task is complex or requires specialized features or architecture that may not be applicable to other tasks. While these models can achieve high performance for their specific task, they may not generalize well to other tasks.

Given that only a subset of studies has undergone testing across multiple modalities, it is crucial to acknowledge that arriving at definitive conclusions regarding the most optimal generative model for medical image generation is not straightforward. Furthermore, the selection of the most suitable medical image generation model depends on a multitude of factors, including dataset dimensions, complexity, and the specific requirements imposed by various downstream tasks [17,34].

Within the scope of this comprehensive review, only a limited number of studies have rigorously evaluated their methodologies across multiple modalities and anatomical regions. In contrast, the majority of research efforts have concentrated on analyzing single modalities or specific anatomical areas. In practice, the significant differences among various modalities, as well as the substantial variations observed across different organs, tissues, and pathological conditions, present a considerable challenge. This challenge arises because a single, universal model would struggle to consistently outperform task-specific models across the diverse range of modalities and anatomical domains. Therefore, we recommend that researchers carefully tailor generative models to meet the unique requirements of various downstream tasks, image modalities, and anatomical or pathological contexts.

7.1.2. GAN or Diffusion Model?

Since debut in 2014, GANs have seen widespread adoption in the field of medical image synthesis. This popularity can be ascribed to GANs’ utilization of adversarial learning strategy, which has consistently showcased superior generative capabilities in comparison to alternative image synthesis methods like VAEs and Flow-based models [279]. However, it’s important to acknowledge that GANs frequently face challenges during their training process, and their stability relies on the careful selection of hyperparameters and regularization terms. Nonetheless, GANs maintain their prominent position within the domain of medical image synthesis [21].

In the year 2020, the landscape of image synthesis underwent a transformation with the advent of DDPM [280]. This novel framework exhibited remarkable image synthesis prowess, swiftly outperforming GANs in various aspects. Empirical research has provided compelling evidence that Diffusion models exhibit a heightened capacity to capture a broader spectrum of sample diversity compared to GANs [281]. While we strongly believe in the potential of diffusion models as a promising direction in generative modeling, it is important to note that their reliance on multiple denoising steps can result in a slower sampling process compared to the faster sampling capabilities of GANs. A promising approach to addressing this latency issue involves reducing sampling time while maintaining high image quality [282]. Subsequent research endeavors in this trajectory hold promise for narrowing the sampling speed disparity between diffusion models and GANs without compromising the fidelity of generated images.

Generative models confront a tripartite quandary, as elucidated by Kazerouni et al. [34]. While GANs excel at rapidly generating high-fidelity samples, their effectiveness is constrained by limited mode coverage. In contrast, VAEs and Normalizing Flows exhibit significant diversity in their samples but often at the expense of reduced sampling quality. Addressing these limitations, Diffusion models have emerged as a powerful solution, demonstrating the ability to achieve extensive mode coverage while maintaining high sample quality. However, it is important to recognize that the iterative nature of Diffusion models can lead to prolonged sampling times, making them computationally demanding and highlighting the need for further optimization [281].

In the domain of image creation tasks, both GANs and Diffusion models were initially applied to image generation. However, empirical evidence substantiates that the synthesis quality of Diffusion models distinctly outshines that of GANs [283]. Conversely, in the realm of image translation, recent advancements in GAN technology have solidified their position as the dominant choice over Diffusion models. Notably, there is currently no established baseline model based on Diffusion models that parallels the widely recognized Pix2Pix and CycleGAN frameworks. As a result, the efficacy of Diffusion models in image translation tasks remains unproven and requires further investigation.

7.1.3. Translation with Prior Knowledge

Due to the significant differences in information content among medical images from various modalities—often being entirely distinct—the importance of incorporating prior knowledge into medical image generation tasks becomes clear. Integrating prior knowledge is a crucial step toward improving the quality, authenticity, and clinical relevance of the generated images. This incorporation serves multiple purposes, guiding the generative process and ensuring that the resulting images maintain anatomical accuracy and clinical utility [284,285].

One strategy in the integration of prior knowledge involves the design of custom loss functions, engineered specifically to impose constraints rooted in prior knowledge [162,258]. A tangible illustration of this entails the incorporation of penalties or regularization terms into the loss function. These augmentations serve to incentivize the generated images to closely adhere to known anatomical structures or established clinical guidelines.

Furthermore, a critical aspect of leveraging prior knowledge involves preprocessing the training data. This preprocessing aims to highlight or extract specific features or anatomical structures of interest. Techniques such as image segmentation, registration, and other image processing methods can be strategically applied to enhance the quality of the input dataset, thereby providing the model with more robust and informative data.

In the pursuit of fortifying the model’s capacity to leverage prior knowledge, recourse to pre-trained models or knowledge derived from related medical imaging tasks is a valuable strategy. Transfer learning emerges as a potent technique, allowing the model to glean insights from prior knowledge encoded within models trained on analogous tasks or datasets.

7.1.4. Other Possible Optimization Strategies for Training

In addition to adversarial training, alternative optimization strategies such as gradient perturbation and knowledge distillation have been proposed to enhance model robustness [286,287]. Gradient perturbation techniques improve the robustness and generalization of generative models by introducing controlled noise during training [288]. This strategy strengthens the model’s ability to adapt to unseen data and reduces overfitting to specific training distributions [289]. Knowledge distillation optimizes medical image synthesis by transferring knowledge from complex teacher models to lightweight student models [290]. It enhances the model’s generalization across multiple tasks, enabling it to perform diverse medical image synthesis tasks such as denoising, super-resolution, and modality translation [291]. By leveraging the stable generative capabilities of the teacher model, knowledge distillation mitigates the instability issues commonly encountered during generative model training [292,293].

7.2. Limitations and Future Research

This review paper summarizes recent advancements in deep learning-based medical image synthesis methods. However, several limitations should be acknowledged. First, it was not feasible to aggregate or statistically compare the performance of different generative models due to the heterogeneity of the included studies. These studies employed diverse imaging modalities, reported varied evaluation metrics, and applied models to different downstream tasks, making direct comparisons challenging.

Second, our classification approach was unidimensional. In Section 5, we categorized studies based on downstream tasks, while in Section 6, we grouped them by target modality for image generation. This approach may not facilitate a comprehensive comparative analysis of generative model methods. Future reviews could adopt a more nuanced classification framework, such as categorizing studies by image generation techniques, which might provide deeper insights into the strengths and weaknesses of different synthesis methods.

Third, while diffusion models have rapidly advanced in natural image generation since 2022 [283] and are increasingly being applied to medical image synthesis [34], most relevant papers are currently in preprint form. Due to the absence of peer review, these preprints were excluded from this review, resulting in limited coverage of diffusion models. As these models gain traction in medical image synthesis, future reviews should incorporate peer-reviewed studies to provide a more comprehensive assessment.

The study selection process may also have introduced biases. Reliance on specific databases and keywords might have excluded relevant studies from other sources, such as conference proceedings and technical reports. Additionally, the focus on English-language literature and timeframe restrictions may have led to the omission of significant non-English studies or earlier research. These limitations highlight the need for broader search strategies in future reviews.

Regarding gaps in the literature, several underexplored areas were identified. For instance, the application of physics-based generative models in medical image synthesis has received limited attention. Similarly, research on the generalizability of generative models across different disease stages or imaging devices remains scarce. Addressing these gaps could enhance the comprehensiveness of future assessments.

Finally, several challenges were encountered during the review process. Standardizing data extraction was difficult due to variations in metrics and reporting methods across studies, which hindered direct comparisons. The heterogeneity of studies—such as differences in dataset size, quality, and evaluation metrics—further complicated the synthesis of findings. Additionally, insufficient methodological detail in some studies made it challenging to fully interpret their results and methodologies. These challenges underscore the need for more standardized reporting practices in future research.

8. Conclusions

Medical image synthesis based on deep learning is an emerging and rapidly growing field. In this review, we categorize medical image synthesis into two main types: creation and translation. Creation focuses on generating new images from potential conditional variables, while translation involves mapping images from one or more modalities to another, preserving semantic and informational content.

Currently, medical image synthesis based on deep learning primarily utilizes three models: VAE, GANs, and Diffusion models. Each of these models has distinct characteristics, and the choice among them depends on the specific requirements of the task. The diffusion model, in particular, has demonstrated outstanding performance in natural image generation, earning widespread recognition. It is reasonable to anticipate that future research will increasingly focus on medical image creation and translation using diffusion models. Exploring methods to reduce the training and inference time of diffusion models while maintaining high generation quality may represent a promising research direction in the years to come.

Through an analysis of the literature included in this review, it is evident that deep learning-based medical image creation is a powerful technique for enhancing the performance of downstream tasks. It addresses data limitations, improves generalization, reduces overfitting, and increases model robustness to diverse imaging conditions and variations. Modality translation, on the other hand, can supplement missing modalities, aiding in diagnosis and enhancing downstream tasks such as attenuation correction for PET/MRI, radiotherapy planning without CT images, and CBCT denoising. Additionally, translation between contrast-enhanced and non-contrast images primarily aims to reduce costs, both in terms of time and financial resources, while also offering the potential for more accurate diagnoses, particularly for patients with kidney diseases. Overall, medical image translation based on generative models is multifaceted, highly effective, and holds significant promise for the future.