MRI-GAN: MRI and Tissues Transfer Using Generative Adversarial Networks

We study the brain segmentation by dividing the brain into multiple tissues. Given possible brain segmentation by deep, machine learning can be efficiently exploited to expedite the segmentation process in the clinical practice. To accomplish segmentation process, a MRI and tissues transfer using generative adversarial networks is proposed. Given the better result, we pro-pose the transfer model using GAN . For the case of the brain tissues, white matter ( WM ) , gray matter ( GM ) and cerebrospinal fluid ( CSF ) are segmented. Empirical results show that this proposed model significantly improved segmentation results compared to the stat-of-the-art results. Furthermore, a dice coefficient ( DC ) metric is used to evaluate the model performance.


Introduction
In the last decade, a great growth of medical imaging application has been witnessed as the need for segmentation and classification increased. This growth encouraged the development of new models in order to enhance the segmentation process as similar with the process of human learning in clinical practice (Sun et al., 2020), (Yi et al., 2019). To this end, brain segmentation using deep learning, in which the brain is divide in multiple tissues emerged as a good solution to improve the accuracy of segmentation process (Helaly et al., 2021), (Kazeminia et al., 2020).
Different segmentation models were proposed in the literature for brain segmentation to improve the segmentation process accuracy. One model is exploited to generate global information in the first stage and local information in the second stage, which imposes a good result by fusing the global information and local information. Another multistage model that can provide better results is designed to solve the problem of information loss in the one stage (Iqbal & Ali, 2018), (Shen et al., 2017), (Puttagunta & Ravi, 2021), (Chen et al., 2021). Unlike traditional methods, the deep learning algorithms are less studied in the literature. A fewshot 3D multi-modal medical image segmentation using GAN model, consisting of U-net, generator and encoder was proposed in (Mondal et al., 2018). Despite the merits of this model, it did not consider achieving a good results compare with stat-of-the-art-models. First, the authors generate fake images by using generator. Second, the authors use the labeled data, unlabeled data and fake data to train the discriminator to distinguish between generated data and true data. While the encoder is used to compute the predicted noise mean and log-variance. Few-shot model (Mondal et al., 2018) was inspired by recent studies showing a new technique for few-shot learning. This interesting model motivated the idea of few-shot learning (i.e., where a few labeled data are available) which could be done with very low datasets or in one or two datasets. A large body of literature (e.g., (Luo et al., 2021), (Gu et al., 2020), (Emami et al., 2021), (Afifa Khaled, 2020) and references therein) has studied different techniques of brain segmentation in the brain tissues in order to enhance the accuracy of the segmentation process (i.e., to enhance the accuracy of segmentation results to be fairly close to the manual reference). Adversarial learning technique was studied in (Mondal et al., 2018), (Singh et al., 2021), in which a few labeled data are available. This techniques enhance a networks to produce a good segmentation output. However, very limited models have addressed infant brain segmentation of white matter(W M ), grey matter (GM ) and cerebrospinal fluid (CSF ). In , A model was proposed to segment the infant brain in order to improve the segmentation results from the iSEG Grand MICCAI Challenge dataset. It also introduced the use of direct connections between layers from the same and different path. These direct connections were thus used to help the learning process. However, the deeper networks with fewer number of filters per layer was not considered. Moreover, individual weights from dense connections were not investigated. In this paper, we consider a brain segmentation, in which (W M ), (GM ) and (CSF ) are segmented. M RI encoder, with ground truth encoder compress the features of their input into low-dimensional M RI and tissues vectors. Each encoder can compress one or more inputs to their vectors. Layers where the information pass to the de-coder are designed. The main aim of this study is to design model to transfer the tissues information from the ground truth to the M RI images using GAN . The remaining of this paper is organized as follows. Section 2 presents the novel M RI and tissues transfer model. Section 3 presents our experimental. The result and discussion on the performance of our proposed model presents in Section 4. Finally, Section 5 concludes this work and discusses directions for future work.

Background on Encoder-Decoder
Encoder decoder model emerged more than a decade ago as the concept of describing an image (?). One wellknown study of encoder-decoder, namely auto encoderdecoder (Rehman et al., 2019) and its further well investigated encoder and decoder based on pixel-wise classification. Furthermore, this model enabled nonlinear up sampling and the small number of parameters used for the training, which makes the computational process more higher than other deep learning architectures. However, most of the studies on encoder decoder considered dense block into standard encoder decoder model. Applying encoder-decoder model to our M RI and tissues transfer model is efficient. To achieve the aforementioned paper target (i.e., accurate segmentation for brain images ), we first develop a new encoder decoder model that compress the feature of the inputs, and also design layers that pass the tissues information to the decoder. Empirical results show that this proposed model exhibits fairly close to the manual reference, and a significant reduction in training time as compared to the state-of-the-art-models. Furthermore, Dice coefficient is applied to better demonstrate the significance of our model.

Encoder-Decoder
M RI and tissues transfer has an encoder network and a decoder network, followed by a GT -Layers. Figure.1 show the proposed model. Together, the M RI encoder, ground truth encoder, tissues mapping, boundary detection network and decoder make up the generator of our model.

GT-Layers
We use R mapping function that maps a tissue y into (µ, σ) parameters.
R composed of multiple fully connected layers, takes tissues latent code as input.

DISCRIMINATOR LOSS FUNCTION
The discriminator in our model has unlabeled data loss, labeled data loss and GT images loss. And the overall loss function is as follows: l discriminator = λ labeled l labeled + λ unlabeled l unlabeled + λ fake l fake , Where λ labeled , λ unlabeled andλ fake are hyper-parameters. We set the hyper-parameters in Equation (2) to λ labeled = 1.0, λ unlabeled = 1.0andλ fake = 2.0.
For labeled data, we use the same loss function in the standard segmentation network. (Mondal et al., 2018) it was shown that by using l i,k+1 as subtracted function, the softmax function changed as follows: To be more specific, the idea is to introduce unlabeled loss and fake loss, which have analogues to the two components of discriminator loss in the standard GAN and the labeled loss is the cross-entropy. for more information we refer reader to read (Mondal et al., 2018).

GENERATOR LOSS FUNCTION
We introduce a novel generated loss to encourage G to generate real data. Let x and z denote to the real data and noise, respectively.
In our paper, f (x) contains the activation of the last layer.
By minimizing this loss, we force the generator to generate real data in order to match our data and the correspond K classes of real data, which can be define as: Classes = 1, ..., K.

MICCAI ISEG DATASET
The aim of the evaluation framework introduced by the M ICCAIiSEG organizers is to compare segmentation models of W M , GM and CSF on T 1 and T 2. The M ICCAIiSEG dataset contains 10 images, named subject-1 through subject-10, subject T 1 : T 1 -weighted image, subject T 2 : T 2-weighted, and a manual segmentation label used as a training set. The dataset also contains 13 images, named subject-11 through subject-23, used as a testing set. An example of the M ICCAIiSEG dataset (T 1, T 2, and manual reference contour) is shown in Figure.1. On the other hand, Table 1 shows the parameters used to generate T 1 and T 2. The dataset has two different times (i.e., longitudinal relaxation time and transverse relaxation time), which are used to generate T 1 and T 2. The dataset has been interpolated, registered, and skull-removed by the M ICCAIiSEG organizers.

MRBR A I N S DATASET
The MRBrainsdataset contains 20 subjects for adults for segmentation of (a) cortical gray matter, (b) basal ganglia, (c) white matter, (d) white matter lesions, (e) peripheral cerebrospinal fluid, (f) lateral ventricles, (g) cerebellum, and (h) brain stem on T 1, T 2, and FLAIR. Five (i.e., 2 male and 3 female) subjects are provided to the training set and 15 subjects are provided for the testing set. On the evaluation of the segmentation, these structures merged into gray matter (a − b), white matter (c − d), and cerebrospinal fluid (e − f ). The cerebellum and brainstem were excluded from the evaluation.

Experimental Setup
The proposed model was performed Python on a P C with N V IDIAGP U with a Ubuntu 16.04 operating system. Training our model took 30 hours in total. Testing our model took 5 minutes for the images employed for the testing set.

Result and Discussion
To better demonstrate the significance of our model, We train and test M RI − GAN model on two datasets of different ages (i.e., infants and adults). Table 1

Conclusion
In this paper, the MRI and tissues transfer using generative adversarial networks is proposed. Applying the encoder and decoder algorithms separately was shown to result in brain segmentation, which increased the accuracy of the segmentation. Consequently, we proposed a novel M RI and tissues transfer model that increase the accuracy of the segmentation.