AI-Driven Multimodal Deep Learning for COVID-19 Prediction: A Comparative Analysis of Pre-Trained vs. Custom Models Using Cough, X-ray, and CT Scan Datasets

1. Introduction

The Covid-19 pandemic resulted from the novel SARS-CoV-2 virus that emerged in late 2019 [1]. Its emergence was accompanied by a record number of deaths across the globe. Because of the loss of lives, lockdowns were imminent, leading to businesses shutting down for months to contain the virus. The pandemic strained the healthcare sector to an unprecedented level, as recorded in Europe and the Americas, for example. This development points to the fact that there is a need for a robust diagnostic system for the early detection of the virus [2]. Its early detection would allow medical personnel to effectively contain it because affected persons would be isolated, contact tracing would start in earnest, and patients would be promptly treated [2]. Furthermore, early detection could potentially save lives [2]. The traditional approach for testing patients for COVID-19 has been through the Reverse Transcription Polymerase Chain Reaction (RT-PCR) [3]. This test is usually time-consuming; in addition, it requires specialized skills to carry out the test on patients) [3]. This development therefore requires that experts be trained on how to effectively administer this process. Also, the process gives a lot of conflicting results, potentially allowing COVID-19 patients to go home due to error [4]. In addition, there have been cases of false positives recorded via RT-PCR [5].

In response to this, researchers have come up with artificial intelligence (AI)-based solutions for the detection of COVID-19 from X-rays [5], CT scans [6], and cough sounds [7]. This is now possible because of the availability of datasets. Cough sound has been used to detect COVID-19, as seen in [8]. They combined this with the symptoms of the patients to make a prediction. The use of a cough signal is useful because a COVID-19 cough produces a distinctive pattern, differentiating it from a normal cough [8]. This distinctive pattern arises as the virus attacks the lungs, thereby damaging its structure [8].

Authors have researched several methods for the detection of COVID-19 from images. For example in [5], they developed a deep learning model that extracts key features from X-ray images and then leverages a pre-trained model (VGG19) for the classification of COVID-19. A pre-trained model reflects an already existing model that has been trained on thousands of images. Therefore, its usage is usually appreciated in image classification tasks – since it has learned images sufficiently from different domains. There is a high likelihood of improved performance when in use in different classification tasks, including medical image classification tasks. Several pre-trained models have been used as seen by authors in [9]. They used several pre-trained models for their proposed model. Some of the pre-trained models (transfer learning models) are ResNET50, ResNET101, and InceptionV3, the pre-trained models were also finetuned on the collected X-ray images, where the model trained on ResNET101 proved to have outperformed other models. In addition, other research using deep learning models for X-ray COVID-19 detection has also been done as seen in the following papers [10] [11]. In [12], authors used the VGG-19 pre-trained model for the classification of X-ray images with improved performance.

The CT-scan images have also been solely analyzed for the detection of the presence of the COVID-19 virus. For example, authors in [13] used a similar approach proposed by authors in [9]. However, they used pre-trained models built on the VGG19 and then compared them to other pre-trained models such as Xception Net, and CNN. From their analysis, it is observed that their model performed satisfactorily. In addition, authors in [14] developed a deep learning model that improves on traditional deep learning methods. Their model incorporates two key innovations – the ability to reason based on the passed data, and the ability to learn. This means the model does not need input from humans in setting parameters. The model achieved an F1-score of 0.9731 on CT-scan images [14].

Furthermore, in [15] they showed that the adoption of a pre-trained model for CT-scan can further improve its classification output compared to models that do not use pre-trained models. On the other hand, researchers have also solely used cough datasets for the prediction of COVID-19. This is because patients suffering from the virus have a distinct way of coughing. After all, their lung structure has been altered [16] [17]. Authors in [18] used a Support Vector Machine (SVM) to classify cough sounds. An ensemble model has also been proposed by authors in [19]. The model consists of a CNN layer for feature extraction, and then another classification model. The authors went further to develop an application called "AI4COVID-19” where users can interface with their tool.

The papers reviewed above point to the fact that pre-trained models can potentially improve the output of COVID-19 classification. More recently, the use of multimodal deep learning models has been adopted to improve COVID-19 accuracy. For example, in [20], authors developed a multimodal system that uses X-ray and CT-scan images for classification. They went further to experiment with different transfer learning architectures such as the MobileNetV2 VGG16, and ResNET50 [20]. Similarly, authors in [21] used two pre-trained models, one for the CT scan and another for the X-ray. The outputs of these different deep learning layers are then fused to give an improved classification output. Many researchers have also combined datasets from different sources; for example, authors in [22] combined datasets of X-ray and cough for improved classification. Consequently, two models were developed, and then the outputs of these models were fused. In addition, their cost function gives more weight or relevance to the model with the least error [22].

Authors in [23] also used several pre-trained models for the detection of the virus in X-ray images as well as CT-scan images. From their experiment, it was established that VGG19 gave the best result in terms of classification accuracy. It was also established that X-ray images are more accurate in the detection of the virus compared to CT-scan images.

While COVID-19 detection with multimodal architectures exists, as discussed, to the best of our knowledge, no multimodal architectures take advantage of cough, X-ray, and CT scans together for the classification of the COVID-19 virus. In addition, no research has been carried out to investigate the relevance of pre-trained models on multimodal architecture using these three datasets. Our research therefore aims to fill this gap by researching the relevance of a multimodal system using three datasets in the classification of COVID-19 with or without a pre-trained model. In addition, we also explored the contribution of pre-trained models on unimodal systems for cough, X-ray, and CT-scan datasets.

Many researchers have exhausted using unimodal pre-trained models to improve the classification of COVID-19 via X-ray images, CT-scan, and cough, in addition, to using multimodal designs for different combinations such as X-ray and CT scan. No research has been done, from our knowledge, using multimodal designs that combine cough, CT-scan and X-rays. In addition, we investigate further the impact of the VGG19 pre-trained model on unimodal classifications such as for cough, CT-scan, and X-ray, and then the effect of the VGG19 on a multimodal design that leverages on three datasets – cough, X-ray and CT scan. Therefore, this research investigates the impact of pre-trained models (VGG19, ResNET, and MobileNetV2) on a multimodal system that combines three datasets of cough, X-ray and CT scan. This research is important in that it provides an in-depth analysis of how the combination of three datasets can be used to improve COVID-19 classification. In addition, it provides a comparative study of how selected pre-trained models can influence COVID-19 classification from the perspective of unimodal and multimodal systems. Furthermore, this research also provides a platform to look at medical diagnosis from the multimodal perspective as it may potentially improve classification. The motivation behind the combination of these datasets is hinged on the fact that COVID-19 patients usually exhibit these symptoms – lung abnormalities, and cough [24]. Lung abnormalities can be detected from X-ray images as well as CT scans. There are several advantages to the proposed model of using three datasets. One of them is the ability to harness complementary information - the sound from a cough could depict respiratory disease while images from X-rays and CT scans could reveal the structure of the lungs [25]. In addition, X-rays and CT-scan can also reveal the extent of damage to the lungs while the cough sounds might indicate early signs of COVID-19 [25]. In addition, there is the potential to increase sensitivity (correctly identifying persons with COVID-19) and specificity (correctly identifying persons without COVID-19) [25]. Lastly, data from one source might be unreliable for exams due to poor X-ray or CT-scan images [25].

2. Materials and Methods

This section discusses the architecture of the proposed model as well as the data preprocessing stages for the datasets used. Recall that we used three datasets: cough sound, X-ray, and CT scan. We therefore need to preprocess the data before passing it into the proposed multimodal architecture.

2.1. Data Pre-Processing

The first dataset is the cough dataset found in [26]. This dataset has audio that lasts up to 9 seconds, however, on average, each audio has approximately two seconds segment of cough. Therefore, we needed to extract this segment. To do this, each cough file is loaded using the PyTorch torch audio. Load () function. This function then reads the audio waveform (wf) as seen in Eq. (1).

$$wf, sample\_rate=\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}.\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}\left(\mathrm{a}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o} \_\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{h}\right)$$

(1)

In Eq. (1), wf is the cough waveform (wf). The next stage is resampling. If the sample rate of the audio is not at 16 kHz, we then resample (Eq. 2).

$$\mathrm{w}\mathrm{f}=\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}.\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{s}.\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(sample\_rate,16000\right) \left(wf\right)$$

(2)

The conversion of the waveform to a mono channel is next (Eq 3). The conversion is necessary because, the critical information (cough) can be found in one channel, another advantage of using one channel is for noise reduction.

$$\mathrm{w}\mathrm{f}=\mathrm{w}\mathrm{f}\left[0:1,:\right]$$

(3)

Next, the detection of the cough segment kicks in and it is then extracted. This is done by using the Short-Time Fourier Transform (STFT) [27]. The STFT breaks the cough waveform into small time windows that would enable signal change detection [27]. STFT converts the waveform from a time domain into the frequency domain (F) for each frame $\mathrm{F}=\left|\mathrm{S}\mathrm{T}\mathrm{F}\mathrm{T}\right(wf\left)\right|$ [27]. The next is to calculate the energy of each frame as seen in Eq. (4)

$$energy=\sum (F^2, axis=0)$$

(4)

where $F^2$ squares the size or magnitude of the Short-Time Fourier Transform for each frame. In the next stage, we normalized the energy (Eq. 5).

$$nor{m}_{energy}={\displaystyle \frac{energy-\mathrm{m}\mathrm{i}\mathrm{n}\left(energy\right)}{\max \left( energy\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(energy\right)}}$$

(5)

The normalized energy ($nor{m}_{energy})$ is then analyzed based on a set threshold of 0.5. If an energy’s frame exceeds 0.5, the algorithm then selects the first two seconds of the waveform, effectively collecting the cough portion into$X_{audio}$.

We used the X-ray dataset found in [28], [29], [30] and [31] while for the CT scan, we used the dataset in [14]. The pre-processing for X-ray and CT-scan images is similar. Each image is opened using the PIL image. Open () function, and read as coloured images. The images are further resized into 224 by 224 – since some pre-trained models take images in this size. The preprocessing X-ray images are stored in $X_{xray}$ while $X_{ct-scan}$ is for CT-scan.

2.2. Model Design

Figure 1 depicts the design of the multimodal system which has five layers, excluding the pre-processing layer. For C1 (which houses the first convolution layer for cough). We have the pre-pressed cough sound (X_audio) being fed into the model via block C1. This block takes the processed cough sounds as input as seen in the equation below. In Eqs. (6), (7), and (8) encapsulate the processes in block C1. In Eq. (7) p is the dropout value, set to 0.5.

$$\begin{gathered} C_1=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}1\mathrm{D}\right(X_{audio,}{W}_1,b_1,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=1,\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}=1\left)\right) \\ {C}_1=\mathrm{Dropout}(C_1,p=0.5) \end{gathered}$$

(6)

$$C_1=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(C_1, p=0.5\right)$$

(7)

$$C_1=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}1\mathrm{D}\left(C_1,{\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{e}\mathrm{l}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}}=2,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=2\right)$$

(8)

For block C2, we have Eqs. (9), (10) where $C_1$ is the output from the previous layer while $W_2, \mathrm{a}\mathrm{n}\mathrm{d} b_2$ are the weights and biases of the block.

$$\begin{gathered} C_2=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{u}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}1\mathrm{D}\left(C_1,W_2,b_2,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=1,\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}=1\right) \\ {C}_2=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(C_2, p=0.5\right) \end{gathered}$$

(9)

$$C_2=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}1\mathrm{D}\left(C_2,{\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{e}\mathrm{l}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}} =2,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=2\right)$$

(10)

For X-ray pre-processed images, we have Eqs. (11), (12), (13), (14), (15), and (16) for blocks X1 and X2

$$X_1=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}2\mathrm{D}\left(X_{xray,}{W}_1,b_1,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=1,\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}=1\right)\right)$$

(11)

$$X_1=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(X_1, p=0.5\right)$$

(12)

$$X_1=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}2\mathrm{D}\left(X_1,{\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{e}\mathrm{l}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}} =2,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=2\right)$$

(13)

$${ X}_2=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{u}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}2\mathrm{D}\left(X_1,W_2,b_2,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=1,\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}=1\right)$$

(14)

$$X_2=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(X_2, p=0.5\right)$$

(15)

$$X_2=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}2\mathrm{D}\left(X_2,{\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{e}\mathrm{l}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}} =2,\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{d}\mathrm{e}=2\right)$$

(16)

For CT-scan images, we have the following Equations – (17), (18), (19), (20), (21), and (22).

$${CT}_1=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}2\mathrm{D}\left(X_{ct-scan,}{W}_1,b_1,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=1,\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}=1\right)\right)$$

(17)

$${CT}_1=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left({CT}_1, p=0.5\right)$$

(18)

$${CT}_1=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}2\mathrm{D}\left({CT}_1,{\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{e}\mathrm{l}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}} =2,\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{d}\mathrm{e}=2\right)$$

(19)

$${CT}_2=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{u}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}2\mathrm{D}\left({CT}_1,W_2,b_2,\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{e}=1,\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}=1\right)$$

(20)

$${CT}_2=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left({CT}_2, p=0.5\right)$$

(21)

$${CT}_2=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}2\mathrm{D}\left({CT}_2,{\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{e}\mathrm{l}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}} =2,\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{d}\mathrm{e}=2\right)$$

(22)

For FC1, in Figure 1, we have Eq. (23). $\text{'}Combined\text{'}$ combines the outputs from C2, X2, and CT2, as seen in Eq. (24)

$${FC}_1=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(W_{fc1,}.Combined+b_{fc1}\right)$$

(23)

$$Combined=\left[\begin{array}{c}{C}_2\\ X_2\\ {CT}_2\end{array} \right]$$

(24)

$W_{fc1}$ has 164,913408 trainable weights while $b_{fc1}$ has 164 trainable parameters. ${FC}_1$ is the fully connected layer, as seen in Figure 1. The second fully connected layer is in Eq. (25). $W_{fc2,}$ has 1164 trainable parameters while $b_{fc2}$ has 1 parameter. The output of this goes into a sigmoid activation in Eq. (26), and then the output is seen in Eqs. (27) and (28).

$${FC}_2=\left(W_{fc2,}.{FC}_2 +b_{fc2}\right)$$

(25)

$$Ouput=\mathsf{\sigma }\left({FC}_2\right)$$

(26)

$$if Ouput\ge T \left(T=0.5\right):Positive \left(Covid-19 present\right)$$

(27)

$$if Ouput<T \left(T=0.5\right):Negative \left(Covid-19 absent \right)$$

(28)

2.3. Training

For the training process, we used the Eq. (29). We trained for 30 epochs. The inputs to the proposed model go into the model at the same time. This means that a sample of positive COVID-19 cough, X-ray and CT scan goes into the model at once. This also goes for negative COVID-19. We used batch processing of 8 during the training process.

$${\theta }_{t-1}={\theta }_t-lr {\displaystyle \frac{\delta L\left(M(X_{audio,},X_{xray,},X_{ct-scan,}\right), y)}{\delta \theta }}$$

(29)

In Eq. 29, $\theta$ is the model parameter, while $\left(M(X_{audio,}{X_{xray,}, X}_{ct-scan,}\right)$ is the forward pass of the model. $lr$ is the learning rate. Lastly$, L\left(M(X_{audio,},X_{xray,},X_{ct-scan,}\right), y$) is the loss function.

Figure 2 provides the structure of the dataset for training, testing and validation.

2.4. Validation

At the end of each epoch, we validate the model, and if the loss value on the validation dataset is less than the previous epoch, the current training model is saved. For each batch, i of 8 data points (eight cough sounds, eight X-ray images, and eight CT-scan), do the following in Eqs (30) to (35):

• Initialize model’s best loss $L_{best}= \infty$
• Forward pass

  $$y_i^{^}=X_{audio,},X_{xray,},X_{ct-scan,}$$

  (30)
• Compute loss

  $${loss}_i^{^}=L\left(y_i^{^},y_i\right)$$

  (31)
• Compute accuracy

  $$y_{i, predicted }^{^}=1\left(y_i>0.5\right)$$

  (32)

  $$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}={\sum }_{i=1}^{N}1 L\left(y_{i, predicted}=y_i\right)$$

  (33)
• We then calculate the total loss $L^{\left(t\right)}$ on the validation dataset for each epoch.

  $$L^{\left(t\right)}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N \left({loss}_i^{^}\right)$$

  (34)
• $if L^{\left(t\right)}< L_{best}, then$

  $$L_{best}=L^{\left(t\right)}$$

  (35)

2.5. Unimodal Systems

To test the robustness of the multimodal design, we developed two different models for each dataset, one with a VGG19 pre-trained model (Figure 4) and another without a pre-trained model (Figure 3). We decided to concentrate on VGG19 as it is claimed in [21] that its performance outperforms other pre-trained models. For the unimodal system in Figure 3, for cough the T1 and T2 blocks replicate the C1 and C2 blocks in Figure 1. Also, for the unimodal system for X-ray T1 and T2 replicates the X1 and X2 blocks. The same goes for the CT-scan - CT1 and CT2 for T1 and T2. Same as FC1 and FC2.

In Figure 4, the pre-trained model is the VGG19. FC1 and FC2 reflect the same architecture as seen in Figure 1. The training, and testing validation of the unimodal systems follow the same process prescribed for the multimodal systems, earlier. Except that the combination of datasets is not implemented.

2.6. Multimodal Pre-Trained Systems

Furthermore, we also developed a multimodal pre-trained model. We experimented on three pre-trained models – they are the VGG19, ResNET and MobileNetV2. We used these models as they are popular pre-trained models that have been used in the literature.

3. Evaluation

For the evaluation of the proposed multimodal system, we passed samples of the test dataset in batches of 8. Each data sample contains a collection of Cough C, X-ray X, and CT-scan CT (C, X, CT). This input is then fed into the model to produce a prediction $y_i$ = $M(C_I, X_I, {CT}_I)$. Here, M reflects the trained model. This evaluation is also carried out on the multimodal system equipped with pre-trained models, and also the unimodal system with or without pre-trained models. The following evaluation metrics are used (Eqs 36-40). Where TP is the true positive, TN is the true negative, FP is the false positive and FN is the true negative.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\ast 100$$

(36)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(37)

$$Specificity={\displaystyle \frac{TN}{TN+FP}}$$

(38)

$$F1=2\ast {\displaystyle \frac{Precision\ast Recall}{TN+FP}}$$

(39)

4. Results

The results highlight several key insights into the use of deep learning models. First for the unimodal systems for cough, X-ray, and CT-scan. The result from Table 1 shows the accuracy score as well as the F1-score of the unimodal system for cough. Looking at the graphs of the unimodal and pre-trained unimodal in Figure 6 and Figure 7, it is observed that the cough with a pre-trained model improves the validation loss, as against the graph in Figure 6 without a pre-trained model. It therefore means that the traditional deep learning model cannot learn the dataset patterns, hence the poor outcome. In Table 2, there is an improved performance on the accuracy and the F1-score. This shows the importance of transfer learning on the cough dataset to deliver an improved outcome. The architectural complexity of the VGG-19 might have assisted in delivering an improved outcome.

The same outcome is also recorded for unimodal deep learning system for X-ray, Table 3 and Table 4, Figure 8 and Figure 9. The pre-trained model for the VGG-19 outperformed the traditional CNN deep learning model (Table 3 and Table 4). It can also be observed that the over-fitting is minimized as the training loss curve and the validation loss curve are relatively close.

However, this is not the case for the CT scan. The traditional unimodal deep learning model outperformed the pre-trained unimodal for CT-scan (Table 5 and Table 6). An explanation to this could be that the images learned from the pre-trained model may not be useful in the CT-scan scenario. Even though the pre-trained model and the traditional model attempt to solve the over-fitting problem encountered during the training phase (Figure 10 and Figure 11).

Moving on to multimodal systems (Table 7, Table 8, Table 9 and Table 10), it is observed that the model that leveraged on a pre-trained model and that without a pre-trained model have the same accuracy value of 98% (Table 7 vs Table 8). However, using the F1-score, the non-pre-trained model outperformed the pre-trained model. An explanation for this could be that since the multimodal system has learned from three different datasets, it has aggregated complementary information from different datasets. As a result, it is possible to learn the unique attributes from these datasets to deliver an improved COVID-19 classification model. We also extended the experiment to two additional pre-trained models (ResNET and Mobilenetv2), in Table 9 and Table 10 however, they could not outperform the multimodal system developed without the pre-trained model.

Using the F1-score it is also true that the non-pre-trained multimodal performs better than the unimodal CT scan proposed by [14] which gave an F1-score of 0.9731. The multimodal system, that we proposed, is enriched by learning from three diverse datasets. In addition, analyzing the training graphs (Figure 12 and Figure 13) shows that the non-pre-trained multimodal system (Figure 12) exhibits less overfitting compared to the VGG19-based pre-trained multimodal.

5. Discussion

The results highlight several key insights into the use of deep learning models. First for the unimodal systems for cough, X-ray, and CT-scan. The result from Table 1 shows the accuracy score as well as the F1-score of the unimodal system for cough. In Table 2, there is an improved performance on the accuracy and the F1-score. This shows the importance of transfer learning on the cough dataset to deliver an improved outcome. The same outcome is also recorded for unimodal deep learning systems for X-rays. The pre-trained model for the VGG19 outperformed the traditional CNN deep learning model (Table 3 and Table 4). However, this is not the case for the CT scan. The unimodal deep learning model outperformed the pre-trained unimodal for CT-scan (Table 6 and Table 6). An explanation for this could be that the images learned from the pre-trained model may not be useful in the CT-scan scenario.

Moving on to multimodal systems, it is observed that models that leveraged on pre-trained model and that without pre-trained model are close - regarding the F1-score. Using the F1-score, the model without the pre-trained model outperformed the pre-trained model. An explanation to this could be that since the multimodal system has learned from three different datasets, aggregating complementary information from different datasets. As a result, there is a high possibility that it would have learned the unique attributes from these datasets to deliver an improved COVID-19 model. We also extended the experiment to two additional pre-trained models (ResNET and Mobilenetv2) in Table 9 and Table 10, however, they could not outperform the multimodal system developed without the pre-trained model.

The multimodal system is enriched by learning from three diverse datasets. From Figure 13, we can observe that the multimodal system with a pre-trained model over-fits. This also shows that VGG19 pre-trained model was just learning almost completely from its own training dataset, and not learning from the new dataset. This contrasts with the non-pre-trained multimodal model (Figure 12).

6. Conclusion

This study developed a multimodal deep learning system for classifying COVID-19 through three datasets – cough, X-ray and CT scan. Using a pre-trained model such as ResNet, VGG19, and MobileNetV2, the results show that a multimodal system that combines these datasets could deliver an impressive performance without any pre-trained models. This is possible as the multimodal system has learned sufficiently from different heterogeneous datasets, making it robust enough to perform well without a pre-trained model. However, this is not the case for unimodal models developed for cough and X-ray – it was found that their pre-trained models outperformed their unimodal deep learning models. An explanation for this could be that the images learnt from VGG19 are indeed useful for the training process. On the flip side, this was not the case for the unimodal deep learning model developed for the CT scan. It was discovered that the pre-trained model did not outperform the non-pre-trained unimodal deep learning model for the CT scan. An explanation for this could be that the images learned from the pre-trained model might not be useful for the CT scan. The results conclude that the non-pre-trained multimodal system had an F1-score of 0.9804 while that of a pre-trained model had 0.98.

This research points to the need to investigate the characteristics of datasets before engaging in transfer learning from pre-trained models. While pre-trained models might be useful for tasks such as unimodal COVID-19 classification for cough and x-tray, they might be counterintuitive for unimodal CT-scan and multimodal COVID-19 classification.