An Improved Machine Learning Model for Pulmonary Embolism Detection and Segmentation

1. Introduction

Pulmonary embolism (PE) is the obstruction of blood arteries in the lungs caused by a blood clot[1]. Peripheral edema is third in prevalence among cardiovascular illnesses. The disease has a death rate of 30% [2,3,4]. A delay in diagnosing the condition raises the likelihood of impairment and mortality [5]. Early diagnosis is crucial for treating the disease effectively [6,7]. CTPA is the preferred method for diagnosing PE due to its quick and detailed imaging capabilities [8,9]. Blood arteries show bright on contrast-enhanced CT scans due to the contrast material. The embolism appears dark because it does not absorb the contrast agent. Figure 1 displays pulmonary embolism in a high-quality computed tomography scan.

Detection of PE from CTPA images is performed manually by experienced radiologists. Therefore, it can be time consuming and sometimes difficult [10]. Some studies have shown that there is a 13% discrepancy between overnight and daytime assessments for the detection of PE [11,12,13]. In addition, in some emergency situations, rapid and accurate assessment of PE is of great importance [14]. Some semi-quantitative methods can be used to measure the degree of vascular occlusion and determine the severity of PE. The most common of these methods are the Mastora score and the Qanadli score, known as the Vascular Obstruction Index (VOI), measured by an expert [15,16]. However, there is inconsistency between different experts in the use of these methods [17]. Therefore, the researchers used computer-assisted systems to automatically detect PE.

In this study, an improved mask R-CNN method was proposed for the detection and localisation of pulmonary embolism from CT images. The relationship between the loss function and the performance of the Mask R-CNN algorithm was determined. A local dataset was used in the study. Details of the proposed method are given in the second part of the article. Experimental results are presented in the third part.

1.1. Related Works

Thanks to the developing technological infrastructure, the performance in PE detection is increasing by using different algorithms. Early studies of pulmonary fixation included limited clinical applications and showed poor performance. In these studies, clinical findings were used instead of CT images as material. As a method, feature extraction was done with simple artificial neural networks [18,19,20]. However, in recent years, there are also studies using up-to-date methods for the automatic detection of PE [21,22]. These studies have shown that computer aided systems are successful in detecting PE. It has also been shown that these systems can accurately detect small PEs that may escape the eye of the specialist [22,23]. In a study, PE detection was made automatically by classifying the features of PE with K-Nearest Neighbors (KNN), Artificial Neural Network (ANN) and Support Vector Machine (SVM) algorithms. A sensitivity of 98% was obtained [24]. Machine learning algorithms have been widely used in recent years and have achieved high performance. There are many studies in which MR, CT images are analyzed with deep learning algorithms and much higher performances are obtained [25,26,27]. With convolutional neural networks, high performance can be achieved in PE detection via CTPA images [28,29]. Pham et al combined natural language processing and machine learning in diagnosing thromboembolic disease [30]. CT-based deep learning and automated PE studies have distinct challenges compared to their counterparts in other locations. For example, PE data represent only a small fraction, next to the size of the baseline CT data. There are also signal-to-noise problems when the intravenous contrast injection protocol and patient's breath-hold instructions are not followed [31,32]. For this reason, the correct creation of the neural network model and dataset used has a high effect on the performance. In particular, it is of great importance that the ground truths in the dataset are created correctly. Examples of recent studies conducted with deep learning-based techniques for the detection of PE are presented in Table 1.

2. Material and Methods

2.1. Material

The dataset was obtained from the Radiology Department of Kahramanmaraş Sütçü İmam University. Computed tomography images of 50 patients, 27 of whom were female and 23 male, were used. The age range of the patients is between 28 and 95. Images were obtained with TOSHIBA AQUILION ONE 320/640 Slice instrument. A raw dataset was created with a total of 430 1212*1212 images in .tif format by taking only PE-containing sections. In this study, the sections in which PE is displayed as the largest surface area are used . The CT scans are 8 bit (0-255) gray level images. An average of 9 PE-containing sections were taken from each patient (min=8 , max=12). Ethics committee report was obtained for the data set. All of the PEs in the images in this dataset were labeled by Kamil DOĞAN, one of the authors of the study, a 15-year expert radiologist, using the Matlab ImageLabeler toolbox.

2.2. Data Augmentation

The study only included patients with pulmonary embolism and no other accompanying tissues such lymph nodes. Images were acquired from a restricted sample of 50 patients. These images underwent data augmentation techniques. The most often utilized improvement methods include image rotation, contrast adjustment, vertical and horizontal mirroring, and zooming. This study utilized horizontal and vertical rotation (±90°) as well as horizontal and vertical mirroring approaches. The number of images has been multiplied by five.

2.3. Preprocessing

The dimensions of the raw images in the dataset are 1212 x 1212. PE is located in the middle region of all images in raw dataset. We obtained a 448x448 sub-image from the middle region of the raw image to cover the sections with pulmonary embolism. Figure 2 shows the raw image and sample subimage with PE.

2.4. Mask R-CNN Network Architecture

Region-based Convolutional Neural Networks (R-CNN) [33], Faster R-CNN [34] and Mask R-CNN [35] have high performance in object detection. Unlike other models, Mask R-CNN performs both detection and segmentation. This network is the extended version of Faster R-CNN network. as follows; Mask R-CNN also has an extra segmentation pattern (segmentation mask). There are two phases in Mask R-CNN. In the first stage, feature extraction is performed for the regions. In the second stage, bounding box detection, class detection and segmentation are performed according to the extracted features. The Mask R-CNN architecture includes a backbone network. Figure 3 shows the Mask R-CNN architecture.

As seen in Figure 4, Mask R-CNN has an FPN network that provides deep feature extraction. ResNet50 [36] was used as the backbone network in the study. The Feature Pyramid Network (FPN) is designed according to the pyramid concept. It is a network structure that stands out in speed as well as accuracy. It has a multi-scale feature map. It has an upstream and downstream network structure. The upstream path is a convolutional neural network for feature extraction. As the number of upstream layers increases, the semantic value increases. High level structures are detected. Since ResNet has a multi-layer structure, its training speed and estimation performance are quite high. In the basic structure of ResNet, there are hops between the front and back layers to assist back propagation in the training process of deep networks. After the FPN network comes the RPN network. Thanks to this network, regions containing possible objects are detected. The RPN region proposition is a deep convolutional neural network. The RPN takes any size of data as input and finds a bounding box bid based on the object score. It makes this suggestion by shifting a small mesh over the feature map generated by the convolutional layer. After the RPN network is ROIAlign. ROI alignment performs the same process as ROI pooling. However, Roi Align solves the problem of unnecessary offsets in segmentation problems by using bilinear interpolation. In this way, it achieves results much faster. Thanks to this layer, the image dimensions are rescaled and transmitted to the fully connected layer. After the fully connected layer, the class and bounding box information of each region is obtained.

2.5. Improved Mask R-CNN Network Architecture

In the study, Mask R-CNN structure was modified. Thus, the developed Mask was trained as an R-CNN network. In order to improve the detection of pulmonary embolism, two weighting parameters λ₁ and λ₁ are used in the loss function. The loss function consists of the sum of classification loss, positioning loss and segmentation loss as shown in equation 1.

$$L=L_{cls}+{{\lambda }_{1}L}_{box}+{{\lambda }_{2}L}_{mask}$$

(1)

L_cls corresponds to what level the classes detected incorrectly. In multi-object detection, this value should be increased if more than one object is not detected. Lbox is an adjustable parameter for the correct determination of the object's boundaries. This parameter can be increased if the bounding box is incorrectly positioned when the object is detected. On the other hand, λ₁, λ₂ weights the segmentation positioning losses of the object and thus the network performance can be improved. L_mask indicates how accurately the object is segmented. As shown in Equation 2

$$L_{mask}=-\frac{1}{m^2}\sum _{1\le i,j\le m}[y_{ij}{log\widehat{y}}_{ij}^k+\left(1-y_{ij}\right)y_{ij}{\mathrm{l}\mathrm{o}\mathrm{g}(1-\widehat{y}}_{ij}^k)]$$

(2)

is expressed as. Here y and $\widehat{\mathrm{y}}$ denote the label and predictive value, respectively.

2.6. U-Net

The performance of the improved Mask R-CNN algorithm is compared with the U-Net algorithm. For this reason, PE segmentation was also done with the U-Net algorithm. The working principle of the U-Net model is similar to the operation of autoencoders. The purpose of these models is to compress the input data and make the best learning with the least loss [37]. In this architecture, the number of neurons in the input and output layers is equal. However, the number of neurons in the hidden layer can vary. A U-Net based neural network is shown in Figure 4. If the structure of the network is examined, U-Net is a segmentation architecture built on a fully connected convolutional neural network.

Since it resembles the letter U in shape, it got its name from this letter. There are two parts in this architecture. The first part is known as contraction (encoding) and the second part is known as expansion (decoding). The first part, the coding part, is a traditional CNN architecture and the image size is slowly reduced using some convolution and maximum pooling layers. The convolution layer consists of 3x3 filters, and the maximum pooling layer consists of 2x2 filters. The second part, the decoding part, is completely symmetrical to the first part. In this part, the feature map is enlarged step by step by deconvolution towards the actual size of the image. Each convolution layer in the architecture is followed by an activation layer.

2.7. Evaluation Metrics

Sensitivity, specificity, accuracy, Dice and Jaccard were used to determine the performance of Mask R-CNN in detecting pulmonary embolism. Sensitivity refers to the proportion of PE within the detected pixels. A low sensitivity value indicates that true lesions are not adequately detected. On the other hand, high sensitivity values indicate that the system is able to detect a high proportion of the regions that are recognised as lesions. Evaluating different criteria together gives more accurate results. These criteria are expressed as follows. Figure 5 shows the true positive (TP), true negative (TN), false positive (FP) and false negative (FN) areas of the pixel groups obtained by automatic and manual segmentation. The equations for the performance parameters used are given in equations 3, 4, 5, 6 and 7 below.

$$Sensitivity=\frac{TP}{TP+FN}$$

(3)

$$Specifity=\frac{TN}{TN+FP}$$

(4)

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(5)

$$Dice=\frac{2\ast TP}{FP+2\ast TP+FN}$$

(6)

$$Jaccard=\frac{TP}{FP+TP+FN}$$

(7)

3. Experimental Results

In this study, thorax CT images of 50 patients with pulmonary embolism were used. Images containing a total of 430 pulmonary embolisms were created by taking the sections in which the pulmonary embolism was seen from these images. The study was carried out in four stages: data augmentation, preprocessing, PE segmentation, and performance evaluation. Preprocessed images were used as input to the Mask R-CNN network. Images of 36 patients (1505) were used for training, and images of 14 patients (645) were used for testing. Feature extraction for PE detection was performed with the ResNet50 convolutional neural network pre-trained with the COCO dataset. Both hold-out validation (70% train 30% test) and 10-fold cross-validation were used for performance evaluation. Figure 6 shows the automatically and manually segmented PE’s. The PE manually segmented by the expert doctor is shown in red and the PE automatically segmented by the proposed system is shown in blue. Mask R-CNN performs both detection and segmentation. The detection of PE is shown with yellow bounding box.

Manual and automatic segmentations are shown in colour in Figure 7. The regions identified by the expert but not detected by the system are shown in red. The regions detected by the system as PE but not by the doctor are shown in green. The pixels belonging to the PE detected in both ways are shown in yellow. It is seen that the proposed system detects PE with a high performance.

The average sensitivity, specificity, and accuracy values obtained from the test data with the proposed method for hold-out validation are given in Figure 8. The average sensitivity value was 96.4%, the specificity value was 93.5% and the accuracy value was 96.1%. It is seen that the proposed improved Mask R-CNN has a high performance in PE detection.

Dice and Jaccard similarity indices were also used in this study to determine the similarity between automatic and manual PE detection. The minimum and maximum Dice values were found to be 0.95 and 0.97 respectively. The minimum and maximum Jaccard values were found to be 0.88 and 0.91 respectively. Figure 9 shows the graph of the Dice and Jaccard values obtained for the test images. The mean values obtained for the test data are given in Table 2. The mean values of the Dice and Jaccard similarity indices were obtained as 0.96 and 0.90 respectively. The performance of the study was tested using both hold-out validation and 10-fold cross-validation. Table 2 shows the PE segmentation performance of Mask R-CNN and Enhanced Mask R-CNN. Enhanced Mask R-CNN was found to be more successful, especially when 10-fold cross-validation was performed.

The performances of improved Mask R-CNN and U-Net and classical Mask R-CNN in PE detection are compared as shown in Table 3. The results showed that the improved Mask R-CNN provided higher sensitivity, specificity and accuracy values than the classical Mask R-CNN and U-Net models. In particular, the fact that the sensitivity value was higher with the enhanced Mask R-CNN indicates that it has a higher success rate in detecting the pixels of the lesion compared to the other two methods.

The accuracy values obtained at different λ values are given in Table 4. The results obtained show that the highest performance for this study is λ₁=0.9, λ₂=0.8. The case of λ₁=1 and λ₂=1 is valid for classical Mask R-CNN. Here, λ₁ is the coefficient of the loss function of the bounding box (L_box) and λ₂ (L_mask) is the coefficient of the loss function of the mask.

The performance of the proposed improved Mask R-CNN and other methods in detecting PE is compared in Table 5. As can be seen, the proposed improved Mask R-CNN shows a high performance in detecting PE.

4. Conclusions

The location and size of the pulmonary embolism, known as a pulmonary artery occlusion, is important for the treatment of the disease. The study developed a computer-based system for detecting pulmonary embolism from computed tomography scans. The system uses a deep neural network trained on images in the local dataset. Modified Mask R-CNN was used as the network model. This modification process involves weighting the loss functions in the Mask R-CNN algorithm with different coefficients. The results obtained showed that the Mask R-CNN algorithm developed according to the classical Mask R-CNN and U-Net was more successful in removing the PE boundaries. In this study, only CT images with PE and without PE-like lesions such as lymph nodes were selected. Axial sections with PEs were used from these images. Segmentation of PEs in these sections was performed. It is possible that other regions may be detected as PEs in axial sections without PEs. The performance values obtained represent the correct extraction performance of the pixels of the PE. The study has highlighted that the performance of the modified mask R-CNN can be improved in object segmentation applications. Since the system determines the location and boundaries of the PE, it can help specialists make a decision about the PE.