Automated Quantification of Pneumonia Infected Volume in Lung CT Images: A Comparison With Subjective Assessment of Radiologists

1. Introduction

Computed tomography (CT) is the most popular medical imaging modality used in the clinical practice to detect lung diseases (i.e., lung cancer, chronic obstructive pulmonary disease, interstitial lung diseases, pneumonia and others). In order to more accurately assess severity of many lung diseases and predict patients’ prognosis, estimation of disease infected volume and/or its percentage to the total lung volume plays an important role. However, subjective estimation of disease infected regions or volume by radiologists is quite difficult, tedious, and inaccurate (due to the large intra- and inter-reader variability), which make it often infeasible in the busy clinical practice. Thus, in order to help solve this clinical challenge, developing computer-aided detection (CAD) schemes or methods has been attracting broad research interest. For example, the CAD-generated lung density mask has been well developed and tested to quantify percentages of emphysema infected lung volume [1] or degree of lung inflammation [2]. However, quantifying other lung diseases such as pneumonia infected lung volume has not been well developed and evaluated. Thus, we investigated feasibility of developing new CAD schemes that can automatically segment pneumonia infected regions depicting on CT image slices and quantify the percentage of the diseased lung volume, which has potential to more accurately and efficiently assist radiologists in disease diagnosis and assessment of disease severity.

The Coronavirus disease (COVID-19) was first diagnosed in late 2019, this respiratory disease originated from SARS-CoV-2 virus has infected millions of people globally [3] and also produced pneumonia-type diseases. Chest X-ray radiography and CT are two radiological imaging modalities to assist diagnosis of COVID-19 induced pneumonia, and more importantly to monitor its severity and spread of the infection [4]. While chest X-ray images are easier and faster to taken with low cost, CT scan is highly preferred mainly due to its three-dimensional nature and additional information to improve diagnostic accuracy [5]. Thanks to the availability of test kits, the diagnosis of COVID-19 is not a challenging task, but, on the other hand, the assessment of the infection severity and spread to lung regions and cause pneumonia-type infection is very demanding, which can help determine severity of infection and predict disease prognosis or long-term consequence or residual side-effect. However, manual delineation of lung infections and distinguishing between the opacity and texture of infected regions is challenging, time consuming, and highly subjective, which can be influenced by the radiologist’s bias and clinical experiences [5]. Additionally, it is also difficult for radiologists to distinguish between COVID-19 and other viral pneumonia [6].

In developing CAD schemes of lung CT images, deep learning (DL) models have been well recognized and widely used recently to perform the tasks of segmenting the regions of interest (ROIs) and detecting variety of lung diseases using the automatically extracted image features. However, DL algorithms or models generally need large datasets for training to achieve high accuracy and robustness [7]. Unfortunately, it lacks available image datasets with pixel-level annotation of disease regions in medical image research field since the reliable manual annotation of medical images is extremely time-consuming and difficult without heavy involvement of the experienced radiologists [5]. Hence, most of previous studies have focused on developing DL models to classify between COVID-19 and normal or other types of pneumonia cases [8,9,10], which has been found not clinically acceptable [11]. Thus, in order to overcome the disadvantages of previous “black-box” type CAD or DL models, developing new CAD or DL models to automatically segment the pneumonia infected regions due to the COVID-19 with interactive graphic user interface (GUI) is important to increase transparency and allow radiologists to visually inspect the segmented the infected lesions or regions, which can provide radiologists a visual-aid DL tool to assist their decision-making in order to more accurately and robustly quantify the severity and spread of COVID-19 generated pneumonia infection.

In order to test the feasibility and potential clinical utility of quantifying COVID-19 infected pneumonia volume using a novel DL approach or model, the overarching objective of this study is to investigate and compare the agreement between manual and automated segmentation of COVID-19 infected regions, ratings on disease severity, and acceptance level of radiologists to DL-generated lesion segmentation results. Following sections reports brief information of this study.

2. Materials and Methods

2.1. Datasets

In this study, three chest CT image datasets were used, which include two public datasets namely, “COVID-19 CT scans” and “COVID-19 CT segmentation dataset” [12]. The first public dataset includes 20 CT scans of patients diagnosed with COVID-19 from two sources, Coronacases [13] and Radiopedia [14]. Although numerous COVID-19 image datasets are publicly available, one unique characteristic of the dataset selected in this study is that all CT images have been annotated by experts providing three separate masks for the left lung, right lung, and infection. The second public dataset contains 100 axial CT images acquired from more than 40 COVID-19 patients. A mask with three labels is provided by a radiologist for each CT image indicating ground-glass opacity (GGO), pleural effusion and consolidation regions. These two datasets were used to build and/or train the DL model of segmenting and qualifying the disease infected regions or volumes. Additionally, another independent testing dataset is also assembled to test and evaluate the trained DL models, which includes 80 CT scans of COVID-19 patients acquired from” Hospital Regional III Hanorio Delgado” Arequipa, Peru.

2.2. Image preprocessing

In order to achieve higher reliability or robustness of the DL model, several image preprocessing techniques were employed to initially remove clinically unrelated images and normalize remaining images. First, the “COVID-19 CT scans” dataset includes whole CT images of COVID-19 patients. However, some slices of each CT scan (i.e., in the beginning and near the end of scan) usually contain very little lung area, thus not providing helpful information. Including these CT slices in the training data leads to a more unbalanced dataset. Thus, we removed up to 10% of CT images at the beginning and near the end of each CT scan. Generally, all lung infection datasets are unbalanced since the number of infection mask pixels are significantly less than the pixels of the healthy lung and other normal tissues presented in the image. In order to create a more balanced training dataset, we removed all healthy CT slices with no infection mask.

Second, image normalization is another important preprocessing step when training deep neural networks. Thus, all CT images were normalized by clipping the intensities outside the range [600, -1024] HU, (if x > max, then, x' = max. if x < min, then, x' = min ), and the remaining values were scaled between zero and one (x' =(x-x_min)⁄(x_max-x_min )).

Third, we applied the data augmentation technique to generalize and enlarge the dataset and mitigate overfitting. The main augmentation method adopted in this study is Elastic Transform [15] which is commonly applied in biomedical image analysis. The python library Albumentations [16] was used to perform the Elastic Transform and other affine transformations. Along with the elastic Transform, we also applied horizontal flip, Random rotate, and vertical flip. Figure 1 demonstrates changes of a CT slice after applying an augmentation method in this study.

Last, another preprocessing technique was adding different filters as a channel to the CT image. Several filters have been tested with various channel arrangements to pronounce different textures and structures and consequently achieve better discrimination between healthy and infected regions. Contrast Limited Adaptive Histogram Equalization (CLAHE) is one of the filters that has been applied as a channel to the CT images. CLAHE is a variant of adaptive histogram equalization that limits contrast amplification to reduce noise amplification. This filter performs histogram equalization in small patches with high accuracy and contrast limiting. Figure 2 illustrates the effect of applying a CLAHE filter on a CT image.

2.3. Image segmentation models and output

Several common deep neural network models were used in this study, including UNet [17], Feature Pyramid Network (FPN) [18], and Attention Residual UNet [19]. The Segmentation Models library [20] available on GitHub was used to test various segmentation models with different backbones and parameters more conveniently. For each model, many parameters have been tested and modified, including loss functions, fixed and variable learning rates, encoders and decoders, and dropout rates.

2.3.1. Lung segmentation

The first step is to segment lung area depicting on each CT slide. For this purpose, a publicly available model for lung parenchyma segmentation was used to create lung masks and segmenting the lung area [21]. In brief, this model used the U-net with the only adaption being batch normalization after each layer. Figure 3 demonstrates an example of the created lung mask and the lung segmentation result using this mask.

2.3.2. Infection area segmentation

The next step is to segment the disease infected lung regions (from fuzzy ground glass to consolidation patterns), regardless of the severity and development stage, which is one of the primary objectives of this study. Therefore, various object detection and segmentation models with different hyper-parameters have been employed to achieve the highest accuracy. First, the Attention Residual-UNet (AR-UNet) is selected to build the ensembled model in this study. AR-UNet model is an end-to-end infection segmentation network, which embeds attention mechanism and residual block simultaneously into the U-Net architecture. Hence, this model efficiently balances limited training data. In this model, Attention path employs the attention mechanism to capture spatial feature details. Residual block involves the semantic information flow through a 1×1 convolution [22].

Based on literature search and our experiments, we recognize that among many tested loss functions, the Binary cross-entropy loss and the Tversky loss [21] led to the best predictions. Binary cross-entropy is calculated as the following formula (1) [23]. Where t_i is the truth value of either 0 or 1, p_i is the SoftMax probability for the ith class [23].

$$L_{BCE}=-{\sum }_{i=1}^2{t}_i\log \left(p_i\right)$$

(1)

In order to compute the Tversky loss function, a SoftMax along each voxel is applied [21]. Let P and t be the predicted and truth binary labels, respectively. The Dice similarity coefficient (D) between two binary volumes is identified and computed using formula (2):

D (P, t) = 2|Pt|/ (|P|+|t|)

(2)

Since in most cases non-lesion voxels outnumbers the lesion voxels, one of the main challenges in medical imaging is imbalance data especially in lesion segmentation. Therefore, using the unbalanced data in training lead to predictions that are severely biased towards low sensitivity (recall) and high precision, which is not desired particularly in medical applications where false-positives (FPs) are much tolerable than false-negatives (FNs). To achieve an optimum balance between sensitivity and precision (FPs vs. FNs) we used a loss layer based on the Tversky index. This index allows us to put emphasis on FNs and leads to high sensitivity. Using the formula (2) in a training loss layer, it equally weighs recall and precision, FN and FP, respectively [21]. To weigh FNs more than FPs in the training of a network with highly imbalanced data where small lesions’ detection is essential, a loss layer based on the Tversky index is efficient. The Tversky index is computed as the formula (3) [21]:

Ti(P,t,α,β)=|Pt|/(|Pt|+β|P⁄t|+α|t⁄P| )

(3)

where α and β control the magnitude of penalties for FNs and FPs, respectively. Hence, the finally used Tversky loss function is defined as follows using formula (4) [21]:

$$L_T\left(\alpha ,\beta \right)=\frac{{\sum }_{i=1}^N{p}_{0i}{v}_{0i}}{{\sum }_{i=1}^N{p}_{0i}{v}_{0i}+\beta {\sum }_{i=1}^N{p}_{0i}{v}_{1i}+\alpha {\sum }_{i=1}^N{p}_{1i}{v}_{0i}}$$

(4)

In the above equation, p_0i and p_1i are the probability of voxel i lesion and non-lesion, respectively. Additionally, v_0i is 1 for a lesion and 0 for a non-lesion voxel and vice versa for the v_1i [21].

Since image segmentation accuracy and robustness depend on choosing and use of DL models along with optimal training parameters, in order to more accurately and robustly segment disease infection areas or blobs depicting on chest CT images, we have developed, tested and compared five models based on AR-UNet with different training parameters as summarized in Table 1. Additionally, based on the hypothesis that if five models contain complementary prediction scores of pixels belonging to disease infected area, fusion of the predictions of all five selected models can further improve image segmentation results (i.e., prevent under-segmentation as much as possible). While involving several models comes with a longer processing time, the more reliable and precise prediction is worth the extra time.

2.3.3. Segmentation of GGO and Consolidation patches

Moreover, besides the overall infected region segmentation, it is of great importance to distinguish between different stages of COVID-19-infected pneumonia developments in lung and provide better assistance to radiologists. The “COVID-19 CT segmentation dataset” provides manual annotations with 3 infection types, GGO, pleural effusion and consolidation. Since the pleural effusion type is not of great interest in this study, we only included the GGO and consolidation labels into the training dataset. Like the infection segmentation model, we tested various neural network architectures and hyper parameters aiming to achieve the best predictions. We have applied an FPN model to categorize different stages of the COVID-19 in the infected area. This model has 23,915,590 trainable parameters. As depicted in Figure 4 the patch segmentation is based on ResNet and FPN model. ResNet34 is a backbone and FPN is the feature extractor network.

Although, the staging model tends to over-segment the GGO regions, the consolidation segmentation is very accurate. In order to prevent the over-segmentation on GGO area, the infection segmentation model is used to constrain the staging model. This model classifies each patch to three classes of Background, GGO and Consolidation.

2.3.4. Integrated model and GUI

Therefore, three common deep neural network architectures have been employed in this study. For lung segmentation, we have applied a publicly available model for lung parenchyma segmentation based on the UNet model. Additionally, FPN models along with AR-UNet, were developed for infection segmentation since the attention blocks have been shown to be very beneficial in image segmentation [19]. Moreover, an FPN model applied to categorize the severity of the COVID-19 infected area. For each model, many parameters have been tested and modified, including loss functions, fixed and variable learning rates, different encoders and decoders, and dropout rates. All models are written in Python and the TensorFlow library is used to train and test the models.

After extracting the lung and infected lesions by the two segmentation models, the percentage of the infected lung volume is reported along with the average Hounsfield units (HU) inside infected region, which can indicate the density of the lesion of interest and hence the severity of infection. This information is reported for the left and right lungs for each CT slice as well as the whole CT.

Finally, in order to assist radiologists in diagnosis of COVID-19 infected pneumonia using the DL model generated quantitative results or predictive scores, we also designed a stand-alone graphical user interface (GUI) as a “visual-aid” tool, which can be installed on any windows-based computers without the need for any specific programing language or library. Figure 5 illustrates the flow diagram of the developed DL model method and GUI tool.

2.4. Image preprocessing

After observing the output of the lung segmentation model, it was noted that in several cases with sever disease infection, small percentage of the lung may be missing from the segmentation as shown in Figure 5-a, which are typically represent disease infection area. In order to include or recover the missed lung area if the lung segmentation error is visually observed from our GUI, the user (i.e., radiologist) can call an image a specially-designed post-processing function that applies a unique conventional image processing algorithm inspired by the rolling ball algorithm [24] to automatically correct segmentation error. This algorithm starts with extracting the lung contours followed by several steps and morphological filters such as disk drawing, filling holes, median, and erosion operations. As shown in Figure 6, it can convert a jagged and rough lung boundary as shown in Figure 6-a to a smooth one which covers the previously missed lung area as shown in Figure 6-b. While it might lead to a small over-segmentation in some cases, the previously missed area contains very important infected lesions that can significantly affect the assessment of sever cases.

2.5. Evaluation

In order to objectively evaluate new DL model performance, the model was first tested “as is” using an independent dataset of 80 new test CT scans. Next, we asked two expert chest radiologists to retrospectively read and review these 80 sets of CT images. Each radiologist read and examined half of the CT scans (40 patients) and reported the patient infection spread in percentage based on their judgment of the percentage of infected lung volume. These subjectively assessed values are then collected and compared to the values generated by the DL model. It is important to note that in this new testing image dataset of 80 clinical cases, there are no manually annotated lung and disease infection area segmentation marks. Thus, no Dice coefficients can be computed, and we only compared the agreement between radiologists and DL model in predicting percentage of disease infected lung area (or volume) based on the predicted result of infection area ratio or spread scores between radiologists’ assessment and DL models.

Moreover, in order to test radiologists’ confidence level to accept DL-generated infection area segmentation results, we also show radiologists the DL segmentation results displayed on the developed GUI and ask them to rate acceptance level of the infection area segmentation of each CT slice with a score of 1 (poor segmentation) to 5 (excellent segmentation).

Last, we also ask radiologist to assign each patient to two groups of mild infection cases that are dominated with GGO and severe infection cases that have significant fraction of consolidation areas or blobs. We then compared agreement between DL-model generated case classification results and radiologists’ classification results. A corresponding confusion matrix was generated for the comparison and diagnostic accuracy computation.