Efficient Lung Ultrasound Classification

1. Introduction

Artificial Intelligence (AI) and specifically Computer Vision, is having remarkable developments in recent years, allowing software programs to obtain meaningful information from digital images. Medicine is a sector in which the experimentation and use of this technology are experiencing strong growth [1,2,3]. Also, considering that in the year 2020, only in the United States of America, a production of 600 million images was reported [4] and that this number seems to grow steadily, it is increasingly necessary to process this data using robust and trustworthy algorithms developed through the collaboration of medical staff, engineers, and physics at different levels. The SARS-CoV-2 pandemic has again highlighted the need for a quick and safe response, as well as cheap, and to this aim, one of the most peculiar case studies that emerged is the use of point-of-care ultrasound (POCUS) to detect SARS-CoV-2 pneumonia, that is the use of sonography exams in loco instead of dedicated facility [5,6]. In general, the preferred methods to assess pulmonary diseases are both X-Ray or Computed Tomography (CT) due to their high image quality and diagnostic power. Nonetheless, ultrasound appears to detect signs of pathologies in the lung as well or better than CT [7,8,9,10,11]. Even, in some cases, there is a directed mapping between evidence found by ultrasound and CT [12]. The necessities dictated by the SARS-CoV-2 pandemic moved the attention from a precision diagnostic approach to one that aims to maximize a trade-off between accuracy, time, and safety, crucial aspects in “a real emergency situation”. Physicians and researchers around the world adapted their point-of-view towards a direction where the employment of lung POCUS seemed a natural solution for both quarantine subjects and hospitalized ones [8,13,14]. Naturally, CT and Magnetic Resonance Imaging (MRI) are far more precise and reliable exams, but both have downsides which cannot be ignored. Specifically for CT, a notable downside is the associated ionizing radiation, while both exams are more time-consuming and affected by subject cooperation. In the case of POCUS, the trade-off between pros and cons is hugely in favour of the former: it is portable, safe, cheap and repeatable for a lot of medical fields. As a downside, the generated images are noisy and deeply affected by operator experience so interpretation can be less direct than CT or MRI. Artificial Intelligence can be used to narrow this gap; before 2019 [15], deep learning methods were used on ultrasound datasets efficiently [16,17], but not specifically on lung disease detection. In the aftermath of the SARS-CoV-2 pandemic, an increase in articles about this topic can be noted. The underlying reasons are evident: necessity, scientific purpose and a new abundance of data.

In medicine, deep learning algorithms applications are mainly used for classification and segmentation tasks, and, as emerge from recent literature reviews [15,18], they seem to perform efficiently in the field of lung ultrasound. The study about classification focuses above all on the use of AI to discriminate between chest ultrasound images of healthy and pneumonia-afflicted subjects [19]. The vast majority of these studies investigate the possibility of discerning between SARS-CoV-2 and bacterial pneumonia [20]. Among the most performing approaches [11] employs a transfer learning approach, pre-training a network with a more consistent lung CT/MRI dataset. Segmentation approaches studies mostly focus on the detection of bio-markers [21] and signs that are important in POCUS diagnosis (A-lines, B-lines, etc...)[22] Most recent and interesting works provide solutions to discriminate LUS among healthy, pneumonia and SARS-CoV-2 conditions with good results [23,24] and the work described in [25] also provides an explanation of results. Since the classification of medical images entails decisions and actions involving human beings, it is crucial that the system is secure and understandable, that it is used as a second reading and that the final decision must be taken by a clinician. To this aim, an explanation of the results obtained by an automatic system must also be provided. Indeed, the use of POCUS as a diagnostic tool was an argument of debate [9,26,27].

In this paper, we aim to introduce new methods and models for the analysis of LUS images to distinguish between SARS-CoV-2, pneumonia and healthy conditions. The proposed new model, using EfficientNet-b0 [28] as a core, is based on a recent strategy for ensembling at the deep features level [29,30]. It reaches a state-of-the-art (SOTA) accuracy of 100% on a well-known public reference dataset. The proposed model is computationally efficient with a relatively low number of parameters and Floating Point Operations (FLOPs), making it applicable in principle for real-time operation in the point-of-care scenario. As an additional contribution, the explainability of the ensemble model is investigated with a preliminary analysis of activation maps. Such research contributes to understanding models and providing visual explanations of the obtained results.

The paper is organized as follows: in Section 2 we introduce the used dataset and the network architecture proposed, current results are reported in Section 3 and discussed in Section 4, while Section 5 ends the paper with conclusions and ideas for future work.

2. Materials and Methods

2.1. Dataset

The dataset used for this task is, to our knowledge, the largest publicly available LUS (Lung UltraSound) dataset1, comprising ultrasound videos and images from 261 different patients among 41 different sources. The data have been collected, cleaned and reviewed by medical experts. In particular, for the sake of this work, we used the frame-based version when the frames of each video are considered. In Table 1 data distribution is described and some examples for each class are shown in Figure 1. More details about the whole dataset (e.g. patient distribution, acquisition technique, sources) are described both in the article [25] and in the GitHub repository of the project.

2.2. Validation pipeline

First of all, end-to-end training using transfer learning [31] from ImageNet pre-trained models [32] is performed, and then the ensemble is fine-tuned. The description of details follows.

2.2.1. EfficientNet

Since ultrasound has the advantage to be processed in real-time, the efficiency of the processing plays a crucial role. For this reason, in this work EfficientNet-b0 [28] is used as a core model because, in our opinion and based on previous data [29], it is the architecture with the best accuracy/complexity trade-off. The efficiency of this architecture is given by two main factors: (i) the inverted bottleneck MBConv (first introduced in MobileNetV2) as a core module, which expands and compresses channels reducing the complexity of convolution and (ii) the compound scaling by which input, width and depth scaling are performed in conjunction since it is observed that they are dependent.

2.2.2. Hyperparameters

After a previous investigation, in order to reduce the search space, some hyperparameters have been fixed:

Input size:

set to 512×512 because it is the best resolution in order to have less computational cost without losing image details.

Batch size:

set to the maximum available using our GPU (32GB RAM) which is 50 for end-to-end and 200 for fine-tuning.

Regularization:

to prevent overfitting early-stopping is used, and patience is set at 10, because deep models have relatively fast convergence and they usually start overfitting early, no more patience is needed.

Optimizer:

AdaBelief [33] with learning rate $5·{10}^{-4}$, betas (0.9, 0.999), eps ${10}^{-16}$, using weight decoupling without rectifying, to have both fast convergence and generalization.

Validation metric:

Weighted F1-score which better takes into account both errors and data imbalance.

Dataset split:

75/10/15 respectively for train/valid/test subsets.

Standardization:

data are processed in order to belong to a distribution with values around the mean and unit standard deviation, improving stability and convergence of the training.

Then a grid search on the following hyperparameters is performed:

Seeds:

5 seeds are used for the end-to-end training (seeds affect both subset splitting and classification layer initialization) and 5 seeds for ensemble the fine-tuning (affecting the combination layer initialization only);

Interpolation:

bilinear, bicubic and Lanczos for both end-to-end and fine-tuning.

2.2.3. Efficient Adaptive Ensemble

Ensembling is the way of combining two or more models (i.e. weak models) in order to have a new combined model better than the weak ones [34]. In this work, ensembling is performed in this way:

• select the best two end-to-end trained models (i.e. weak models);
• remove the classification layers of the weak models;
• freeze the parameters of the weak models;
• initialize a fresh combination linear layer;
• train the ensemble model (i.e. fine-tune the combination layer) by usual gradient descent.

This kind of ensemble is (i) efficient because only the combination layer is fine-tuned and there are just two weak models that can be executed in parallel since their processing is independent and (ii) adaptive since the ensemble and, especially, its combination layer are trained according to the data.

3. Results

For the sake of robustness and comparability with the SOTA, experiments have been performed using Stratified 5-fold Cross Validation, keeping the frames of a single video belonging to one fold only and having the number of videos per class similar in all folds. In this way, every fold is treated as an independent task as described in Section 2.2.

The result on the single folds are reported in Table 2, Table 3, Table 4, Table 5 and Table 6 showing that Ensemble obtains 100% accuracy at every run in all folds, then since now we focus on end-to-end weak models results. Fold 1 (Table 2) seems to be the hardest having no runs with 100% at every subset, the Fold 2 (Table 3) is the easiest with 100% accuracy at every run while the remaining folds are in the "average".

Last but not least, according to Table 8, the proposed method improves the SOTA, reducing the number of parameters and FLOPS.

4. Discussion

Dataset issues were extensively discussed in its official presentation paper [25]. The method here presented for classifying COVID-19, Pneumonia and Healthy lung ultrasounds outperforms the SOTA: its importance is in terms of the number of parameters and on complexity that are lower than other methods, pointing the way to an efficient and fast classification system that can be embedded in real-time scenarios. On one side, the dataset used is undoubtedly the biggest one publicly available; on the other side, it would be better to test the proposed method on even larger and more heterogeneous datasets to prove its validity.

To better understand the behaviour of our model, we applied the Gradient-weighted Class Activation Mapping (GradCAM)[35] algorithm producing visual explanations: as the saliency maps show, our model seems to produce reasonable explanations since in all the cases it is focusing on meaningful areas. On the other side, when we apply the same method to weak models which didn’t obtain the 100% of accuracy, it is clear that the model concentrates on areas of the LUS that are less or not important. Indeed, we generally noticed that an accurate classification focus on (see a representative example in Figure 2):

• “evidence” usually at the upper side of the image and concentrated activations in case of COVID-19;
• “evidence” everywhere (mainly lower part) with relaxed activation in case of Pneumonia;
• mainly the healthy part of the lung (black) with very expanded activation in the case of Healthy.

A sample of saliency maps of every possible combination of wrong-correct predictions are shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8.

5. Conclusions

An artificial intelligence method was presented to classify LUS videos into healthy, COVID-19 or Pneumonia automatically: the proposed method matches both advantages (e.g. portability, safety) and disadvantages (e.g. challenging interpretation) of LUS. An efficient adaptive ensembling model based on two EfficientNet-b0 weak models achieved an accuracy of 100% on the largest publicly available LUS dataset, improving the performance with respect to the previous SOTA, maintaining the same order of magnitude of an EfficientNet-b0 model. Saliency maps were presented, showing that the proposed network uses significant areas of the LUS images to discern between the analysed class. Future investigations will focus on further improving our adaptive efficient ensembling model and applying it to classify other important signs in LUS we are acquiring in an ongoing telemedicine project [36].