The Multi-Branch Deep-Learning-Based Approach to Heart Dysfunction Classification

1. Introduction

Cardiovascular diseases (CVDs) are one of the most common causes of death globally. An electrocardiogram (ECG) signal representing the heart’s electrical activity is widely used to detect and classify cardiac arrhythmias. Visual inspection of the ECG waveform allows anomalies to be detected. It is crucial in the early stages of heart disease when symptoms occur only occasionally. In such cases, continuous monitoring by wearable devices and an automatic classification process increase the chances of proper detection. To increase the efficiency of ECG analyses, methods for automatic detection of ECG abnormalities are used instead of a physician visually reviewing the ECG waveform periodically. This way, when the algorithm detects disease signals, it classifies them, and the doctor can be notified.

Originally, algorithms based on morphological features and classical signal-processing techniques were commonly used. However, these solutions turned out to be insufficient, as the ECG waveform’s morphological characteristics can have high variability between different patients and are susceptible to interference during measurement [1,2]. Nowadays, deep-learning-based algorithms using recurrent neural networks (RNNs) and convolutional neural networks (CNNs) dominate the field [1,3,4,5]. Recent publications prove hybrid or extended variations of said architectures [6,7] to excel in the ECG analysis task. Another approach gaining traction is the use of attention-based models, where by employing convolutional layers combined with attention, the CNN autonomously extracts detailed morphological and temporal patterns from raw multi-lead ECG data – often removing the need for handcrafted features while delivering higher accuracy and robust cross-dataset performance [8,9].

As ECG classification and prediction are problems in time-series analysis, one of the most commonly used classifiers in that field is the Long Short-term Memory (LSTM) network [10]. The LSTM network is known for its ability to learn long-term dependencies from sequential data, which allows it to be leveraged to profit architectures focusing on both temporal changes and detailed signal artefacts [11]. Still, time-series forecasting continues to be a pivotal challenge in the field of machine learning, and despite the rapid evolution of contemporary algorithms, classical statistical methodologies are still commonly used. One of the interesting new solutions is a hybrid approach integrating a neural residual / attention-extended LSTM architecture with the conventional Holt-Winters statistical model augmented by learnable parameters that have gathered increasing attention [12]. Another new solution is Neural Basis Expansion Analysis for Interpretable Time Series (N-BEATS) [13,14] – one of the newest neural networks used primarily for forecasting time series. In [15], N-BEATS, which was previously used only for forecasting, was proposed to diagnose cardiac problems and compared with commonly used LSTM and GRU networks for such problems. N-BEATS achieved similar results in the experiments and worked exceptionally well for a low number of ECG leads.

In [16,17], a two-branch architecture with two blended sub-networks and wavelet analysis, was shown as superior to common N-BEATS, LSTM and GRU architectures. In this article, this idea is further developed into a new architecture model and instead of adding domain knowledge or other information as a single input with ECG signal, a multi-branch architecture is proposed with up to 6 branches containing different inputs, like raw signal, domain knowledge, the wavelet transform of the signal or drift removal. Multi-branch architecture containing independent branches proved to obtain better results in all tested scenarios, improving the overall performance of neural networks. The paper uses three basic neural networks (LSTM, GRU, N-BEATS) used interchangeably. For the sake of clarity they will be denoted as core networks.

2. Proposed Approach

2.1. Multi-Branch Architecture

In this paper, we follow the approach from [17], where a concept consisting of core networks with two blended sub-networks and wavelet analysis was introduced. Here we propose a novel architecture model. Instead of combining domain knowledge or other information with the ECG signal into a single input, multiple inputs are prepared individually (as shown in Figure 1), and a complex network architecture with six independent branches is proposed, each containing a core network and dedicated input (as shown in Figure 2). The branches of the model are denoted further in this paper as $\alpha ,\beta ,\gamma ,\delta ,ϵ,\zeta$.

Popular solutions from state-of-the-art publications were used as inputs for individual architectural branches. The Raw ECG signal used as input on every branch was subject to denoising, which was performed using wavelet thresholding. The wavelet of choice is Daubechies, the most popular choice for denoising ECG signals [18]. Signals from different datasets were resampled to a common target frequency of 500 Hz (more on the process in Section 3.1). As shown in Figure 1 for computing inputs to $\beta ,\gamma ,\delta ,ϵ$ branches, baseline drift was removed from a denoised signal with the SNIP algorithm. In order to compute the input to the $\zeta$ branch, nonlinear baseline wandering was removed from the denoised signal. The inputs are presented in more detail later in this section.

The data individually preprocessed make the input for dedicated branches consisting of one or more core network blocks with FC (fully connected) layers. Dropouts are added in order to prevent overfitting and improve generalisation [19]. Outputs of those branches are concatenated and used as input for a fully connected neural network layer (FC) to produce probabilities of all output classes. The model of the multi-branch architecture is shown in Figure 2.

Different modalities of the original signal, obtained by described above preprocessing, extends the information resource used to identify the particular signal sample. The details concerning branches is given below.

• The $\alpha$ branch contains the processed ECG signal. As the region surrounding the R peak is critical for diagnosing cardiovascular diseases (CVDs) [16], an R peak detection algorithm is employed for signal segmentation. In this study, we utilized the commonly used Pan-Tompkins algorithm [20] to facilitate both R peak detection and the subsequent segmentation process. Digital ECG recordings were divided into 3-second-long windows surrounding detected peaks, with exactly 1s of the signal prior to peak appearance and 2s afterwards.
• The $\beta$ branch contains information that encapsulates both time and frequency domain information. 4th level wavelet decomposition using type second Daubechies wavelet [21] was performed on the downsampled (by a factor of 2) ECG signal. Obtained approximation and detail coefficients are concatenated to form an input vector $\beta =(A4,D4,D3,D2,D1)$, where $A4$ is an approximation vector and $D1-4$ are detail coefficients.
• The $\gamma$ branch represents professional expertise and domain knowledge of medical authorities [22] defined as indexes and statistics calculated to gain insights, which could potentially be hard to learn by the network. We employed McPhie, Romhilt, Cornell, Lewis and Sokolov-Lyon indexes as well as Cornell product, cardiac rhythm origin (split into two indicators of horizontal and vertical directions), electrical heart axis, and QRS complex statistics like QRS duration, R-S slope duration, notched QRS complex analysis, missing QRS, missing P-wave analysis and heart beats per minute [23,24,25,26]. As most of the aforementioned indicators relate to a single QRS complex across different leads and multiple heartbeats may appear during the 3-second window, we pad each index vector to allocate up to 25 beats. It forms an input matrix of shape 12x256, as shown in Figure 3.
• The $\delta$ branch contains the principal component analysis (PCA) [27] computed on the vector of the combined downsampled version of input $\alpha$ (a raw ECG signal), concatenated with input $\beta$ (wavelet coefficients) and input $\gamma$ (domain knowledge representation).
• The $ϵ$ branch contains ECG signal (as in $\alpha$ branch) with slow baseline drift [28] removed in preprocessing. Slow baseline drift removal removes inherent human- and hardware-related predicaments from the recordings.
• The $\zeta$ branch contains ECG signal (as in $\alpha$ branch) with nonlinear baseline wandering [29] removed in preprocessing. Nonlinear baseline wandering is a technique used to remove sudden signal disruptions and other anomalies (for example, moving the electrode) caused by the patient or the medical personnel from the recordings.

The architecture leverages the advantages of the bootstrap mechanism combined with different input perspectives. By merging the outputs of independent branches, which do not share insights in respective inputs of data, in the Blend layer, the proposed multi-branch architecture enhances the cumulative performance, improving classification outcomes with minimal increase in the total complexity of the solution.

2.2. Core Networks

Each branch consist of the core network, which is the main processing engine (see Figure 2). As the core network one may use different deep-learning models. Data processing by the core network can be performed multiple times on more than single block of the same network, which increases model efficiency, which was shown in [17]. In our research we tested three models used as the core network: LSTM, GRU and NBeats.

The LSTM architecture, proposed in 1997 by Hochreiter and Schmidhuber [30], is a variant of recurrent neural network in which each memory cell holds its state with a self-connection (the constant error carousel), while trainable input and output gates decide when to write new data into the cell and when to release its contents. By guaranteeing uninterrupted gradient flow over extended sequences, this design overcomes the vanishing gradient problem and lets the model selectively retain or discard information, making it exceptionally well-suited for capturing long-range dependencies in sequential data.

Gated recurrent unit (GRU) was first introduced in 2014 by Cho et al. [31]. While retaining LSTM’s gated approach to managing information flow, the GRU relies on just two gates - an update gate that blends the previous hidden state with a newly proposed activation and a freshly introduced reset gate to control incorporation of previous state—eliminating separate memory cells, lowering parameter count, easing training, yet still ensuring robust gradient propagation for capturing long-range dependencies.

In [13], Oreshkin et al. introduced the N-Beats architecture, designed explicitly for interpretable time series forecasting for the architectural model. The cornerstone of this framework is a multi-layer fully connected (FC) network, wherein non-linearities are introduced via the ReLU activation function. This network is tasked with predicting the basis expansion coefficients in both the forward (forecast) and backwards (backcast) directions. Individual blocks are systematically arranged into stacks through a doubly residual stacking strategy, potentially incorporating layers that share both backcast and forecast functionalities. The learned signal waveform represented by backcast is then removed from the original signal, and the result is passed further to the next block. Thanks to that, successive blocks learn to interpret exclusive parts of the signal. This hierarchical aggregation of forecasts facilitates the construction of a remarkably deep neural network while preserving the interpretability of its output and leveraging the benefits of the recurrent-like architecture.

3. Experiments

3.1. Datasets

Our experimental analysis utilized combined training dataset comprising over 88,000 annotated twelve-lead ECG recordings originating from six publicly available sources [32,33,34,35,36,37] as shown in Table 1. These recordings exhibit considerable variability in duration, ranging from 5 seconds to 30 minutes, and in sampling frequencies, including 250 Hz, 257 Hz, 500 Hz, and 1000 Hz. Notably, the majority of the samples are 10 seconds and recorded at 500 Hz. As such, signals with a sampling frequency lower than 500Hz were interpolated, and signals with a sampling frequency of 1000Hz were downsampled to match the target of 500Hz. Each of the six datasets encompasses 26 scored classes, comprising normal rhythm and 25 cardiac abnormalities, and up to 104 unscored classes. The heterogeneity in label quality and the presence of inaccuracies within these datasets contribute to the complexity of the challenge. The discussed sets have a signal from 12 ECG leads, and the analysis of such signals is used in the work because it provides the best results. However, due to the different quality of the sets, in order to protect the core networks from incorrect signals (where some leads are zeroed or lost in part of the signal), at the stage of pre-processing within the domain knowledge vector, marking of erroneous QRS readings and distances between peaks is used. The applied pre-processing allows the use of the architecture of both sets, containing full 12-lead ECG signals as well as those subject to external interference, leading to unsuccessful reads.

For comparative analysis of results, we employed cross-validation techniques by partitioning the entire dataset into five folds for training and testing purposes. Network weights were initialized using a fixed seed to ensure consistent randomization of the validation dataset for each training iteration, facilitating the selection of the optimal number of training epochs. The early stopping technique applied to the validation dataset guided the determination of epochs for a given training set. Subsequently, the model was trained across all training data within each fold, utilizing the initialized network weights from the fixed seed. As accuracy is not a feasible measure for the problem of multi-label classification, model performance was evaluated by calculating the average F-measure, and the challenge metric (see Section 3.3 for details) used in the CinC 2021 challenge.

3.2. Scenarios

To thoroughly verify the capabilities of the proposed solution, we introduced six scenarios described below.

• Scenario A — The first set of experiments was performed to establish the benchmark results and obtain a starting baseline for further comparison. We used a single-branch architecture and tested all branches separately, using only the LSTM (as the most often used) architecture as the core network. The basis of this scenario is an extension of the work published previously [17]. In our experiments, we employed a pruned ECG signal sample of length 0.7s (0.2s before the peak and 0.5s after the R peak).
• Scenario B — The second experiment tests possible differences between core network architectures working on different branches. As such, for each of the core networks, tests of single branches were performed for four branches: $\alpha ,\beta ,\delta$ and $ϵ$. Omitted in the experiment were the branches $\gamma$, as domain knowledge alone proved to be inefficient, and $\zeta$, because $ϵ$ branch baseline drift removal worked better. An enhanced input signal window (3s of signal) was used, and dropout layers were added.
• Scenario C — core network multi-branch architectures were tested with dropout layers embedded in the model. This comparison aimed to check if the combination improves model performance despite some branches offering no performance increase alone when all inputs are presented independently. This way, it can be checked if different perspectives (in the form of differently prepared independent inputs) can provide some improvement, even when such input alone works worse than the raw ECG signal.
• Scenario D — During this set of experiments, we measured and compared the performance of deeper architectures with an increased number of neurons in hidden layers. Multi-branch architecture was used for each of the core networks.
• Scenario E — In the third scenario, we test a multi-branch architecture with LSTM only with and without dropout layers and different pruning approaches. The goal was to establish a strategy to increase the network’s generalizing capability and improve overall performance while reducing the risk of overfitting.
• Scenario F — This experiment is an ablation study. The aim is to verify whether the exclusion of a given single branch from the initial complete set of six branches would change the overall performance of the architecture. Thus, it can be checked if any information from one of the independent branches could be considered excessive and removed to increase the architecture’s performance.
• Scenario G — Based on the results from scenarios D and E, additional tests on all three larger core networks were conducted, using different pruning strategies to check if the results from scenario D can be improved by applying different pruning techniques, especially for the underperforming networks.

3.3. Measuring the Performance

Following related papers submitted during the CinC (Computing in Cardiology) 2021 Challenge (now George B. Moody PhysioNet Challenge), to score results in a way that would allow comparing results to other state-of-the-art approaches, we apply a metric proposed by the CinC Challenge organizer.

As in paper [40], the following notation is used: $C={\left[c_i\right]}_{i=1}^m$ is a set of m distinct diagnoses for a database of n recordings. $A=\left[a_{ij}\right]$ is a multi-class confusion matrix where $a_{ij}$ is the normalized number of recordings in a database that were classified to class $c_i$ but originally belonged to class $c_j$. Different entries have assigned different weights $W=\left[w_{ij}\right]$ defined in [40] based on the similarity of treatments or differences in risks. The score $s={\sum }_{i=1}^m{\sum }_{j=1}^m{w}_{ij}{a}_{ij}$ is a generalized version of the traditional accuracy metric. Finally, this score is normalized so that a classifier that always outputs the true class or classes receives a score of 1, and an inactive classifier that always outputs the normal class receives a score of 0:

$$s_{normalized}={\displaystyle \frac{s-s_{inactive}}{s_{true}-s_{inactive}}},$$

where $s_{inactive}$ is the score for the inactive classifier and $s_{true}$ is the score for the ground-truth classifier.

The challenge metric was proposed for the complex multi-label classification problem of 26 different CVDs. It includes a matrix of weights that states how harmful misclassification is between each pair of diseases. One of the aims stated in the [40] is that a classifier that returns only positive outputs will typically receive a negative score, i.e., a lower score than a classifier that returns only negative outputs, reflecting the harm done by false alarms.

3.4. Results

The experiments were performed on computational servers with a 24-core processor clocked at 3.2 GHz, 256 GB RAM clocked at 3.2 GHz, and four NVIDIA Quadro RTX 5000 ADA 32 GB GPUs.

To thoroughly test the proposed approaches we performed experiments in 6 scenarios, which are described below.

• Scenario A – a test of single-branch architecture with LSTM as the core network. Table 2 shows results for six different branches (different data inputs) tested separately. The core network was configured with two hidden layers, with seven neurons each. As can be seen, branches $\gamma ,\delta ,\zeta$ perform worse than the baseline raw signal ($\alpha$ branch), while branches $\beta$ and $ϵ$ obtain better results.
• Scenario B – tests possible differences between core network architectures working on different branches. Single branch tests were performed for branches $\alpha ,\beta ,\delta ,ϵ$, comparing LSTM, GRU, and N-BEATS architectures, and the results are provided in Table 3. Omitted in the experiment were the branches $\gamma$, as in Scenario A, domain knowledge alone proved to be inefficient, and $\zeta$, because $ϵ$ branch baseline drift removal worked better. Each of the core networks was configured with two hidden layers, with seven neurons each. An enhanced input signal window (3s of signal) was used, and dropout layers were added. Dropout is a regularization technique for neural networks that prevents overfitting by randomly setting a subset of activations to zero during training, aiming to reduce co-adaptation of neurons, improving generalization [41]. Similarly to the experiment from scenario A, also in this scenario, the best results were achieved for the $\beta$ and $ϵ$ branches. The obtained results were relatively consistent across all core networks, but it is worth noting that GRU tended to have the highest variance for branches, which gave the best results. Also, the addition of a dropout layer (not used in Scenario A) and larger signal window improved the LSTM network’s results of the $\alpha$ branch so that they were almost the same as for the $ϵ$ and for this network, only $\beta$ gave much better results.
• Scenario C – test comparing the multi-branch architectures of core networks. LSTM, N-BEATS, and GRU networks were used with a full six-branch architecture with a dropout layer and 5% pruning each epoch. Each of the core networks was configured with two hidden layers, with seven neurons each. The experiment shows that a full multi-branch architecture obtains better results for the 12-lead ECG signal (Table 4) than an architecture based on any of the single branches (Table 3), for each of the core networks. As can be seen, both GRU and LSTM achieved the same results in this scenario, with GRU being slightly faster and having more consistent results.
• Scenario D – tests how core networks performed after size increase. The network from Scenario C is modified, and two hidden layers of size 7 were changed into two hidden layers of size 11. The same dropout layer and 5% pruning each epoch are used. As can be seen in Table 5, despite an increase in network size, the time needed for classification was almost the same, while both the F-measure and challenge score for the two networks (N-BEATS and LSTM) significantly improved. An interesting observation is the worsening of the results of the GRU network, which obtained slightly worse results. The same result was obtained when the experiment was repeated. A careful inspection of the network weights showed that the weights for two branches ($\delta$ and $\zeta$) were practically zeroed within this architecture.
• Scenario E – test prune and dropout strategies on LSTM core network. The test is conducted on the multi-branch network with all six branches, and results are shown in Table 6. The core network was configured with two hidden layers, with seven neurons each. The best results were achieved for two configurations, both with an added dropout layer, in one case without pruning, and in the other with 20% L1 regularisation performed once. Other solutions, with the exception of one-time random 20% pruning, which considerably worsened the results, obtained the same or only slightly better results than the lack of prune and dropout.
• Scenario F — Results of an ablation study (single branch exclusion to check if any information can be considered excessive and degrading performance) tested on LSTM as the core network with a dropout layer from Scenario E. The results are provided in Table 7, and as can be seen, no branch removal improved the architecture, but each worsened the obtained results.
• Scenario G – extended version of scenario D with incorporated information about prune and dropout strategies from scenario E. The results are provided in Table 8, no pruning strategy was more effective than used in scenario D. As in scenario D, in each case N-BEATS improved performance, but was the worst solution, while GRU worked worse than for a smaller architecture.

4. Discussion

The results obtained during the experiments confirm that the multi-branch architecture with independently preprocessed inputs provides better results than the single-branch architecture in the complex problem of multi-label classification of CVDs.

Based on scenarios A and B, it can be seen that $\beta$ and $ϵ$ are the most important single inputs, i.e. 4th level wavelet decomposition using type second Daubechies wavelet of the ECG signal and ECG signal with slow baseline drift removed, which were the only branches to obtain better results than the raw ECG signal.

The results obtained in the paper in scenarios B, C and D are consistent with the previous results from [17], indicating the advantage of LSTM and GRU over N-BEATS in detecting cardiovascular diseases for full 12-lead ECG signals. In all cases N-BEATS was outperformed by both GRU and LSTM, suggesting that this network performs better for simpler classification problems (like the 2-lead ECG signal classification presented in [17]).

Scenario E showed that the presented architecture for the problem did not have excessive tendencies to overfit, and there is no need to provide complicated pruning mechanisms. In the experiments, it turned out that adding only a dropout layer or a dropout layer and a one-time prune with 20% L1 regularization is an entirely sufficient solution, while for larger networks 5% L1 pruning each epoch obtained the best results as shown in Scenario G.

As the experiments from scenarios A and F have shown, providing the model with additional information on subsequent input branches, even such information that alone causes worse classification results, improves the overall performance of the classifier.

The ablation study conducted in scenario F showed the impact of excluding individual branches. In no case did removing branches, even the least useful ones in scenario A, such as $\gamma$, improve the classification result, but only slightly worsened it. Removing the $\gamma$ branch was also the only case where a significant time gain was observed for the classification of a single time window (from 0.40 ms to 0.24 ms). In cases where classification speed is important, it is possible to consider removing this branch. Otherwise, it can be assumed that the proposed architecture works best with complete available information.

Comparing LSTM networks on the exact network sizes, the best result obtained for the single-branch architecture is 0.441 (at 0.09 ms for a single window), 0.465 (at 0.40 ms) for the multi-branch architecture or 0.456 (at 0.24s) for the multi-branch architecture with the removed $\gamma$ branch.

The results from scenario D are interesting, wherewith the increase of the network size, a significant improvement in the classification result was observed for two of the three tested networks (from 0.456 to 0.486 and from 0.335 to 0.390) while maintaining practically the same classification time window in the case of LSTM (from 0.40ms to 0.41ms) or even its reduction in the case of N-BEATS and GRU (from 0.53ms to 0.51ms and from 0.37 ms to 0.36ms). This shows that the multi-branch architecture offers greater optimization possibilities for the selected problem and the possibility of further improving the classification result for the same core networks in cases where the single-branch architecture cannot obtain better results.

The proposed architecture will work well in complex problems, where, in addition to the signal, there is also a lot of additional information and/or it is possible to clean/process the data in different ways. It is a universal structure that offers easy extensibility with additional branches to make use of all available information.

As can be seen in Table 9, the proposed architecture applied to RNN allowed for improvement of obtained results, but for the challenge problem, CNNs still work better. As such, further research includes generalizing the architecture to other types of neural networks (including networks with attention blocks), starting with CNN variants used in state-of-the-art for the challenge problem.