A Temporal Difference and Cross-variate Fusion Network for Multivariate Time Series Classification

1. Related Works

1.1. Pattern-Based Multivariate Time Series Classification Algorithm

Distance-based methods, which typically use Euclidean distance (ED) or dynamic time warping (DTW) distance between time series pairs as a similarity measure, use traditional classifiers (e.g., SVM and NN) to classify MTSs. NN-DTW is proposed in [15], which combines DTW and NN classifiers, and has been shown to be a very strong baseline. The effectiveness of distance-based approaches is directly influenced by the choice of distance measures, which requires professional design of appropriate distances.

Feature-based methods classify time series by identifying words patterns[16] or shapelet patterns[17,18] in the time series. Such methods were originally proposed for UTS classification, but can be easily transformed and directly applied to MTS classification. However, the use of these feature-based methods heavily relies on the knowledge of domain experts and requires extensive preprocessing and feature engineering works[27]. Thus, applying feature-based methods to different time series datasets can be challenging.

Ensemble-based approaches achieve classification by ensembling multiple classifiers that focus on different perspectives[19,20]. The final classification result is obtained through weighted voting. Such models greatly improve the classification efficiency of traditional methods.

1.2. DL-Based Multivariate Time Series Classification Algorithm

Deep learning algorithms, which do not require complex data preprocessing or extensive feature engineering, can solve the problems of traditional methods. Specifically, deep learning methods have shown promising results for TSC. For instance, OS-CNN [28] uses Conv1D to extract local features of time series, while ROCKET and MiniROCKET employs numerous stochastic convolutional kernel transformations to extract features relevant to TSC[29,30]. InceptionTime [31] utilises inception structures to investigate multiscale time series representations. Wang et al. developed a technique based on recurrent neural networks and proposed the Adaptive Differential Evolution (ADE) algorithm to derive classifiers for time series[32]. Subsequently, there have been attempts to use a dual-network-based model, which consists of a local-feature extraction network and a relation extraction network in parallel. MLSTM-FCN uses a combined LSTM layer and an FCN layer augmented by squeezing and excitation modules to extract time series features from MTS and a softmax layer is then applied to predict the labels[26]. [33] proposed RAN, a residual attention net that consists of a ResNet-based feature network and a transformer-based relation network. This approach also shows promising performance on UCR datasets. A robust temporal feature network (RTFN) is proposed for extracting features in time series classification[34]. The network consists of a temporal feature network (TFN) with a residual structure, multiple convolutional layers, and a long short-term memory (LSTM) based attention network (LSTMaN). However, existing research fails to adequately capture the relationships between variables and overlook the explicit modeling of temporal differences in time series data. In next section, we will propose TDCFNet, which help us to harness the valuable insights from sequence segments with significant temporal difference and cross-variate relationship, enabling the extraction of more discriminative features to bolster subsequent classification tasks.

2. Method

2.1. Problem Statement

Given a set of multivariate time series $X=\left\{X_1,X_2,X_3,\dots ,X_N\right\}$ of N instances, we have multivariate time series data $X_i=\left\{x_1,x_2,x_3,\dots ,x_T\right\}\in {\mathbb{R}}^{T\times D}$, where T denotes the total sampling length of the sequence, and D is the number of variables of each sampling point $X_i$, which can be expressed as a matrix as follows:

$$X_i=\left(\begin{array}{cccc}{x}_{(1,1\mid i)}\hfill & x_{(2,1\mid i)}\hfill & \hfill & x_{(T,1\mid i)}\hfill \\ x_{(1,2\mid i)}\hfill & x_{(2,2\mid i)}\hfill & \cdots \hfill & x_{(T,2\mid i)}\hfill \\ \hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ x_{(1,D\mid i)}\hfill & x_{(2,D\mid i)}\hfill & \cdots \hfill & x_{(T,D\mid i)}\hfill \end{array}\right)$$

(1)

Note that univariate time series data are situations where D equals 1, while multivariate time series data are situations where D exceeds 1.

The goal of TSC is to accurately predict the class labels $y=\left\{1,2,3,\dots ,c\right\}$ from class c. Therefore, we construct a classifier F, thus we have:

$${\widehat{y}}_i=F\left(X_i\right)$$

(2)

where ${\widehat{y}}_i\in \{1,2,\dots ,c\}$. F can be trained by minimising the discrepancy between the predicted class ${\widehat{y}}_i$ and the ground-truth class $y_i$ . Here, we adopt a cross entropy loss which is a commonly used loss function for multi-class classification to reflect this discrepancy.

2.2. Model Architecture Overview

The proposed model framework is illustrated in Figure 2 It comprises of three components: the two Stream Diff-LSTM module, for extracting time-series features that contain information about dynamic differential changes, the cross-variate feature extraction network module, for extracting relational feature representations between variables, the feature fusion and multi-class classification module, which fuses the features of the first two components and passes the combined results to a fully connected layer for classification learning. To understand the algorithmic process of our model better, we present the pseudo-code as shown in Algorithm 1. In the following, we describe the implementation details of each part in detail.

Algorithm 1:The pseudo-code of TDCFN.

1. Input: A set of Multivariate time series X 2. Output: The probability P of each category 3. Initialize: Parameters of TDCFN 4. Forward Pass: 5. Obtain two levels of temporal feature representations $\mathbf{H},\widehat{\mathbf{H}}$ by applying Two Stream Diff-LSTM on X 6. Compute the output ${\mathbf{X}}_{embed}$ using the embedding layer: ${\mathbf{X}}_{embed}=F_{embedding}\left({\mathbf{X}}_i\right)$ 7. Compute the cross-variate feature extraction network module output $\mathbf{F}$ by applying depthwisen, channel-wise and dimension-wise convolutions and skip connection operations: $\mathbf{F}=F_{dim}\left(F_{channel}\left(F_{depth}\left({\mathbf{X}}_{embed}\right)\right)\right)+{\mathbf{X}}_{embed}$ 8. Acquire feature ${\mathbf{P}}_{LSTM}$ and ${\mathbf{P}}_{CFN}$ by global average pooling operation on $(\mathbf{H},\widehat{\mathbf{H}})$ and $\mathbf{F}$ respectively 9. Concatenate ${\mathbf{P}}_{LSTM}$ and ${\mathbf{P}}_{CFN}$ and obtain ${\mathbf{F}}_{con}$ 10. Fed ${\mathbf{F}}_{con}$ into fully connected layers to obtain the low-dimensional embedding 11. Apply softmax function to obtain the probability distribution P 12. Backward Pass: 13. Compute the loss using the predicted probabilities and the ground truth labels 14. Perform backpropagation to update the parameters of the entire network

2.3. Two Stream Diff-LSTM Module

The Recurrent Neural Network (RNN) is presently the most widely used deep neural network framework for processing sequence data. Unlike traditional feed-forward deep neural network models, RNN networks preserve the internal hidden state in the constituent network cells to handle sequential dependencies in capturing sequences. The standard RNN network expresses the update of the hidden state $\mathbf{h}$ as:

$${\mathbf{h}}_t=g\left(\mathbf{U}{\mathbf{h}}_{t-1}+\mathbf{W}{\mathbf{X}}_t\right)$$

(3)

where g represents a smooth and bounded activation function, such as the sigmoid function. $\mathbf{W}$ denotes the weight parameters from the input layer to the hidden layer at time step t. $X_t$ represents the input received by the model at time at time step t. $\mathbf{U}$ stands for the weight parameters of the hidden layer corresponding to time step $t-1$, while ${\mathbf{h}}_t$ and ${\mathbf{h}}_{(t-1)}$ denote the hidden state from the current time step t and previous time step $t-1$ respectively.

The Long Short-Term Memory (LSTM) network is a variant of Recurrent Neural Networks (RNNs) that aims to address the issue of long-term dependencies in traditional RNNs. It is a more sophisticated RNN model unit. Similar to RNNs, LSTM is composed of memory cells that store information about the input sequence. However, LSTM uses gate units to control the flow of information in and out of the memory cells. The structure of the LSTM memory cell is illustrated in Figure 3. Let ${\mathbf{x}}_t$ denote the input sequence at the current time step, and m represents the number of variables. ${\mathbf{h}}_t\in {\mathbb{R}}^n$ denotes the hidden state of the LSTM cell, and n is a hyperparameter of the LSTM model indicating the number of storage cells in the hidden layer. ${\mathbf{i}}_t\in {\mathbb{R}}^n$ denotes the input gate of the storage cell, ${\mathbf{f}}_t\in {\mathbb{R}}^n$ denotes the forgetting gate of the storage cell, and it is the output gate. ${\mathbf{g}}_t\in {\mathbb{R}}^n$ is the intermediate computation of ${\mathbf{c}}_t$ and ${\mathbf{c}}_t\in {\mathbb{R}}^n$ denotes the main information of the storage cell. $\mathbf{W}$ and $\mathbf{U}$ are trainable parameters. $\sigma$ and tanh denote the sigmoid activation function and the tanh activation function, respectively. ⊙ denote the elementwise product operation.The LSTM memory cell can be implemented as the following equation:

$${\mathbf{f}}_t=\sigma \left({\mathbf{W}}_f·\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_f\right)$$

(4)

$${\mathbf{k}}_t={\mathbf{c}}_{t-1}\odot {\mathbf{f}}_t$$

(5)

$${\mathbf{i}}_t=\sigma \left({\mathbf{W}}_i·\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_i\right)$$

(6)

$${\mathbf{g}}_t=tanh\left({\mathbf{W}}_g·\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_g\right)$$

(7)

$${\mathbf{j}}_t={\mathbf{g}}_t\odot {\mathbf{i}}_t$$

(8)

$${\mathbf{c}}_t={\mathbf{j}}_t+{\mathbf{k}}_t$$

(9)

$${\mathbf{o}}_t=\sigma \left({\mathbf{W}}_o·\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_o\right)$$

(10)

$${\mathbf{h}}_t=tanh\left({\mathbf{c}}_t\right)\odot {\mathbf{o}}_t$$

(11)

Two-Stream Network is a common approach in deep learning. Inspired by the Two-Stream CNN proposed by Karpathy et.al [35] and the Temporal Difference Network proposed by [36], we try to adapt the LSTM to Two Stream Diff-LSTM. Figure 4 shows our Two Stream Diff-LSTM cell structure. As shown in the figure, the traditional LSTM cell is extended into a two-stream network, including the difference stream and the original stream. In the difference stream, the feature representation of the difference is obtained by processing the difference time series with the update equation of LSTM. Given the time series $\mathbf{x}=\left\{{\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3,\dots ,{\mathbf{x}}_N\right\}$, we preprocess the time series to obtain the difference time series $\widehat{\mathbf{x}}=\left\{{\widehat{\mathbf{x}}}_1,{\widehat{\mathbf{x}}}_2,{\widehat{\mathbf{x}}}_3,\dots ,{\widehat{\mathbf{x}}}_N\right\}$, in the following way:

$${\widehat{\mathbf{x}}}_t=\left\{\begin{array}{c}0,\mathrm{if}t=1\\ {\mathbf{x}}_t-{\mathbf{x}}_{t-1},\mathrm{if}t=2,3,\dots ,N\end{array}\right.$$

(12)

where ${\mathbf{x}}_t,{\widehat{\mathbf{x}}}_{\mathbf{t}},{\mathbf{x}}_{\mathbf{t}-\mathbf{1}}\in {\mathbb{R}}^D$. After the difference processing, we obtain the temporal input containing information about the dynamic changes in the temporal dimension, which is important for learning temporal feature representation. Next, we use it to guide the operation of the gate mechanism of the LSTM in the difference stream, updating the cell as a way to model using (4)–(11). Specifically, for each time step of the difference data ${\widehat{\mathbf{x}}}_t$, we compute the state of the hidden layer of the cell ${\widehat{\mathbf{h}}}_t$ considered as a differential temporal feature describing dynamically changing information in the temporal dimension.

At the same time, we use this feature to guide the LSTM update of the original stream. Specifically, for the data ${\widehat{\mathbf{x}}}_t$ at each time step, we utilized ${\widehat{\mathbf{h}}}_t$ and the hidden layer ${\widehat{\mathbf{h}}}_{t-1}$ of the previous time step to participate in the learning of the LSTM cell’s representation by guiding the operation of the LSTM’s gate mechanism, including the input gate, forget gate, and the output gate, and updating the cell’s state. The original flow is formulated as follows:

$${\mathbf{f}}_t=\sigma \left({\mathbf{W}}_f·\left[{\mathbf{h}}_{t-1},{\widehat{\mathbf{h}}}_t,{\mathbf{x}}_t\right]+{\mathbf{b}}_f\right)$$

(13)

$${\mathbf{i}}_t=\sigma \left({\mathbf{W}}_i·\left[{\mathbf{h}}_{t-1},{\widehat{\mathbf{h}}}_t,{\mathbf{x}}_t\right]+{\mathbf{b}}_i\right)$$

(14)

$${\mathbf{o}}_t=\sigma \left({\mathbf{W}}_o·\left[{\mathbf{h}}_{t-1},{\widehat{\mathbf{h}}}_t,{\mathbf{x}}_t\right]+{\mathbf{b}}_o\right)$$

(15)

Eventually, we obtained two levels of temporal feature representations $\mathbf{H}=\left\{{\mathbf{h}}_1,{\mathbf{h}}_2,{\mathbf{h}}_3,\dots ,{\mathbf{h}}_n\right\}$ and $\widehat{\mathbf{H}}=\left\{{\widehat{\mathbf{h}}}_1,{\widehat{\mathbf{h}}}_2,{\widehat{\mathbf{h}}}_3,\dots ,{\widehat{\mathbf{h}}}_n\right\}$, where n is a hyperparameter of the LSTM model indicating the number of storage cells in the hidden layer.

2.4. Cross-Variate Feature Extraction Network Module

In this convolution module, we use the idea of depthwise separable convolution architecture [38] to implement the modelling of relationships across variables to extract temporal features. The key idea behind depthwise separable convolution is to split feature learning. The convolutional operation is decomposed into two distinct steps: the depthwise convolution and the pointwise convolution. In the depthwise convolution, each channel of the input is convolved separately, preserving the depth dimension. This operation allows for spatial convolution to be performed without altering the depth of the input. The resulting output from the depthwise convolution is then passed through the pointwise convolution, where $1\times 1$ convolutions are applied to adjust the number of input channels from the depthwise convolution to a desired new channel depth. The purpose of this architecture is to perform independent computations for each channel, allowing different representations of the data to be learned within each channel, which can better capture the complex relationships and characteristics of the data. Therefore, we use this idea of separate and independent representation learning to help us process time series.

First, we apply the input MTS to the embedding layer of the 1D convolution with padding. The embedding layer is able to effectively transform the input sequence data into embedding vectors with more advanced representation capabilities through the convolutional embedding process. This transformation can help neural networks to better understand and process sequence data. After embedding, the number of channels of the output tensor and the number of features per channel provide another multilevel representation of features, simultaneously, we preserve the temporal sequence of data across multivariate dimensions, as follows:

$${\mathbf{X}}_{embed}=F_{embedding}\left({\mathbf{X}}_i\right)\in {\mathbb{R}}^{D\times C\times L}$$

(16)

where D is the number of dimensions, L denotes the number of channels and L represents the length after embedding.

Then, inspired by the depthwise separable convolution architecture, we further decompose the convolutional operation into three distinct steps: depthwise, dimension-wise, and channel-wise convolutions. Specially, each step is implemented using group convolution, which divides the input channels into multiple groups, allowing for parallel processing of feature maps within each group. This parallelization strategy not only reduces computational complexity but also enhances feature diversity and representation capability.

In detail, firstly the depthwise convolution treats each dimension independently, performing convolutions along the temporal dimension while preserving the channels and depth dimensions. This step allows for capturing temporal patterns across different channels. Following the depthwise convolution, the channel-wise convolution focuses on convolving across the channel direction. This operation ensures that features are extracted efficiently within each channel while maintaining temporal coherence. Lastly, the dimension-wise convolution applies convolutions along the dimensions, enabling the model to capture inter-variate dependencies and interactions. This step enhances the model’s ability to learn complex relationships between different dimensions, leading to improved representation learning. Importantly, residual connections are incorporated by connecting the output of the last step to the final output, allowing the network to learn identity mappings more easily and effectively learn features. The structure is illustrated in Figure 5.

$$\mathbf{F}=F_{dim}\left(F_{channel}\left(F_{depth}\left({\mathbf{X}}_{embed}\right)\right)\right)+{\mathbf{X}}_{embed}$$

(17)

By breaking down the convolutional operation into these three steps, our model can effectively extract multi-level features from the input data, incorporating temporal, dimensional, and channel-wise information. This comprehensive feature extraction process contributes to the model’s robustness and performance across various tasks.

2.5. Feature Fusion and Multi-Class Classification Module

We fuse the features obtained from the previous two modules as inputs to the classification module. First, we use $F_{pool}$ represents the global average pooling operation, applied to the output of the two modules respectively, as follows:

$${\mathbf{P}}_{LSTM}=F_{pool}\left(\mathbf{H},\widehat{\mathbf{H}}\right)$$

(18)

$${\mathbf{P}}_{CFN}=F_{pool}\left(\mathbf{F}\right)$$

(19)

Then, we concatenate the pooling result ${\mathbf{P}}_{LSTM}$ and ${\mathbf{P}}_{CFN}$ to obtain the fuse feature ${\mathbf{F}}_{con}$, which can be represented as follows:

$${\mathbf{F}}_{con}=Concat\left({\mathbf{P}}_{LSTM},{\mathbf{P}}_{CFN}\right)$$

(20)

where $Concat(·)$ is the concatenate operation. Finally, we fed ${\mathbf{F}}_{con}$ into a fully connected layer and softmax function to do the multi-classify task. We used the cross-entropy loss function to train our model.

3. Experimental Results and Analysis

3.1. Dataset

The dataset we choose is the latest UEA multivariate time series classification archive [37], to evaluate the performance of the proposed model. The UEA multivariate time series classification archive, which serves as our benchmark,collects data from various real-world application domains, including ECG, EEG/MEG signal, Human Motion Recognition(Motion) , Human Activity Recognition (HAR), Audio Spectra (AS), and Other. The datasets vary in length from 8 (PEMS-SF) to 3000 (MOTORIMAGERY), in dimension from 2 (AtrialFibrilation, PenDigits) to 963 (PEMSSF), and in category from 2 (MotorImagery) to 39 (Phoneme). And the train sizes range from 12 (StandwalkJump) to 7494 (PenDigits). The details of the datasets are shown in Table 1.

3.2. Evaluation Metrics

This paper evaluates the performance of our proposed model and some other MSTC algorithms for each dataset using classification accuracy, average accuracy, and the number of Wins/Ties as evaluation metrics.

This section may be divided by subheadings. It should provide a concise and precise description of the experimental results, their interpretation as well as the experimental conclusions that can be drawn.

3.3. Experimental Performance

We evaluate our proposed method against eight benchmark approaches commonly used in multivariate time series classification. These benchmarks include traditional distance-based classifiers, recent models such as the bag-of-patterns approach as well as a deep learning method. Here are the details of the benchmark methods we consider:

• 1NN-ED and 1NN-ED(norm): These classifiers are traditional distance-based model, which utilize the one-nearest neighbor algorithm with Euclidean distance, both with and without data normalization. The Euclidean distance metric measures the straight-line distance between two points in a multidimensional space. This baseline method is widely used in time series classification tasks due to its simplicity and effectiveness.
• 1NN-DTW-i and 1NN-DTW-i(norm): These classifiers are traditional distance-based model,which employ dimension-independent Dynamic Time Warping (DTW) with and without normalization. The DTW algorithm computes distances based on the sum of DTW distances for each dimension of the time series data. DTW is a flexible and robust measure that accounts for variations in the temporal alignment of sequences, making it suitable for time series classification tasks.
• 1NN-DTW-D and 1NN-DTW-D(norm): These classifiers are traditional distance-based model, which utilize dimension-dependent Dynamic Time Warping (DTW-D) with and without normalization. Unlike the dimension-independent DTW approach, DTW-D directly computes the DTW distance based on multidimensional points, rather than treating each dimension separately. This variant of DTW is particularly useful when the dimensions of the time series data are interdependent and require joint consideration during distance computation.
• WEASEL+MUSE: This framework is feature-based model based on the bag-of-patterns (BOP) approach and represents the latest advancement in multivariate time series classification[16].
• MLSTM-FCN: This algorithm is one of state-of-the-art deep learning based framework designed for multivariate time series classification. The model architecture includes an LSTM layer, stacked CNN layers, and a Squeeze-and-Excitation block for feature extraction. We utilize the source code provided by the authors and execute the approach using default parameter settings, as described in the original work[26].

We evaluate the performance of each method based on three key metrics: classification accuracy, average accuracy, and the frequency of Wins/Ties across different datasets. The results are summarized in Table 2. Notably, TDCFN demonstrates superior classification accuracy, outperforming other methods on 11 datasets. Furthermore, TDCFN achieves the highest average classification accuracy, reaching 0.67. Regarding the number of Wins/Ties, TDCFN exhibits 11 instances of Wins/Ties, while ensemble model WEASEL+MUSE records 4 Wins/Ties. These results underline the effectiveness and robustness of TDCFN compared to alternative methods.

Specifically, our model outperforms or achieves comparable accuracy scores to existing methods across various datasets, demonstrating its effectiveness in multivariate time series classification tasks. For instance, on the AF dataset, our TDCFN model achieved a notable accuracy improvement compared to traditional methods such as ED-1NN and DTW-1NN-D. While the WEASEL+MUSE algorithm attained a higher accuracy score of 0.333, our TDCFN model demonstrated a significant enhancement with an accuracy of 0.400. This indicates the effectiveness of our model in capturing complex patterns within the AF dataset, which may involve intricate temporal dynamics and cross-variate interactions. Similarly, on the CT dataset, our TDCFN model showcased substantial improvement over other algorithms, achieving an accuracy score of 0.986. Although the WEASEL+MUSE algorithm also attained a high accuracy score of 0.990, our TDCFN model demonstrated competitive performance, outperforming conventional methods like ED-1NN and DTW-1NN-I. This suggests that our model effectively captures the distinctive characteristics of the CT dataset, which may involve nuanced temporal variations and inter-variable dependencies.Moreover, on the FD dataset, our TDCFN model exhibited significant advancement in accuracy compared to baseline methods. With an accuracy score of 0.566, our model surpassed the performance of ED-1NN and DTW-1NN-D, indicating its capability to discern subtle patterns and anomalies within the FD dataset, which might encompass diverse temporal patterns and complex interrelations among variables. Additionally, on the HMD dataset, our TDCFN model demonstrated remarkable accuracy enhancement, achieving a score of 0.391. While the WEASEL+MUSE algorithm also performed well with an accuracy of 0.365, our TDCFN model showcased superior performance compared to conventional methods like ED-1NN and DTW-1NN-D. This suggests that our model effectively captures the unique characteristics of the HMD dataset, which may involve intricate temporal structures and cross-variate dependencies inherent in human motion data.

Furthermore, in this experiment, we observed the loss reduction during the training process. By plotting the training loss curves for these datasets, we can clearly see a continuous decrease in the model’s loss, indicating effective convergence on these datasets. These observations provide strong evidence for the validity of our model and the stability of the training process. Figure 6 illustrates the results obtained from the two datasets.

3.4. Ablation Experiments

The above experiments show that TDCFN is a feasible solution for the MTSC. Moreover, ablation experiments are essential in assessing the individual contributions of components within a model. In our study, we conducted ablation experiments to investigate the impact of removing two key modules from our model: the Two Stream Diff-LSTM module and the Cross-variate feature extraction network module. Table 3 shows the three comparisons of the ablation experiments.

• Multilayer Perceptron(MLP) Classifier: In this ablation experiment, we constructed a model comprising solely a fully connected layer classifier, which is also called Multilayer Perceptron(MLP) Classifier, devoid of any additional feature extraction or processing modules. This experiment aims to serve as a baseline comparison to evaluate the significance of the feature extraction components present in the full TDCFN model.
• Without Two Stream Diff-LSTM module: This experiment involved removing the Two Stream Diff-LSTM module, which is responsible for capturing temporal features containing dynamic differential information. By conducting this ablation experiment, we aimed to understand the influence of this module on the overall performance of the model.
• Without Cross-variate feature extraction network module: In this experiment, we excluded the Cross-variate feature extraction network module, which is designed to extract relational features among variables. The purpose was to assess the significance of this module in enhancing the model’s ability to capture inter-variable relationships and representation.

Through these ablation experiments, we aimed to gain insights into the individual contributions of each module to the overall performance of our model. This analysis helps validate the effectiveness of our model design choices and provides guidance for further refinement and optimization. The experimental details are shown in Table 3. The table presents experimental results across different types of datasets, including ECG, HAR, AS, EEG/MEG, and Motion. The observed results demonstrate that our model outperforms all other configurations across all types of datasets. The average accuracy rate further corroborates the effectiveness of our model, achieving the highest average accuracy across all dataset types.

Across all datasets, we observed consistent improvements in classification accuracy when incorporating both the Two Stream Diff-LSTM and Cross-variate feature extraction network into our TDCFN model. For instance, on the AF dataset, which represents ECG data, the TDCFN model achieved a significantly higher accuracy of 0.400 compared to alternative configurations. This underscores the crucial role played by both the Two Stream Diff-LSTM and Cross-variate feature extraction network in effectively capturing temporal dynamics and cross-variate dependencies inherent in ECG signals. Similarly, on the BM dataset (HAR), our TDCFN model attained perfect accuracy (1.000), outperforming all alternative configurations. This highlights the indispensability of the proposed model architecture in accurately discerning complex human activity patterns, which may involve intricate temporal sequences and inter-variable relationships.

Overall, the ablation study results demonstrate the pivotal role played by the Two Stream Diff-LSTM and Cross-variate feature extraction network in enhancing the classification performance of our proposed TDCFN model across diverse multivariate time series datasets. Our ablation study provides valuable insights into the efficacy of individual model components and highlights the robustness of our approach in addressing complex time series classification tasks.

4. Conclusion

In this article, we proposed a novel deep learning algorithm for multivariate time series classification task, named TDCFN. The key idea of TDCFN is to fuse two stream Diff-LSTM and cross-variate feature extraction network, aiming to leverage both temporal dynamics and inter-variable relationships in multivariate time series data. Through extensive experiments on various benchmark datasets, we demonstrated the effectiveness and robustness of TDCFN in capturing complex patterns and making accurate predictions across different application domains. Furthermore, we conducted ablation studies to analyze the contributions of each component in our model. TDCFN presents a promising solution for multivariate time series classification, offering a comprehensive framework that effectively integrates temporal dynamics and inter-variable relationships.