Reevaluating the Potential of a Vanilla Transformer Encoder for Unsupervised Time Series Anomaly Detection

1. Introduction

Time series anomaly detection is to identify unusual patterns or events in time series data [1,2]. This is an important problem across many domains like finance, manufacturing, and healthcare [3,4]. Due to its importance, this problem has been a focus of research for several decades. Initially, statistical methods [5,6,7,8] and machine learning methods [9,10,11,12] were studied. Recently, with advancements in deep learning, there has been increasing interest in applying deep learning models to time series anomaly detection [13]. In particular, unsupervised time series anomaly detection has attracted significant attention due to the issue of collecting abnormal data in real-world scenarios [14,17,18,19].

In unsupervised time series anomaly detection, numerous models have been proposed. Some studies have focused on capturing temporal dependencies using RNNs and LSTMs [15,20,21], while others have used CNNs and TCNs to extract temporal features locally and hierarchically [22,23,24]. Additionally, Graph Neural Networks have been employed to model complex relationships in multivariate time series data [25,26,27]. Generative models, such as Variational Autoencoders, Generative Adversarial Networks, Normalizing Flows, and Diffusion models, have also been explored. These models detect anomalies by analyzing reconstruction errors or deviations from learned data distributions [16,28]. Autoencoders, in particular, are widely adopted due to their simplicity and effectiveness in capturing temporal features [30].

Transformers, originally introduced in natural language processing (NLP), demonstrated superior performance compared to previous models [31]. Since then, it has been successfully applied across diverse domains, proving its effectiveness. Recently, the success of Transformers has driven extensive research into their application for unsupervised time series anomaly detection [32,33,34,35]. In particular, the Transformer encoder has gained attention for its ability to simultaneously capture temporal dependencies and complex patterns within time series data. However, due to the inherent complexity of time series data, many studies have attempted to improve a vanilla Transformer encoder by introducing novel techniques. These improvements have achieved superior performance on various benchmarks [32,36,37,38].

In contrast to these trends, this study revisits the potential of the vanilla Transformer encoder for unsupervised time series anomaly detection. We argue that, despite its simplicity, the vanilla Transformer encoder, when combined with thoughtful preprocessing techniques and design choices, can achieve competitive performance with state-of-the-art models. This research proposes a minimalist framework that leverages a vanilla Transformer encoder and a simple linear decoder to perform anomaly detection effectively. Our experiments demonstrate that this approach highlights the often-overlooked potential of the vanilla Transformer encoder in handling diverse time series anomlay detection tasks.

2. Materials and Methods

This work aims to revisit the potential of a vanilla Transformer encoder in unsupervised time series anomaly detection. To evaluate the capabilities of a vanilla Transformer encoder, we propose an autoencoder-based framework equipped with a vanilla Transformer encoder and a linear layer decoder(see Figure 1). This framework makes use of a simple architecture while enabling a vanilla Transformer encoder to effectively capture temporal dependencies. This framework takes time series segments, generated from (preprocessed) time series data, as input and reconstructs the taken time series segments. The proposed framework consists of the following modules: preprocessing time series, generating time series segments, segment-level normalization-denormalization, time series segment embedding, and an asymmetric encoder-decoder.

2.1. Preprocessing Time Series Data

Preprocessing plays a vital role in the development of a deep learning framework. We are concerned with normalization techniques for preprocessing in the context of unsupervised learning-based time series anomaly detection. They are used to convert the scale of time series data into a specified interval, which usually significantly impacts the performance of the anomaly detection process.

Representative normalization techniques include min-max normalization and z-score normalization. Let $X\in {\mathbb{R}}^{T\times V}$ represent time series data, where $x_{ij}$ denotes the value at the i-th time step of the j-th variable. Its normalized time series data $\tilde{X}\in {\mathbb{R}}^{T\times V}$ can be expressed in $\tilde{X}=\varphi \left(X\right)$, where $\varphi (·)$ is a normalization method which is defined as follows:

$$\varphi \left(x_{i,j}\right)={\displaystyle \frac{x_{i,j}-{min}_k\left\{x_{i,k}\right\}}{{max}_k\left\{x_{i,k}\right\}-{min}_k\left\{x_{i,k}\right\}}}\max \text{-}\min \mathrm{normalization}$$

(1)

$$\varphi \left(x_{i,j}\right)={\displaystyle \frac{x_{i,j}-{\mu }_{·,j}}{{\sigma }_{·,j}}}\mathrm{z}\text{-}\mathrm{score}\mathrm{normalization}$$

(2)

where ${\mu }_{·,j}$ and ${\sigma }_{·,j}$ represent the mean and standard deviation of the j-th variable, respectively.

This study emphasizes selecting the most suitable normalization method for each dataset rather than applying a uniform approach, to better capture temporal features. In model development, we consider the following three options: no use of normalization, min-max normalization, and z-score normalization. We choose the best option by examining the performance of each option for the given dataset.

Some datasets are composed of multiple sub-datasets (see Table 1 and Table 2). For instance, the multivariate dataset MSL consists of 27 sub-datasets, and the univariate dataset ABP consists of 42 sub-datasets. For datasets consisting of multiple sub-datasets, we apply two levels of normalization sequentially: first at the sub-dataset level, followed by the dataset level, with three normalization options available at each level. Sub-dataset level normalization is performed on each sub-dataset individually, while dataset-level normalization is applied across all sub-datasets as a whole. The total nine combinations of normalization methods for the sub-dataset level and the dataset level are examined to select the best one. At inference time, parameters such as min, max, mean, and standard deviation, computed during training, are utilized.

2.2. Generating Time Series Segments

Time series datasets often consist of long sequences dynamically collected in real-world environments. These characteristics can make it difficult to directly feed raw time series data into an outlier detection model. To address this, techniques for generating fixed-length segments from the original data are employed, facilitating easier analysis and processing by the model.

The windowing technique is a commonly used method for generating time series segments in unsupervised anomaly detection for time series data. It involves copying fixed-length segments from an original time series data at regular intervals using a sliding window. Varying the window length and stride enables the generation of different sets of time series segments from the data. The process of generating a full set of time series segments from preprocessed data using the windowing technique can be outlined as follows:

$$\mathbf{x}=\mathcal{W}\left(\tilde{X}\right)$$

(3)

where $\mathcal{W}(·)$ is a window function, $\mathcal{W}:{\mathbb{R}}^{T\times V}\to {\mathbb{R}}^{N\times L\times V}$, $\mathbf{x}$ and $\tilde{X}$ denote the time series sigement and its preprocessed dataset, respectively.

The optimal window length and stride size for capturing temporal dependencies effectively can vary across datasets. To address this, we adopt window lengths and stride sizes tailored to each dataset. Furthermore, stride size can be applied differently during training and evaluation. During evaluation, the stride size matches the window length, following the non-overlapping approach used in prior studies such as [20,37,38,39]. In contrast, during training, the stride size is selected based on whether an overlapping or non-overlapping approach is more effective for capturing temporal dependencies. This approach ensures efficient temporal pattern extraction during training.

2.3. Segment-Level Normalization and Denormalization

Real-world time series data are often non-stationary, with statistical properties that change over time. This non-stationarity poses significant challenges in extracting meaningful temporal features. Since time series segments are derived from non-stationary data, they inherently retain this non-stationarity. To address these challenges, we use a segment-level normalization and denormalization technique, designed to effectively capture temporal dependencies in non-stationary time series segments. As shown in Figure 1, the normalization is applied to time series segments prior to their embedding. The embedded time series segments pass through the encoder and the corresponding time series segments are reconstructed by the decoder. The reconstructed one is denormalized into the scale of original time series segments.

We employ a reversible instance normalization method [40], originally proposed for time series forecasting to address distribution shifts between training and testing datasets, for normalizing and denormalizing time series segments. Given batched time series segments ${\mathbf{x}}^{\prime }\in {\mathbb{R}}^{B\times L\times V}$ with batch size B, the sequence length L and the number of variables V, the normalization method is applied to multivariate time series segments as follows:

$${\tilde{\mathbf{x}}}_{i,:,j}={\alpha }_j\left({\displaystyle \frac{{\mathbf{x}}_{i,:,j}^{\prime }-{\mu }_{i,j}}{{\sigma }_{i,j}}}\right)+{\beta }_j$$

(4)

where $\tilde{\mathbf{x}}\in {\mathbb{R}}^{B\times L\times V}$ represents the normalized time series segments, ${\mu }_{i,j}$ and ${\sigma }_{i,j}$ denote the mean and standard deviation of the data points of the j-th variable within a time series segment i, and ${\alpha }_j$ and ${\beta }_j$ are learnable parameters used for scaling and shifting of the j-th variable. This normalization might mitigate the impact of non-stationality, while preserving the temporal characteristics of time series segments.

The encoder and decoder are trained to reconstruct the normalized time series segments at the output of the decoder. For normalized time series segments reconstructed by the decoder, denormalization is carried out as follows:

$${\widehat{\mathbf{x}}}_{i,:,j}={\sigma }_{i,j}\left({\displaystyle \frac{{\overline{\mathbf{x}}}_{i,:,j}-{\beta }_j}{{\alpha }_j}}\right)+{\mu }_{i,j}$$

(5)

where $\widehat{\mathbf{x}}\in {\mathbb{R}}^{B\times L\times V}$ represents the final denormalized reconstructed time series segments produced by the model, while $\overline{x}$ represents the normalized reconstructed time series segments generated by the decoder.

2.4. Time Series Segment Embedding

Time series segment embedding is needed to transform time series segments into a format suitable for the proposed autoencoder-based architecture which consists of the encoder and decoder shown in Figure 1. The embedding module converts the data points of dimension V within time series segments into real-valued vectors of dimension F. To capture both individual data point information and short-term temporal dependencies within time series segments, we use a 1D convolutional layer for embedding. That is, the embedding method generates an embedding vector e by considering individual data points along with their neighboring data points for a time series segment $\overline{x}$.

$$\mathbf{e}=\mathrm{Conv}1\mathrm{d}\left(\tilde{\mathbf{x}}\right)$$

(6)

where $\mathbf{e}\in {\mathbb{R}}^{B\times L\times F}$ is a batch of B sequences, where each sequence consists of L embedding vectors with a feature dimension of F. The 1D convolutional layer consists of F kernels, each with a length of 3, a stride of 1, and no bias term, to generate an embedding vector of dimension F. Zero padding is applied to ensure that the sequence length of the embedded time series segment matches that of the original time series segment.

Additionally, we choose not to use positional encoding (PE), which is commonly employed in Transformers to incorporate positional information for extracting position-based features. This decision is based on two key considerations. First, positional encoding works by injecting additional positional information into the data. However, this process may lead the model to interpret the sequential flow of the data based on the positional encoding rather than its original order. As a result, the model may fail to properly learn the natural sequential relationships in the data. In reconstruction-based methods, preserving the original sequence is crucial, and such distortions can negatively impact reconstruction performance. Second, a reconstruction model, such as an autoencoder, aims to reproduce the input data as accurately as possible. Adding positional encoding introduces new patterns to the data, which might cause the model to learn these patterns inappropriately or to overlook essential structural information from the original data by overemphasizing positional differences. This may impair the model’s reconstruction performance, resulting in degrading its anomaly detection capabilities. To assess the impact of positional encoding, we have conducted an ablation study as part of our analysis (see Subsection 3.4).

2.5. Asymmetric Autoencoder

Autoencoders are widely used in unsupervised time series anomaly detection because of their simplicity and capability to reconstruct input data. An autoencoder consists of two components: an encoder and a decoder. The encoder extracts features from the input data, while the decoder reconstructs the input data from these extracted features. According to Wang et al. [41], an overly powerful decoder can hinder the encoder’s ability to effectively capture meaningful features. This is because the decoder may accurately reconstruct the input data during training, even when relying on uninformative features such as random noise.

To mitigate this issue, we propose an asymmetric autoencoder design where the encoder is more complex and parameter-rich, while the decoder is kept simpler with fewer parameters. This asymmetry ensures that the decoder focuses solely on reconstruction, allowing the encoder to effectively capture rich and informative temporal features from the time series segments. Specifically, we use a vanilla Transformer encoder for the encoder and a linear layer for the decoder.

Given a batch of embedding vector sequences $\mathbf{e}\in {\mathbb{R}}^{B\times L\times F}$, a vanilla Transformer encoder produces a batch of hidden vector sequences $\mathbf{h}\in {\mathbb{R}}^{B\times L\times F}$ as follows:

$$\mathbf{h}=\mathrm{VanillaTransformerEncoder}\left(\mathbf{e}\right)$$

(7)

where $\mathbf{h}$ is the output of the last encoder block in a vanilla Transformer encoder. Then, a decoder reconstructs normalized time series segments for given hidden vector sequences.

The decoder, implemented as a linear layer, transforms each hidden vector $h\in {\mathbb{R}}^F$ from the encoder into a data point $\overline{x}\in {\mathbb{R}}^V$ within a normalized time series segment, as follows:

$$\overline{x}=Wh$$

(8)

where $W\in {\mathbb{R}}^{V\times F}$ is a weight matrix for the decoder. The decoder processes each hidden vector generated by the encoder to reconstruct the normalized time series segment corresponding to the input time series segment provided to the encoder.

2.6. Training

The proposed framework is an autoencoder-based model designed to reconstruct given time series segments. To achieve this, we employ a reconstruction-based training approach that minimizes the discrepancy between the input and the reconstructed output. Specifically, the model is trained to reduce the difference between the input time series segments and the corresponding reconstructed segments generated by the framework. Let ${\mathbf{x}}^{\prime }\in {\mathbb{R}}^{B\times L\times V}$ represent the input time series segments, and let $\widehat{\mathbf{x}}\in {\mathbb{R}}^{B\times L\times V}$ denote the reconstructed segments. The model is optimized using the following loss function:

$$\ell ({\mathbf{x}}^{\prime },\widehat{\mathbf{x}})={\displaystyle \frac{1}{B}}\sum _{b=1}^B{\parallel {\mathbf{x}}_b^{\prime }-{\widehat{\mathbf{x}}}_b\parallel }_2^2$$

(9)

where B is batch size, and ${\mathbf{x}}_b^{\prime }$ and ${\widehat{\mathbf{x}}}_b$ denote the b-th segments of ${\mathbf{x}}^{\prime }$ and ${\widehat{\mathbf{x}}}_b$, respectively.

2.7. Evaluation

We use an evaluation protocol commonly used in reconstruction-based methods. The protocol consists of three steps: first, computing anomaly scores; second, predicting whether each timestamp is normal or abnormal; and third, evaluating the predictions using standard evaluation metrics.

Various methods exist for computing anomaly scores in reconstruction-based approaches. We utilize the anomaly scoring method proposed in the literature [36,37,38,39]. For a given data point $x_t^{\prime }\in {\mathbb{R}}^V$ from a batched time series segment $X^{\prime }$ and its reconstructed counterpart ${\widehat{x}}_t\in {\mathbb{R}}^V$ produced by the trained model, the anomaly score $s_t$ at timestamp t is computed as follows:

$$s_t={\parallel x_t^{\prime }-{\widehat{x}}_t\parallel }_2$$

(10)

To predict the anomaly status ${\widehat{y}}_t$ at each timestamp t within the time series data $\tilde{X}$, we compute it as follows:

$${\widehat{y}}_t=\left\{\begin{array}{c}1,s_t\ge \theta \hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(11)

Here, ${\widehat{y}}_t$ represents the anomaly status prediction, where ${\widehat{y}}_t\in 0,1$, and $\theta$ is the threshold used to determine whether a data point is anomalous or normal. The threshold is a critical parameter for accurate predictions. Among the various methods available for threshold selection, we adopt the approach used in the previous comparative studies [37,38,39].

To evaluate the model’s performance, we utilize a range of metrics. We employ Precision, Recall, and F1 score for experiments on multivariate datasets, which are widely used in unsupervised time series anomaly detection. Additionally, we use Point Adjustment (PA) [42], a post-processing method that considers an entire anomalous segment correctly identified if at least one data point within the segment is detected as anomalous. PA-based metrics are applied to both the proposed framework and the comparative models [36,37,38,39] for fair evaluation.

However, PA-based metrics have limitations in fully capturing model performance [43,44]. To address these issues, we adopt the affiliation metrics [45] in experiments with univariate datasets. These metrics, designed to overcome the shortcomings of PA-based metrics, include affiliation Precision, affiliation Recall, and affiliation F1 score, providing a more robust evaluation.

3. Comparative Experiments and Results

3.1. Benchmark Datasets

This section outlines the datasets utilized to evaluate the proposed framework. The evaluation includes both multivariate and univariate time series datasets widely used.

3.1.1. Multivariate datasets

We use six multivariate time series anomaly detection benchmarks, as described in Table 1. The MSL [15] contains telemetry data with labeled anomalies from the Mars Science Laboratory rover. The SMAP [15] includes telemetry and anomaly labels from the Soil Moisture Active Passive satellite. The SMD [16] is a labeled multivariate time series dataset containing metrics like CPU and memory usage from 28 server machines over 5 weeks. The PSM [46] features anonymized server metrics from eBay spanning 13 weeks of training and 8 weeks of testing. The GECCO [2] focuses on drinking water quality monitoring, providing multivariate time series data with physical and chemical properties. The SWAN-SF [2] includes multivariate time series data on extreme space weather conditions for space weather analysis.

Table 1. Description of multivariate time series anomaly detection benchmark datasets.

	# sub-datasets	# dimensions	# training data points	# test data points	Anomaly ratio (%)
MSL	27	55	58,317	73,729	10.53
SMAP	54	25	138,004	435,826	12.84
SMD	28	38	708,405	708,420	4.16
PSM	1	25	132,481	87,841	27.76
GECCO	1	9	69,260	69,261	1.05
SWAN-SF	1	38	60,000	60,000	32.60

Table 1. Description of multivariate time series anomaly detection benchmark datasets.

	# sub-datasets	# dimensions	# training data points	# test data points	Anomaly ratio (%)
MSL	27	55	58,317	73,729	10.53
SMAP	54	25	138,004	435,826	12.84
SMD	28	38	708,405	708,420	4.16
PSM	1	25	132,481	87,841	27.76
GECCO	1	9	69,260	69,261	1.05
SWAN-SF	1	38	60,000	60,000	32.60

3.1.2. Univariate datasets

We employ nine univariate time series anomaly detection benchmark datasets, described in Table 2. These datasets are derived from the UCR Anomaly Archive [47], following the categorization by Goswami et al. [48]. The UCR Anomaly Archive consists of 250 sub-datasets, each originating from a specific domain. Goswami et al. [48] grouped these sub-datasets into nine distinct domains: ABP, Acceleration, Air Temperature, ECG, EPG, Gait, NASA, Power Demand, and RESP. To construct the datasets, we concatenated all sub-datasets within each domain.

Table 2. Description of univariate time series anomaly detection benchmark datasets.

	# sub-datasets	# training data points	# test data points	Anomaly ratio (%)
ABP	42	1,036,746	1,841,461	0.37
Acceleration	7	38,400	62,337	1.71
Air Temperature	13	52,000	54,392	0.82
ECG	91	1,795,083	6,047,314	0.38
EPG	25	119,000	410,415	0.45
Gait	33	1,157,571	2,784,520	0.38
NASA	11	38,500	86,296	0.86
Power Demand	11	197,149	311,629	0.61
RESP	17	868,000	2,452,953	0.12

Table 2. Description of univariate time series anomaly detection benchmark datasets.

	# sub-datasets	# training data points	# test data points	Anomaly ratio (%)
ABP	42	1,036,746	1,841,461	0.37
Acceleration	7	38,400	62,337	1.71
Air Temperature	13	52,000	54,392	0.82
ECG	91	1,795,083	6,047,314	0.38
EPG	25	119,000	410,415	0.45
Gait	33	1,157,571	2,784,520	0.38
NASA	11	38,500	86,296	0.86
Power Demand	11	197,149	311,629	0.61
RESP	17	868,000	2,452,953	0.12

3.2. Baselines

To assess the effectiveness of the proposed framework, we compare it with recent advanced deep learning methods for unsupervised time series anomaly detection. For multivariate datasets, the baseline methods include Anomaly Transformer [37], DCdetector [39], MEMTO [38], and AnomalyLLM [36]. For univariate datasets, the comparisons feature TS-TCC [49], THOC [20], NCAD [50], and AnomalyLLM [36].

3.3. Implementation Details

The proposed framework was implemented using Python 3.8.19 and PyTorch 2.1.0. Training and evaluation were conducted on a single NVIDIA A100 GPU, utilizing the Adam optimizer [51]. The hyperparameter configurations for the multivariate and univariate datasets are detailed in Table A1 and Table A2, respectively. Key hyperparameters were determined through a grid search, while other parameters were set to commonly used default values.

3.4. Main Results

Table 3 presents the F1 scores comparing the proposed vanilla Transformer encoder-based model with state-of-the-art time series anomaly detection methods, including Anomaly Transformer, DCdetector, MEMTO, and AnomalyLLM, across the six multivariate time series datasets.

The results for the proposed model were obtained following the aforementioned experimental procedures, while the results for the other models were taken from their respective publications. The findings reveal that the proposed model achieved competitive performance across most datasets, with superior results on some. Specifically, it recorded the highest F1 scores of 0.975 and 0.920 on SMAP and GECCO, respectively. For SMD, PSM, and SWAN-SF, it achieved the second-highest performance, following AnomalyLLM. In particular, on GECCO, the proposed model outperformed both DCdetector and AnomalyLLM by a substantial margin.

These results highlight that, despite its relatively simple architecture, the proposed model delivers performance comparable to or better than more advanced models. Through thoughtful design choices, the vanilla Transformer encoder-based model effectively captures the temporal dependencies inherent in time series data. Further details on the experimental results are provided in Table A3.

Table 4 presents the Affiliation F1 scores comparing the proposed vanilla Transformer encoder-based model with state-of-the-art time series anomaly detection methods, including THOC, NCAD, and AnomalyLLM, across the nine univariate time series datasets. The results of the proposed model were obtained using the outlined experimental procedures, while those for the other models were sourced from the experiments in AnomalyLLM [36].

The findings show that, with few exceptions, the proposed model demonstrated competitive or superior performance in most datasets. Notably, it achieved the highest scores of 0.966, 0.802, and 0.763 on Acceleration, ECG, and RESP, respectively. For datasets such as ABP, Air Temperature, Gait, NASA, Power Demand, and Average, the model delivered the second-best performance. Although the proposed model showed a noticeable performance gap compared to AnomalyLLM on ABP and EPG, it performed competitively or even better on the remaining datasets.

As with the multivariate dataset experiments, these results confirm that the proposed model, despite its relatively simple architecture, achieves performance comparable to or exceeding that of advanced methods. This shows that with thoughtful design choices, a vanilla Transformer encoder-based model can effectively capture temporal features. Additional details on the experimental results are provided in Table A4.

3.5. Ablation Study

We conducted an ablation study to evaluate the impact of the key design principles underlying the proposed framework. These principles include: (1) segment-level normalization and denormalization, (2) the use of positional encoding, and (3) preprocessing of time series data.

3.5.1. Segment-level normalization and denormalization

To evaluate the effectiveness of segment-level normalization and denormalization, we assessed the model’s performance with and without RevIN. For this analysis, three multivariate datasets (MSL, SMAP, and SMD) and three univariate datasets (ECG, Gait, and RESP) were selected. The F1 score was used as the evaluation metric for the multivariate datasets, while the affiliation F1 score was applied for the univariate datasets.

Table 5 shows the F1 scores for the multivariate datasets, and Table 6 presents the affiliation F1 scores for the univariate datasets. In the experiments with the multivariate datasets, incorporating RevIN consistently led to improved performance across all datasets. Similarly, in the univariate dataset experiments, RevIN generally achieved superior performance. These findings demonstrate that employing RevIN can enhance the performance of both multivariate and univariate anomaly detection tasks.

RevIN has been develped to address distribution shift issues that occur between training and testing datasets in time series forecasting. In this study, we used RevIN not only to mitigate distribution shifts between training and testing datasets but also to alleviate distribution shifts between time series segments in window-based anomaly detection. Such segment-level distribution shifts can prevent a model from effectively learning general temporal patterns during training. By addressing this issue, RevIN facilitates the efficient capture of general temporal patterns.

3.5.2. Positional Encoding

The analysis revealed that incorporating positional encoding (PE) does not consistently improve the model’s performance in time series anomaly detection. Experiments on multivariate datasets, such as MSL, SMAP, and SMD, produced mixed results when absolute positional encoding (APE) or learnable PE was applied. Notably, the vanilla Transformer encoder achieved the highest F1 scores without any form of PE, as shown in Table 5. This indicates that preserving the temporal context in its original form may be more effective for reconstruction-based anomaly detection tasks.

Reconstruction tasks often rely on maintaining the integrity of the temporal context. Adding positional information can cause the model to differentiate features unnecessarily, potentially leading to overfitting on anomalous data. For univariate datasets, such as ECG, Gait, and RESP (see Table 6), the exclusion of PE also resulted in higher affiliation F1 scores. This aligns with the hypothesis that PE may introduce additional complexity, counterproductive for models focused on reconstructing temporal patterns.

Overall, these findings underscore that positional encoding is not essential for the vanilla Transformer encoder in this task. Instead, direct modeling of temporal dependencies can yiedld better results.

3.5.3. Preprocessing Time Series Data

Preprocessing strategies played a crucial role in determining the model’s performance. The experiments examined various preprocessing combinations of normalization at both the dataset and sub-dataset levels, including min-max normalization and z-score normalization. The results in Table 7 and Table 8 provide the following key insights: For all the multivariate datasets, highest F1 scores are achieved without using any normalization methods at sub-dataset-level and dataset-level normalizations. For the univariate datasets, different normalization methods at sub-dataset-level and dataset-level normalizations produced the best affiliation F1 scores.

Choosing the preprocessing techniques like normalization that align with the specific characteristics of a dataset is crucial for preserving the integrity of its temporal structure. Proper preprocessing strategies, for multivariate and univariate datasets, enhance the model’s ability to distinguish normal patterns from anomalies, thereby improving reconstruction accuracy. These findings advocate for a tailored approach to preprocessing in time series anomaly detection, emphasizing its significant impact on model performance.

3.6. Hyperparameter Sensitivity

We performed a hyperparameter sensitivity analysis for the key parameters of our proposed framework. Figure 2 presents the results of this analysis.

For the training window stride, the results indicate that the framework demonstrates robust performance within a specific range. However, performance declines significantly as the stride decreases, especially in the MSL and SMD datasets. This drop may result from variations in the time series segments generated according to the training window stride.

Regarding the window length, the framework exhibits high sensitivity to this parameter. For the MSL and SMAP datasets, performance tends to decrease as the window length increases. Conversely, in the SMD dataset, shorter window lengths lead to poorer performance. Additionally, in the SMAP dataset, performance declines when the window length deviates from 50, either shorter or longer. Since window length directly affects the generation of training and testing time series segments, it is a critical hyperparameter. Given that the optimal window length varies across datasets, selecting an appropriate value is crucial for achieving the best results.

For the number of heads in the Transformer-based encoder, this parameter appears to have minimal impact on model performance. While employing multiple attention heads allows the model to capture diverse dependencies, the results suggest that increasing the number of heads does not enhance performance. This indicates that effective anomaly detection can be achieved with a smaller number of dependencies.

The model dimension size shows varying effects depending on the dataset. For MSL and SMAP, performance decreases as the model dimension size increases. In contrast, for SMD, performance deteriorates when the model dimension size is either smaller or larger than 128. In PSM, performance remains unchanged across all model dimension sizes. These results highlight the need to carefully tune the model dimension size for optimal performance.

The impact of the number of encoder blocks varies depending on the dataset. For SMAP and PSM, performance remains almost unchanged regardless of the number of encoder blocks. In contrast, SMD and PSM exhibit greater performance fluctuations, indicating that this parameter can affect model performance in specific datasets.

The learning rate results show trends similar to the number of encoder blocks. In SMAP and PSM, performance is largely unaffected by changes in the learning rate. However, in SMD and PSM, significant variations in performance are observed, indicating that the learning rate is also an important hyperparameter.

Lastly, the dropout ratio results suggest that the framework achieves robust performance across different dropout ratios. Even in MSL, where the largest performance variation is noted, the changes are relatively minor.

4. Discussion

This work highlights the potential of a vanilla Transformer encoder with carefully selected design choices for unsupervised time series anomaly detection. Despite its simplicity, the proposed framework delivers performance that is competitive with, and in some cases exceeds, state-of-the-art models across various benchmark datasets. Notably, our framework achieved overall performance comparable to the SOTA model, AnomalyLLM, which relies on abnormal data injection [50,52,53] — a technique that synthesizes abnormal data from normal data during training. In contrast, our framework uses only normal data for training. These findings underscore the viability of a vanilla Transformer encoder as an effective model for unsupervised time series anomaly detection.

Transformers require effective training strategies due to their limited inductive bias [54]. To address this, prior studies have introduced various architectural modifications. For instance, Anomaly Transformer added an extra branch to the Transformer encoder to distinguish dependencies between normal and abnormal data [37]. DCdetector developed a two-tower model with multi-head attention to simultaneously capture two types of dependencies: point-level and patch-level [39]. MEMTO incorporated a memory module to encode prototypical features [38], while AnomalyLLM implemented several architectural adjustments to efficiently model temporal patterns [36].

In contrast to these approaches, we propose a straightforward yet effective framework built on a vanilla Transformer encoder without any architectural changes. Instead, our approach emphasizes optimizing design choices such as preprocessing, time series segmentation, omitting positional encoding, segement-level normalization and demormalization, and fine-tuning hyperparameters. Comprehensive evaluations on multiple benchmark datasets for unsupervised time series anomaly detection demonstrate the effectiveness of our proposed framework.

While our proposed framework demonstrated promising results, some limitations remain. First, unsupervised time series anomaly detection is inherently challenging due to the absence of abnormal data in the training dataset [33]. With only normal data available for training, it becomes difficult to optimize hyperparameters, like the threshold $\theta$ in Eq.(11), that are sensitive to abnormal data.

Second, our evaluation of multivariate datasets in the experiments was conducted using the point adjustment (PA) method [42], which is known to potentially overestimate model performance [43,44]. To ensure fair comparisons with other models, we applied the same PA method as used in prior studies. However, for univariate datasets, we employed affiliation metrics [45], which are independent of the PA method, to provide a more reliable assessment of our framework’s performance in comparison to other models.

Third, we evaluated our proposed framework using a non-overlapping window evaluation protocol, where the window step size matches the window length. All the compared methods followed the same protocol for performance evaluation. In addition to the non-overlapping window protocol [20,36,37,38,39], some studies use overlapping window evaluation protocols. For example, a step size of 1 is employed in certain methods [26,27,33,55,56,57]. These differences in window step sizes during evaluation make direct comparisons across studies more challenging.

Lastly, we found that adjusting the window step size during training proved effective. However, keeping the step size fixed during the generation of time series segments restricts the model’s ability to capture the diverse temporal features present in time series data. This limitation may have adversely contributed to the significant performance gap observed compared to AnomalyLLM on certain univariate datasets.

Future research should address the limitations identified in this study. First, the reliance on training datasets containing only normal data restricts the model’s ability to generalize to diverse anomaly patterns. Incorporating semi-supervised or self-supervised learning techniques could improve the model’s capacity to handle unseen anomalies. Additionally, the evaluation methodology, which relies on the point adjustment method and non-overlapping window protocols, could be enhanced by exploring alternative metrics and approaches to ensure more comprehensive and equitable model comparisons. Investigating dynamic or adaptive windowing strategies may further enhance the detection of temporal patterns. Lastly, extending the framework to accommodate more diverse datasets and application scenarios, including resource-constrained environments, could improve its practicality and scalability.

5. Conclusions

This study revisited the potential of a vanilla Transformer encoder for unsupervised time series anomaly detection. By designing a straightforward yet effective autoencoder-based framework, we demonstrated that a vanilla Transformer encoder, when paired with thoughtful design choices, can deliver performance that is competitive with or even surpasses advanced state-of-the-art models across various univariate and multivariate benchmark datasets. Our findings highlight that the key design components, such as segment-level normalization and denormalization, omission of positional encoding, and careful hyperparameter configuration, play crucial roles in the model’s success. Despite its simplicity, the proposed framework effectively captures temporal dependencies for time series anomaly detection, highlighting the often-underestimated value of vanilla Transformer encoders.