TGCformer: A Transformer-Based Spatiotemporal Fusion Framework for Power Load Anomaly Detection

1. Introduction

With the continuous growth in electricity demand, the issue of Non-Technical Losses (NTL) caused by abnormal electricity consumption behavior has become increasingly prominent, threatening the secure operation of distribution networks and causing significant economic losses [1,2]. As a crucial technique for identifying such anomalies and ensuring the stable operation of power grids, power load anomaly detection has attracted increasing research attention in recent years. Existing approaches can be broadly categorized into three groups: traditional methods, machine learning methods, and deep learning methods.

Traditional anomaly detection methods primarily include strategies based on statistical analysis, signal processing, density estimation, and clustering. While statistical and signal processing methods offer fundamental modeling of load data, they are often inadequate for handling non-stationary, high-dimensional, and highly nonlinear sequences [3]. Density-based and clustering methods can identify potential anomalies in an unsupervised manner; however, they are inherently limited in explicitly modeling long-term temporal dynamics and complex sequential dependencies [4,5,6]. Although hybrid forecasting-based methods and residual analysis techniques have improved detection accuracy to some extent [7], traditional approaches overall struggle to capture the complex temporal dependencies and spatial correlations inherent in power load data, limiting their performance in practical applications.

Machine learning methods enhance the modeling of complex features and improve generalization through supervised or semi-supervised mechanisms. Support Vector Machines (SVM) have demonstrated good performance in handling nonlinear features and imbalanced datasets [8,9]. Ensemble learning algorithms, such as Random Forest (RF), Extreme Gradient Boosting (XGBoost), and Gradient Boosting Machines (GBM), significantly boost detection robustness and accuracy through multi-model fusion [10,11,12]. However, these methods typically depend on manually engineered features, making it difficult to fully exploit the dynamic temporal patterns and latent spatial dependencies within load data.

Deep learning methods offer substantial advantages in time series modeling and high-dimensional feature representation. Architectures like Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), and Convolutional Neural Networks (CNN) can effectively capture nonlinear relationships and long-term dependencies [13,14]. Recently, the Transformer architecture, with its self-attention mechanism, has shown exceptional performance in long-sequence modeling and has been widely applied to power load forecasting and anomaly detection [15]. Studies indicate that combining Transformer with techniques like clustering, residual analysis, or Generative Adversarial Networks (GAN) can significantly enhance detection performance [16]. Furthermore, improved Transformer variants designed for multivariate and multi-scale time series exhibit greater stability and generalization in complex scenarios [17]. Nevertheless, most existing methods focus predominantly on temporal modeling, paying insufficient attention to the potential spatial-topological features within power systems and to mechanisms for effectively fusing multi-modal features.

To address the aforementioned limitations, this paper proposes an anomaly detection framework named TGCformer, based on the dynamic fusion of multi-scale temporal features and graph-level features. The framework aims to achieve joint spatiotemporal modeling and multi-source feature fusion for power load data, thereby improving detection accuracy and model interpretability. The main contributions of this paper are as follows:

(1) Multi-scale Temporal Feature Extraction: We employ the TSFresh automatic feature extraction method to mine statistical, time-domain, and frequency-domain features from load sequences, constructing a rich multi-scale temporal representation.

(2) Graph-level Embedding Feature Extraction: We utilize an improved GATv2 network to model sparse spatial correlations among grid nodes. By enhancing interactions between key nodes via a self-attention mechanism, the method achieves unified modeling of local topology and global dependencies, improving the expressiveness and robustness of graph-level features.

(3) Cross-attention-based Dynamic Feature Fusion Mechanism: We design a cross-attention interaction module to enable deep coupling and dynamic weighting between multi-scale temporal features and graph-level features. This allows the model to adaptively focus on critical feature channels while suppressing redundancy from irrelevant information, significantly enhancing the accuracy and stability of anomaly detection.

In summary, this paper achieves spatiotemporal feature fusion and information interaction for power load data through the collaborative modeling of TSFresh and a Sparse GATv2 network, coupled with a cross-attention mechanism. The proposed framework provides an efficient, scalable, and interpretable solution for anomaly detection in complex power systems.

2. TGCformer

2.1. Overall Framework

To address the issues of insufficient representation of multi-scale temporal patterns and difficulty in effectively modeling dynamic spatial correlations in power load data, this paper proposes the spatiotemporal fusion anomaly detection framework TGCformer. This method improves anomaly detection performance by achieving joint spatiotemporal modeling of multi-scale time statistical features and dynamic graph-level features. The overall structure is shown in Figure 1 and mainly includes four modules: temporal statistical feature extraction, sparse graph feature encoding, spatiotemporal feature cross-fusion, and classification decision-making. The method primarily consists of three parts:

Firstly, Multi-scale Temporal Feature Modeling: Unlike traditional Transformers that rely solely on implicit positional encoding, TGCformer explicitly constructs spatiotemporal structural features to directly model temporal evolution and spatial topological relationships.

Secondly, Dynamic Spatial Feature Modeling: Constructs a sparse adjacency graph based on KNN, introduces GATv2 to adaptively encode spatial dependencies between nodes, explicitly preserving the spatial topological relationships between load nodes.

Thirdly, Spatiotemporal Feature Dynamic Fusion: Designs a cross-attention fusion mechanism, using temporal features as Query and spatial features as Key and Value, to achieve heterogeneous information interaction and complementary fusion while suppressing redundant information. The Transformer encoder refines deep semantics, and the final fully connected layer outputs the anomaly detection result.

2.2. Spatiotemporal Structural Feature Extraction and Encoding

2.2.1. Multi-Scale Time Series Statistical Feature Extraction

In the temporal dimension, TGCformer utilizes the TSFresh library [18] to automatically extract time-domain, spectral-domain, and frequency-domain features, enabling explicit characterization of the sequential and evolutionary patterns of time series. Some features are shown in Figure 2. Typical features include:

absolute_sum_of_changes (sum of absolute values of first-order differences), used to reflect sequence volatility:

$$ASOC=\sum _{m=0}^{n-1} \left|x_{i+1}-x_i\right|$$

(1)

fft_coefficient (average coefficient of Fourier transform), used to capture the periodicity and frequency characteristics of time series:

$$A_k=\sum _{m=0}^{n-1} a_m\mathrm{e}\mathrm{x}\mathrm{p}\left(-2\pi i{\displaystyle \frac{mk}{n}}\right)$$

(2)

C_3 (nonlinear measure based on physics for time series data), used to characterize the self-similarity and higher-order statistical information of time series:

$$C\_3={\displaystyle \frac{1}{n-2\mathrm{lag}}}\sum _{i=0}^{n-2\mathrm{lag}} x_{i+2\mathrm{lag}}^2{x}_{i+2\mathrm{lag}}{x}_i$$

(3)

As shown in Module1 of Figure 1, after TSFresh extracts multi-scale features, feature selection is required. Univariate linear regression F-test [19] is used to screen significant features:

$$SSR=\sum _{j=1}^n {\left({\widehat{y}}_j-\overline{y}\right)}^2,SSE=\sum _{j=1}^n {\left(y_j-{\widehat{y}}_j\right)}^2$$

(4)

$$F={\displaystyle \frac{SSR/1}{SSE/(N-2)}}$$

(5)

$$p=P(F_{1,N-2}\ge F_{obs})$$

(6)

Subsequently, the Benjamini-Yekutieli (BY) procedure [20] is applied to control the false discovery rate:

$$p_{\left(i\right)}^{BY}S=\mathrm{m}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{g\bullet p\left(i\right)}{i}}c\left(e\right),1\right),c\left(g\right)=\sum _{j=1}^n {\displaystyle \frac{1}{j}}$$

(7)

where $g$ is the total number of features, and $p\left(i\right)$ is the original $p$-value sorted in ascending order. If the corrected $p_{\left(i\right)}^{BY}<0.05$, the feature is considered significant and retained.

2.2.2. Graph-Level Embedding Feature Extraction Method Based on Sparse GATv2

In the spatial dimension, to characterize the dynamic dependencies of sequences, a dynamic graph feature extraction module based on sparse GATv2 is designed (Module2 in Figure 1). First, a sparse adjacency graph is constructed via KNN to retain key temporal dependencies. Then, GATv2 is used to perform adaptive weighting and dynamic aggregation of node features, enhancing information interaction between nodes. Finally, graph-level embedding vectors are generated through average pooling, achieving efficient representation of the global topology and dynamic structure of sequences.

1)

Sparse Neighbor Graph Construction

Treat the time series as graph-structured data, with each time point corresponding to a node. To reduce the computational complexity of a fully connected graph, KNN is used to construct a sparse adjacency matrix: for time point $i$, the $k$ temporally closest points are selected as neighbor nodes, forming the sparse graph $G=(V,E)$. The initial features of each node consist of three parts: (1) the original value $x_i$; (2) the mean ${\mu }_i^{\left(s\right)}$ and standard deviation ${\sigma }_i^{\left(s\right)}$ under different window scales; (3) sine-cosine positional encoding $PE\left(i\right)$:

$${\mu }_i^{\left(s\right)}={\displaystyle \frac{1}{s}}\sum _{t=i-⌊{\displaystyle \frac{s}{2}}⌋}^{i+⌊{\displaystyle \frac{s}{2}}⌋} x_t,{\sigma }_i^{\left(s\right)}=\sqrt{{\displaystyle \frac{1}{s}}\sum _{t=i-⌊{\displaystyle \frac{s}{2}}⌋}^{i+⌊{\displaystyle \frac{s}{2}}⌋} x_t^2-{\left({\mu }_i^{\left(s\right)}\right)}^2}$$

(8)

$${PE\left(i\right)}_{2p}=\mathrm{s}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{i}{{10000}^{\frac{2p}{d_{model}}}}}\right),{PE\left(i\right)}_{2p+1}=\mathrm{c}\mathrm{o}\mathrm{s}\left({\displaystyle \frac{i}{{10000}^{\frac{2p}{d_{model}}}}}\right)$$

(9)

2)

Dynamic Graph Attention Modeling

GATv2 [21] incorporates a nonlinear mapping before computing attention weights, allowing attention scores to adjust dynamically based on node features, thereby better capturing nonlinear dependencies.

For node $i$ at layer $l$, its feature $h_i$ and the feature $h_j$ of neighbor node $j$ are linearly transformed and concatenated, then input into a $LeakyReLU$ function, followed by a nonlinear mapping to obtain the attention score (as shown in Figure 3):

$$q_{ij}=a^{T}LeakyReLU\left(\left[W{h}_i\left|\right|W{h}_j\right]\right),j\in N_i$$

(10)

where $a$ is a learnable attention vector, $W$ is a learnable weight matrix, and $\left|\right|$ denotes vector concatenation. The attention score $q_{ij}$ reflects the importance of neighbor node $j$ to node $i$.

The $\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}$ function is used for normalization within the neighbor set to compute attention weights:

$${\alpha }_{ij}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}{(q}_{ij})}{\sum _{k\mathsf{ϵ}{N}_i} \mathrm{e}\mathrm{x}\mathrm{p}{(q}_{ik})}}$$

(11)

where $N_i$ denotes the set of neighbors for node $i$. GATv2 employs a multi-head attention mechanism to enhance feature representation capability:

$$h_i^{(l+1)}={\left|\right|}_{k=1}^K\sigma \left(\sum _{jϵN_i} {\alpha }_{ij}^k{W}^k{h}_j^{\left(l\right)}\right),k\in (\mathrm{0,1},2,\dots ,\mathrm{K})$$

(12)

where $\sigma (\bullet )$ is the activation function, and $W^k$ denotes the feature projection matrix for the $k$-th attention head. The multi-head mechanism yields richer and more robust node feature representations. Residual connections and layer normalization are applied after each GATv2 layer to improve training stability and convergence. After multiple stacked layers, the final node graph embedding representation is obtained.

3)

Graph-level Feature Aggregation

To obtain global dynamic graph features for the entire time series, an average pooling operation is introduced after the sparse GATv2 layers, aggregating the embedding features of all nodes into a graph-level representation vector:

$${Avepool(h}_G)={\displaystyle \frac{1}{N}}\sum _{i=1}^N h_i^{\left(L\right)}$$

(13)

where L denotes the number of GATv2 layers. The resulting graph-level feature vector $h_G$ comprehensively characterizes the dynamic dependency structure of the load sequence, providing a dynamic graph semantic representation foundation for subsequent cross-attention fusion.

2.3. Spatiotemporal Feature Fusion and Encoding Based on Multi-Head Cross-Attention

In power load anomaly detection, time series features and spatial topological features are heterogeneous, exhibiting significant differences in semantic expression and statistical distribution. While the self-attention mechanism of the standard Transformer can effectively capture global dependencies within time series, its cross-modal information interaction capability is limited. It can only model dependencies within a single modality, making it difficult to explicitly model the relationship between the temporal and spatial domains, leading to insufficient spatiotemporal information fusion. To address this, this paper proposes the introduction of a cross-attention mechanism [22,23] based on the standard Transformer framework, aiming to achieve deep interactive fusion of multi-scale temporal features and graph embedding features (as shown in Module3 of Figure 1).

Specifically, TGCformer takes the multi-scale time series features $Y_{tsf}$ extracted by TSFresh and the graph embedding features $Y_{gat}$ generated by the sparse GATv2 network as dual-channel inputs. The cross-attention mechanism enables explicit relational modeling between these heterogeneous features. As shown in Figure 4, $Y_{tsf}$ serves as the Query vector, and $Y_{gat}$ serves as the Key and Value vectors, obtained via linear projection:

$$Q_{tsf}=Y_{tsf}{W}^Q,K_{gat}=Y_{gat}{W}^K,V_{gat}=Y_{gat}{W}^V$$

(14)

Cross-attention generates fused features by computing the similarity between Query and Key and performing a weighted sum over Value:

$$Attention\left(Q_{tsf},K_{gat},V_{gat}\right)=softmax\left({\displaystyle \frac{Q_{tsf}{K_{gat}}^T}{\sqrt{d_k}}}\right)V_{gat}$$

(15)

Compared to self-attention, which focuses only on dependencies within the same source, cross-attention achieves directed interaction between features from the temporal and spatial domains. It can adaptively aggregate graph structure information most relevant to the temporal features, achieving cross-modal feature alignment. Furthermore, to enhance multi-semantic relational modeling capability for spatiotemporal features, a Multi-Head Cross-Attention (MHCA) mechanism is introduced to learn complementary relationships in parallel across different subspaces:

$$H^{\left(f\right)}=MultiHead\left(Q_{tsf},K_{gat},V_{gat}\right)=Concat({head}_1,{head}_2,\dots {head}_n)W^O$$

(16)

where $W^O$ is a learnable linear projection matrix. The multi-head mechanism learns in parallel across different feature subspaces, thereby capturing complementary semantic relationships between temporal and graph structural features, significantly enhancing spatiotemporal dependency modeling capability and feature expression richness.

To improve deep encoding stability, TGCformer retains the Transformer's residual connection and layer normalization structure, ensuring stable gradient propagation and enhancing feature representation efficiency. The inter-layer computation is as follows:

$$X^{\mathrm{\text{'}}}=LayerNorm\left(X_{input}+MultiHead\left(Q_{tsf},K_{gat},V_{gat}\right)\right)$$

(17)

$$Z=LayerNorm\left(X^{\mathrm{\text{'}}}+FFN\left(X^{\mathrm{\text{'}}}\right)\right)$$

(18)

The Feed-Forward Network (FFN) adopts a standard two-layer linear transformation structure with GELU activation and Dropout to prevent overfitting:

$$FFN\left(x\right)=Dropout\left({Linear}_2\left(GELU\left({Linear}_1\left(x\right)\right)\right)\right)$$

(19)

where ${Linear}_1\in R^{d_{model}\times d_{ff}}$,${Linear}_2\in R^{d_{ff}\times d_{model}}$, and $d_{ff}\gg d_{model}$ to enhance nonlinear representation capability.

The fused feature vector $H_{fused}$ obtained after stacking multiple cross-attention layers is input into a Multilayer Perceptron (MLP) for anomaly detection (as shown in Module4 of Figure 1). The designed MLP has four hidden layers, each containing linear projection, batch normalization, GELU activation function, and Dropout to enhance nonlinear representation capability and reduce overfitting risk[24]. The forward pass can be represented as:

$$h^{\left(l\right)}=Dropout\left(GELU\left(BN\left(W^{\left(l\right)}{h}^{(l-1)}+b^{\left(l\right)}\right)\right)\right),l=\mathrm{1,2},\mathrm{3,4}$$

(20)

$$GELU\left(x\right)=x\bullet \mathsf{\Phi }\left(x\right)=x\bullet {\displaystyle \frac{1}{2}}\left[1+erf\left({\displaystyle \frac{x}{\sqrt{2}}}\right)\right]$$

(21)

where $\mathsf{\Phi }\left(x\right)$ is the cumulative distribution function of the standard normal distribution, and $erf(\bullet )$ is the error function.

The output layer maps the last hidden layer features to a single neuron and generates the probability of a sample belonging to the anomalous class via the Sigmoid activation function:

$$z=\left(W^{\left(0\right)}{h}^{\left(4\right)}+b^{\left(0\right)}\right),l=\mathrm{1,2},\mathrm{3,4}$$

(22)

$$y=\sigma \left(z\right)={\displaystyle \frac{1}{1+e^{-z}}}$$

(23)

During training, the network uses Focal Loss to optimize the anomaly detection task:

$${\mathcal{L}}_{focal}=-\alpha (1-y{)}^{\gamma }\mathrm{l}\mathrm{o}\mathrm{g}(y)$$

(24)

where $\alpha$ and $\gamma$ are tuning factors to mitigate training bias caused by class imbalance.

3. Experiments and Validation

3.1. Data Description

This paper uses the actual electricity consumption data from the Irish Electricity Research Center (Irish CER Smart Meter Dataset) [25] as experimental samples. The dataset contains smart meter records from 5,633 residential and commercial users, with a sampling interval of 1 hour, totaling 12,002 time series samples. Since the dataset originates from normal users who gave informed consent to participate in the study,it is therefore assumed that the original data represents normal electricity consumption behavior. To construct anomaly samples, this paper refers to literature [26,27,28] and combines typical patterns of Advanced Metering Infrastructure (AMI) attacks[29], designing six electricity theft methods from perspectives such as reducing consumption, setting to zero, and random perturbation, corresponding to different types of non-technical theft scenarios. Subsequently, anomaly samples are mixed into the original data according to proportions to construct three types of anomalous datasets with anomaly ratios of 5%, 10%, and 15%, used for multi-scenario model evaluation and robustness verification. Figure 5 shows the hourly electricity distribution for normal samples and under the six fraud methods. It can be observed that the overall distribution of normal samples is relatively broad, with the median and extreme values significantly higher than those of fraudulent samples. The distributions of various anomaly samples show downward shifts and contraction trends, with varying degrees of contraction across different theft methods. This indicates that the designed anomaly samples are significantly different in statistical characteristics from normal loads, effectively simulating multiple types of electricity theft behavior.

3.2. Evaluation Metrics

To comprehensively evaluate model performance in the power load anomaly detection task, this paper selects Accuracy, Precision, Recall, and F1-Score as evaluation metrics. The calculation methods are as follows:

$$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(25)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(26)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(27)

$$F1\_score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(28)

3.3. Overall Performance Evaluation of TGCformer

Comparative evaluation of TGCformer's overall performance was conducted using datasets with different anomaly ratios. The results are shown in Table 1. As the anomaly proportion increases, TGCformer exhibits high stability across all metrics. When the anomaly proportion is low (5%), the model's overall accuracy is high, but the recall rate is relatively low, indicating some missed detections under conditions of scarce anomaly samples, mainly influenced by sample imbalance. As the anomaly proportion increases to 10% and 15%, the model's Recall and F1 scores significantly improve, indicating stronger identification capability and better balanced performance when anomaly samples increase. Overall, the model maintains excellent detection effectiveness and strong robustness under different anomaly proportions.

To validate the effectiveness of TGCformer, it was compared with two traditional time series anomaly detection methods, InceptionTime [30] and XceptionTime [31]. Experimental results show that TGCformer significantly outperforms the comparison models in ACC, Precision, Recall, and F1_score across all anomaly ratios. Under low anomaly ratios (5%), traditional models show low Recall, indicating limited ability to identify minority-class anomalies, while TGCformer's Recall significantly improves, demonstrating stronger capture capability for anomaly samples in imbalanced data scenarios. As the anomaly ratio increases to 15%, TGCformer's metrics remain at high levels, showing stability and robustness, whereas traditional models exhibit larger fluctuations in F1 scores, indicating unstable performance when anomaly samples increase. Figure 6 presents bar charts comparing the performance of different models under various anomaly ratios, visually demonstrating that TGCformer outperforms comparison models across all metrics, with the advantage in identifying minority-class samples being most pronounced under low anomaly ratios.

3.4. Ablation Study

To verify the effectiveness of the model design, ablation experiments were conducted on the test set with an anomaly ratio of 10%, comparing detection performance with different feature inputs. The results are shown in Table 2. When using only multi-scale temporal features (Only-TSFresh) modeled via self-attention, the model achieves high precision but relatively low recall, indicating missed detections still exist when relying solely on temporal statistical information. When using only graph embedding features (Only-GATv2), performance is weakest, with Precision notably decreased and false alarm rate high, suggesting that utilizing only spatial dependencies cannot provide sufficient discriminative power. The complete TGCformer, which simultaneously incorporates temporal and spatial features and performs explicit fusion via cross-attention, achieves the best performance in Precision, Recall, and F1_score. This validates that dual-channel collaborative modeling and the cross-attention mechanism can effectively enhance anomaly pattern recognition capability and detection stability.

Figure 7 visualizes the cross-attention weights. The average attention distribution (left) shows the model consistently focuses on a few key topological nodes across multiple interaction layers. The 8×8 attention matrix from the final layer (right) shows some nodes receive higher weights during feature fusion. This indicates the model can adaptively identify the feature dimensions and nodes most contributing to anomaly detection, achieving efficient information integration.

As shown in Figure 8, after PCA dimensionality reduction of TSFresh features (left), normal users (blue dots) are mainly concentrated in the central region, but some points extend along horizontal and vertical directions forming a "long tail," increasing the difficulty of anomaly detection. Anomalous users (red dots) are partly distributed at the edges or far from the main cluster, and while some overlap with normal samples, there are also discrete samples. This feature distribution provides effective information for anomaly detection. Scatter plots based on high-variance features (right) further show that anomaly samples are mostly concentrated at the edges or in discrete regions, providing discriminative clues for the model.

4. Conclusions

This paper proposes TGCformer, a novel dual-channel model for dynamic spatiotemporal feature fusion, designed for power load anomaly detection. TGCformer integrates multi-scale temporal features with graph-level spatial features, aiming to fully leverage global temporal patterns and the spatial correlations among grid nodes. Experimental results demonstrate that TGCformer consistently achieves excellent performance across various anomaly ratio scenarios. The model's overall F1-score is significantly higher than those of state-of-the-art deep temporal baselines, including InceptionTime and XceptionTime, with substantial average improvements. This validates the model's robustness and superiority in identifying anomalies within imbalanced datasets. However, the study also reveals a limitation: when the model relies heavily on the cross-attention mechanism for fusion, it may underutilize discriminative features from individual channels, potentially hindering fine-grained recognition of specific anomaly types. Future work will explore methods to enhance the model's adaptive utilization of diverse features, preserving its fusion strengths while better harnessing the discriminative power of heterogeneous information.