Time Series Foundation Model for Improved Transformer Load Forecasting and Overload Detection

1. Introduction

Accurate load forecasting is critical for the safe operation and maintenance of transformers, preventing overloads and extending equipment life. Load forecasting models help identify potential overload risks and take preventive measures to ensure a balance between supply and demand, reduce energy waste and optimize power dispatch. At the same time, it supports trading decisions in the electricity market, improves grid efficiency and reliability, and reduces overall costs [1,2,3,4]. However, due to the nonlinear relationship between power load and factors such as weather, economic conditions, and population changes, predicting transformer load remains highly complex. Moreover, special events, such as the Spring Festival, can lead to sudden spikes in electricity demand, especially in rural areas, potentially resulting in overloads and voltage instability.

Traditional forecasting methods often rely on time series classification or short-term prediction algorithms [5,6,7,8,9,10], typically combining LSTM and XGBoost [11,12,13,14,15]. While XGBoost requires extensive feature engineering, LSTM suffers from issues like error accumulation in long-term predictions and difficulty in modeling long-distance dependencies. Moreover, these models struggle with large datasets and fail to provide fine-grained insights into “which overloaded when.”

Recent research has enhanced Transformer and MLP architectures to handle time series’ unique properties, such as non-stationarity [16,17,18], periodicity [19], multi-scale characteristics [20,21,22], and trends [23,24,25]. These models effectively address error accumulation and long-range dependencies, with applications in fields like finance [26,27,28,29], energy [30,31,32], transportation [33,34,35], environment [36,37], and industry [38,39,40]. While these models are efficient and fast to train, they struggle with scaling. Modeling thousands of transformers simultaneously can lead to overfitting and resource-heavy computations, making individual transformer models more practical but increasing maintenance burdens.

Foundation Models (FMs) pre-trained on large-scale time series data aim to address these challenges [41,42,43,44,45,46]. FMs offer generalization and can manage tasks involving thousands of transformers, but they are not always suited for complex real-world time series problems. For example, predicting load surges during the Spring Festival requires ultra-long-range dependency modeling, as current loads depend on historical patterns from previous years. This challenge is compounded by the need to incorporate covariates like temperature and population movement, which FMs typically struggle with in pre-training and require careful fine-tuning to integrate.

To overcome the limitations of Foundation Models (FM) and enhance load forecasting, we propose the FreqMixer model, a novel approach for power load forecasting. FreqMixer integrates covariate learning and frequency-domain learning within a multitask fine-tuning framework, improving forecasting accuracy, particularly in special event scenarios like the Spring Festival. Our contributions are as follows:

• We design a Side Network utilizing the PEFT (Parameter Efficient Fine-Tuning) method with covariates to address long-term dependencies. Ablation experiments show that Side Net outperforms traditional methods like full trimming, head trimming, adapter, and LoRA in both time and frequency domains.
• We introduce frequency-domain features combined with PEFT during fine-tuning, enhancing model generalization with fewer parameters and improving performance in diverse scenarios.
• We implement a multi-task learning framework to boost model generalization in Smart Grid Management, mitigating overfitting by leveraging complementary knowledge across tasks, resulting in improved performance across multiple tasks.

2. Related Work

2.1. Foundation Models for Time Series Analysis

In recent years, significant progress has been made in time series forecasting models, particularly in pre-training and cross-domain forecasting for large-scale data. The Lag-Llama [47] model, a decoder-based Transformer, enhances zero-point forecasting by using lags as covariates, effectively capturing long-range dependencies. TimeGPT-1 [45], an encoder-decoder Transformer, improves zero-point prediction efficiency and generalization across domains. TTMs [41], based on TSMixer [48], offer a general framework for multi-domain forecasting without relying on domain-specific data. TimesFM [42], like Lag-Llama, focuses on lag-based pre-training for zero-point prediction. Moirai [44], a masked encoder model trained on the LOTSA dataset (27 billion observations), excels in large-scale, cross-domain forecasting. LLM4TS [49] and TEMPO [50] leverage fine-tuning of GPT-2 for time series forecasting, showing LLMs’ potential in predicting non-linguistic data. LLMTime [51] introduces labeling strategies to enhance zero-point prediction, while Time-LLM [46] integrates text and time series data, increasing flexibility without additional computational cost. Timer [43] and UniTS [52] further develop generalized architectures for multi-task time series analysis.

These models demonstrate the power of Transformer- and LLM-based architectures in time series forecasting and cross-domain applications. However, foundational models focus on generalized pre-training, often lacking comprehensive support for covariates and efficient adaptation to downstream tasks. The challenge remains in effectively integrating task-specific covariates during fine-tuning to meet task requirements.

2.2. Parameter Efficient Fine-Tuning

The most straightforward Parameter Efficient Fine-Tuning (PEFT) method involves modifying a specific subset of parameters in the FM, such as bias and layer specifications, which are critical for downstream tasks. Linear Probe [53] introduces a linear layer on top of the FM as a classifier, freezing FM parameters to explore pre-training capabilities and serving as a baseline for PEFT methods. BitFit [54] shows that optimizing only the bias terms is effective, while DP-BiTFiT [55] focuses on bias terms and scaling factors in specific layers, improving training speed and reducing storage costs. The adapter [56], first used in NLP and later in computer vision (CV), is extended by AdaptFormer [57], which demonstrates the effectiveness of parallel adapter insertion in vision tasks. LoRA [58] approximates weight updates by injecting a low-rank matrix, targeting attention queries and value projection matrices.

Side-Tuning [59] offers a different PEFT approach by employing an independent side network in parallel with the FM, while SAN [60] uses a two-branch side-adapter for mask proposal and attentional bias. ViT-Adapter [61] integrates spatial priors and feature interaction in Visual Transformers, particularly suited for intensive prediction tasks. LST [62] reduces GPU memory consumption by separating trainable parameters from the backbone using a small Transformer, and DTL [63] designs a compact side network for ViT to improve parameter and memory efficiency.

Compared to Linear Probe, adapters, and LoRA, sidetuning introduces prior knowledge and covariates, offering more adaptability to downstream tasks while maintaining PEFT efficiency.

2.3. Multitask Learning

Multi-task learning (MTL) enhances task-specific performance by training models on multiple related tasks simultaneously, capturing generalized and complementary knowledge. It expands datasets and reduces overfitting by incorporating unsupervised or self-supervised tasks, making it ideal for data-scarce scenarios. MTL improves model performance through shared knowledge across tasks and enables transfer learning by adding task-specific parameters, suitable for delay-sensitive applications. Models like MultiNet [64], Mult [65], and UniT [66] use shared backbone networks and task-specific heads, with encoders learning a generic representation and task heads making predictions.

A key challenge in MTL is balancing the learning of all tasks to prevent one task from dominating. Uncertainty Weighting [67] addresses this by optimizing model weights and noise parameters, giving lower weights to tasks with higher uncertainty. GradNorm [68] balances the learning rates of tasks by normalizing gradient magnitudes, ensuring stable training. Dynamic Weight Averaging (DWA) [69] adjusts task weights dynamically, while Dynamic Task Prioritization (DTP) [70] prioritizes tasks with higher losses. MGDA [71] treats MTL as a multi-objective optimization problem, seeking a Pareto optimal solution for task balance.

In smart grid tasks like station load forecasting and overload prediction, MTL improves accuracy, reduces overfitting, enhances robustness, and handles multiple tasks within a single model.

3. Methodologies of FreqMixer

3.1. Problem Formulation & End-to-End multitask Finetuning

We evaluate our approach on three time series tasks: forecasting, classification, and multi-task learning. Each task has its own challenges, from predicting future values to classifying time series and managing multiple tasks simultaneously. Below, we outline the key aspects for each task, including quantile loss for forecasting, cross-entropy loss for classification, and GradNorm for balancing multi-task learning.

Time Series Forecasting Task. We define a multivariate time series as $x\in {\mathbb{R}}^{T\times C}$, where T is the time series length, and C is the number of channels (features). The task is a sequence-to-sequence prediction: given a history sequence x, we predict the future sequence $y\in {\mathbb{R}}^{H\times C}$ using a model $f(·)$, resulting in a forecast $\widehat{y}=f\left(x\right)\in {\mathbb{R}}^{H\times C}$, where H is the forecast horizon.

We train the model by minimizing the quantile loss (MQLoss), which measures asymmetrical errors between y and $\widehat{y}$ for a given quantile q:

$$\begin{array}{cc}\hfill \mathrm{MQLoss}(y,\widehat{y})=& {\displaystyle \frac{1}{H\times C}}\sum _{i=1}^H\sum _{c=1}^C(q·{(y_{i,c}-{\widehat{y}}_{i,c})}^{+}\hfill \\ \hfill & +(1-q)·{({\widehat{y}}_{i,c}-y_{i,c})}^{+})\hfill \end{array}$$

where ${\left(x\right)}^{+}=max(x,0)$. The parameter q adjusts the weight for over- or under-predictions, making the loss suitable for capturing uncertainties.

Time Series Classification Task. A multivariate time series is represented as $x\in {\mathbb{R}}^{T\times C}$, where T is the length of the time series and C is the number of channels (or variates). In the BizITObs domain, these channels include both business KPIs and IT events. The classification task aims to predict a class label $y\in {\mathbb{R}}^K$ from a sequence x using a model $f(·)$, resulting in a prediction $\widehat{y}=f\left(x\right)\in {\mathbb{R}}^K$, where K is the number of predefined classes.

The model is trained by minimizing the Cross-Entropy Loss:

$$\mathrm{Cross}-\mathrm{Entropy}=-\sum _{i=1}^C{w}_i·y_{i}log\left(p_i\right)$$

(1)

where $y_i$ is the true label (one-hot encoded), $p_i$ is the predicted probability, and $w_i$ is the weight for class i, typically balancing class frequency: $w_i={\displaystyle \frac{N}{n_i}}$ with N being the total number of samples and $n_i$ the number of samples in class i. This loss encourages the model to correctly classify by aligning the predicted probabilities with the true label distribution.

Multi-Task Learning with GradNorm for Task Balancing. In multi-task learning (MTL), we predict multiple related tasks simultaneously from a univariate time series $x\in {\mathbb{R}}^T$, where T is the time series length. For each task $k=1,2,\cdots ,K$, the model predicts target sequences $y_k\in {\mathbb{R}}^{H_k}$, with $H_k$ being the output length for task k.

The model outputs a set of predictions for all tasks:

$$\widehat{Y}=f\left(x\right)=\{{\widehat{y}}_1,{\widehat{y}}_2,\cdots ,{\widehat{y}}_K\}$$

(2)

where $\widehat{Y}$ are the predicted outputs for the tasks. A key challenge in MTL is balancing the learning speed of different tasks, which may have varying scales and difficulties. GradNorm addresses this by adjusting task weights ${\lambda }_k$ to equalize the gradient norms of different tasks during training, ensuring all tasks learn at similar rates.

The total loss is a weighted sum of individual task losses:

$$L=\sum _{k=1}^K{\lambda }_k{\mathcal{L}}_k(y_k,{\widehat{y}}_k)$$

(3)

where ${\lambda }_k$ is a learnable weight, and ${\mathcal{L}}_k$ is the loss for task k. GradNorm dynamically adjusts ${\lambda }_k$ based on the gradient norms $g_k$ to balance training by equalizing the normalized gradient norms of all tasks. This prevents any single task from dominating, promoting balanced learning across all tasks.

3.2. Background of TTMs

Our approach builds on the large-scale pre-trained foundation model TTMs. Tiny Time Mixers (TTMs) are compact models (starting from 1 million parameters) designed for efficient transfer learning on publicly available time series datasets. Based on the lightweight TSMixer architecture, TTMs use methods like adaptive chunking, diverse resolution sampling, and resolution prefix tuning to handle multiresolution pre-training with limited capacity. TTMs capture inter-channel correlations and incorporate exogenous signals during fine-tuning, making them highly effective for zero- and few-shot prediction tasks, improving performance by 4%-40% over popular benchmarks while reducing computational requirements. TTMs can even run on CPU-only devices, increasing usability in resource-limited environments. The TTMs workflow consists of two phases:

Pre-training phase: TTMs are trained on public datasets focusing on univariate time series. Multivariate datasets are converted into independent univariate series for preprocessing and backbone network input. Multiresolution pretraining captures temporal dynamics using Mean Square Error (MSE) loss. Multivariate correlation handling is deferred to the fine-tuning phase.

Fine-tuning phase: TTMs adapt to the target domain with zero-, few-, or full-sample prediction. During fine-tuning, the backbone network is frozen, and the decoder handles multivariate tasks by enabling the channel mixer to capture inter-channel correlations.

3.3. End-to-End Multitask Finetuning Framework

We explore Parameter Efficient Fine-Tuning (PEFT), which aims to surpass full fine-tuning with minimal parameter changes, allowing the model to efficiently adapt to new tasks while preserving the knowledge in the frozen backbone network. The backbone, responsible for processing low-level or generic features learned during pre-training, remains unchanged. Instead, fine-tuning focuses on the added companion network, the decoder, and the multi-task prediction head, enabling adaptation to the transformer load prediction task. Our fine-tuned module includes a FeedForward (FF) module and a frequency domain learning module, FreqMLP, as shown in Figure 1.

Embedding. The model input consists of the historical window of load data $x\in {\mathbb{R}}^{(T\times c)}$, future covariates $\in {\mathbb{R}}^{(T+h)\times c_1}$, and other historical covariates, such as weather and population flow data $e\in {\mathbb{R}}^{(T\times c_2)}$. The historical window passes through the embedding layer of the Backbone, resulting in an embedding $h\in {\mathbb{R}}^{d_{\mathrm{model}}\times c}$, where $d_{\mathrm{model}}=n\times h_f$. In details, at the i-th layer of the Backbone, the patch splitting block of the Backbone reshapes the input from the previous layer $X_h^{i-1}\in {\mathbb{R}}^{c\times n\times h_f}$ into the current layer’s input $X_h^i\in {\mathbb{R}}^{c\times (n-K_i)\times (h_f/K_i)}$, where $K_i=2^{(L-i)}$. This increases the number of patches by $K_i$-fold and reduces the patch size by the same $K_i$-fold. The future covariates and historical covariates are transformed through linear layers, where the historical covariates are mapped via the weight matrix $W\in {\mathbb{R}}^{d\times t}$, and the future covariates are mapped via the weight matrix $W^{\prime }\in {\mathbb{R}}^{d\times (t+h)}$. To allow interaction between each channel of the historical variables and covariates, we replicate the covariate embeddings C times and concatenate them with xalong the last dimension, resulting in an embedding $e\in {\mathbb{R}}^{c\times (c_1+c_2)·d}$.

FeedForward. In the feedforward layer, the input embedding vector e is first mapped to a lower-dimensional space $d_s$ via the weight matrix $W_{ff1}\in {\mathbb{R}}^{d_s\times (c_1+c_2)·d}$, followed by a nonlinear activation function. Then, the mapped vector is transformed back to the original $(c_1+c_2)·d$ dimension via the weight matrix $W_{ff2}\in {\mathbb{R}}^{(c_1+c_2)·d\times d_s}$.

FreqMLP. Previous work [72] found that the frequency domain MLP learning mode is favorable for timing prediction. MLP in frequency-domain learning has two main advantages: first, it captures the global dependencies of the signal through the spectrum, which leads to a better understanding of the overall structure of the data; and second, it improves the learning efficiency and reduces the noise interference by concentrating the energy of the signal in the key parts of the frequency domain. Specifically, the input vector $x\in {\mathbb{R}}^{c\times (c_1+c_2+1)·d}$ is transformed into the frequency domain via a Fourier transform, resulting in a complex representation with real and imaginary parts. The real and imaginary components are then projected to a lower-dimensional space $d_s$ using separate weight matrices $W_{\mathrm{down}}\in {\mathbb{R}}^{d_s\times (c_1+c_2)·d}$, effectively reducing the dimensionality of the input. After projection, the real and imaginary parts are activated using a non-linear function such as ReLU and normalized, followed by dropout to prevent overfitting. The resulting low-dimensional vectors are then projected back to the original dimension $(c_1+c_2)·d$ using the weight matrix $W_{\mathrm{up}}\in {\mathbb{R}}^{(c_1+c_2)·d\times d_s}$. Subsequently, a sparse activation function (e.g., Softshrink) is applied to the combined real and imaginary parts to retain the most relevant features while reducing redundancy. Finally, the data is transformed back to the time domain via an inverse Fourier transform, yielding the processed embedding $x\in {\mathbb{R}}^{T\times (c_1+c_2)·d}$, which models the relationship between the load ratio and the covariates.

SideNet. In each layer of the SideNet, the output of the corresponding layer in the backbone network is incorporated to refine the hidden state. Specifically, the process is as follows: Let $h_l^{\prime }$ denote the output of the l-th layer in the backbone network, where $h_l^{\prime }=\mathrm{Backbone}(h_{l-1};\theta )$, with $h_{l-1}$ being the input from the previous layer. The SideNet then computes a residual update $\Delta h_l$ and an auxiliary embedding $e_l$ by processing the concatenation of the backbone’s hidden state $h_{l-1}$ and the embedding from the previous layer of the SideNet, $e_{l-1}$, as follows:

$$\Delta h_l,e_l=\mathrm{SideNet}\left(\mathrm{concat}\left[h_{l-1},e_{l-1}\right]\right)$$

The hidden state for the current layer is then updated by adding the residual $\Delta h_l$ to the previous hidden state, yielding the final output for the current layer:

$$h_l=h_l^{\prime }+\Delta h_l$$

This process allows the SideNet to iteratively refine the hidden state by incorporating information from both the backbone and auxiliary features at each layer, enhancing the model’s representation power and capacity to learn complex relationships between the inputs.

Task-specific Head. After the embedding $h\in {\mathbb{R}}^{d_{\mathrm{model}}\times c}$ passes through the decoder, its shape remains unchanged, serving as a shared representation for all tasks. This shared representation is then fed into multiple task-specific heads. For the regression task, the shared representation is processed by a linear layer with a weight matrix $W_{\mathrm{reg}}\in {\mathbb{R}}^{H\times d_{\mathrm{model}}}$, producing the regression output as $y_{\mathrm{reg}}=W_{\mathrm{reg}}h$, where H represents the output dimension for the regression task. Similarly, for the classification task, the shared representation is fed into a classification head with a weight matrix $W_{\mathrm{cls}}\in {\mathbb{R}}^{\mathrm{num}\_\mathrm{class}\times d_{\mathrm{model}}}$, yielding the classification logits as $y_{\mathrm{cls}}=W_{\mathrm{cls}}h$, where num_class represents the number of classes in the classification task. These task-specific heads allow the model to adapt the shared representation h to different tasks, such as regression and classification, while maintaining a consistent dimensionality throughout.

4. Experiments

4.1. Data

For experimental purposes, we selected electric load data from 2021 to present (2024-September) for more than 160 transformers in a region of southern China. The data were sampled every 15 minutes. Each point-in-time data is labeled according to the official rubric for whether it is in a heavy or overloaded state. The data before 2024 is taken as the training set and the data after 2024 is taken as the test set. These transformers have unstable load characteristics during the Chinese New Year period every year due to the homecoming wave, which creates a big obstacle for the management and maintenance of the equipment. All data are normalized by the features of the training set to ensure the stability of training. Due to the sensitive nature of the data, it cannot be publicly disclosed.

4.2. Experimental Setting

To evaluate the performance of our FreqMixer model, we benchmark it against several well-established multivariate time series forecasting methods. We focus on comparing FreqMixer to deep learning models and mixer models, excluding traditional approaches like XGBoost and ARIMA, as they struggle with complex features and large-scale data.

Forecasting Benchmarks. We compare FreqMixer with various multivariate time series prediction models: (i) mixer models (including FreqMixer and TTMs), (ii) deep learning models such as PatchTST [22], DLinear [25], iTransformer [73], tsMixer [48], and timemixer [74], and (iii) LSTM+MLP [75], MLP [76]. We do not include traditional methods like XGBoost or ARIMA due to their limitations in capturing complex time series patterns, especially for long-term prediction.

Model Configuration. All models were trained with a context length $T=512$ and prediction length $H=288$, except for FreqMixer, which uses autoregression to predict 96 steps at a time. For FreqMixer, the batch size was $b=512$ and the hidden feature size for the SideNet was 32. Baseline models were implemented using the Neuralforecast and Time Series libraries with recommended hyperparameters. All models were trained with early stopping based on validation performance, using a quantile loss function (MQloss) with a quantile of 0.8. Our training environment included 72-core CPUs, 256GB of RAM, and two H100 80GB GPUs.

Metrics. We evaluate performance using regression metrics (MSE, MAE, MAPE) and classification metrics (Accuracy, Precision, Recall, F1-score). Classification metrics are particularly important for transformer load prediction tasks, as they reflect whether the predicted values correctly indicate the equipment’s operational state, such as overload conditions. We determine overloads by checking if the load rate exceeds a predefined threshold.

4.3. Load Forecasting in 2024 Spring Festival

In this section, we demonstrate the potential of the model for load factor prediction through various empirical evidence.

Load Forecasting. In our experiments, we compared the FreqMixer models with current benchmark models, using data from the past 5 days to predict the load for the next 3 days. To fully evaluate model performance, we assessed each model at different lead times. Table 1 clearly shows that FreqMixer significantly outperforms traditional LSTM hybrid methods and MLP schemes in power load forecasting, particularly for multi-day lead time predictions.

Specifically, FreqMixer achieves a 24% reduction in MAPE, an 87% improvement in recall, and a 72% increase in accuracy. These improvements highlight that FreqMixer not only better captures the relationship between historical and future data, but also excels at handling complex time-dependent features. Notably, FreqMixer achieves 3.5 times the recall of the SOTA benchmark and doubles the Precision when the lead time is 3 days. These advancements are largely due to FreqMixer’s ability to model long-range dependencies, overcoming the limitations of traditional models, and delivering more consistent and accurate results, especially in capturing complex long-distance dependencies. Figure 2 shows the forecasting results during the Spring Festival, with a load surge observed in a transformer in southern China (a), comparing the performance of FreqMixer with other methods.

These results demonstrate that FreqMixer not only improves prediction accuracy in power load forecasting, but also effectively handles the impact of special events, like the Chinese New Year, on transformer loads. Leveraging the power of a large model and multidimensional covariates, FreqMixer shows strong performance in addressing long-distance dependencies and complex time series forecasting tasks.

Overload Prediction. In this section, we show the applicability of the pre-trained model to different types of downstream tasks (i.e., ). We use 5 days of historical data to predict whether the next day’s will be heavily overloaded or not. As shown in Table 2, our method significantly reduces the false alarm rate and improves the accuracy of the prediction compared to other baseline methods. On the one hand, this is due to the fact that our method is better able to capture the complex relationship between historical load data and future heavy overload events; on the other hand, it is the fact that the pre-training step helps to improve the generalization to multiple downstream tasks, highlighting the potential of the time series base model.

4.4. Ablation Study

To better understand the effectiveness of our proposed model and its components, we conducted a series of ablation studies. These experiments were designed to isolate and evaluate the contributions of different architectural choices, including the use of the SideNet module, covariate handling strategies, and the fine-tuning approaches. In the following sections, we present the results of these ablation experiments to highlight the importance of each design decision in improving model performance and parametric efficiency.

Applicability to Multiple Downstream Tasks. We tested various fine-tuned architectures in both multitasking and single-tasking scenarios and justified our design. We used load rate data from 10 transformers for testing and evaluated the performance of the model in a load prediction task. Table 3 shows the experimental results, where Full represents full fine-tuning, head represents only the task head being fine-tuned, and adapter and lora as well as side net all use our proposed SideNet module to support covariates, with the difference being in the different locations and whether or not to pass the covariate as a state quantity with the network layer. Full, head, and lora do not pass covariates to the next layer; they use the same covariate embedding for each layer. Experimental results demonstrate that this approach is suboptimal, due to the fact that the covariate embeddings of adapter and SideNet allow for a more abstract representation at deeper layers. Finally, the parallel SideNet shows the best performance over the serial adapter in both single and multitasking situations. This conclusion is not surprising, as previous work [57] has demonstrated that parallel structures always have better performance for the same structure.

Parametric Efficiency. The parametric efficiency of our approach cannot be captured in this experiment because the TTMs pre-training model itself is small. We tested the efficiency of the fine-tuning module on the pre-trained Transformer Timer by performing a temporal prediction task on the ETTh1 [77] dataset, using 512 historical sequences to predict 96 long sequences. We use the FF-only module and the FreqMLP-only module as well as the hybrid module. The experimental results in Table 4 demonstrate that when fine-tuning in the frequency domain, performance can be obtained with about 1/100 of the parameters over the full amount of fine-tuning methods. Despite the higher efficiency of FreqMLP-only, the reason we still need hybrid is that FreqMLP cannot handle discontinuous signals, such as step signals.

5. Conclusions

In this paper, we propose the FreqMixer framework for adapting time-series foundation models to power-related tasks. Through extensive experiments, we show that using covariates for downstream tasks achieves a 20% reduction in MAPE, 80% recall improvement, and 70% precision enhancement in transformer load prediction during Chinese New Year. We demonstrate that incorporating frequency-domain features during fine-tuning improves efficiency, with our module fine-tuned on Timer achieving optimal performance on the ETTh1 dataset using only 1/232 of the parameters. Additionally, multi-task fine-tuning adapts a single model for multiple tasks, improving recall by 8% and precision by 100% in load classification. Our next steps includes real-time system monitoring and addressing system limitations.