A Hybrid Stacking Model for Enhanced Short-Term Load Forecasting

1. Introduction

2. Literature Review

3. Methodology and Framework

3.1. Related Methodology

3.1.1. XgBoost

XgBoost [34] stands for Extreme Gradient Boosting, which is a machine learning system based on Gradient Boosting design with higher performance. Moreover, it demonstrates a robust capacity for accurately capturing dynamic trends in short-term power loads, proving particularly effective in tackling STLF problems [10]. It mainly uses the gradient-boosting decision tree to improve speed and performance. It tries to correct the residuals of all previous weak learners by adding new weak learners because one learner may not be enough to get good results. When these learners are combined together for the final prediction, the accuracy is higher than that of individual learners. The final prediction is the sum of each learner’s scores [35,36].

The computational procedure of the XgBoost is as follows [35]:

1: The sample weights and model parameters are first initialized by assigning the same weights to all samples in the training set.

2: We need to use Equation (1) to calculate the classification error rate at each iteration:

$$er{r}_c=\frac{\sum w_{i}I(y_i\ne G_c{x}_i)}{\sum w_i}$$

(1)

where $w_i$ is the weight of the i-th sample, $G_m$ is the c-th classifier.

3: In the $c+1_{t}h$ iteration, we need to calculte the weight of the $i+1_{t}h$ sample as Equation (2):

$$w_{i+1}=w_i{e}^{a_{m}I\left(y_i\ne G_m{x}_i\right)}$$

(2)

where $a_c$ = log((1-$er{r}_c$)/$er{r}_c$)

4: Training XgBoost involves reducing the loss of the dataset’s goal function. To balance the decay of the goal function with the model’s complexity, a second-order Taylor expansion is performed on the loss function, and a regular term is added to the objective function to prevent overfitting. Use of Equation (3) to compute the objective function:

$$Obj=\sum _{i=1}^{n}L(y_i,{\widehat{y}}_i)+\sum _{k=1}^K\Omega \left(f_k\right)$$

(3)

where n is the dimension of the feature vector, L($y_j$,${\widehat{y}}_i$) is the loss function, $y_i$ is the true value, ${\widehat{y}}_i$ is the predicted value, $\Omega$($f_k$) is a regular term, used to control the complexity of the tree structure.

5: As for the complexity, we use Equation (4) to calculate:

$$\Omega \left(f\right)=\gamma N+\frac{1}{2}\lambda \sum _{i=1}^N{w}_i^2$$

(4)

where N is the number of leaf nodes, $\gamma$ is the decreasing value of the minimum loss function for node splitting, which is used to control the degree of conversation. The representation of the weight L2 regularization is $\lambda$. The square L2 modulus of w is used to regulate the tree’s complexity and prevent overfitting.

3.1.2. LSTM

LSTM is a recurrent neural network (RNN) architecture to address the challenges posed by processing long-time sequences in RNNs. LSTM tackles the gradient explosion or disappearance problem encountered by traditional RNNs due to their single-state hidden layer [37]. LSTM introduces a memory cell and three types of gates: input gate, output gate, and forget gate. The memory cell serves as a long-term storage location capable of retaining information over extended periods, while the gates regulate the flow of information into and out of the cell. This architecture allows LSTM to store and propagate errors backward through time and layers, facilitating the learning process across multiple time steps [38]. In the STLF problem, LSTM can handle input sequences of variable lengths, where historical load data have different lengths [13]. Moreover, LSTM is adept at capturing sequential patterns and long-term dependencies in historical load data, including daily cycles and seasonal trends [39]. The architecture of LSTM cell is shown in Figure 2. The calculation process of each cell in LSTM is shown below[37].

Forget Gate:

It determines what information to discard and how much useful information to retain from a prior cell state by assigning a value between 0 and 1 to the previous cell state in comparison to the current cell input with the help of the sigmoid activation function.

$$G_t=\sigma (W_G·\left[h_{t-1},x_t\right]+b_G)$$

(1)

where $\sigma$ is the sigmoid function,$G_T$ signifies the forget gate’s weight matrices,$b_G$ indicates the bias terms of the forget gate, and [$h_{t-1}$,$x_t$]denotes combining two vectors into a single vector.

Input Gate:

Two vectors of the current unit are generated by applying the sigmoid activation function and the tanh activation function, respectively, to the output information of the previous unit and the input information of the current unit, and then these vectors are multiplied together to generate the final input information.

$$\begin{array}{cc}\hfill i_t& =\sigma (W_i·\left[h_{t-1},x_t\right]+b_i)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\tilde{c}}_t& =tanh(W_c·\left[h_{t-1},x_t\right]+b_c)\hfill \end{array}$$

(3)

where $h_{t-1}$ is the output information,$x_t$ is the input information, $W_i$ and $W_c$ represent the input gate and cell state weight matrices, respectively, and $b_i$ and $b_c$ denote the bias terms of the input gate and cell state.

Cell Status:

Long-term memory updates by mixing previous data with new inputs. It updates by dot-producting the prior state and the forget gate, then adds the product of the input gate and current cell’s state.

$$c_t=G_t\ast c_{t-1}+i_t\ast {\tilde{c}}_t.$$

(4)

Output Gate:

It assigns values between 0 and 1 to regulate information flow in its current state, using the sigmoid function for previous cell output data and processing current cell input data inversely. Next, it applies the tanh function to the cell state’s output, right before executing a dot product operation."

$$\begin{array}{cc}\hfill o_t& =\sigma (W_o·\left[h_{t-1},x_t\right]+b_o)\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill h_t& =o_t\ast tanh\left(c_t\right)\hfill \end{array}$$

(6)

where $h_{t-1}$ is the output information of the previous cell, $x_t$ is the input information of current cell,$W_o$ indicates the output gate weight matrices, and $b_o$ denote the bias terms of output gate.

3.1.3. Stacked LSTM

Multiple layers of LSTM units comprise the stacked LSTM model, an extension of the LSTM model. Each layer of a stacked LSTM consists of multiple LSTM units, with the output of one layer as the input for the following layer [15]. The layers are layered atop one another to form the architecture of a deep neural network. The architecture of the stacked LSTM is shown in Figure 3. Using a stacked LSTM is intended to enhance the capacity and ability of the model to learn complex patterns in sequential data. Each stack layer can capture distinct levels of abstraction and learn distinct temporal dependencies [20]. Lower layers can concentrate on capturing local patterns and short-term dependencies, whereas higher layers can learn more abstract and long-term dependencies.

3.1.4. Bi-LSTM

The bidirectional LSTM (Bi-LSTM) is an extension of the traditional LSTM model that incorporates both forward and backward information flow. Bi-LSTM consists of a forward LSTM layer and a backward LSTM layer. During the forward pass, the forward LSTM processes the input sequence from the beginning to the end, capturing past information. Simultaneously, the backward LSTM processes the input sequence in reverse, capturing future information [13]. The outputs of the two layers are combined to obtain the final output sequence. It allows the model to capture contextual information from both the forward and backward perspectives, reducing reliance on any single time step and improving model robustness [14].

Figure 4. The architecture of the Bi-LSTM

Figure 4. The architecture of the Bi-LSTM

3.1.5. Lasso Regression

The Lasso regression algorithm is a linear regression approach adapted to L1-regularization function, it performs shrinkage and variable selection simultaneously for better prediction [40]. As for feature selection, it can change the weight of useless features to zero to solve the multicollinearity problem among various features. The Lasso regression aims to identify the subset of important features that minimize response variables’ prediction error. The Lasso regression aims to minimize the loss function:

$$\begin{array}{cc}\hfill L\left(\beta \right)& =\Sigma {\left(Y_i-\Sigma {\beta }_j\ast X_{ij}\right)}^2+\lambda \Sigma \left|{\beta }_j\right|\hfill \end{array}$$

(1)

where $Y_i$ is the observed outcome for the $i-th$ observation, $X_{ij}$ is the $j-th$ feature value for the $i_{th}$ observation, ${\beta }_j$ is the coefficient for the $j-th$ feature, and $\lambda$ is the regularization parameter. By increasing the value $\lambda$, the coefficients of less important features are shrunk towards zero, and the model will become sparser and more interpretable.

3.1.6. Stacking Technique

The stacking technique is an ensemble learning method combining multiple models to improve machine learning performance. It is used to leverage the strengths of different models and minimize the weaknesses of each model. It is usually composed of two layers, a series of base models considered as the first layer, and a meta-model, which is usually only one and considered the second layer. The output of the base model will be input to the meta-model as new features, and the output of the meta-model is considered the final prediction result. As for the first layer, more base models are helpful for feature learning because they can obtain the learning ability of different models for features. As for the second layer, the regression algorithm is demonstrated to be effective [20].

4. Performance Evaluation

5. Discussion

6. Conclusion

Abbreviations

References

1. Peng, L.; Lv, S.X.; Wang, L.; Wang, Z.Y. Effective electricity load forecasting using enhanced double-reservoir echo state network. Engineering Applications of Artificial Intelligence 2021, 99, 104132. [Google Scholar] [CrossRef]
2. Fan, G.F.; Zhang, L.Z.; Yu, M.; Hong, W.C.; Dong, S.Q. Applications of random forest in multivariable response surface for short-term load forecasting. International Journal of Electrical Power & Energy Systems 2022, 139, 108073. [Google Scholar]
3. Yang, D.; e Guo, J.; Sun, S.; Han, J.; Wang, S. An interval decomposition-ensemble approach with data-characteristic-driven reconstruction for short-term load forecasting. Applied Energy 2022, 306, 117992. [Google Scholar] [CrossRef]
4. Habbak, H.; Mahmoud, M.; Metwally, K.; Fouda, M.M.; Ibrahem, M.I. Load Forecasting Techniques and Their Applications in Smart Grids. Energies 2023, 16. [Google Scholar] [CrossRef]
5. Maldonado, S.; González, A.; Crone, S. Automatic time series analysis for electric load forecasting via support vector regression. Applied Soft Computing 2019, 83, 105616. [Google Scholar] [CrossRef]
6. Sharma, S.; Majumdar, A.; Elvira, V.; Chouzenoux, E. Blind Kalman filtering for short-term load forecasting. IEEE Transactions on Power Systems 2020, 35, 4916–4919. [Google Scholar] [CrossRef]
7. N, S.G.; Sheshadri, G.S. Electrical load forecasting using time series analysis. In Proceedings of the 2020 IEEE Bangalore Humanitarian Technology Conference; 2020; pp. 1–6. [Google Scholar]
8. Nepal, B.; Yamaha, M.; Yokoe, A.; Yamaji, T. Electricity load forecasting using clustering and ARIMA model for energy management in buildings. Japan Architectural Review 2020, 3, 62–76. [Google Scholar] [CrossRef]
9. Ye, N.; Liu, Y.; Wang, Y. Short-term power load forecasting based on SVM. Proceedings of the World Automation Congress 2012, 2012, pp. 47–51. [Google Scholar]
10. Xu, J. Research on power load forecasting based on machine learning. In Proceedings of the 2020 7th International Forum on Electrical Engineering and Automation (IFEEA); 2020; pp. 562–567. [Google Scholar]
11. Deng, X.; Ye, A.; Zhong, J.; Xu, D.; Yang, W.; Song, Z.; Zhang, Z.; Guo, J.; Wang, T.; Tian, Y.; Pan, H.; Zhang, Z.; Wang, H.; Wu, C.; Shao, J.; Chen, X. Bagging–XGBoost algorithm based extreme weather identification and short-term load forecasting model. Energy Reports 2022, 8, 8661–8674. [Google Scholar] [CrossRef]
12. Hossain, M.S.; Mahmood, H. Short-term load forecasting using an LSTM neural network. Proceedings of the 2020 IEEE Power and Energy Conference at Illinois (PECI). IEEE, 2020, pp. 1–6.
13. Mughees, N.; Mohsin, S.A.; Mughees, A.; Mughees, A. Deep sequence to sequence Bi-LSTM neural networks for day-ahead peak load forecasting. Expert Systems with Applications 2021, 175, 114844. [Google Scholar] [CrossRef]
14. Atef, S.; Eltawil, A.B. Assessment of stacked unidirectional and bidirectional long short-term memory networks for electricity load forecasting. Electric Power Systems Research 2020, 187, 106489. [Google Scholar] [CrossRef]
15. Ren, H.; Li, Q.; Wu, Q.; Zhang, C.; Dou, Z.; Chen, J. Joint forecasting of multi-energy loads for a university based on copula theory and improved LSTM network. Energy Reports 2022, 8, 605–612. [Google Scholar] [CrossRef]
16. Zhao, X.; Li, Q.; Xue, W.; Zhao, Y.; Zhao, H.; Guo, S. Research on ultra-short-term load forecasting based on real-time electricity price and window-based XGBoost model. Energies 2022, 15. [Google Scholar] [CrossRef]
17. Aly, H.H. A proposed intelligent short-term load forecasting hybrid models of ANN, WNN and KF based on clustering techniques for smart grid. Electric Power Systems Research 2020, 182, 106191. [Google Scholar] [CrossRef]
18. Li, C.; Chen, Z.; Liu, J.; Li, D.; Gao, X.; Di, F.; Li, L.; Ji, X. Power load forecasting based on the combined model of LSTM and XGBoost. Proceedings of the 2019 the International Conference on Pattern Recognition and Artificial Intelligence; Association for Computing Machinery: New York, NY, USA, 2019; PRAI’19, pp. 46–51. [Google Scholar]
19. Tan, Z.; Zhang, J.; He, Y.; Zhang, Y.; Xiong, G.; Liu, Y. Short-term load forecasting based on integration of SVR and stacking. IEEE Access 2020, 8, 227719–227728. [Google Scholar] [CrossRef]
20. Moon, J.; Jung, S.; Rew, J.; Rho, S.; Hwang, E. Combination of short-term load forecasting models based on a stacking ensemble approach. Energy and Buildings 2020, 216, 109921. [Google Scholar] [CrossRef]
21. Barman, M.; Dev Choudhury, N.; Sutradhar, S. A regional hybrid GOA-SVM model based on similar day approach for short-term load forecasting in Assam, India. Energy 2018, 145, 710–720. [Google Scholar] [CrossRef]
22. El-Hendawi, M.; Wang, Z. An ensemble method of full wavelet packet transform and neural network for short term electrical load forecasting. Electric Power Systems Research 2020, 182, 106265. [Google Scholar] [CrossRef]
23. Li, J.; Deng, D.; Zhao, J.; Cai, D.; Hu, W.; Zhang, M.; Huang, Q. A novel hybrid short-term load forecasting method of smart grid using MLR and LSTM neural network. IEEE Transactions on Industrial Informatics 2021, 17, 2443–2452. [Google Scholar] [CrossRef]
24. Tayab, U.B.; Zia, A.; Yang, F.; Lu, J.; Kashif, M. Short-term load forecasting for microgrid energy management system using hybrid HHO-FNN model with best-basis stationary wavelet packet transform. Energy 2020, 203, 117857. [Google Scholar] [CrossRef]
25. Azeem, A.; Ismail, I.; Jameel, S.M.; Harindran, V.R. Electrical load forecasting models for different generation modalities: A review. IEEE Access 2021, 9, 142239–142263. [Google Scholar] [CrossRef]
26. Akhtar, S.; Shahzad, S.; Zaheer, A.; Ullah, H.S.; Kilic, H.; Gono, R.; Jasiński, M.; Leonowicz, Z. Short-Term Load Forecasting Models: A Review of Challenges, Progress, and the Road Ahead. Energies 2023, 16. [Google Scholar] [CrossRef]
27. Yin, C.; Mao, S. Fractional multivariate grey Bernoulli model combined with improved grey wolf algorithm: Application in short-term power load forecasting. Energy 2023, 269, 126844. [Google Scholar] [CrossRef]
28. Massaoudi, M.; Refaat, S.S.; Chihi, I.; Trabelsi, M.; Oueslati, F.S.; Abu-Rub, H. A novel stacked generalization ensemble-based hybrid LGBM-XGB-MLP model for short-term load forecasting. Energy 2021, 214, 118874. [Google Scholar] [CrossRef]
29. Moradzadeh, A.; Mansour-Saatloo, A.; Nazari-Heris, M.; Mohammadi-Ivatloo, B.; Asadi, S. Introduction and literature review of the application of machine learning/deep learning to load forecasting in power system. In Application of Machine Learning and Deep Learning Methods to Power System Problems; 2021; pp. 119–135. [Google Scholar]
30. Xiao, X.; Mo, H.; Zhang, Y.; Shan, G. Meta-ANN—A dynamic artificial neural network refined by meta-learning for short-term load forecasting. Energy 2022, 246, 123418. [Google Scholar] [CrossRef]
31. Solyali, D. A comparative analysis of machine learning approaches for short-/long-term electricity load forecasting in Cyprus. Sustainability 2020, 12. [Google Scholar] [CrossRef]
32. Wu, Z.; Zhao, X.; Ma, Y.; Zhao, X. A hybrid model based on modified multi-objective cuckoo search algorithm for short-term load forecasting. Applied Energy 2019, 237, 896–909. [Google Scholar] [CrossRef]
33. Moradzadeh, A.; Zakeri, S.; Shoaran, M.; Mohammadi-Ivatloo, B.; Mohammadi, F. Short-term load forecasting of microgrid via hybrid support vector regression and long short-term memory algorithms. Sustainability 2020, 12, 7076. [Google Scholar] [CrossRef]
34. Chen, T.; Guestrin, C. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Association for Computing Machinery: New York, NY, USA, 2016; KDD’16, pp. 785–794. [Google Scholar]
35. Suo, G.; Song, L.; Dou, Y.; Cui, Z. Multi-dimensional short-term load forecasting based on XGBoost and fireworks algorithm. In Proceedings of the 2019 18th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES); 2019; pp. 245–248. [Google Scholar]
36. Nobre, J.; Neves, R.F. Combining principal component analysis, discrete wavelet transform and XGBoost to trade in the financial markets. Expert Systems with Applications 2019, 125, 181–194. [Google Scholar] [CrossRef]
37. Lv, L.; Wu, Z.; Zhang, J.; Zhang, L.; Tan, Z.; Tian, Z. A VMD and LSTM based hybrid model of load forecasting for power grid security. IEEE Transactions on Industrial Informatics 2022, 18, 6474–6482. [Google Scholar] [CrossRef]
38. Patterson, J.; Gibson, A. Deep learning: A practitioner’s approach; O’Reilly Media, Inc., 2017. [Google Scholar]
39. Hossain, M.S.; Mahmood, H. Short-term load forecasting using an LSTM Neural network. In Proceedings of the 2020 IEEE Power and Energy Conference at Illinois (PECI); 2020; pp. 1–6. [Google Scholar]
40. Tibshirani, R. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological) 1996, 58, 267–288. [Google Scholar] [CrossRef]
41. Kozlov, A. Daily electricity price and demand data, 2020.
42. Jhana, N. Hourly energy demand generation and weather, 2019.
43. Li, S.; Kong, X.; Yue, L.; Liu, C.; Khan, M.A.; Yang, Z.; Zhang, H. Short-term electrical load forecasting using hybrid model of manta ray foraging optimization and support vector regression. Journal of Cleaner Production, 2023; 135856. [Google Scholar]
44. Li, Y.; Ni, M.; Lu, Y. Insulator defect detection for power grid based on light correction enhancement and YOLOv5 model. Energy Reports 2022, 8, 807–814, 2022 The 5th International Conferenceon Electrical Engineering and Green Energy. [Google Scholar] [CrossRef]