A Case Study on Targeting a Practically Simple Pre-Training Method to Enhance Machine Learning-Based Climate Prediction Models

1. Introduction

Statistical methods are essential tools for climate prediction, especially for longer lead times, during which the predictive skill of climate models tends to be considerably low [1,2,3,4,5,6,7,8,9,10]. Compared with traditional linear statistical methods (e.g., multivariate linear regression), machine learning methods that can capture complex nonlinear relationships are increasingly used in climate prediction studies [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]. Indeed, some machine learning-based climate prediction models have already been applied to practical operations, and the development and utilization of machine learning methods have emerged as an inevitable trend [27,28,29,30,31,32,33].

The volume of sample data required by machine learning-based statistical models is significantly larger than that demanded by linear statistical models, owing to the large number of tunable parameters. In the context of most climate research, the observation period is typically too short (less than 100 years) to yield an adequate quantity of samples used for model training. For this reason, simple machine learning models (i.e., those with only dozens of parameters) are preferred in some studies [34,35,36,37,38]. In other studies using relatively complex machine learning models (i.e., those with parameters far exceeding 1,000), numerical simulation results from different climate models are often utilized to expand the training sample size. Specifically, this approach requires climate models to have a certain level of reasonable predictive skill for such climate events, such as the ENSO index over the next 18 months [39,40], as well as upcoming monthly mean temperature [41] and seasonal precipitation [42]. In other machine learning application fields (e.g., image recognition and medical detection), a variety of mature techniques have been developed to tackle the small sample size issue [43,44,45,46,47,48]. However, the field of climate prediction has to date rarely seen research that explores technical methods for addressing small sample size challenges.

This study aims to develop a simple and applicable method for addressing small sample size challenges by integrating climate knowledge into existing techniques from other application fields. Notably, data augmentation and transfer learning are two primary approaches to tackling such issues in machine learning [49,50,51,52,53]. In climate prediction, composite analysis of abnormal events is a widely used method for mechanistic studies [54,55,56,57,58]. A single composite map can serve as an artificial composite sample to characterize the main features of such events [59,60,61,62]. In this study, data augmentation is first performed using these composite maps. The climate prediction model is then pre-trained on these augmented artificial samples, following the transfer learning framework. Finally, two approaches to utilizing the pre-trained models and their enhancement effects on the final predictive model are explored.

This study focuses on a climate prediction case: predicting July precipitation in the middle and lower reaches of the Yangtze River using prophase winter sea surface temperatures (SST). To enhance efficiency, a simple shallow neural network (NN) with two hidden layers is used to build the predictive model. The structure of this paper is organized as follows: Section 2 introduces the training process of the NN model and the corresponding sample data; Section 3 presents the effects of pre-training methods; Section 4 discusses the experience in using this pre-training method; and Section 5 provides the conclusions.

2. Data and Methods

2.1. The NN Predictive Model

This study focuses on demonstrating the pre-training method and its effects, while other aspects of the NN model are directly set and not discussed. Specifically, the input layer and two hidden layers contain 6, 8, and 3 neurons respectively. At the initial moment, all biases are set to zero, and the weights are random values following a normal distribution controlled by one random number seed (hereafter referred to as initialization seed). For model training, we adopt the early stopping method. This method is widely used to address the small sample sizes challenges, as it eliminates the need for an independent validation set [63,64,65]. The dropout technique is not used. The initial learning rate is set to 0.01, and it will be adaptively adjusted based on the gradient during the model training process [66].

2.2. Observed Samples and Composite Samples

The original station-observed monthly precipitation data are provided by the National Climate Center of the China Meteorological Administration. Based on this data, we first average the July precipitation amounts from all stations within the region into a single value. Next, we remove the decadal trend, calculate precipitation anomaly percentages, and finally moderately modify the abnormal precipitation while confining its range to -1 to 1. For brevity, this indexed precipitation will hereafter be denoted as Rain. The original monthly SST data were downloaded from the IRI/LDEO Climate Data Library [67]. Consistent with the processing of Rain, their decadal trend has also been removed. Here, 30 potential SST predictors are first extracted via Empirical Orthogonal Function (EOF) analysis on gridded SST data over the Pacific and Indian oceans, which provide predominant indicative signals [6][70]. Finally, only six EOF modes with the strongest correlation with Rain are selected. From 1951 to 2021, there are a total of 71 observed samples (one per year), with each sample containing one Rain index and six SST predictors (upper panel of Figure 1)

Inspired by the concept of data enhancement [71,72], we artificially generated additional samples based on composite analysis (hereafter referred to as composite samples). Among the 71 observed samples, 15 are high-Rain samples and 20 are low-Rain samples. Their composite maps are presented in the middle panel of Figure 1. In general, the high- and low-Rain composite maps share a consistent SST spatial distribution but exhibit opposing anomaly features. Correspondingly, their Rain index values are 0.771 and－0.646, respectively, and these two values are roughly opposites in magnitude. Previous studies have proposed many physical mechanisms to explain these composite maps [68,73,74,75]. Based on these mechanistic analyses, it is reasonable to infer that the enhancement of the prophase SST signal leads to increased rainfall. Therefore, some composite samples used for model training were artificially produced by scaling the composite maps (six SST predictors and corresponding Rain index). Here, both the high- and low-Rain composite maps were used to generate 20 composite samples each (40 in total), as shown in the lower panel of Figure 1. In the high- and low-Rain composite samples, the signal of the SST predictor increases linearly throughout, but the Rain growth rate slows down after reaching a certain level and ceases to increase when its absolute value exceeds 0.8. This nonlinear response of Rain to SST is consistent with climate background knowledge to a certain extent; it avoids the adverse impact of excessive linearization of composite samples on the training of NN models.

2.3. Model Training Procedures and Evaluation Metrics

In the field of machine learning, Loss (the difference between model's predicted values and actual labels) is commonly used to evaluate model performance. Different from this Loss, climate prediction models are typically evaluated using the linear correlation coefficient between their predicted values and observed data (hereafter referred to as Cor), which pays more attention to the sign correspondence between the two sequences. In this study, both Loss and Cor are adopted as evaluation metrics. Due to small sample sizes, the performance of trained models is often unstable and sensitive to the initialization seed [7][80]. Therefore, we demonstrate a group of the trained predictive model through 10 or 30 repeated experiments, where only the random number seeds (e.g., initialization seed) are varied. Accordingly, the Loss and Cor scores are represented by the mean values of these 10 or 30 repetitions, as well as their corresponding standard deviations.

Here, the NN model is first pre-trained on composite samples to yield a pre-trained model that can capture the common characteristics of high- and low-Rain samples. At this step, the Loss threshold for early stopping is set to 0.02. Subsequently, the pre-trained model undergoes further tuning based on the training set derived from observed samples, with training ceasing once the Loss falls below 0.4. It is noteworthy that the performance of the finally trained predictive models is sensitive to their corresponding pre-trained models. To enhance the performance of the final predictive model, one optimal pre-trained model is selected from all repeated experiment members via the Cor score on the training set right after the pre-training step; all final predictive models undergo further tuning based on training set and this optimal pre-trained model.

To demonstrate the effect of the pre-training method, we conducted three categorys of experiments. The first category is Without Pre-trained Models: the predictive NN model is trained directly on the training set. The second category is Different Pre-trained Models (i.e., without selecting the optimal pre-trained model): in the repeated experiments at pre-training step, one initialization seed corresponds to one pre-trained model, which in turn corresponds to one finally trained predictive model. The third category is One Fixed Pre-trained Model: one optimal pre-trained model is selected from a large number of repeated experiments right after the pre-training step, and this fixed pre-trained model is used in all subsequent further tuning experiments. The difference between the second and third categorys of experiments demonstrates the significance of selecting such a fixed optimal pre-trained model.

3. Results and Analysis

3.1. Performance of Pre-Trained Models

At the pre-training step, there is no need to worry about the overfitting issue. Since these 40 composite samples almost exhibit linear variation characteristics, the NN model can approximate them after only a small number of training iterations. As shown in the upper panel of Figure 2, the Rain values calculated by the 30 pre-trained models are nearly identical to those of the composite samples, with an average Loss of 0.019. This low Loss is attributed to the pre-defined training termination threshold, which is set to 0.02. Notably, although the 30 pre-trained models yield almost identical Rain values, their internal parameters (i.e., weights and biases) differ significantly (not shown). This parameter discrepancy explains why the performance of these models varies substantially on the observed samples (see the lower panel of Figure 2). Quantitatively, the Cor scores of these pre-trained models on the observed samples range from a minimum of 0.091 to a maximum of 0.440. For subsequent training with the training set (e.g., the first 45 observed samples), only the optimal pre-trained model, selected from these 30 models based on its linear correlation with the training set (the one corresponding to the red line in Figure 2), is used for the third category of experiment (i.e., One Fixed Pre-trained Model) in the following subsection.

3.2. Performance of Predictive Models on Training and Test Sets

Figure 3 illustrates the performance of the predictive models corresponding to the three experimental categorys introduced earlier. The first 45 observed samples were employed as the training set for pre-trained model further tuning. Given that the Loss threshold for early stopping is set to 0.4, the Loss scores of these three experimental categorys on the training set (i.e., 0.397, 0.396, and 0.396) are closely clustered and marginally below 0.4. Furthermore, the use of pre-trained models (the second and third experimental categorys) does not yield improvements in the Cor score with respect to training set performance. Compared to experiments without the pre-training method (where the Cor score is 0.651±0.009), those with a fixed pre-trained model exhibit a slightly lower Cor score (0.649±0.007); however, this difference falls within the uncertainty range. In terms of stability, predicted Rain values from the 30 predictive models vary significantly in those experiments without pre-trained models and with different pre-trained models. In contrast, experiments with a fixed pre-trained model demonstrate superior stability. This indicates that differences in the initial states of NN models are a key contributor to the instability of the final trained models.

In Figure 3, the last 26 observed samples are used as the test set, and this paragraph analyzes the performance of the final trained models on this set. It is evident that the final predictive models derived from the second and third experimental categorys, which utilize pre-trained models, exhibit better performance on the test set, particularly in terms of the Cor score. The reason for this is that pre-trained models capture the common characteristics of both high- and low-Rain samples; these characteristics, which can be taken as theoretical understanding, are applicable to both the training set and the test set. On the other hand, the process of further tuning pre-trained models with the training set may also disrupt the common characteristics that the pre-trained models originally captured. The more training iterations there are, the more these common characteristics are lost. In the 30 experiments with different pre-trained models (i.e., the second experimental category), most training iterations fall within the range of 5 to 15. In the 30 experiments with one fixed optimal pre-trained models (i.e., the third experimental category), the training iterations are only 3, 4, or 5. This explains why the third category of experiment can better improve the performance of the final predictive model on the test set.

Multi-model ensemble is a common-used method to enhance the stability of predictive models [37,81,82]. Figure 4 shows the results from model ensemble experiments. Each ensemble experiment comprises 30 model members (i.e., the average value of the 30 members is reported), with 10 groups of such ensemble experiments included in each category of experiment. As expected, the results of these model ensemble experiments demonstrate excellent stability, as shown in Figure 4, where the outcomes of the 10 groups of ensemble experiments almost completely overlap. Compared with individual models (Figure 3), the model ensemble also shows significant performance improvements on both the training set and the test set (Figure 4). Taking the second category of experiment as an example, the Loss and Cor scores on the training set improved from 0.396 and 0.653 (Figure 3) to 0.371 and 0.735 (Figure 4), while those on the test set improved from 0.470 and 0.382 (Figure 3) to 0.449 and 0.422 (Figure 4). Notably, in the third category of experiment, the use of one fixed pre-trained model leads to a relatively low uncertainty in the individual model results (Figure 3). Consequently, the improvement contributed by the multi-model ensemble is relatively weak (Figure 4). However, the third category of experiment still performs the best on the test set. Here, according to the chronological order of the observed samples, ～2/3 of the earlier samples are divided into the training set, and the remaining ～1/3 are used as the test set. When the sample size is small, the split of the training and test sets can significantly impact model performance. That is to say, the model evaluation results presented herein depend on the current splitting method and thus lack robustness.

3.3. Evaluation of Model Predictive Skill via Cross-Validation Experiments

Cross-validation is a statistical method used to evaluate the performance and generalization ability of predictive models. Leave-One-Out (LOO), a special case of cross-validation, involves using one sample as the test set and the remaining samples as training data in each iteration [83–86]. To enhance the robustness of evaluating the predictive model’s performance, we conducted experiments using the LOO method (Figure 5). In the first category of experiments (i.e., without using the pre-training method), the Loss and Cor scores are 0.651 and 0.338, respectively. In the second category (i.e., using different pre-trained models), the predictive skill improved significantly, with Loss dropping to 0.475 and Cor rising to 0.376. In the third category (i.e., using one fixed optimal pre-trained model), the predictive skill improved even more, with Loss decreasing to 0.461 and Cor increasing to 0.440. Additionally, the stability of the model’s predictions has improved. In the first category of experiments, the standard deviations of Loss and Cor are 0.005 and 0.021, respectively. In the second category, their standard deviations dropped to 0.002 and 0.009, and in the third category, they decreased further to 0.001 and 0.002. In summary, the pre-training method not only improves the model’s predictive skills but also enhances the stability of its predictions, particularly when the “one fixed pre-trained model” approach is adopted.

4. Discussion

From a statistical perspective, composite maps for generating composite samples should be derived solely from abnormal events in the training set, not all observed samples. Notably, composite maps representing the common characteristics of both high and low anomaly events can be regarded as theoretical knowledge to a certain extent. This theoretical knowledge is not limited by samples and has extensibility. Correspondingly, composite maps from the training set are generally consistent with those from all observed samples. Thus, in this study’s experiments, the composite maps are fixed and not adjusted with changes in the training set. Additionally, the number of generated composite samples has little impact on the pre-training method’s performance, as these samples mainly follow linear scaling characteristics.

The pre-training method proposed herein is not only simpler but also more widely applicable as compared to that based on climate model simulation results. From the perspective of practical service needs, climate prediction focuses on abnormal events, which typically involve the excess or deficit of a specific climate variable (e.g., July precipitation). In most cases, composite analysis of high and low anomaly events reveals clear pre-event signals and influence mechanisms, with those for the two event types generally opposing each other. For climate events with such characteristics, the aforementioned method can be used to construct a pre-trained model that integrates climate theoretical knowledge. In contrast, climate models often fail to simulate abnormal events reasonably. Consequently, expanded samples derived from climate model simulations might have significant flaws and even be unfit for pre-training.

5. Conclusions

The main purpose of this study is to develop a simple and practical pre-training method for machine learning-based statistical models, taking advantage of the unique characteristics in climate research. This method adapts to the small sample size of climate events while enhancing the models’ predictive skills. Given the instability caused by small sample sizes, all figures present results from 10 or 30 repeated experiments (only the model’s random number seeds varied) to improve the robustness of model performance. Additionally, to eliminate the impact of training-test set division, the cross-validation method is adopted to further enhance the reliability of conclusions.

The pre-training method developed here involves two key steps: artificial sample generation and selection of the optimal pre-trained model. Artificial samples serve to incorporate the theoretical knowledge of climate events (i.e., generalizable common characteristics) into the pre-trained model. During subsequent tuning of the pre-trained model using the observational training set, some of these incorporated common characteristics will be lost. To mitigate such loss, the optimal pre-trained model that most closely matches the observational training set is selected from numerous repeated experiments. Experimental results demonstrate that pre-training with artificial samples significantly enhances the final predictive model’s performance on the test set. Additionally, selecting a fixed optimal pre-trained model not only further improves predictive performance but also substantially reduces prediction instability. In short, this pre-training method is a simple and practical approach to addressing the small sample size problems and enhancing predictive model performance.