5.1. Performance Evaluation Metrics
To assess models, we employed the root mean square error (RMSE), mean absolute percentage error(MAPE), population stability index(PSI) which were very widely used for assessment in prediction. The formula are as follows:
In formulas (7) and (8), n represents the number of evaluated samples, yi represents the true value of the samples, i.e. actual man-hour, and ŷi represents the predicted value of the samples, i.e. estimated man-hour. The closer RMSE and MAPE are to 0, the better the predictive performance of the model. In formula (9),represents the mean of the sample, the meanings of n and yi are the same as those in formula (7) and (8). The closer R2 is to 1, the better the model performance; the closer it is to 0, the worse the model performance. In formula (10),is the actual proportion of the sample within the partition boundaries, and is the predicted proportion of each partition sample in the test dataset. PSI is used to measure the difference in data distribution between test samples and modeling samples, and is a common indicator of model stability. It is generally believed that model stability is high when PSI is less than 0.1, average when PSI is between 0.1 and 0.25, and poor when PSI is greater than 0.25.
5.2. Comparative Analyses
In this paper, the steel cutting hours were predicted using four models, (SVR), BPNN, LR, RFR, respectively.A total of 600 datas were used in the test set for prediction, and the parameters of each model were optimized using network search.Finally, experimental comparisons are conducted on the four prediction schemes,and the four man-hours prediction schemes are shown in
Table 5.
Figure 3 shows histograms comparing the performance of four models, where Figure (a) shows the histograms of RMSE, MAPE, and R
2.Due to the significant difference in magnitude between PSI and the other three metrics, in order to better demonstrate the comparative relationship of PSI, a separate comparison is made in Figure (b).
Table 5 and
Figure 3 show that the RMSE, MAPE, R
2, and PSI of the RFR model are superior to the other three models. The RMSE of RFR is 0.69 lower than SVR, 0.98 lower than BPNN, and 1.18 lower than LR; The R
2 of RFR is 0.03 higher than SVR, 0.04 higher than BPNN, and 0.05 higher than LR, indicating that the predicted man-hours of the RFR model are closer to the actual man-hours and have the smallest error.For MAPE , RFR is 1.52% lower than SVR, 2.18% lower than BPNN, and 2.61% lower than LR, which indicates that the RFR model has the highest prediction accuracy.The PSI value of all four models are less than 0.1, indicating that the stability of all four models is high, among which the PSI value of the RFR model is significantly lower than that of the other three models by one order of magnitude, indicating that the RFR model has the highest stability.
Because PSI is a metric of model stability, in order to better analyze the stability of the RFR model, we divided the samples into 10 intervals.
Table 6 shows the detailed data for each interval of the RFR model, where actual represents the number of real samples in the interval, predict represents the number of predicted samples in the interval, actual_rate represents the percentage of actual samples in the interval to the total sample, and predict_rate represents the percentage of predicted samples in the interval to the total sample.As can be seen from
Table 6,except for the difference of 11 between the number of predicted samples and the actual number of samples in the 3rd interval, there is not much difference between the number of predicted samples and the actual number of samples in the other 9 intervals, such as interval 8 and interval 10, where the difference is only 2 samples. This indicates that the RFR model has high stability in the prediction of man-hours.
The prediction result of test set by RFR is shown in
Figure 4.The RFR model exhibited outstanding performance on the test set, achieving a coefficient of determination (R
2) as high as 0.9447. This metric signifies the model's capability to elucidate the variability in the target variable. The R
2 value of the RFR model underscores its considerable advantage in capturing the intricate relationship among steel processing hours. Furthermore, there exists a robust linear correlation between the model predictions and the actual observations, underscoring the RFR model's high level of accuracy and reliability in predicting steel longitudinal cutting processing time.
In order to further validate the performance of the model, we also presented SVR,BPNN and LR to make predict experiment. The forecasting results are shown in
Figure 5,
Figure 6 and
Figure 7.
Figure 5 shows the prediction result by SVR.The R
2 value of the SVR model stands at 0.9125, slightly lower than that of the RFR model but still within an acceptable range, indicating the effectiveness of SVR in addressing nonlinear problems. SVR efficiently captures the nonlinear features of the data by identifying the optimal hyperplane in the high-dimensional space. Despite its slightly inferior prediction accuracy compared to RFR, SVR's robustness in handling small samples or high-dimensional data suggests its potential applicability in specific contexts.
The forecasting result of BPNN can be seen in
Figure 6.The R
2 value of the BPNN model stands at 0.9043, indicating a certain degree of accuracy in modeling the nonlinear relationship between the steel processing time. BPNN, as a deep learning model, can learn complex data mapping relationships through training with the back-propagation algorithm. Despite the potential requirement for more tuning parameters and computational resources, the BPNN's robust capability in handling large-scale datasets should not be overlooked, particularly in scenarios where data features are rich and model complexity requirements are high.
Figure 7 shows the prediction result of LR.The LR model yields R
2 value of 0.8914, representing the weakest performance among the four models. However, this does not diminish LR's practical value. As a linear model, LR remains effective in handling simple linear relationships or serving as a benchmark model. Its simplicity and interpretability render it a reliable choice in certain scenarios, particularly in studies with limited datasets or stringent requirements for model interpretability.
The relative error of the RFR model is depicted in
Figure 8, showcasing an average relative error of -1.7956%. This indicates that the model's predicted values on the test set generally tend to be lower than the actual values, exhibiting a slight negative bias. This bias could stem from the model's inadequate comprehension or overfitting of specific data features during the training process. However, given the high R
2 value of the RFR model, the influence of this bias on the overall prediction results may be constrained.
Table 7 assesses the comparison of the prediction results by RFR,SVR,BPNN and LR.After comparison, it is obvious that the prediction result of RFR is much better than the other three models. It proves that RFR can be effectively applied to the prediction problem of man-hours while maintaining sufficient accuracy.
In summary, the findings of this study reveal that the RFR model exhibits the most effective performance in predicting steel longitudinal cutting processing time, followed by the SVR, BPNN, and LR models. These results offer a valuable tool for enhancing production optimization within the steel processing industry, with potential benefits including increased productivity and cost reduction. Future research endeavors could delve into optimizing model strategies, such as employing feature selection and dimensionality reduction techniques to enhance prediction accuracy. Additionally, integrating learning methods could combine the strengths of various models, thereby further improving prediction accuracy and robustness.