Do Machine Learning Models Beat Simple Benchmarks When Forecasting Emergency Department Performance? A Rolling-Origin Evaluation in New South Wales Public Hospitals

Objective: Machine learning (ML) is increasingly proposed for forecasting emergency department (ED) performance but is rarely compared with simple benchmarks. We tested whether ML forecasts of hospital-level ED timeliness measures in New South Wales (NSW) outperform simple benchmarks. Methods: We used public quarterly Bureau of Health Information data for NSW public hospitals with complete series for two measures: the percentage of patients starting treatment on time (42 hospitals, 2010–2026) and the percentage of patients leaving the ED within four hours (44 hospitals, 2010–2024). Six methods were compared: naive (last value), seasonal naive, rolling four-quarter mean, exponential smoothing (ETS), seasonal ARIMA (SARIMA) and a global XGBoost model trained across all hospitals. Forecasts one and four quarters ahead were evaluated over 12 rolling origins by mean absolute error (MAE). Results: One quarter ahead, MAEs for treatment on time ranged from 3.90 percentage points (XGBoost) to 5.92 (seasonal naive), with the naive forecast at 4.05. XGBoost improved on naive by 0.15 points (95% CI 0.01 to 0.29; p = 0.015), which was not significant after correction for multiple comparisons. For the four-hour measure, the naive forecast was most accurate (MAE 2.72), and XGBoost was less accurate (+0.28 points; 95% CI 0.06 to 0.49). Four quarters ahead, no method beat the naive forecast for either measure. Conclusions: For quarterly hospital-level ED timeliness, a naive forecast was as accurate as, or more accurate than, ML and classical time-series models. Any proposed ML forecasting tool for ED performance should be benchmarked against simple methods before adoption.