From Microdata to Macro Forecasts: A Comparative Study of Aggregation Strategies and Benchmark Models in Social Protection Spending

1. Introduction

Social protection agencies, fiscal authorities, statistical offices and, generally, governments, must forecast (benefit) expenditure months or even years ahead to support budget allocation, risk and actuarial assessments, along with policy evaluation. The methods to perform such tasks are numerous and have been researched to a greater extent over this and the past century, as a systematic review of four decades of budget-forecasting research documents [1]. Child benefit programmes may be somewhat a representative case for several reasons which can be contracted into the following. First, payments are numerous, individually quite small and are governed by legislated eligibility rules. Moreover, they are periodically reindexed by policy decision, which results in the fact that the aggregate expenditure series, in the end, combine demographic dynamics with policy induced level shifts. In Republika Srpska (Bosnia and Herzegovina), child benefits are administered through the Public Child Protection Fund and indexed annually to the minimum wage [2,3,4]. This way, an aggregate series is produced that is short in the statistical sense, a little over a hundred monthly observations in the frame this paper shall deal with. However, these observations are underpinned by more than a million individual payment records. This combination of a short target series and rich administrative microdata is common in the public sector and raises a practical question that this paper tries to address directly: Does the availability of transaction-level microdata change which forecasting method a public-finance statistician should use?

The natural reference points for such a question are usually the benchmark forecasting models. These models usually include but are not limited to: the seasonal naive method, ETS model, seasonal ARIMA with or without exogenous drivers, and the Theta method [5]. Decades of empirical evidence, most prominently from the M-competition series [6,7,8], show that these simple and transparent models are more often than not difficult to beat, and that individually trained machine learning (ML) models frequently underperform them on short series [9], in spite of different other findings [10,11] as well. The M4 competition, however, also showed that ML can become competitive when it is hybridised with statistical structure or supplied with additional information [12,13], though can face overfitting risks. Whether administrative microdata constitute exactly the kind of external information that allows ML to overtake benchmarks on a short public finance series is, to our knowledge, an open question.

Answering it credibly in this setting requires confronting two methodological hazards that are very easy to overlook, and are, in fact, often overlooked. The first concerns measurement. The administrative payment calendar produces recurring months of near-zero payments (affected by system resets and holidays), during which percentage error metrics such as the mean absolute percentage error (MAPE) become numerically unstable and can exceed several thousand percent without reflecting any genuine forecast failure [14]. This means that these anomalies must be handled explicitly, otherwise the reported ranking of models is going to be driven by the metric rather than by forecast quality. The second hazard, also present in forecasting, concerns information leakage. Many of the most natural microdata-derived predictors, such as the number of recipients in a month, or the distribution of payment amounts within, they are all contemporaneous with the target, i.e. by accounting identity, the mean payment multiplied by the recipient count reproduces the aggregate almost exactly, which might be tolerable in accounting nowcasting, but can seriously violate basic forecasting principles. Supplying such quantities to an ML model yields forecasts of spurious accuracy, because those quantities are unknown at the moment the forecast must be made. Leakage of this kind is a documented and widespread failure mode in ML-based science [15,16], and controlling for it is what allows an honest comparison between microdata-enriched ML and the benchmark models.

We compare four different forecasting strategies, each using an increasing amount of information. This is all done under a strictly leakage controlled evaluation. The strategies are evaluated using expanding-window cross-validation [17,18], with the weighted absolute percentage error (WAPE) and mean absolute scaled error (MASE) as our main performance measures, since they are less sensitive to situations when individual targets are close to zero. Specifically, the strategies are: Top-down approach, where we directly forecast the aggregated series; Bottom-up approach, where we forecast each series individually, and combine the category-level forecasts; Feature-enriched aggregate, where we use ML models and feed them with lagged microdata features that are known at the forecast origin; Hybrid-minimum trace (MinT) reconciliation, where we use the approach of Wickramasuriya et al. [19].

Our central finding reinforces, rather than overturns, the benchmark forecasting literature. Once every predictor is constrained to be observable at the forecast origin, exponential smoothing (in this case ETS), is the single best model in every anomaly treatment (WAPE $13.9\%$, MASE $0.755$ under the exclusion treatment), and MinT reconciliation is statistically indistinguishable from it; the feature-enriched ML models do not match either. The apparent promise of microdata-enriched ML does not survive a leakage-controlled design: once the microdata-derived predictors are restricted to information available at the forecast origin, the feature-enriched strategy fails to match either the benchmarks or the reconciliation approach. The choice of anomaly treatment is equally consequential: naive percentage-error evaluation is uninterpretable, whereas exclusion-based and regime-aware treatments yield stable rankings.

Against this background, this manuscript offers three contributions. First, it provides a systematic comparison of four aggregation strategies: top-down, bottom-up, feature-enriched aggregate and hybrid minimum-trace reconciliation. These strategies are compared across a common set of benchmark and ML models, including the naive method, exponential smoothing, ARIMA, and the Theta method alongside LASSO, XGBoost, and LightGBM, on a short public-finance series. The comparison is conducted under expanding-window design in which every predictor is, by construction, observable at the forecast origin, so that no strategy benefits from information unavailable at the time a forecast would be made (a leakage controlled design). Second, it proposes and compares four treatments of administrative calendar anomalies: naive, exclusion, calendar adjustment and regime-aware classification. This shows that the evaluation framework is decisive for correctly judging model performance when the series contains months of near-zero payments. Third, it clarifies the conditions under which simple benchmark models remain preferable to machine learning on short administrative series, and when the additional information carried by transaction-level microdata does, and does not, translate into improved aggregate forecasts. The remainder of our paper is organised as follows. Section 2 reviews the relevant literature. Section 3 describes the institutional context and data. Section 4 presents the strategies, models, feature construction, anomaly treatments, and evaluation framework. Section 5 reports results, Section 6 discusses them, and Section 7 concludes.

2. Literature Review

This section reviews the methodological building blocks of the study and states, where that proves useful, their formal definitions. We consider a monthly aggregate series ${\left\{y_t\right\}}_{t=1}^T$ with seasonal period $m=12$, and denote by ${\widehat{y}}_{T+h\mid T}$ an h-step-ahead forecast made at origin T.

2.1. Benchmark Forecasting Models

Benchmark models are valued for combining parsimony with sustained empirical performance. This makes them well suited to short series where richer and more complex model may under- or overfit [5]. The seasonal naive model is the simplest such reference and repeats the most recent seasonal observation,

$${\widehat{y}}_{T+h\mid T}=y_{T+h-m(k+1)},k=⌊(h-1)/m⌋,$$

(1)

and, despite its simplicity, is the reference against which the scale-free error measures below are defined [14].

Exponential smoothing (ETS - Error, Trend, Seasonal) is formalised within the innovations state-space framework [20]. For the aditive trend-seasonal case, the measurement and state-transition equations are

$$\begin{array}{cc}\hfill y_t& ={\ell }_{t-1}+b_{t-1}+s_{t-m}+{\epsilon }_t,\hfill \\ \hfill {\ell }_t& ={\ell }_{t-1}+b_{t-1}+\alpha {\epsilon }_t,\hfill \\ \hfill b_t& =b_{t-1}+\beta {\epsilon }_t,s_t=s_{t-m}+\gamma {\epsilon }_t,\hfill \end{array}$$

(2)

with smoothing parameters $(\alpha ,\beta ,\gamma )$ and h-step forecast ${\widehat{y}}_{T+h\mid T}={\ell }_T+h{b}_T+s_{T+h-m(k+1)}$. The specific error, trend, and seasonal components (each additive, multiplicative, or absent, possibly damped) are selected automatically by minimising a corrected Akaike information criterion [21].

ARIMA models the series through autoregressive and moving-average polynomials with seasonal and non-seasonal differencing [22],

$${\varphi }_p\left(B\right){\mathrm{\Phi }}_P\left(B^m\right){(1-B)}^d{(1-B^m)}^D{y}_t=c+{\theta }_q\left(B\right){\mathrm{\Theta }}_Q\left(B^m\right){\epsilon }_t,$$

(3)

where B is the backshift operator and $(p,d,q){(P,D,Q)}_m$ are chosen by the Hyndman-Khandakar stepwise algorithm [21].

The Theta method [23] decomposes the deseasonalised series into theta-lines $Z_t\left(\theta \right)$ whose second differences are scaled by a coefficient $\theta$,

$${\nabla }^2{Z}_t\left(\theta \right)=\theta {\nabla }^2{y}_t,t=3,\dots ,T.$$

(4)

The standard method extrapolates the linear line $Z_t\left(0\right)$ by ordinary least squares and the curvature line $Z_t\left(2\right)$ by simple exponential smoothing, then averages the two extrapolations and reseasonalises. Hyndman and Billah [24] show this is equivalent to simple exponential smoothing with drift, which explains its robustness on short series and its strong showing in the M-competitions.

2.2. Machine-Learning Methods and the Statistical-ML Debate

Two families of ML models are used here. The LASSO estimates a sparse linear predictor by penalising the ${\ell }_1$ norm of the coefficients [25],

$$\widehat{\mathit{\beta }}=\arg \underset{\mathit{\beta }}{\min }{\displaystyle \frac{1}{2N}}{\displaystyle \sum _{i=1}^N}{\left(y_i-{\beta }_0-{\mathbf{x}}_i^{\top }\mathit{\beta }\right)}^2+\lambda {\displaystyle \sum _{j=1}^p}\left|{\beta }_j\right|,$$

(5)

where the design vector ${\mathbf{x}}_i$ collects lagged values of the target and calendar indicators, and $\lambda$ is selected by cross-validation.

Gradient-boosted trees [26,27] fit an additive ensemble ${\widehat{y}}_i={\sum }_{k=1}^K{f}_k\left({\mathbf{x}}_i\right)$ of regression trees $f_k\in \mathcal{F}$ by minimising a regularised objective,

$$\mathcal{L}=\sum _{i}L\left(y_i,{\widehat{y}}_i\right)+\sum _k\mathrm{\Omega }\left(f_k\right),\mathrm{\Omega }\left(f\right)=\gamma \left|f\right|+\frac{1}{2}\lambda {\parallel \mathbf{w}\parallel }^2,$$

(6)

where $\left|f\right|$ is the number of leaves and $\mathbf{w}$ the leaf weights.

Multi-step forecasts from (5) (6) are produced recursively.

The relative performance of these methods against statistical benchmarks tends to be a central theme of the forecasting literature. Makridakis et al. [9] found that popular ML methods were dominated by traditional statistical ones across accuracy measures and horizons on the M3 series. But at what cost? In larger datasets, and even those smaller, this cost is a substantially higher in computational terms. The M-competitions reinforced this on short, monthly data [6,7], while also showing that ML becomes competitive hybridised with statistical structure [12] or combined through feature-based model averaging [13]. Januschowski et al. [28] offer criteria for classifying methods that clarify when global cross-learning ML is expected to be of use, and Petropoulos et al. [29] synthesise the consensus that ML’s disadvantage on short series reflects information scarcity rather than an intrinsic limitation. Our study tests whether transaction-level microdata provide the additional information needed to close this gap.

2.3. Hierarchical Forecasting and Reconciliation

When a target aggregates disaggregate series, the forecasts should be coherent. In other words, the disaggregate forecasts should sum to the aggregate. Stacking all n series of the hierarchy in ${\mathbf{y}}_t$ and the $n_b$ bottom-level series in ${\mathbf{b}}_t$, coherence is expressed through a summing matrix $\mathbf{S}\in {\{0,1\}}^{n\times n_b}$,

$${\mathbf{y}}_t=\mathbf{S}{\mathbf{b}}_t.$$

(7)

Given a vector of incoherent base forecasts ${\widehat{\mathbf{y}}}_{T+h}$, reconciliation maps them to coherent forecasts through a matrix $\mathbf{G}$,

$${\tilde{\mathbf{y}}}_{T+h}=\mathbf{S}\mathbf{G}{\widehat{\mathbf{y}}}_{T+h}.$$

(8)

Hyndman et al. [30] obtained $\mathbf{G}$ by generalised least squares, and Wickramasuriya et al. [19] derived the minimum-trace (MinT) solution that minimises the trace of the reconciled forecast-error covariance, yielding minimum-variance unbiased coherent forecasts,

$$\mathbf{G}={\left({\mathbf{S}}^{\top }{\mathbf{W}}_h^{-1}\mathbf{S}\right)}^{-1}{\mathbf{S}}^{\top }{\mathbf{W}}_h^{-1},$$

(9)

where ${\mathbf{W}}_h$ is the covariance matrix of the base forecast errors, estimated in practice by shrinkage. Early evidence in the literature showed that reconciled and bottom-up approaches consistently outperformed top-down forecasting on structured data [31], and the framework has ever since been extended to temporal and cross-temporal settings [32,33]. Athanasopoulos et al. [34] provide a comprehensive review and Athanasopoulos and Kourentzes [35] caution that evaluating hierarchical forecasts is itself non-trivial. Whether top-down, bottom-up, or reconciled modelling is preferable depends on the cross-sectional correlation and relative noise of the disaggregate series. This question is examined by our Strategies B and D for a short series with three highly correlated categories.

2.4. Forecast Accuracy Measurement and Model Comparison

The choice of accuracy measure directly shapes benchmark comparisons. The mean absolute percentage error,

$$\mathrm{MAPE}={\displaystyle \frac{1}{H}}{\displaystyle \sum _{h=1}^H}\left|{\displaystyle \frac{y_{T+h}-{\widehat{y}}_{T+h}}{y_{T+h}}}\right|,$$

(10)

is undefined when $y_{T+h}=0$ and grows without bound as $y_{T+h}\to 0$, which is precisely the pathology that arises in administrative payment series with zero and near-zero months. To avoid this, Hyndman and Koehler [14] propose the mean absolute scaled error, which scales errors by the in-sample seasonal naive error,

$$\mathrm{MASE}={\displaystyle \frac{\frac{1}{H}{\sum }_{h=1}^H|y_{T+h}-{\widehat{y}}_{T+h}|}{\frac{1}{T-m}{\sum }_{t=m+1}^T|y_t-y_{t-m}|}},$$

(11)

so that $\mathrm{MASE}<1$ indicates outperformance of the seasonal naive benchmark. The weighted absolute percentage error, which pools errors and actuals across the forecast block and is therefore stable near zero, and the squared-error analogue RMSSE, are defined in Section 4.

Furthermore, to introduce a traditional conventional comparison, we implement the Diebold-Mariano test [36] which operates on the loss differential $d_t=g\left(e_{1t}\right)-g\left(e_{2t}\right)$ of two competing forecasts under a loss $g(·)$ (here squared error), with statistic

$$\mathrm{DM}={\displaystyle \frac{\overline{d}}{\sqrt{\widehat{Var}\left(\overline{d}\right)}}}\stackrel{d}{\to }\mathcal{N}(0,1),$$

(12)

to which Harvey et al. [37] apply a small-sample correction and refer the statistic to a t distribution. For simultaneous comparison across all models, the Model Confidence Set [38] iteratively tests the hypothesis of equal predictive ability across a set of models and eliminates the worst-performing until the hypothesis is not rejected, and thus returns the subset ${\mathcal{M}}_{1-\alpha }^{*}$ that contains the best model with probability at least $1-\alpha$. Together these procedures separate genuine accuracy differences from sampling variations.

2.5. Evaluation Design and the Validity of the Information Set

Credible benchmark comparisons require out-of-sample evaluation that respects the temporal ordering of the data [17]. We use an expanding-window design with rolling origins. At each origin the model is re-estimated on all data up to that point and further evaluated on the subsequent H-month block. Bergmeir and Benítez [18] and Bergmeir et al. [39] clarify the conditions under which cross-validation is valid for autoregressive prediction, and Cerqueira et al. [40] document the sensitivity of comparisons to the performance estimation procedure. The explanding window is also convenient because it fixes the information set available at each origin, which any predictor must respect: features computed from data unavailable at the origin would otherwise inflate apparent accuracy, as discussed previously, which is a consequence of information leakage [15,16]. Accordingly, all predictors used in this study, including those in microdata, are constructed so as to be observable at the forecast origin.

2.6. Forecasting Social Protection and Public Expenditure

Empirical forecasting studies specific to social transfer programmes remain scarce with an early exception of Plotnick and Lidman [41] who conducted research in forecasting welfare caseloads with the goal of improving budget planning. Another exception includes Fantazzini [42] who supported/augmented monthly food-stamp data in the US with online search data, as well as Hayashi [43] who does the same with Japanese public-assistance programme. However, these authors forecast programme caseloads with augmentative signals rather than aggregate expenditure. None of them forecast aggregate expenditure from transaction microdata, while also adhering to leakage control principle with feature enriched models; this is the gap we are trying to address. The broader public finance literature has concentrated on things such as VAT and generally tax revenues and aggregate budget projections and on the distributional analysis of transfer systems [44]. An example of a more comparable nowcasting task is the nowcasting of administrative unemployment-insurance claims in the context of the COVID-19 disruption [45]. However, the focus of that work is on producing contemporaneous nowcasts, whereas we are interested in multi-step-ahead forecasts with leakage control. Standard seasonal adjustment methods such as those implemented in JDemetra+ [46] are designed to remove effects such as trading-day and weekday effects, but not the sorts of anomalies due to the institutional payment processing cycle typical of administrative payment systems. As far as we know there are no examples of using administrative microdata at the transaction level to inform aggregate transfer forecasts in the benchmarking literature. Our Strategy C aims to do precisely this. This represents a gap that our paper seeks to address.

3. Data and Institutional Context

3.1. Institutional Background

Child benefit payments in Republika Srpska are regulated by the Law on Child Protection [47] and its subsequent amendments [48]. They are administered through the Public Protection Fund [2]. In the administrative data they fall into three categories that reflect distinct entitlement rules and benefit rates rather than birth order alone:

• K1 (Category 1): the standard child allowance, paid for the first, second and fourth child in a household. It is subject to a household-income test and paid at the base rate.
• K2 (Category 2): the allowance for the third child, which is also subject to the same income test but paid at a higher, pronatalist rate than K1.
• K3 (Category 3): the allowance for children in vulnerable categories (for example: children with a certified disability or health impairment, and children under guardianship or without parental care), granted regardless of birth order or household income; paid at the highest rate.

All three rates are fixed as percentages of a base linked to the minimum wage and are re-indexed annually [3,47,48]; following the January 2022 reform, the K1, K2, and K3 rates stood at 18%, 26% and 32% of the base, respectively. Each monthly payment record belongs to exactly one category and the aggregate forecasting target is their sum.

3.2. Forecasting Target

The forecasting target is monthly aggregate child benefit expenditure, in Bosnian convertible marks (KM),

$$y_t=y_{\mathrm{K}1,t}+y_{\mathrm{K}2,t}+y_{\mathrm{K}3,t},$$

(13)

where $y_{k,t}$ is total spending in category k during month t. The aggregate $y_t$ is the object of primary interest for budget monitoring; the three category series $y_{k,t}$ (see Figure 1) serve as the disaggregate level for the bottom-up and reconciliation strategies of Section 4. Because the three categories share the same indexation base, administrative payment calendar, and broad beneficiary population, their series are strongly correlated (see Figure 1). This is a feature that proves decisive for the comparison of aggregation strategies in Section 5. Throughout, we forecast expenditure at short horizons of up to six months ahead. We do not construct a single long-range annual projection for reasons discussed in Section 4.5.

3.3. Data

The dataset comprises all individual child benefit payment records administered by the Fund for the period June 2017 to December 2025. The transaction-level file contains $N=\mathrm{1,181,239}$ payment records. Each record carries a unique applicant identifier, the payment amount and date, on of the three benefit categories (K1, K2 or K3), and administrative attributes including the number of children in the household, monthly household income per member, disability and guardianship status, as well as payment method (via post or via bank account). Linking payments to administrative request record yields $305,338$ distinct benefit requests.

A benefit request may be renewed over time and may change category across renewals as circumstances within beneficiaries’ households change. In our data a non-trivial share of request change category at least once. More specifically, about $30\%$ of all requests ($34,033$ of $113,953$) change category at least once. This share rises up to cca. $71\%$ among requests that are renewed at least once. This means that assigning payment to a recipient’s most recently or median (time-wise) observed category would thus misattribute part of past or any spendings. We instead attribute each payment to the category that was administratively in force on its transaction date, using a point-in-time (as-of) join in which the applicable decision is the most recent positive decision dated on or before the payment (see Figure 1. This leaves the aggregate $y_t$ unchanged but produces category series $y_{k,t}$ that are free of attribution drift.

Over the sample the aggregate series shifts from a lower pre-reform level to a markedly higher one after the January 2022 indexation change, with a pronounced December peak each year and recurring near-zero months (Figure 2). These near-zero months arise from the administrative payment cycle: at the turn of each calendar year the system is reset before new-year payments begin, producing genuinely near-zero payments that are correctly recorded rather than missing or erroneous. To flag them without look-ahead, we calibrate a threshold on the initial training window only (the first $W_0=60$ months, which precede the earliest forecast origin): we take the median of the observations above their lower quartile, which is a robust estimate of the normal monthly level that discards the reset months themselves, and set the threshold at five percent of it,

$$\tau =0.05\times median\left\{y_t:t\le W_0,y_t>Q_{25}\left(y_{1:W_0}\right)\right\},$$

(14)

labelling month t anomalous whenever $y_t<\tau$. This yields $\tau \approx \mathrm{KM}\mathrm{59,417}$ and identifies twelve anomalous months ($11.7\%$ of observations), predominantly in January; the remaining 91 normal months form the basis for the exclusion-based evaluation (T2).

The near-zero months recur predominantly in January. An unsupervised Isolation Forest, applied without knowledge of the threshold rule, independently flags a majority of the same months (see Figure 2). Truth be told, this is would not be too convincing had we not conducted a PELT changepoint analysis which located the January 2022 indexation break. Together these confirm that the anomalies are genuine structural features rather than artefacts of the labelling rule.

4. Methodology

Figure 3 gives a schematic overview of the workflow described in this section: point-in-time data attribution, feature construction, the four forecasting strategies, the four anomaly treatments, the expanding-window evaluation, and the three statistical comparison procedures.

4.1. Forecasting Strategies

We compare four strategies that differ in the level of aggregation at which models are estimated and in the information set they use:

• Strategy A - Top-Down (TD). All models are estimated directly on the aggregate series $y_t$.
• Strategy B - Bottom-Up (BU). Models are estimated independently on each category series $y_{k,t}$, and the category forecasts are summed, ${\widehat{y}}_{t+h}={\sum }_k{\widehat{y}}_{k,t+h}$ [30].
• Strategy C - Feature-Enriched Aggregate (FEA). The aggregate $y_t$ remains the target, but the redictor matrix supplied to the machine-learning models is augmented with lagged features derived from the transaction-level microdata (Section 4.3).
• Strategy D - Hybrid MinT Reconciliation. Category-level base forecasts (from ETS and from ARIMA) are reconciled with the aggregate through minimum-trace reconciliation with a shrinkage covariance estimator [19], following Equations (7) to (9), independently at each of the origins.

4.2. Forecasting Models

Within each of the strategies we estime the benchmark and ML models defined in Section 2. This means that we estimate the outputs provided by the seasonal naive method, exponential smoothing (ETS) with automatic selection, ARIMA with automatic order selection, and the Theta method as benchmarks. On the other side, as ML models we estimate LASSO, XGBoost and LightGBM. The regularised-regression (LASSO) and gradient-boosting models (XGBoost, LightGBM) use a lag-feature matrix comprising lags $1,2,3,12,13$ of the target, a 12-month rolling mean, and 12 month-of-year indicators, with multi-step forecasts produced recursively. Under Strategy C these models additionally receive the lagged microdata features below.

ML hyperparameters are held fixed through all origins and treatments, as well as strategies, in order to keep the model comparison fully transparent and comparable and also to avoid per-origin tuning that a series of this length cannot support. They are visible in the Table 1.

Because gradient-boosted trees are not inherently time-series oriented models, meaning they are not inherently sequential (unlike ETS or ARIMA), they treat each row of predictors independently and have zero time notion; the forecasting task is cast as a strictly supervised tabular-regression problem through a time-delay embedding, which was explained in the previous paragraph and will be discussed further in the study. In other words, this embedding, which helps tabular learners adapt to forecasting tasks [49], combined with the expanding window design in which every feature is observable at the origin (see Section 4.3), is what allows these non-sequential learners to be applied to a time-series problem.

4.3. Microdata-Derived Features

From the individual payment records we construct monthly quantities describing the recipient population. This means that we encompass the following: the number of active recepients, the number of new entrants, of exiting recipients; we also include the distributional moments of individual payment amounts (mean, standard deviation, selected percentiles), and demographic composition (mean children per household, mean household income per member, disability share, and bank-transfer share). Because these quantities are, by construction, contemporaneous with the target, they are used only in lagged form: every non-deterministic predictor of $y_{t+h}$ is lagged so as to be observable at the forecast origin t (with the exiting count, which embeds one further month of hindsight, lagged by one additional month), while only deterministic calendar terms enter contemporaneously (Figure 4). All strategies are evaluated under this common constraint, so that no strategy uses information unavailable when a forecast would be produced.

4.4. Anomaly Treatment Approaches

Because the near-zero anomalous months distort percentage-error metrics, we compare four ways of handling those when scoring the same forecasts. T1 (Naive) scores all months, including anomalous ones. T2 (Exclusion) trains on all months but computes accuracy over non-anomalous test months only, following the spirit of Hyndman and Koehler [14]. T3 (Calendar Adjustment) normalises the series to a daily average by working days before modelling and then back-transforms afterwards. T4 (Regime-Aware) adds a stage one ${\ell }_1$-penalised logistic classifier, trained only on origin-available features, that predicts whether each target month is anomalous; finally, predicted anomalous months receive the historical mean of observed anomalous expenditure, and all others the standard model forecast. The classifier is retrained at each origin.

4.5. Evaluation Framework

We use expanding-window cross-validation with rolling origins [17,18]. Starting from an initial training window od $W_0=60$ months, the window grows by one month per origin, and at each origin t we forecast horizons $h=1,\dots ,H$ with $H=6$ months (see Figure 5). This gives us 38 evaluation origins over 2022:05–2025:06; the feature-enriched Strategy C, which requires a full year of lagged microdata before its first usable origin, is evaluated over 26 origins. Accuracy is measured by the weighted absolute percentage error (WAPE) and the mean absolute scaled error (MASE) as primary metrics, with MAPE and RMSSE reported alongside. Each metric is averaged over the six horizons within a block and the over the origins. WAPE and RMSSE are defined as

$$\mathrm{WAPE}={\displaystyle \frac{{\sum }_{h=1}^H|y_{t+h}-{\widehat{y}}_{t+h}|}{{\sum }_{h=1}^H\left|y_{t+h}\right|}},\mathrm{RMSSE}=\sqrt{{\displaystyle \frac{\frac{1}{H}{\sum }_{h=1}^H{(y_{t+h}-{\widehat{y}}_{t+h})}^2}{\frac{1}{T-m}{\sum }_{s=m+1}^T{(y_s-y_{s-m})}^2}}},$$

(15)

complementing MAPE (10) and MASE (11), with $m=12$. Pairwise accuracy differences are tested with the Diebold-Mariano test [36,37], and the set of statistically indistinguishable best models is identified by the Model Confidence Set at $\alpha =0.10$ [38].

This design targets short-horizon forecast accuracy rather than a single long-range projection. In other words, each model is judged on how well it forecasts up to six months ahead across many origins, which mirrors the operational task of in-year expenditure monitoring and cash-outflow planning. A longer projection can be produced by refitting the selected model on the full series and extrapolating twelve months ahead with certain prediction intervals; we comment on this operational use in the discussion but do not treat it as an evaluation target, since a single unreplicated projection cannot be cross-validated and would not support model comparison.

4.6. Software Utilisation

All analyses were conducted in Python, with IDE of Visual Studio Code and using statsforecast (ETS, ARIMA, Theta, seasonal naive), scikit-learn, xgboost, and lightgbm (regularised regression and gradient boosting), and hierarchicalforecast (minimum-trace reconciliation). Point-in-time attribution uses an as-of merge. All computations were run on a consumer machine (Apple MacBook Pro, M2 Pro, 16 GB), which is itself relevant to the benchmark comparison. The benchmark models (ETS, Theta, ARIMA, seasonal naive) fit in well under a second per origin, whereas the recursive machine-learning models such as LASSO, XGBoost, and LightGBM across all strategies and treatments, dominate the runtime, taking approximately 40 minutes for the full expanding-window evaluation. The parsimony and low computational cost of the benchmark models is therefore a further practical advantage over the machine-learning alternatives on a series of this length, and, given that the benchmarks can in various cases be the most accurate, reinforces the cost-versus-benefit concern raised by Makridakis et al. [9]: the machine-learning models are both slower and sometimes no better.

5. Results

5.1. Impact of Anomaly Treatment on Measured Accuracy

Figure 6 reports mean WAPE for every strategy-model combination under each of the four anomaly treatments, and Figure 7 the best WAPE attained by each strategy under each treatment. The choice of treatment dominates the measured accuracy. Under the naive treatment (T1), the percentage error is uninterpretable because of the twelve near-zero months enter the denominator of (10), which results in the mean MAPE of even the best model (ETS) exceeding $5,000\%$, which is exactly the degeneracy that motivated the scaled measures of Hyndman and Koehler [14]. The weighted and scaled errors remain stable throughout. Under T1 the best WAPE is $0.211$ and the best MASE is $0.979$. Excluding the anomalous months from evaluation (T2) lowers the best WAPE to $0.139$ and the best MASE to $0.755$, and yields the only ranking that is stable across all four accuracy metrics.

Calendar adjustment (T3) does not improve on T1 (best WAPE $0.219$, MASE $1.016$), confirming that the anomalies arise from administrative processing cycles rather than month-long effects, which standard calendar adjustment is simply not designed to capture [46]. The regime-aware treatment (T4), on the other hand, is effective. Its best WAPE is $0.166$ and best MASE $0.733$, second only to exclusion. The stage-one regime classifier, trained only on origin-available features, predicts for each month whether it is an administrative reset. When evaluated against the structural threshold labels across the expanding-window origins, it identifies the administrative resets with precision $1.00$ and recall of $0.75$ over the evaluation window, meaning that every month it flags as a reset is a true reset and it correctly identifies three of every four resets (see Figure 11); under T4 the benchmark models lead again, with several ML variants at or above $\mathrm{MASE}=1$ (see Figure 10).

5.2. Strategy and Model Comparison Under Exclusion (T2)

Table 2 and Figure 8 present the main accuracy results under T2, the preferred evaluation framework. Exponential smoothing under the top-down strategy attains the lowest error on every metric (WAPE of $13.9\%$, MAPE of $11.9\%$ and MASE of $0.755$), with minimum-trace reconciliation (D·ETS-MinT) a close second (WAPE $14.5\%$) and bottom-up ETS third. The two reconciliation variants and the leading benchmarks achieve $\mathrm{MASE}<1$, and thus outperform the seasonal naive reference. Regularised regression is competitive with top-down LASSO ranking fifth (WAPE $17.6\%$, MASE $0.948$), ahead of top-down Theta and of both gradient-boosting families.

Crucially, the feature-enriched models do not surpass the benchmarks. The best Strategy C model (LightGBM-FEA, WAPE $20.3\%$) ranks thirteenth and XGBoost-FEA (WAPE $22.2\%$) seventeenth. All three Strategy C variants record $\mathrm{MASE}>1$. Top-down gradient boosting without microdata features performs at least as well (XGBoost, WAPE $18.6\%$). This indicates that the lagged microdata features add little once every predictor is required to be observable at the forecast origin. Bottom-up modelling does not improve on top-down for any model. This is consistent with the high cross-sectional correlation of the K1/K2/K3 series (Figure 1). The two low-order autoregressive benchmarks bracket this ordering instructively: the seasonal AR(1)(1)₁₂ is competitive but sits below every exponential-smoothing and Theta variant (WAPE $23.3$–$26.9\%$), while the plain AR(1), lacking any seasonal term, is the least accurate model in the study (WAPE $45.7$–$48.0\%$). Low-order autoregression alone is thus insufficient on this strongly seasonal series, whereas the seasonal and smoothing benchmarks remain hard to beat. The per-origin WAPE distribution (Figure 8) shows ETS and MinT with both the lowest medians and the tightest spread, while the feature-enriched and gradient-boosting models show wider, heavier-tailed distributions. Example six-step forecast paths for the best model appear in Figure 9.

Figure 8. Per-origin WAPE distribution by strategy–model combination under Treatment T2, ordered by median.

Figure 8. Per-origin WAPE distribution by strategy–model combination under Treatment T2, ordered by median.

Figure 9. Six-step-ahead forecasts (orange) against actuals (dark) for the best Treatment-T2 model (top-down ETS) at two illustrative expanding-window origins.

Figure 9. Six-step-ahead forecasts (orange) against actuals (dark) for the best Treatment-T2 model (top-down ETS) at two illustrative expanding-window origins.

Figure 10. Forecast skill (MASE) under Treatment T4. Values below the dashed line ($\mathrm{MASE}=1$) outperform the seasonal naive benchmark.

Figure 10. Forecast skill (MASE) under Treatment T4. Values below the dashed line ($\mathrm{MASE}=1$) outperform the seasonal naive benchmark.

Figure 11. Regime classifier performance. Red triangles mark true administrative resets; green circles mark months predicted anomalous by the origin-available classifier (precision $1.00$, recall $0.75$).

Figure 11. Regime classifier performance. Red triangles mark true administrative resets; green circles mark months predicted anomalous by the origin-available classifier (precision $1.00$, recall $0.75$).

5.3. Statistical Comparison: Diebold-Mariano, Model Confidence Set and Friedman Test (Paired with Post-Hoc Nemenyi Analysis)

We now conduct the Diebold-Mariano (DM) statistical testing adjusted for the sample size, through the small-sample t-correction [37]. This way, we compare each model against the best performer (top-down ETS). Under the exclusion treatment (T2), a total of eleven of the nineteen comparable models are significantly worse at the $5\%$ level. The two plain AR(1) model benchmarks are rejected with a DM statistics of roughly 10-14 with a p-value of $p<{10}^{-12}$. These benchmark models are then followed by the gradient boosting and seasonal naive variants. Under the regime-aware treatment (T4), a total of twelve out of seventeen models are significantly worse than top-down ETS. Given the pairwise nature of the DM test, we establish that ETS beats the weaker specifications individually but does not, on its own, characterise the full field.

For a simultaneous comparison we opt for the Model Confidence Set (MCS) [38]. This framework retains the subset of models that cannot be statistically distinguished from the best at a given confidence level (refer to Table 3). At $\alpha =0.10$ the confidence set is broad. It retains twenty models under the naive treatment and eighteen under exclusion, including every top-down and bottom-up benchmark and ML model, the seasonal AR(1)(1)₁₂ autoregression, as well as both MinT reconciliation variants. Two exclusions might be informative. First, no feature-enriched (Strategy C) model enters the confidence set under any treatment. Second, the plain AR(1) is excluded under the preferred exclusion treatment while the seasonal AR(1)(1)₁₂ is retained. This indicates that a low-order autoregression must carry seasonal structure to remain competitive on this series. The MCS is therefore, in a way, perimissive among the aggregate and category-level models, which are hard to distinguish from ETS on per-origin WAPE. However, as shown, the MCS draws a sharp line around the feature-enriched models and the non-seasonal autoregression.

Finally, as a third and complementary layer we apply the non-parametric procedure of Demšar [50], who applies a Friedman omnibus test followed by a Nemenyi post-hoc analysis and a critical-difference diagram (see Figure 12). The Friedman test rejects equality of mean ranks under exclusion (${\chi }^2=219.0$, $p=2.1\times {10}^{-34}$). It does so by restricting attention to the origins common to all models, so that the rank blocks are complete. The mean-rank ordering reproduces the WAPE ranking of Table 2. Top-down ETS attains the best mean rank of $6.12$ and is followed by ETS-MinT reconciliation and bottom-up ETS with mean ranks of $6.19$ and $7.04$ respectively. The two plain AR(1) benchmarks are, as shown, by a wide margin, the worst ($21.2$ and $22.0$). At $\alpha =0.10$ the Nemenyi critical difference is large ($\mathrm{CD}=6.39$). This is, to be fair, a direct consequence of comparing many models over a limited number of common origins. For this reason, it does not finely separate the leading group, but it does place both AR(1) specifications, the ARIMA and LightGBM variants as well as the feature-enriched models significantly below top-down ETS.

Taken together, the three procedures, in the end, lead to a consistent conclusion. Exponential smoothing and MinT reconciliation lead in this very case, and the broad benchmark class (including seasonal autoregression) remains statistically competitive, whereas the non-seasonal low-order autoregression is the weakest approach. Finally, microdata feature enrichment never lifts the ML models into the leading group.

5.4. Accuracy over Time

We also checked whether the ranking is driven by any single period. Table 4 reports WAPE under T2 by the calendar year of the forecast target. We may notice that the ordering is relatively stable over time. In short, the top-down ETS and MinT reconciliation are the best repeatedly through all years. All models turn out to be least accurate in 2022, and over time improve monotonically. What is also interesting, and again proven, is that the feature enriched models do not improve with time; if anything their yearly WAPE goes only upward (Strategy C has no 2022 column because its first usable origin falls in 2023).

6. Discussion

6.1. When Do Simple Benchmark Models Remain Competitive?

The results of this research provide us with an insight that is very frequently discussed in forecasting literature. Complex models may provide flexibility and statistical power, but when faced with data and empirical constraints, they rarely are able to compete with simple benchmark statistical models. In other words, on a single short administrative series, simple benchmark models are clearly competitive. But besides the fact that they are competitive, they are also preferable. This is proven on the following: exponential smoothing (ETS) consistently, under all anomaly treatments, attains the lowest error (for instance T2: WAPE $13.9\%$, MASE $0.755$); minimum-trace reconciliation is competitive to ETS, and with ETS it displays the tightest per-origin error distributions, as shown in Figure 8. Three different procedures prove this statistically: pairwise Diebold-Mariano tests [36,37], the Model Confidence Set [38] and a Friedman-Nemenyi rank analysis [50]. These procedures show that Theta and ETS lead and no machine-learning configuration displaces them. The central and much replicated finding, which is common in business and retail series that dominate the M-competitions, is again repeated here, in a public social-transfer domain far from these sector, which is that well-specified statistical benchmarks are hard to beat on short series [6,7,29]. What is interesting is that this result recurs in so different a setting, which itself, is very informative. It suggests that the benchmark advantage on short series is a structural property of the data-generating regime (few observations, strong seasonality, a policy-induced level shift), rather than an artefact of any particular application domain.

The two low-order autoregressive benchmarks may appear to contradict the above conclusion. The plain AR(1), without a seasonal term, is the least accurate model in the entire study with a WAPE between $45.7$ and $48.0\%$, and is rejected by every test, whereas the seasonal AR(1)(1) is far more competitive, but still below every ETS and Theta specification (WAPE $23.3$-$26.9\%$). This shows that the autoregressive structure within these models is not the one responsible for the benchmark superiority. What is responsible, on the other hand, is the parsimonious joint modelling of trend and seasonality embodied in ETS and Theta. This distinction is very much transparent in the Theta method, which is equivalent to simple ETS with drift [24]. However, this reading is nonetheless warranted: the Model Confidence Set is broad with eighteen models under exclusion, so we do not claim unique statistical superiority for exponential smoothing. The practical case rests on its lowest point error, tightest dispersion, parsimony and negligible computational cost.

6.2. Does Enriching the Models Add Value?

Whether richer information overturns benchmark models’ dominance is a complementary question to the one discussed in the previous subsection. However, the answer appears to be more or less negative. Predictors engineered from $1.18$ million individual payment records, unfortunately fail to improve on the plain aggregate models whatsoever. No feature-enriched model attains $\mathrm{MASE}<1$, none enters the Model Confidence Set under any of the treatments and top-down gradient boosting without microdata performs at least as well, which isolates the marginal value of the microdata as essentially none once every predictor is contrained to be observable at the forecast origin.

This outcome is aligned, and helps further explain, the boundary conditions established in the recent ML forecasting literature. Machine learning tends to beat statistical benchmarks when it can exploit cross-series structure, for instance learning a single global model across many related series [51], or, for example, when the series share exploitable characteristics such as related seasonal patterns and sufficient length [52]. Unfortunately, none of these conditions hold in our setting. Firstly, our forecasting target is a single aggregate series, and the three disaggraegate categories are near-collinear rather than heterogeneous, and thus there are no cross-learning opportunities. The microdata, moreover, is informative largely because it is contemporaneous, so once lagged to respect the information set, their marginal content over the target’s own history and seasonality is small.

6.3. Aggregation, Anomaly Treatment, Cost and Limitations

Three secondary results happen to reinforce our main finding. Regarding aggregation, bottom-up modeling in Strategy B does not improve on top-down (Strategy A) for any model whatsoever. This is beacuse K1, K2 and K3 share indexation base, beneficiary population pool and payment calendar and are (thus) near collinear (see Figure 1). As a result, disaggregation adds estimation noise without any predictive value. Where the hierarchy does help is through reconciliation rather than summation. Minimum trace reconciliation [19] matches the best top-down model and the confidence set retains it, so in general, the practical lesson is to reconcile rather than sum, and to also expect only modest gains when the disaggregate series are highly correlated.

When it comes to anomaly treatment, the crucial aspect is handling of the near-zero and zero anomalous months. Naive scoring inflates MAPE beyond several thousand percent due to the said anomalous degeneracy, which motivated the scale-free measures of Hyndman and Koehler [14]. This is why we emphasize weighted and scaled errors throughout. Calendar adjustment (T3) was proven ineffective because the anomalies are institutional rather than calendar-length in origin, which is yet again impossible to capture by standard adjustment methods [46]. On the other hand, the regime-aware treatment (T4) is effective with the best WAPE of $0.166$, MASE of $0.773$. The way how this was handled was that a simple origin-available classifier detected the resets with precision $1.00$ and recall $0.75$ and assigned them their historical near-zero level. For any series with administrative zero-periods such as payment systems, procurement cycles and seasonal programs, we recommend excluding the anomalous months from scoring. If exclusion is not an option, then we recommend handling them through an explicit regime step and reporting scaled rather than percentage errors.

On a more technical note, the cost and operational use of the software in fitting these models is worth mentioning. The benchmark models fit in well under a second per origin, whereas the recursive ML models ran up to forty minutes for the full evaluation on a Macbook Pro M2. As the benchmarks are simultaneously the most accurate, the ML models here provide no added value, since they are significantly slower and no better than benchmarks. This aligns with the cost concern by Makridakis et al. [9].

Several limitations qualify these conclusions. First, the results pertain to a single government’s child benefit programme, so generalisation might require further evidence. Next, the ML set is here representative and compared to benchmark set. This is why deeper learning architectures are omitted, as they exploit rather long histories or cross-learning across many related series [51,52], and these conditions are absent here. This is why their inclusion would be inappropriate and unnecessary. Moreover, the microdata lagged features often understate the value of genuinely leading indicators (for instance the counts of pending or approved-but-unpaid entitlements) that a richer administrative linkage may supply. Finally, the evaluation spans 38 expanding-window origins on a 103 month series. The overlapping origins mean the Friedman-Nemenyi analysis is read as a verification rather than exact independent inference, and a longer series would show how accuracy changes with the horizon and whether reconciliation helps.

7. Conclusions

The main question of this research was whether simple benchmark models are competitive in forecasting social protection expenditures but also whether richer, feature-enriched information adds predictive value. This was done using monthly child-benefit expenditure in Republika Srpska with a short aggregate series of 103 observations supported by $1.18$ million individual payment records. Four forecasting strategies of increasing informational richness were compared across a common set of benchmark and ML models, under an expanding window design in which every predictor is observable at the forecast origin and also across four treatments of the administrative calendar anomalies characteristic of payment data.

Three conclusions follow. First, the general idea is that, according to the findings resulted from this research, benchmark models remain preferable on short, correlated, strongly seasonal series. Exponential smoothing (ETS), in this case, turns out to be by far the most accurate model under every one of the anomaly treatments (WAPE $13.9\%$, MASE $0.755$ under exclusion), whereas MinT performs just as well. This ordering is confirmed statistically by Diebold-Mariano tests, the Model Confidence Set and a Friedman-Nemenyi analysis. Seasonal low order autoregression runs close and is competitive, wheres plain AR(1) is the least accurate model in the study. This proves that the parsimonous modelling of seasonality and trend makes the benchmark class hard to beat, not really the autoregressive structure alone [7,29]. Second, on this specific short time-series data, the microdata available, once restricted and controlled to the forecast origin, appears to add incremental to no predictive value at the aggregate level. This is visible through the fact that no feature-enriched model enters the Model Confidence Set, and that the gradient boosting without microdata performs at least as well. This stays consistent with the boundary conditions under which ML is known to succeed, namely cross-learning and genuinely leading informating, none of which is present for a single aggregate series with near-collinear components [51,52]. Finally, the treatment of calendar anomalies matters in its own right. Naive percentage-error evaluation is unusable, while the exclusion and regime-aware treatments give stable rankings, which is a caution that applies to any series with such (institutional) anomalies, i.e. institutional zero-periods.