BQEB FoundationBench: A Reproducibility-First Benchmark Framework for Foundation Models in Energy Forecasting

1. Introduction

Time-series forecasting research entered a genuinely new operating regime over 2023–2024. Rather than training a model from scratch on each new dataset, a family of pretrained “foundation models” — Chronos [1], TimesFM [2], Moirai [3], TimeGPT [4], Lag-Llama [6], and Tiny Time Mixers (TTM) [7], among others — now offer zero-shot or few-shot forecasting directly from a pretrained checkpoint, with no dataset-specific training loop required. Independently, PatchTST [5] demonstrated that a channel-independent, patched Transformer trained per-dataset could match or exceed these larger pretrained models on standard benchmarks, without itself being pretrained at foundation-model scale. Both developments matter for energy forecasting specifically: a utility or grid operator evaluating whether a zero-shot model is viable for a new feeder, region, or asset class needs a reproducible way to compare that option against both a trained-from-scratch baseline and a naive reference, under conditions that do not silently favor one paradigm over the other.

This manuscript is Paper 3 of the BQEB research program. Paper 1 established a reproducibility-first architecture for the program — Research Artifact, Reproducibility, Benchmark Evolution, and Scientific Lineage Layers — and populated it with one executed module, ForecastBench, addressing trained-from-scratch forecasting [8]. Paper 2 extended that architecture with two further layers, Threat and Governance-adjacent Benchmark structure specific to cyber-resilience evaluation, and populated the Benchmark Evolution Layer with a second module, SecBench, whose results remain reserved for future execution [9]. It does not redesign that architecture. Instead, it populates the Benchmark Evolution Layer with a third module — BQEB FoundationBench — addressing a task family neither prior paper covers: reproducible evaluation of pretrained foundation models under zero-shot and few-shot conditions.

1.1. Relationship to Papers 1 and 2

FoundationBench introduces no new architectural layer. The governing architecture remains the four-layer design established in Paper 1 — Research Artifact, Reproducibility, Benchmark Evolution, and Scientific Lineage — with SecBench additionally introducing a Threat Layer specific to its own cyber-resilience scope, which this paper does not use or extend. This paper registers within the existing Benchmark Evolution Layer as its third module, alongside ForecastBench (Module 1) and SecBench (Module 2). Where this paper’s own sections describe a mechanism the companion papers already define — the software registry pattern, the reproducibility guarantees, the governance concordance process — the description is intentionally brief and cites the relevant paper for the complete specification. What FoundationBench adds to the Benchmark Evolution Layer registry is a model-registry extension specific to pretrained models (Section 6); what it adds to the Scientific Lineage Layer is a deployment-mode disclosure requirement (Section 13) that neither ForecastBench’s trained-from-scratch models nor SecBench’s planned classifiers required, because neither prior module included a closed-source, API-only candidate the way this module’s registry does.

Task scope is deliberately narrow relative to what it could have been. FoundationBench does not define new forecasting targets: it reuses ForecastBench’s own two tasks — next-hour electricity load forecasting and day-ahead electricity price forecasting — on the same BQEB-Data v1 dataset [10], and evaluates a structurally different class of model against them under a structurally different protocol. This is a deliberate choice to avoid architectural duplication: the interesting research question this module answers is not “what should be forecast” but “how should a zero-shot forecaster be evaluated fairly against the trained-from-scratch baselines Paper 1 already established,” and answering that question does not require inventing new targets.

2. Related Work

Foundation models for time-series forecasting have developed along at least three distinct architectural lines, and this paper treats those lines as genuinely different rather than as interchangeable instances of one category. The first line adapts language-model architectures directly: Chronos tokenizes scaled, quantized time-series values and trains a T5-style encoder–decoder with a standard cross-entropy objective, ranging from 20M to 710M parameters [1]. The second line uses a decoder-only patched-Transformer design pretrained on very large real-world corpora: TimesFM is pretrained on approximately 100 billion real-world time points and produces zero-shot point and quantile forecasts from a 200M-parameter checkpoint [2], while Moirai uses a masked-encoder Transformer pretrained on the 27-billion-observation LOTSA archive spanning nine domains and is explicitly designed for arbitrary frequency and variate count [3]. The third line departs from the Transformer paradigm entirely: TTM uses a lightweight MLP-Mixer-style architecture with a parameter count far smaller than the other models discussed here, specifically targeting fast zero-shot and few-shot inference on multivariate series [7]. TimeGPT, developed by Nixtla, is offered exclusively as a hosted API rather than a downloadable checkpoint [4] — a deployment choice with direct consequences for reproducibility that Section 13 addresses explicitly rather than treating as incidental. Lag-Llama applies a decoder-only Transformer with explicit lag-based covariates to produce probabilistic zero-shot forecasts [6].

PatchTST occupies a different position in this literature and this paper does not flatten that distinction. Its patched, channel-independent Transformer design was introduced as a supervised and self-supervised architecture evaluated per-dataset, not as a model pretrained once and deployed zero-shot across new datasets [5]; where PatchTST appears in foundation-model comparison studies, it typically appears as a strong per-dataset reference point rather than as a like-for-like zero-shot competitor. The model registry (Section 6) records that distinction explicitly rather than listing all seven models as members of one undifferentiated category.

Two existing benchmark efforts evaluate time-series foundation models directly and are the closest points of comparison for this paper’s contribution. GIFT-Eval assembles 23 real-world datasets spanning roughly 144,000 individual series and 177 million observations across seven general domains — including energy as one domain among seven — and evaluates 17 baselines spanning statistical, deep-learning, and foundation-model approaches [11]. A second effort, FoundTS, similarly targets comprehensive, unified benchmarking of foundation models for time-series forecasting across general domains [12]. That breadth is a genuine strength on its own terms: a model that performs well across seven general domains offers stronger evidence of transferability than any single-domain result can, and no energy-specific benchmark, including this one, substitutes for that kind of broad evaluation. What neither resource provides is a governance layer connecting benchmark artifacts to a versioned research architecture, or a disclosed chain of evidence from raw dataset inspection through to reported metric — the gap Table 1 compares directly, without asserting superiority beyond what that comparison actually supports.

Reproducibility of machine learning research more broadly has been addressed as a methodological subject in its own right [13], and this paper extends that discipline — and the specific mechanisms of Papers 1 and 2 [8,9] — to a task family neither prior module addresses: zero-shot and few-shot evaluation of externally pretrained models, several of which the benchmark author does not control the training process, training data, or version history of.

3. Research Gap

Three conditions define the gap this paper addresses. First, no existing energy-specific benchmark evaluates time-series foundation models under a reproducibility and governance discipline comparable to what Papers 1 and 2 established for trained-from-scratch and planned classification tasks respectively; general-purpose suites such as GIFT-Eval and FoundTS cover energy only as one domain among several, not as a dedicated focus with a governed, versioned dataset behind it.

Second, treating foundation models as a single undifferentiated category obscures real differences in reproducibility guarantees. An open-weight, locally executable checkpoint — Chronos, TimesFM, Moirai, Lag-Llama, TTM — supports deterministic re-execution and checksum verification in a way a closed-source hosted API such as TimeGPT structurally cannot. That distinction carries direct governance consequences, and the registry specified in Section 6 is designed to surface it rather than obscure it.

Third, no BQEB-program artifact to date specifies what a fair zero-shot evaluation protocol looks like against the trained-from-scratch baselines Paper 1 already reports. Should the mandatory naive seasonal baseline, for instance, be computed identically to the one implicit in ForecastBench’s own results? Section 9 answers this explicitly rather than leaving it implicit.

4. Contributions

This paper makes five contributions, each scoped to benchmark framework and protocol design rather than to executed model comparisons.

• Contribution 1: A reproducibility-first benchmark framework for evaluating time-series foundation models on energy-forecasting tasks, registered as the third module of the existing Benchmark Evolution Layer rather than as a new architectural layer.
• Contribution 2: A foundation-model registry that records architecture, parameter count, pretraining corpus, deployment mode, and license for each candidate model, explicitly distinguishing open-weight zero-shot models, a closed-source API-only model, and a primarily supervised architecture rather than treating all seven as interchangeable.
• Contribution 3: A reuse-not-duplicate task design that applies a zero-shot and few-shot evaluation paradigm to ForecastBench’s existing forecasting targets, enabling direct comparison against Paper 1’s trained-from-scratch baselines under a shared dataset and partitioning discipline.
• Contribution 4: An evaluation protocol combining point metrics, probabilistic metrics, a mandatory naive-baseline comparison, and bootstrap confidence intervals, designed for the specific evaluation hazards zero-shot forecasting introduces (Section 9).
• Contribution 5: A governance extension addressing reproducibility risks unique to externally pretrained models — checkpoint versioning, API non-determinism, and license constraints on redistribution — that neither Paper 1 nor Paper 2’s governance sections needed to address.

5. FoundationBench Architecture

FoundationBench occupies the existing Benchmark Evolution Layer as its third module, shown in Figure 1. It does not modify the Research Artifact or Reproducibility Layers, and does not use SecBench’s separately-introduced Threat Layer, which has no bearing on a forecasting benchmark. It extends two layers: the Benchmark Evolution Layer, via a model-registry entry type specific to pretrained foundation models (Section 6); and the Scientific Lineage Layer, via a deployment-mode disclosure requirement that records whether a registered model can be locally re-executed, checksummed, and version-pinned, or whether — as with an API-only model — those guarantees cannot be made (Section 13).

Table 1 compares FoundationBench against the two existing general-purpose foundation-model benchmarks identified in Section 2. The comparison is deliberately narrow: FoundationBench does not claim broader dataset coverage than GIFT-Eval or FoundTS, both of which span substantially more domains and series. Its distinguishing property is architectural — a governed, versioned dataset with a disclosed evidence chain, inherited reproducibility guarantees from Papers 1 and 2, and a benchmark specifically scoped to one domain rather than general-purpose coverage.

6. Foundation Model Registry

The registry extends Papers 1 and 2’s software registry pattern with a record type specific to externally pretrained models, shown in Figure 2. Each entry records architecture family, parameter count, pretraining corpus (where disclosed by the model’s own publication), deployment mode, and license, and is classified into exactly one of three deployment modes: open-weight zero-shot, closed-source API, or primarily supervised. This classification is the registry’s central design decision, because it determines what reproducibility guarantee a downstream evaluation using that model can honestly make.

The registry deliberately does not attempt to force a single evaluation protocol onto all three deployment modes. An open-weight zero-shot model can be executed locally under a fixed seed, checksummed, and re-executed independently; a closed-source API model cannot be checksummed against a fixed weight file, and its outputs may change between calls if the provider updates the underlying model without a version change visible to the caller — a real, disclosed limitation Section 13 returns to rather than one this manuscript resolves by assumption. A primarily supervised model such as PatchTST requires a per-dataset training step that neither of the other two modes requires, meaning its inclusion in a “zero-shot” comparison table would misrepresent what was actually evaluated unless the training step is disclosed alongside the result. Table 4 (Section 15) enforces this disclosure by including a deployment-mode column in the planned-results template.

7. Benchmark Tasks

Consistent with the formal specification style established across this research program, FoundationBench is defined as a 5-tuple B = (D, T, M, E, R):

• D — the dataset: BQEB-Data v1’s load and price forecasting fields [10], reused unmodified from Paper 1, with no FoundationBench-specific dataset variant.
• T — the task set {T1, T2, T3}: zero-shot/few-shot load forecasting, zero-shot/few-shot price forecasting, and the PatchTST supervised reference (Section 7.1, Section 7.2 and Section 7.3), reusing ForecastBench’s own two forecasting targets rather than defining new ones.
• M — the model registry: the seven candidate models in Table 2, each classified into exactly one deployment mode (open-weight zero-shot, closed-source API, or primarily supervised).
• E — the evaluation function: MAE, RMSE, and MASE as point metrics (Section 9.1), CRPS where a predictive distribution is available (Section 9.2), and the mandatory naive-baseline comparison (Section 9.3).
• R — the reproducibility artifact set: pinned checkpoint identifiers, deployment-mode disclosure, and the evidence-tier classification inherited from the companion architecture paper’s Reproducibility Layer.

A specific benchmark run is a tuple (d, t, m) ∈ D × T × M producing an evaluation e ∈ E and a reproducibility record r ∈ R. No run of this form has been executed at the time of writing; every symbol above names a specification, not a completed computation.

FoundationBench defines two tasks, both reused directly from ForecastBench rather than newly specified: next-hour electricity load forecasting and day-ahead electricity price forecasting, evaluated on the same BQEB-Data v1 dataset [10] under the same chronological ordering Paper 1 established. What changes is not the target but the evaluation paradigm applied to it.

7.1. Task 1: Zero-Shot Load Forecasting

A registered zero-shot model receives a context window of historical load observations and produces a forecast for the next hour without any gradient update on BQEB-Data v1. Few-shot variants, where explicitly supported by a given model’s interface, may additionally receive a small number of in-context examples drawn only from the training partition, never from the validation or test partitions.

7.2. Task 2: Zero-Shot Price Forecasting

The same paradigm applied to day-ahead electricity price forecasting, using the same context-window and few-shot constraints as Task 1. Consistent with Paper 1’s own finding that price forecasting is the harder of the two tasks for trained-from-scratch models, this task is expected — not reported, since no execution has occurred — to be at least as difficult for zero-shot models given the added absence of any dataset-specific training signal.

7.3. Task 3: Supervised Reference (PatchTST)

Because PatchTST’s primary operating mode is per-dataset training rather than zero-shot inference, it is evaluated as a supervised reference point under the same protocol ForecastBench already established for Paper 1’s baseline models, not folded into the zero-shot comparison tables as though it were directly comparable. This task exists specifically to prevent the registry’s own distinction (Section 6) from being undone by a results table that erases it.

Table 3. Formal task specification for T1–T3, following the B = (D, T, M, E, R) definition above.

Task	Objective	Input	Output	Labels/Targets	Assumptions	Success Criterion	Recommended Metrics
T1	Zero-shot/few-shot next-hour load forecast	Context window of historical load_mw; optional in-context examples from training partition only	Point or quantile forecast for next hour	load_mw (Table, Paper 1)	No gradient update on BQEB-Data v1; few-shot examples never from validation/test	Forecast beats naive same-hour-yesterday baseline (Sec. 9.3)	MAE, RMSE, MASE; CRPS where applicable
T2	Zero-shot/few-shot next-day price forecast	Context window of historical energy_price_usd_mwh; same few-shot constraints as T1	Point or quantile forecast for next day	energy_price_usd_mwh (Table, Paper 1)	Expected at least as hard as T1, given no dataset-specific training signal	Forecast beats naive same-hour-previous-week baseline (Sec. 9.3)	MAE, RMSE, MASE; CRPS where applicable
T3	Supervised reference point for T1–T2 comparison	Same context windows as T1/T2, plus a per-dataset training step	Point forecast (PatchTST’s standard configuration)	Same targets as T1/T2	Reported separately, never folded into zero-shot comparison tables	Establishes the trained-from-scratch reference Paper 1 already reports	MAE, RMSE, MASE (CRPS reported N/A)

Table 3. Formal task specification for T1–T3, following the B = (D, T, M, E, R) definition above.

Task	Objective	Input	Output	Labels/Targets	Assumptions	Success Criterion	Recommended Metrics
T1	Zero-shot/few-shot next-hour load forecast	Context window of historical load_mw; optional in-context examples from training partition only	Point or quantile forecast for next hour	load_mw (Table, Paper 1)	No gradient update on BQEB-Data v1; few-shot examples never from validation/test	Forecast beats naive same-hour-yesterday baseline (Sec. 9.3)	MAE, RMSE, MASE; CRPS where applicable
T2	Zero-shot/few-shot next-day price forecast	Context window of historical energy_price_usd_mwh; same few-shot constraints as T1	Point or quantile forecast for next day	energy_price_usd_mwh (Table, Paper 1)	Expected at least as hard as T1, given no dataset-specific training signal	Forecast beats naive same-hour-previous-week baseline (Sec. 9.3)	MAE, RMSE, MASE; CRPS where applicable
T3	Supervised reference point for T1–T2 comparison	Same context windows as T1/T2, plus a per-dataset training step	Point forecast (PatchTST’s standard configuration)	Same targets as T1/T2	Reported separately, never folded into zero-shot comparison tables	Establishes the trained-from-scratch reference Paper 1 already reports	MAE, RMSE, MASE (CRPS reported N/A)

8. Dataset Requirements

FoundationBench requires no new dataset. It reuses BQEB-Data v1 [10] exactly as Paper 1 validated it: 10,512 records at 15-minute resolution, load_mw and energy_price_usd_mwh as the two forecast targets, and the same chronological ordering. What foundation-model evaluation adds is a context-window requirement that trained-from-scratch evaluation did not need to specify: each registered model’s minimum and maximum supported context length must be recorded in the model registry (Table 2) and checked against the dataset’s available history before a task is attempted, since a model requiring more historical context than a given evaluation window provides cannot be fairly evaluated on that window without disclosure of the shortfall.

BQEB-Data v1 remains synthetic — generated through stochastic simulation rather than field measurement, a status disclosed in the dataset’s own documentation and repeated here because it bears directly on how any foundation-model result on this benchmark should eventually be read. A zero-shot model’s real value proposition is generalization from broad, real-world pretraining to a new deployment context; evaluating that capability against a synthetic simulator is a meaningfully different claim than evaluating it against field-metered utility data, and this manuscript does not present the two as equivalent.

9. Evaluation Protocol

Point-forecast accuracy alone is not sufficient for this benchmark, for a reason distinct from the class-imbalance argument SecBench makes for its own protocol: several of the registered models (Chronos, Lag-Llama, TimesFM) produce probabilistic forecasts natively, and reducing their output to a point estimate before scoring discards exactly the capability a zero-shot deployment decision would most want evaluated. The protocol therefore specifies both metric families.

9.1. Point Metrics

Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE), defined identically to Paper 1’s usage, are reported for every model regardless of whether it natively produces point or probabilistic output; probabilistic models are point-scored using their median forecast.

The Mean Absolute Scaled Error (MASE) is additionally required specifically because it enables comparison across the two forecasting tasks (load and price), which have different units and different naive-baseline difficulty, in a way raw MAE cannot:

MASE = mean(|yᵢ − ŷᵢ|) / mean(|yᵢ − yᵢ₋ₘ|) (1)

where m is the seasonal period used by the mandatory naive baseline (Section 9.3) and the denominator is the in-sample mean absolute error of that naive baseline.

9.2. Probabilistic Metrics

The Continuous Ranked Probability Score (CRPS) is required for every model capable of producing a predictive distribution or quantile forecasts, since it is the standard scoring rule for probabilistic forecasts and rewards well-calibrated uncertainty rather than only central-tendency accuracy. Models that produce point forecasts only (PatchTST in its standard configuration, and TTM’s point-forecast head) report CRPS as not applicable rather than being assigned an estimated or imputed value.

9.3. Mandatory Naive Baseline

Every reported result must be accompanied by a naive seasonal baseline (same-hour-yesterday for load, same-hour-previous-week for price) computed on the identical test window. This mirrors the discipline SecBench applies to its majority-class baseline: a zero-shot foundation model that fails to outperform a naive seasonal baseline has not demonstrated forecasting value regardless of how sophisticated its architecture is, and the protocol requires this comparison be visible in every reported table rather than available only on request.

10. Statistical Evaluation

Zero-shot evaluation introduces a specific statistical hazard trained-from-scratch evaluation does not: a single forecast run reflects one draw from whatever stochasticity the model’s inference procedure contains (sampling temperature, dropout-at-inference where applicable, or API-side non-determinism for TimeGPT), and reporting it as a single point estimate would overstate the precision of the comparison. The protocol specifies bootstrap confidence intervals computed by resampling the test window with replacement across multiple inference runs per model, and requires that any comparison between two models be accompanied by a paired statistical test on the same test instances rather than a bare ranking of point estimates. No confidence intervals, test statistics, or p-values are reported in this manuscript, consistent with Section 15.

11. Software Architecture

FoundationBench’s software design extends the ForecastBench package’s [14] registry pattern (Paper 1) rather than introducing a parallel codebase, following the same precedent SecBench established for Module 2. Figure 3 shows the planned evaluation pipeline. Registries added are: a model registry (Table 2, Section 6); an inference-configuration registry recording context length, quantile levels requested, and few-shot example count per run; and a results registry scoped to the two reused forecasting targets. Experiment tracking, artifact checksumming, and the deterministic-execution guarantees of the companion architecture paper’s Reproducibility Layer are inherited unchanged for every open-weight model; Section 13 states explicitly where they cannot be inherited unchanged for the one closed-source model in the registry.

12. Reproducibility Framework

This framework’s deterministic-execution, partitioning, and artifact-checksumming mechanisms are defined in the companion architecture paper and are not re-derived here. FoundationBench adopts them without modification for every open-weight registered model. What this module adds is a checkpoint-provenance requirement specific to externally pretrained models: every registry entry records the exact checkpoint identifier (Hugging Face revision hash or equivalent) used in any reported run, since a model’s public checkpoint can be updated by its publisher between the time a benchmark is designed and the time it is executed, and a result reported against an unpinned checkpoint identifier is not independently reproducible even in principle.

Figure 4 shows the reporting lifecycle this requirement feeds into. Where two reported runs of the same model and checkpoint disagree — the same category of discrepancy Paper 1’s own Scientific Lineage Layer disclosed for two published ForecastBench results — this framework requires the discrepancy be disclosed in the governance log rather than resolved by preferring one run silently.

13. Governance

Governance responsibilities — identifier management, version concordance, and cross-artifact reconciliation — are inherited from the companion architecture paper rather than redefined. This module’s governance extension addresses one condition neither Paper 1 nor Paper 2 needed to address: a registered model whose weights, training data, and inference behavior are not under the benchmark author’s control. For TimeGPT specifically, the registry (Table 2) records that no local checkpoint exists, that inference occurs through a hosted API whose underlying model version is not independently verifiable by the caller, and that any reported TimeGPT result is consequently reproducible only in the weaker sense of “repeatable against the same API endpoint at a later date,” not in the stronger sense of “independently re-executable from a checksummed artifact” that applies to the other six registered models. This distinction is disclosed here as a governance constraint, not treated as disqualifying — TimeGPT remains a scientifically relevant comparison point precisely because it represents a real, commercially available deployment option grid operators may actually consider.

14. Experimental Plan

Completed

• Identification and verification of seven candidate foundation models against their original publications (Section 2, Table 2).
• Specification of the model registry, deployment-mode classification, and benchmark tasks (Section 6, Section 7 and Section 8).
• Specification of the evaluation protocol, including point and probabilistic metrics and the mandatory naive baseline (Section 9).
• Software and governance extension design (Section 11, Section 12 and Section 13).

Planned

• Registration of all seven models with pinned checkpoint identifiers where applicable.
• Zero-shot and few-shot inference runs for Tasks 1–2 across all open-weight and API-based models.
• Supervised training and evaluation of PatchTST as the Task 3 reference point.
• Statistical validation per Section 10 (bootstrap confidence intervals, paired significance testing, multi-run sensitivity).

15. Benchmark Execution Status and Future Validation

Results Pending Experimental Execution

No foundation model has been executed against BQEB-Data v1 under this protocol at the time of writing. This section is deliberately not titled ‘Results’: it reports execution status, not findings. Table 4 is provided in the exact structure a completed benchmark run will populate, including an explicit deployment-mode column, so that a companion execution study can report results in a directly comparable and honestly-labeled form.

15.1. Future Experimental Study

The following is planned for a companion execution study and is not performed in this manuscript. All seven registered models will be evaluated on both tasks under the protocol specified in Section 7, Section 8, Section 9 and Section 10. Reporting will follow Table 4’s structure exactly, including for any model that fails to outperform its naive baseline — a negative result under this protocol is a valid, reportable outcome, not an omission. This list is a planning specification, not a commitment to any reported outcome for any named model.

16. Discussion

FoundationBench’s contribution at this stage is benchmark infrastructure — a model registry that refuses to flatten real differences between deployment modes, a task design that reuses rather than duplicates existing BQEB targets, and an evaluation protocol built for the specific hazards zero-shot forecasting introduces — not a demonstrated finding about which foundation model forecasts energy load or price best. That question is exactly the one Section 15.1 defers to a companion study. The architectural choice most likely to be scrutinized is the decision to include a closed-source API model in the same registry as six open-weight models rather than excluding it on reproducibility grounds; Section 13’s position is that disclosure, not exclusion, is the more useful response, since excluding TimeGPT would not make it disappear from the set of options a real deployment decision would actually consider.

Generalizability Beyond BQEB-Data

This specification is written against BQEB-Data v1’s two forecasting targets specifically, and that dependency should be stated rather than left implicit. Extending FoundationBench to a future dataset or a new forecasting domain requires no new architectural mechanism: a new dataset registers at the Research Artifact Layer under the pattern Paper 1 already established, and the task set T is extended by adding a new target rather than by redesigning the registry, provided the new target follows the same chronological-partitioning discipline Section 8 requires. New foundation models are, if anything, the extension case this framework is best prepared for: Section 6’s deployment-mode classification (open-weight, closed-source API, or primarily supervised) was designed specifically so that a model released after this paper is written can be registered under the existing three-way classification rather than requiring a new category.

Backward compatibility follows from the same registry discipline: adding a model or a dataset does not alter how a previously registered model’s results are interpreted, since each registry entry’s checkpoint identifier and deployment mode are recorded at registration time and are not retroactively affected by later additions. Governance of this evolution is inherited from the companion architecture paper’s Scientific Lineage Layer rather than redefined here — the same n=1 boundary that paper discloses for its own extensibility claim applies to this module too: the registry mechanism is specified and has processed seven real model entries, but has not yet been exercised by an independent party registering an eighth. Closing that gap is future work, not a claim this manuscript makes about itself.

17. Threats to Validity

• Synthetic data. Every FoundationBench task inherits BQEB-Data v1’s synthetic-generation status; zero-shot performance on this benchmark is a statement about generalization from real-world pretraining to a synthetic simulator, not a validated statement about generalization to field-metered utility data.
• Heterogeneous reproducibility guarantees. The registry spans models with fundamentally different reproducibility properties (open-weight vs. closed-source API); aggregating them into one leaderboard without the deployment-mode column this protocol requires would misrepresent what was actually compared.
• Context-length mismatch. Registered models differ substantially in minimum and maximum supported context length; a model whose minimum context exceeds what a given evaluation window provides cannot be fairly scored on that window, and this threat is only partially mitigated by the disclosure requirement in Section 8.
• Checkpoint drift. Open-weight models’ public checkpoints can be updated by their publishers between benchmark design and execution; the checkpoint-pinning requirement in Section 12 mitigates but does not eliminate this risk for any evaluation that spans a long execution window.
• API non-determinism. TimeGPT results cannot be guaranteed bit-identical across repeated calls, a limitation disclosed in Section 13 rather than resolved.
• Single dataset, single simulated region. No cross-dataset or cross-region validation exists yet for this benchmark.
• PatchTST comparability. Task 3’s supervised reference point is not directly comparable to Tasks 1–2’s zero-shot results, and any future report that ranks all seven models on one table without preserving this distinction would misrepresent the benchmark’s own design.

18. Future Work

• Execution of the experimental plan in Section 14, producing the first real results for this benchmark.
• Extension of the model registry to additional foundation models as they are released, following the same deployment-mode disclosure discipline established here.
• Validation against real or semi-real utility telemetry, where access constraints permit, to test generalization beyond the synthetic simulator.
• Fine-tuning and few-shot adaptation studies, distinguished explicitly from the zero-shot results this protocol prioritizes.
• Extension to additional BQEB-Data v1 targets beyond load and price, should a future module require them — explicitly not undertaken in this paper to avoid unscoped task growth.

19. Conclusion

This paper has specified BQEB FoundationBench, the third module of the BQEB Benchmark Evolution Layer, extending the companion architecture’s existing four-layer design with a model registry and evaluation protocol specific to pretrained foundation models. Two reused forecasting tasks, a deployment-mode-aware model registry spanning seven real, verified models, and a governance extension addressing externally pretrained model risks are fully specified. No results are reported: Section 15 marks them explicitly reserved for future experimental execution, and this disclosure is treated as part of the contribution rather than a gap to be minimized. Figure 5 situates this module within the program’s broader roadmap.