BQEB SecBench: A Reproducible Cyber-Resilience Benchmark Framework for Intelligent Energy Systems

Rakesh Kumar Agrawal; Wasim Mohammed Amin Tambe; Nihar Karra

doi:10.20944/preprints202609.0013.v2

Submitted:

09 September 2026

Posted:

11 September 2026

You are already at the latest version

Abstract

Cyber-physical convergence in modern power systems has expanded the attack surface available to adversaries capable of manipulating sensor data, spoofing measurements, disrupting communications, or injecting false state estimates, yet publicly available benchmark infrastructure for evaluating AI-based cyber-event detection in smart-grid settings remains scarce relative to the benchmark infrastructure available for forecasting and dispatch tasks. This paper introduces BQEB SecBench, a reproducible cyber-resilience benchmark framework extending the four-layer BQEB architecture (Research Artifact, Reproducibility, Benchmark Evolution, Scientific Lineage) established in a companion paper to a fifth, benchmark-specific layer and a new Threat Layer. SecBench targets the security-relevant fields of the BQEB-Data v1 synthetic smart-grid dataset, which have been publicly available since the dataset's release but have not, to our knowledge, been benchmarked in any prior publication. Inventory inspection of the distributed attack_class field shows severe class imbalance: of 10,512 records, 10,331 (98.28%) carry no attack label, while four documented event types — sensor fault (82), spoofing (37), denial-of-service (34), and false data injection (28) — make up the remainder. We treat this imbalance as the central design problem the framework must solve and specify four benchmark tasks, an imbalance-aware evaluation protocol built on precision-recall and threshold-independent metrics rather than accuracy, and reproducibility and governance mechanisms extending the software architecture of the companion paper. This manuscript introduces the benchmark framework and evaluation protocol; empirical model comparisons are reserved for subsequent experimental validation. No detection results, accuracy figures, or statistical claims are reported: every quantity that would depend on model execution is explicitly marked reserved for future experimental validation. We regard this disclosure as part of the framework's contribution, not a limitation to be minimized.

Keywords:

cyber resilience

;

smart-grid cybersecurity

;

AI benchmarking

;

intrusion detection

;

energy AI

;

reproducible research

;

synthetic datasets

Subject:

Computer Science and Mathematics - Artificial Intelligence and Machine Learning

I. Introduction

Power systems now depend on a dense layer of sensing, communication, and control infrastructure whose compromise can translate directly into physical operating consequences: a manipulated measurement can misinform state estimation, an unavailable communication link can stall a protective action, and a falsified sensor reading can trigger an incorrect operational decision at grid scale. AI-based security analytics — detecting false data injection, sensor manipulation, denial-of-service disruption, and related events from operational data — has accordingly become an active research area [1,2,3,4]. What this area lacks, relative to forecasting and optimization, is shared benchmark infrastructure: public, labeled, reproducibly evaluable data against which detection methods can be compared under a common protocol, rather than each study reporting results against its own dataset, its own preprocessing choices, and its own notion of what counts as an attack.

This manuscript is a companion module study within the BQEB research program. The companion architecture paper established a four-layer reproducibility architecture — research artifact, reproducibility, benchmark evolution, scientific lineage — and populated it with one executed module, ForecastBench, addressing forecasting tasks [5]. This paper does not restate that architecture; it is used here only as background, architectural reference, and reproducibility foundation. What this paper contributes is a second module, BQEB SecBench, extending the architecture to cyber-resilience benchmarking, and two new architectural layers — Threat and Benchmark — that paper did not require.

Consistent with the evidentiary discipline established across this research program, this paper distinguishes clearly between what has been designed and what has been executed. The benchmark framework’s tasks, threat model, evaluation protocol, and software and governance extensions are specified completely. No detection model has yet been trained or evaluated against them. Section XV states this explicitly rather than allowing it to be inferred.

This paper makes explicit three questions the sections above answer implicitly, so that a reader can check the paper’s own claims against them directly rather than infer what is being asked:

RQ1: Can a cyber-resilience benchmark be specified precisely enough, under severe class imbalance, that a future evaluation is fully determined by the specification rather than by choices made at execution time?
RQ2: Can the BQEB architecture’s reproducibility and lineage mechanisms extend to a second, structurally different module (classification under extreme imbalance) without modification to their core design?
RQ3: Where a benchmark’s own protocol design is itself a contested methodological choice, can that choice be disclosed and justified before any result exists to defend it?

This paper answers RQ1 and RQ2 directly: Sections V–X specify the benchmark completely, and Section XI–XIII show the architecture’s mechanisms extending without modification. RQ3 is answered by construction throughout — Section IX’s metric choices, Section VII’s split-strategy justification, and Section XVII’s threats are each argued before execution, not after. No result exists yet to make any of these questions easier or harder to answer honestly.

A. Relationship to the BQEB Architecture Paper

The companion architecture paper and this manuscript serve different, non-overlapping purposes. It establishes the research infrastructure architecture — the four-layer design, the software registry pattern, the reproducibility mechanisms, and the governance process — and demonstrates it with one executed benchmark module. This paper instantiates a second module within that already-established architecture; it does not re-argue why the architecture should exist or re-derive how its layers work. Concretely, ForecastBench [5] is the architecture’s forecasting benchmark, evaluating regression tasks on continuous targets; SecBench is the architecture’s cybersecurity benchmark, evaluating classification tasks on a severely imbalanced categorical target. Where this paper’s own sections describe an architectural mechanism the companion architecture paper already defines, the description is intentionally brief and cites it for the complete specification rather than reproducing it.

Positioning: SecBench is not a forecasting paper adapted to security, and it is not a claim of detection capability. It is a benchmark specification — tasks, threat model, split strategy, and metric set — for a class-imbalance problem the companion architecture’s first module (forecasting) never had to solve. Where this paper is silent on a result, that silence is the finding: no number in this manuscript should be read as implying what a future run would show.

II. Background and Related Work

False data injection against power-system state estimation was formalized by Liu, Ning, and Reiter, who showed that an attacker with partial network-topology knowledge can construct injections that evade classical bad-data detection [1]. A substantial subsequent literature addresses detection of this and related attack types — sensor spoofing, denial-of-service, and communication-layer manipulation — using statistical, model-based, and learning-based methods [2,3,4].

The dominant public benchmark resource in this specific space is the Industrial Control System Cyber Attack Power System dataset developed by Mississippi State University and Oak Ridge National Laboratory, generated from a laboratory-scale transmission testbed instrumented with phasor measurement units, relay logs, and Snort network alerts, and released in binary, three-class, and multiclass (up to 37-class) variants [6]. It is, to our knowledge, the most widely cited benchmark resource for smart-grid intrusion detection and is used as the primary comparison point in Table 1.

Severe class imbalance is a recognized, general challenge in intrusion-detection benchmarking: He and Garcia’s survey catalogs the sampling, cost-sensitive, and kernel-based responses to this challenge [8]. SMOTE [7] is the most widely adopted resampling response and informs the evaluation protocol in Section IX.

Task 3 (Section VIII) frames unknown-attack detection as an open-set recognition problem. Scheirer et al. formalize the distinction between closed-set and open-set recognition and argue that closed-set evaluation systematically overstates a classifier’s real-world reliability whenever the deployment environment can present categories absent from training [9].

Reproducibility of machine learning research has been addressed as a methodological subject in its own right [10], and this paper extends that discipline — and the specific mechanisms of the companion architecture paper [5] — to a task family regression-focused benchmark work does not require: imbalanced classification under an extremely small minority class.

Table 1. Comparison of Existing Smart-Grid Cybersecurity Benchmarks.

Resource	Dataset type	Task coverage	Imbalance handling	Software / reproducibility	Governance
MSU-ORNL Power System Attack Datasets [6]	Laboratory transmission testbed (PMU, relay, Snort logs)	Binary / 3-class / up to 37-class	Not standardized; widely reported as a challenge	Not centrally maintained; static files	N/A (static release)
General network-intrusion sets (NSL-KDD / CICIDS family)	General IT network traffic, not grid-native	Binary / multiclass	Varies by study	Not centrally maintained	N/A
BQEB SecBench (this work)	Synthetic smart-grid simulation	Binary, multiclass, open-set, imbalance-focused	Explicit design objective (Sec. IX)	Extends tested, versioned, checksummed registry (Sec. XI)	Explicit Scientific Lineage Layer; disclosed precedent finding (Sec. XIII)

III. Research Gap

Three conditions define the gap this paper addresses. First, BQEB-Data v1’s security-relevant fields have been publicly available since the dataset’s initial release but remain unbenchmarked in any BQEB publication or, to our knowledge, any external one. Second, this is a previously named but unstarted item in the BQEB program’s own public roadmap. Third, no dataset-generation documentation available to us specifies how the attack_class labels were produced — a gap this paper discloses as a threat to validity (Section XVII) rather than resolves by assumption.

IV. Contributions

This paper makes five contributions, each scoped to framework and protocol design rather than to executed detection performance.

Contribution 1: A reproducible cyber-resilience benchmark framework for intelligent energy systems, specifying the tasks, data scope, and artifact structure a cyber-event detection study on BQEB-Data v1 requires.
Contribution 2: An extension of the BQEB architecture (Paper 1) with two new layers — a Threat Layer encoding the attack taxonomy and adversary model, and a Benchmark Layer defining SecBench’s tasks and evaluation protocol.
Contribution 3: An imbalance-aware cybersecurity evaluation protocol, built on precision-recall and threshold-independent metrics with a mandatory majority-class baseline.
Contribution 4: A governance and artifact-reproducibility approach for the cyber-resilience track, extending Paper 1’s identifier-management and cross-artifact reconciliation practices.
Contribution 5: A modular design allowing future integration of additional cybersecurity AI methods by registration within the existing software architecture rather than by parallel implementation.

V. BQEB Secbench Architecture

SecBench extends the companion architecture paper’s four-layer architecture with two additional layers specific to cyber-resilience benchmarking, shown in Figure 1. The Research Artifact Layer is unchanged in mechanism but newly scoped to the dataset’s security-relevant fields. A new Threat Layer sits above it, encoding the attack taxonomy and threat model (Section VI). The Benchmark Layer defines SecBench’s four tasks and evaluation protocol (Sections VIII–IX). The Benchmark Evolution and Scientific Lineage Layers are extended, not replaced: the former inherits the companion paper’s registry and extension pattern, and the latter inherits its identifier-management and version-concordance mechanisms.

VI. Cyber Threat Model

The threat model considers an adversary capable of one or more of four actions documented in the dataset’s attack taxonomy (Table 2): injecting false sensor or measurement data, manipulating individual sensors directly, disrupting communication availability, or otherwise producing an operational anomaly consistent with compromise. The model does not assume a specific attacker capability level beyond what the dataset’s own documentation supports. Figure 3 traces this path from a documented attack type through to the benchmark’s reported metrics, at the level of a single record.

Detection boundaries are defined at the level of the record: a model under this benchmark observes the same operational and environmental features available to ForecastBench’s forecasting tasks (Paper 1, Table 2) and is evaluated on its ability to assign the correct attack_class label to a given record. This is a detection-from-telemetry framing, not a network-traffic or protocol-level intrusion-detection framing, and Section XVII states this distinction as a scope boundary rather than a limitation to be argued away.

Figure 2. Smart-grid cyber-threat taxonomy: observed attack_class distribution in BQEB-Data v1 (direct CSV inventory, third-hand via the companion architecture paper’s source material).

Table 2. attack_class Inventory in BQEB-Data v1.

attack_class value	Row count	Share of dataset
none	10,331	98.28%
sensor_fault	82	0.78%
spoofing	37	0.35%
dos	34	0.32%
false_data_injection	28	0.27%

This is dataset inventory evidence, not an experimental result — no model, classifier, or detection procedure produced these counts; they are a direct tally of the field as distributed.

Figure 3. SecBench threat-to-detection pipeline, from a documented attack type (Table 2) through feature extraction to the benchmark metric set (Table 4) and evidence tier (Table 5). This diagram traces a single record’s path through the benchmark specification; no arrow represents an executed result.

VII. Dataset Characterization

BQEB-Data v1 [13] is synthetic — generated through stochastic simulation rather than field measurement — a status disclosed in the dataset’s own documentation and repeated here because it bears directly on how any detection result on this benchmark should eventually be read. This benchmark framework makes no claim of real utility deployment, real-world intrusion validation, or production cybersecurity capability at any point in this manuscript. Its security-relevant fields comprise attack_class (five-valued categorical), cyber_alert_flag, sensor_fault_flag, anomaly_score, and failure_risk_score.

Inventory inspection of the distributed attack_class field, reported in the BQEB program’s own dataset validation material, establishes the inventory in Table 2: 10,331 records (98.28%) labeled none, and 181 records (1.72%) distributed across four attack types ranging from 82 (sensor_fault) down to 28 (false_data_injection). This inventory is treated as directly observed evidence, carried forward from the companion architecture paper’s source material rather than freshly re-inspected from the raw file by this manuscript’s authors. The corresponding distribution for cyber_alert_flag, sensor_fault_flag, and anomaly_score has not been separately inspected and requires validation from the dataset artifact directly.

With only 28 positive examples in the rarest class, a conventional chronological 70/15/15 split risks leaving single-digit representation in validation and test partitions. Section VIII and Figure 4 treat the resulting split-strategy decision as one requiring explicit justification rather than a silent default.

VIII. Secbench Benchmark Tasks

Consistent with the goal of making this specification directly implementable, SecBench is defined formally as a 5-tuple B = (D, T, M, E, R), following the same structural pattern the companion architecture paper uses for its own module definitions:

D — the dataset: BQEB-Data v1’s security-relevant field subset (Section VII), fixed for this benchmark version.
T — the task set {T1, T2, T3, T4}: binary detection, multiclass classification, open-set unknown-attack detection, and the mandatory imbalance-aware reporting track (Section VIII).
M — the model registry: any classifier registered under Section XI’s model registry, evaluated identically across T1–T3.
E — the evaluation function: the metric set in Table 4, applied per class and in aggregate, with accuracy explicitly excluded as a primary metric.
R — the reproducibility artifact set: checksummed outputs, seed and split configuration, and the evidence-tier classification in Table 5.

A specific benchmark run is then a tuple (d, t, m) ∈ D × T × M producing an evaluation e ∈ E and a reproducibility record r ∈ R. No run of this form has yet been produced; every symbol above names a specification, not a completed computation.

Four tasks are defined. Task 1, binary cyber-event detection, distinguishes any attack-labeled record from none and partially offsets the multiclass sample-size problem by pooling all 181 positive records into a single class. Task 2, multiclass attack classification, retains the five-way distinction in Table 2 and is expected to be the harder task given the smallest class’s size. Task 3, unknown-attack detection, is an open-set framing in which a model trained on a subset of attack types is evaluated on its ability to flag a held-out attack type as anomalous without having been trained to name it. Task 4 is not a separate detection task but an explicit evaluation track: every result reported under Tasks 1–3 must be accompanied by the imbalance-aware reporting specified in Section IX, rather than accuracy alone.

Table 3. Formal task specification, following the B = (D, T, M, E, R) definition above.

Task	Input	Output	Labels	Assumptions	Success criterion
T1: Binary	Record features (Sec. VI)	P(attack)	{none, attack} (181 vs. 10,331)	All 4 attack types pooled as one class	Reported per Table 4; majority-class baseline mandatory
T2: Multiclass	Record features (Sec. VI)	P(class) over 5 classes	5-way (Table 2)	Rarest class n=28 sets the effective floor on any per-class claim	Macro-F1, MCC per Table 4, not accuracy
T3: Open-set	Record features, subset of attack types at train time	In-distribution label or ‘unknown’	Held-out attack type withheld from training	Follows Scheirer et al.’s open-set framing [9]	Correctly flags held-out type as anomalous without naming it
T4: Reporting track	Any T1–T3 result	Imbalance-aware report	N/A — applies to reported results, not raw records	Accuracy alone is never sufficient evidence	Full Table 4 metric set present in any reported result

Task 3 evaluates open-set recognition by withholding an entire attack type for evaluation. This design intentionally creates a challenging generalization scenario but also interacts with the dataset’s inherent class imbalance. Because the smallest documented attack class contains only 28 instances (Table 2), withholding an entire attack type simultaneously reduces the diversity of attack patterns available during training and limits the number of evaluation samples. Consequently, results for Task 3 should be interpreted in the context of these statistical constraints. The benchmark is intended to stimulate research on open-set recognition, few-shot learning, and foundation-model approaches under realistic data scarcity rather than to provide definitive comparative rankings. Future benchmark releases incorporating additional datasets and broader attack taxonomies are expected to improve statistical robustness while preserving reproducibility and comparability across benchmark versions.

IX. Evaluation Methodology

Accuracy alone is not reported as a primary metric in this benchmark: at 98.28% majority-class prevalence, a trivial always-predict-’none’ classifier achieves 98.28% accuracy while detecting no attacks whatsoever. The protocol instead specifies precision, recall, and F1-score per class; macro-F1 and weighted-F1 for aggregate comparison; balanced accuracy; the Matthews correlation coefficient (MCC), which Chicco and Jurman show remains informative in cases where F1 and accuracy both produce misleadingly optimistic scores [11]; and PR-AUC in preference to ROC-AUC, which is known to present an overly optimistic picture under this level of skew.

The Matthews correlation coefficient is defined as

MCC = (TP·TN − FP·FN) / √[(TP+FP)(TP+FN)(TN+FP)(TN+FN)]

(1)

and balanced accuracy as the unweighted mean of per-class recall,

Balanced Accuracy = (1/C) Σᶜ Recallᶜ

(2)

where C is the number of classes. Table 4 lists the complete metric set with the rationale for including each.

Table 4. SecBench Evaluation Metric Set.

Metric	Role
Precision, Recall, F1 (per class)	Class-level detection quality
Macro-F1 / Weighted-F1	Aggregate comparison across classes
Balanced Accuracy	Aggregate comparison robust to skew
MCC	Single-number summary informative under severe imbalance [11]
PR-AUC	Threshold-independent, preferred over ROC-AUC under skew

X. Statistical Evaluation

With as few as 28 positive examples in a class, point-estimate metrics reported without uncertainty bands would materially overstate what the benchmark can support. The protocol specifies bootstrap confidence intervals computed by resampling the test set with replacement; McNemar’s test or an equivalent paired test for comparing two classifiers on the same test instances; and multiple random seeds or split realizations reported alongside, not instead of, a primary seeded run. No confidence intervals, test statistics, or p-values are reported in this manuscript, consistent with Section XV.

Figure 5. Attack-detection evaluation pipeline. Results are reserved for future experimental validation.

XI. Software Architecture

SecBench’s software design extends the ForecastBench package’s [12] registry pattern described in the companion architecture paper rather than introducing a parallel codebase. Three registries are added: a model registry for classifiers and detectors; an attack registry recording the taxonomy in Table 2 and any future attack types as they are added; and a dataset registry entry scoping the software to the security-relevant field subset described in Section VII. Experiment tracking, artifact checksumming, and the deterministic-execution guarantees of the companion architecture paper’s Reproducibility Layer are inherited unchanged.

XII. Reproducibility Framework

This framework’s deterministic-execution, partitioning, and artifact-checksumming mechanisms are defined in the companion architecture paper and are not re-derived here; SecBench adopts them without modification, except that the partitioning choice for this benchmark is stratification-sensitive rather than chronological-only, as justified in Section VII. What SecBench adds beyond Paper 1’s framework is a dataset-versioning requirement specific to the attack registry, so that a given benchmark run can be tied to a specific, dated version of the taxonomy it was evaluated against.

Figure 6. Reproducibility and governance lifecycle, extending the pattern established in Paper 1.

XIII. Benchmark Governance

Governance responsibilities — identifier management, version concordance, and cross-artifact reconciliation — are inherited from the companion architecture paper rather than redefined. Paper 1’s Scientific Lineage Layer has already produced one disclosed finding: an unresolved numerical discrepancy between two published versions of the ForecastBench benchmark result, which remains an open item at the time of this writing. That precedent is relevant here because it demonstrates the governance mechanism functions, and because SecBench, having produced no results yet, has no artifact-reconciliation finding of its own to report.

XIV. Experimental Plan

A. Completed

Identification and characterization of the dataset’s security-relevant fields (Section VII).
Direct inventory of the attack_class distribution (Table 2), carried forward from prior BQEB validation material.
Specification of the five-layer architecture, four benchmark tasks, threat model, and evaluation protocol (Sections V–IX).
Software and governance extension design (Sections XI–XIII).

B. Planned

Model training across the candidate classifiers registered under Section XI for Tasks 1–3.
Evaluation under the metric set specified in Section IX and Table 4.
Comparative analysis across models, including the mandatory majority-class baseline.
Statistical validation per Section X (bootstrap confidence intervals, paired significance testing, multi-seed sensitivity).

XV. Benchmark Execution Status and Future Validation

Results Pending Experimental Execution

No model has been trained or evaluated under this benchmark framework at the time of writing. This section is deliberately not titled ‘Results’: it reports execution status, not findings. The Planned Evaluation Template below is provided in the exact structure the completed benchmark will populate.

Consistent with the disclosure practice this program applies throughout, Table 5 consolidates every claim this manuscript makes against the tier of evidence it currently has, in one place, before the detailed evaluation template below.

Table 5. Evidence tier by claim, consistent with the classification practice established in the companion architecture paper.

Claim	Evidence Tier
Benchmark task, threat model, and evaluation protocol specification	Specified, complete
Attack-class inventory (Table 2)	Verified — direct inventory, carried forward from prior BQEB validation material
Registry extension (model, attack, dataset registries)	Specified, complete; inherits a tested base registry
Deterministic execution, checksummed artifacts	Inherited from Paper 1 unmodified — not independently re-verified by this manuscript for SecBench specifically
Detection performance, any model, any task	Not evaluated — explicitly reserved (Table 6)
Statistical validation (bootstrap CIs, paired tests)	Specified, not executed
Split-strategy justification for extreme minority class	Argued, not yet empirically tested against an alternative split

No row in Table 5 is expected to change status as a result of this manuscript’s own editing — only real execution, reported in a future companion study, moves an item between tiers.

Table 6. Planned Evaluation Template for SecBench Tasks 1–3.

Task	Model	Precision	Recall	F1	MCC	PR-AUC
Task 1: Binary detection	—	Reserved	Reserved	Reserved	Reserved	Reserved
Task 2: Multiclass classification	—	Reserved	Reserved	Reserved	Reserved	Reserved
Task 3: Unknown-attack detection	—	Reserved	Reserved	Reserved	Reserved	Reserved

‘Reserved’ denotes reserved for future experimental validation; no values are estimated or implied.

A. Future Experimental Study

The following is planned for a companion execution study and is not performed in this manuscript. Candidate models span classical and modern tabular-classification approaches: Logistic Regression as an interpretable linear baseline; Random Forest and Gradient Boosting as ensemble baselines consistent with Paper 1’s model choices for the regression case; XGBoost and LightGBM as gradient-boosting implementations commonly reported as strong tabular-data baselines; and neural network baselines (architecture to be specified at execution time). Evaluation will report macro-F1, MCC, PR-AUC, and balanced accuracy per Section IX, together with bootstrap confidence intervals and results across multiple random seeds per Section X. This list is a planning specification, not a commitment to any reported outcome for any named model.

XVI. Discussion

SecBench’s contribution at this stage is benchmark infrastructure — a task design, threat model, and evaluation protocol capable of remaining statistically interpretable under a genuinely difficult imbalance regime — not a demonstrated detection capability. The question this paper answers is why a reproducible cyber-resilience benchmark is needed and how one should be structured given the data’s imbalance, not which classifier is better. That second question is exactly the one Section XV.A defers to a companion study, deliberately, so that a model comparison is never produced under the pressure of also having to justify the benchmark’s own design in the same manuscript.

The most consequential design choice — pairing multiclass and binary task framings so that the rarest class’s sample-size limitation does not silently invalidate the entire benchmark — is itself a testable claim only once real classifiers are run against it.

This paper’s single largest gap, stated directly rather than left for a reviewer to surface, is that Table 5’s evidence tiers are almost entirely specification, not execution — one row (the attack-class inventory) is verified; the rest are design commitments awaiting a companion study. This matters specifically because SecBench’s central claim is that the benchmark remains statistically interpretable under this imbalance regime, and interpretability under imbalance is exactly the kind of property that can look correct on paper and still fail once real per-class sample sizes interact with a real model’s actual error distribution. Closing this gap requires the companion execution study named in Section XV.A; no amount of additional specification in this manuscript substitutes for it.

Generalizability Beyond BQEB-Data

This paper’s specification is written against BQEB-Data v1’s security-relevant fields specifically, and that dependency should be stated rather than left implicit. Applying SecBench to a future dataset requires the same registry-based extension pattern Section VIII already establishes for models: a new dataset registers at the Research Artifact Layer under the architecture’s existing mechanism (Section V), and the four documented attack types (Table 2) are extended, not replaced, since nothing in the task or metric definitions (Section VIII, Table 3) assumes exactly four classes. A fifth or sixth attack type would require re-deriving the class-imbalance analysis in Section IX, not redesigning the benchmark.

Backward compatibility follows directly from this pattern: a dataset extension does not retroactively change what a prior SecBench evaluation reported, since each run’s reproducibility record (Section XII) is checksummed against the specific dataset version it used. Governance of such extensions is not yet formalized as its own procedure — this paper documents the mechanism by which a new dataset or attack type could register, but has not exercised that mechanism against a real second dataset, the same n=1 boundary the companion architecture paper discloses for its own extensibility claim. Closing this gap is future work, not a claim this manuscript makes about itself.

XVII. Threats to Validity

This section is additive to, not a replacement for, the disclosure already made in Sections III and VII: those gaps are restated here as threats, not introduced for the first time.

Synthetic data. Every SecBench task inherits BQEB-Data v1’s synthetic-generation status; detection performance on this benchmark is a statement about the simulator’s attack model, not a validated statement about real intrusion behavior.
Extreme class imbalance. The rarest class (false_data_injection, n=28) limits the statistical power of any per-class claim regardless of model quality.
Single dataset, single simulated region. No cross-dataset or cross-region validation exists yet for this benchmark.
No real utility telemetry. This benchmark cannot, by itself, support a claim about real-world intrusion-detection efficacy.
Undocumented label-generation process. How attack_class labels were produced is not specified in any source available to us.
Detection-from-telemetry framing only. The threat model (Section VI) does not cover network-traffic or protocol-level intrusion detection.

XVIII. Future Work

Execution of the experimental plan in Section XIV, producing the first real results for this benchmark.
Validation against real or semi-real utility telemetry, where access constraints permit.
Federated cyber-resilience benchmarking across multiple simulated or real sites.
Extension toward digital-twin-based threat simulation, explicitly as a research direction rather than a claimed capability.
Evaluation of foundation-model and few-shot approaches for the Task 3 open-set setting.

XIX. Conclusion

This paper has specified BQEB SecBench, the second module of the BQEB benchmark architecture, extending the companion architecture paper’s four-layer design with Threat and Benchmark Layers specific to cyber-resilience evaluation. Four tasks, an imbalance-aware evaluation protocol, and software and governance extensions are fully specified. No results are reported: Section XV marks them explicitly reserved for future experimental execution, and this disclosure is treated as part of the contribution rather than a gap to be minimized.

Acknowledgments

No acknowledgements.

Funding

No external funding was received for this research.

Conflicts of Interest

The authors declare no conflict of interest.

Ethics Statement

This work uses synthetic data only; no human subjects, personal data, or field-collected utility data are involved. As a benchmark for cyber-event detection, this manuscript deliberately avoids describing detector-evasion strategies in operational detail, even against a synthetic system.

Data Availability Statement

BQEB-Data v1 is available via Harvard Dataverse (DOI 10.7910/DVN/VUVTED) and IEEE DataPort (DOI 10.21227/xj9b-2885).

Software Availability Statement

SecBench is planned to extend the ForecastBench software package (MIT license, Zenodo DOI 10.5281/zenodo.21735978); no SecBench-specific software release exists at the time of writing.

Artifact Availability and Lineage

This section states SecBench’s position in the BQEB program’s scientific lineage explicitly, consistent with the Scientific Lineage Layer’s practice in the companion architecture paper. Three artifacts are currently distinguishable: the BQEB Reproducibility Architecture (Zenodo DOI: 10.5281/zenodo.22291930), an independent, citable, archived research artifact; BQEB ForecastBench (Zenodo DOI: 10.5281/zenodo.21735978), the architecture’s first validated downstream implementation; and BQEB SecBench (this manuscript), the architecture’s second module, named in the program’s own roadmap and specified here without an independent software release or DOI of its own at the time of writing. No DOI in this lineage is presented as validating another’s content, and none is evidence of reproducibility on its own — a DOI is a persistent identifier, not a claim of verification, consistent with the language established in the companion architecture paper. When SecBench’s software extension is released, it is expected to receive its own independent identifier following the same separation practice, rather than being folded into ForecastBench’s existing DOI or the architecture’s.

Appendix A. Dataset Card — BQEB-Data v1 Security Fields

Field	Value
Scope of this card	Security-relevant subset only; full dataset card is Paper 1, Appendix A
Fields	attack_class (categorical, 5 values), cyber_alert_flag, sensor_fault_flag, anomaly_score, failure_risk_score
attack_class distribution	none: 10,331 (98.28%); sensor_fault: 82 (0.78%); spoofing: 37 (0.35%); dos: 34 (0.32%); false_data_injection: 28 (0.27%)
Evidence status	Inventory inspection of the distributed attack_class field; carried forward from Paper 1’s source material
Other field distributions	Not yet inspected — requires validation from dataset artifact
Label provenance	Not documented in any source available to us

Appendix B. Cybersecurity Benchmark Card — Secbench

Field	Value
Tasks	Binary detection; multiclass classification; unknown-attack (open-set) detection; imbalance-aware evaluation track
Primary metrics	Precision, Recall, F1 (per class), Macro-F1, Weighted-F1, Balanced Accuracy, MCC, PR-AUC
Excluded as primary metric	Accuracy (misleading at 98.28% majority-class prevalence)
Status	Protocol and architecture specified; execution not started
Software basis	Extension of ForecastBench package registry (MIT license)

Appendix C. Threat Model Card

Field	Value
Attacker capability assumptions	Bounded by dataset documentation; not extended beyond what source data supports
Attack categories covered	Sensor fault, spoofing, denial-of-service, false data injection
Detection framing	Telemetry/record-level; not network-traffic or protocol-level
Explicit non-claims	Not validated against real intrusion behavior; not a production intrusion-detection system

Appendix D. Reproducibility Checklist

Dataset publicly available with a persistent identifier — yes (inherited from Paper 1)
Attack taxonomy versioned as a distinct registry — specified, not yet implemented
Software extends an existing tested registry rather than a new codebase — yes, by design
Deterministic execution (fixed seeds) — specified, inherited from Paper 1
Majority-class baseline reported alongside all results — specified as mandatory (Section IX)
Statistical validation (bootstrap CIs, paired significance tests) — specified, not yet executed
Results independently verifiable via checksummed artifacts — specified, not yet applicable (no results exist)

Appendix E. Replication Protocol

Once executed, replication of this benchmark will require: (1) the BQEB-Data v1 security-field subset, obtained via the identifiers in the Data Availability Statement; (2) the SecBench software extension of the ForecastBench registry (not yet released; tracked as future work); (3) the fixed random seed(s) and split configuration documented at release time; (4) independent verification of published checksums against a freshly obtained copy of any released prediction and metric artifacts.

References

Liu, Y.; Ning, P.; Reiter, M. K. False data injection attacks against state estimation in electric power grids. Proc. 16th ACM Conf. Comput. Commun. Secur. (CCS), 2009; pp. 21–32. [Google Scholar]
Sandberg, H.; Teixeira, A.; Johansson, K. H. On security indices for state estimators in power networks. Proc. 1st Workshop Secure Control Syst. (CPSWEEK), 2010. [Google Scholar]
Kosut, O.; Jia, L.; Thomas, R. J.; Tong, L. Malicious data attacks on the smart grid. IEEE Trans. Smart Grid 2011, vol. 2(no. 4), 645–658. [Google Scholar] [CrossRef]
Karimipour, H.; Dehghantanha, A.; Parizi, R. M.; Choo, K.-K. R.; Leung, H. A deep and scalable unsupervised machine learning system for cyber-attack detection in large-scale smart grids. IEEE Access 2019, vol. 7, 80778–80788. [Google Scholar] [CrossRef]
Agrawal, R. K. BQEB: A reproducibility-first architecture for modular energy AI benchmarking. (Paper 1). Zenodo 2026. [Google Scholar] [CrossRef]
Pan, S.; Morris, T.; Adhikari, U. Developing a hybrid intrusion detection system using data mining for power systems. IEEE Trans. Smart Grid 2015, vol. 6(no. 6), 3104–3113. [Google Scholar] [CrossRef]
Chawla, N. V.; Bowyer, K. W.; Hall, L. O.; Kegelmeyer, W. P. SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 2002, vol. 16, 321–357. [Google Scholar] [CrossRef]
He, H.; Garcia, E. A. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 2009, vol. 21(no. 9), 1263–1284. [Google Scholar] [CrossRef]
Scheirer, W. J.; Rocha, A.; Sapkota, A.; Boult, T. E. Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2013, vol. 35(no. 7), 1757–1772. [Google Scholar] [CrossRef] [PubMed]
Pineau, J.; et al. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. J. Mach. Learn. Res. 2021, vol. 22(no. 164), 1–20. [Google Scholar]
Chicco, D.; Jurman, G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom. 2020, vol. 21, Art.(no. 6). [Google Scholar] [CrossRef] [PubMed]
Agrawal, R. K. “BQEB ForecastBench: A reproducible smart-grid forecasting benchmark framework,” (software, v1.0.0). Zenodo 2026. [Google Scholar] [CrossRef]
Agrawal, R. K. BQEB-Data v1: BIO-Quantum Energy Brain benchmark dataset for smart grid intelligence, renewable forecasting, storage optimization, and cyber resilience. Harvard Dataverse 2026. [Google Scholar] [CrossRef]

Author Biography

RAKESH KUMAR AGRAWAL is a Senior Consultant at Atos specializing in artificial intelligence, platform engineering, cloud engineering, and intelligent enterprise systems. His research interests include trustworthy artificial intelligence, enterprise AI governance, intelligent energy systems, AI benchmarking, digital twins, and large language model operations. He is an IEEE Senior Member and an IETE Fellow, and holds multiple AWS and Microsoft certifications in cloud computing and artificial intelligence. His recent research contributions include the BIO-Quantum Energy Brain (BQEB) research initiative and reproducible benchmarking frameworks for intelligent energy systems.

Figure 1. BQEB SecBench architecture: two new layers (Threat, Benchmark) extend the four-layer architecture defined in the companion architecture paper.

Figure 4. SecBench benchmark workflow, from data to reported artifact. Orange stages mark points requiring an explicit, disclosed design decision.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.