PC-Audit: A Decision-Reliability Framework for Auditing Validation-Based Machine-Learning Model Selection in Weak-Signal Financial Time Series

1. Introduction

Validation-based machine-learning model selection is a routine step in deployment: several candidate models are trained on historical data, one is chosen using a finite validation interval, and the selected candidate is then carried forward to a later operating period. In temporally dependent and weak-signal settings, however, the existence of a numerical validation winner does not by itself establish that the choice is reliable. Candidate errors may be close, observations are serially dependent, regimes can change, and a short validation segment may appear decisive even when it contains little information about the next period.

Financial time series provide a particularly demanding setting for this problem. Daily index-ETF returns have low signal-to-noise ratios, broad-market information is incorporated quickly, and small apparent performance differences can be sensitive to calibration, execution assumptions, and market regime. These features make financial prediction a useful stress test for a broader evaluation question: whether the evidence behind a validation-selected machine-learning action is strong enough to support a deployment interpretation.

Most empirical studies naturally emphasize predictive accuracy, model architecture, hyperparameter choice, or trading outcomes. Considerably less attention is given to the reliability of the selection act itself: whether the model chosen on validation data is statistically distinguishable from alternatives, whether the ordering transfers out of sample, and whether the resulting interpretation survives dependence, multiplicity, cost, execution, and temporal stress. Without such checks, a routine validation winner can be overinterpreted as stable deployment evidence.

This paper addresses that narrower methodological gap. The aim is not to search for a universally strongest predictor or to claim an investable ETF timing rule, but to construct an auditable framework that determines how strongly a validation-selected candidate can be interpreted before deployment. The central idea is to treat model selection as a decision object: the selected model name must be reported together with calibration timing, confidence-set uncertainty, validation-to-test transfer, regret, cost and execution stress, temporal-holdout behavior, and an audit-state record that can inform the next review cycle.

Time-aware partitioning is necessary but not sufficient. A next-day target must be assigned to a split according to the date of the predicted return, not only the date of the feature row. Prediction-scale mismatch also complicates model comparison: a recurrent or tree model may have useful ordering information but an unstable amplitude. Calibration can correct amplitude, yet fitting and scoring a calibration coefficient on the same validation outcomes creates optimistic losses. Finally, selecting the numerically lowest validation error hides uncertainty when several candidates are statistically indistinguishable.

To address these issues, PC-Audit (Prequential-Calibration Audit) evaluates validation-driven model selection as a pre-deployment decision. The framework combines target-date partitioning, chronological prequential calibration, a moving-block-bootstrap Model Confidence Set (MCS), validation-to-test diagnostics, dependence-aware inference, transaction-cost and execution checks, computational-cost reporting, and a fixed 2025 temporal stress evaluation. PC-Select is the operational rule that identifies one candidate using the audit-defined validation loss; it is not an additional forecasting model. The remaining components quantify the uncertainty and practical limits surrounding that action. Thus, the framework is useful even when the evidence is insufficient to support a strong deployment interpretation.

The framework imposes three requirements. First, the selected action must arise from a leakage-controlled validation rule. Second, its uncertainty must be reported through confidence-set and transfer diagnostics. Third, any economic or cost-related interpretation must remain conditional on multiplicity, dependence, execution, sensitivity, and temporal-stress evidence. These requirements turn model selection into a falsifiable decision protocol: the evidence may support, qualify, or reject the proposed deployment action. PC-Audit contributes a prespecified decision-reliability architecture for validation-based model selection. It combines leakage-controlled selection, prequential calibration, uncertainty-set assessment, transfer diagnostics, and forward-only governance within a common information boundary. Its novelty lies in how these components are ordered and constrained around the validation-selected action as the explicit audit object. The constituent statistical procedures are individually established; no claim is made that any single calibration, bootstrap, or confidence-set procedure is new.

The study answers three audit questions:

1.

Does a leakage-controlled validation rule provide sufficient signed-return deployment evidence when confidence-set uncertainty, selection margin, rank transfer, hit rate, regret, and competing baselines are jointly audited?

2.

Are volatility-like risk-surveillance targets more auditable than signed returns under the same protocol, and do they remain bounded under strong rolling/EWMA references and 2025 stress?

3.

Can completed audit outputs be carried forward as monitoring variables and fallback actions for the next deployment review without turning the benchmark into an online learner?

The contribution is threefold. First, the paper treats the validation-selected candidate as an explicit audit object and integrates calibration-timing control, MCS-based uncertainty assessment, selection-margin analysis, validation-to-test transfer diagnostics, dependence-aware inference, and practical stress checks into one reproducible decision-reliability protocol. Second, it shows that the strength of selection evidence differs materially by target type: volatility-like risk-surveillance targets exhibit more stable validation-to-test evidence than signed daily returns under the same leakage-controlled design, while strong rolling and EWMA references remain competitive. Third, it introduces a forward-only audit state and fallback simulation that use completed-year evidence to inform the next scheduled review without tuning the reported test results. Together, these contributions provide an operational evidence framework for assessing when a validation-selected candidate warrants a stronger or weaker deployment interpretation. Raw snapshots, features, prediction caches, scripts, intermediate tables, and file hashes are supplied for verification.

Table 1 summarizes how the design addresses common evaluation risks in weak-signal time-series machine learning. The table is included to make the claim boundary explicit: each protocol component either removes a possible source of optimism or defines when a positive interpretation is not warranted.

2. Related Work

Machine learning has been applied extensively to financial time-series prediction, from regularized linear models and tree ensembles to recurrent and deep architectures [1,2,3,4]. Recent benchmark studies also examine larger time-series and foundation-model families [5,6]. These studies establish the candidate-model background for this work, but architectural breadth does not resolve the evaluation problem created by validation-based selection. PC-Audit therefore focuses on the reliability of the decision produced by model selection rather than on proposing another forecasting architecture.

Evaluation protocol is a central issue for temporally dependent data. Random cross-validation can mix temporal information and misrepresent deployment conditions, whereas rolling-origin or walk-forward evaluation preserves the forward-looking structure of the task [7,8,9]. Leakage-controlled financial benchmarks further emphasize train-only preprocessing, causal feature construction, strong naive references, and conservative conclusions when signed-return prediction remains close to baseline performance [10]. PC-Audit extends this evaluation logic by asking whether the deployment interpretation created by validation-based selection is supported by uncertainty, transfer, and stress evidence. Its audit object is the selection decision itself, not only the forecast error of each candidate.

A second stream studies model-selection risk. Even when several models are reasonable on average, a short validation segment may have low probability of identifying the best next-period forecaster [11]. This problem is related to backtest overfitting, research-protocol discipline, and multiple-testing risk in financial research, where repeated searching can create attractive but fragile results [12,13,14,15]. The present study therefore reports top-one hit rate, validation-to-test rank correlation, selection regret, and permutation evidence rather than treating the lowest validation MAE as self-justifying.

Uncertainty-aware model comparison provides a complementary foundation. The Model Confidence Set retains models that cannot be rejected as inferior under a specified loss and bootstrap scheme [16]. Its financial forecasting applications show why confidence-set reporting can be more informative than simply naming a best model when the data cannot sharply separate alternatives [17]. PC-Audit uses this property as an audit component: MCS membership constrains the strength of a deployment interpretation, but it is not treated as a forecasting model and does not imply economic profitability.

Resource-aware forecasting and multi-objective model selection have also been studied for time-series systems [18]. PC-Audit is related but distinct. It does not optimize a combined accuracy-cost score or search for a new model family. Instead, it evaluates a fixed candidate pool in a controlled CPU-only setting and asks whether the validation-selected action remains defensible after calibration timing, statistical uncertainty, transfer diagnostics, costs, execution delay, sensitivity variants, and temporal stress are audited.

Related work on Machine Learning Operations (MLOps) and ML lifecycle management broadens the evaluation perspective from model development to reproducible deployment, monitoring, versioning, and feedback across the operational lifecycle [19,20]. PC-Audit is complementary to this literature but targets a narrower upstream decision: before a validation-selected candidate is interpreted as deployable, how strong is the evidence supporting that selection? Its audit-state mechanism records completed evidence for the next scheduled review while preserving the information boundary of the current test period. The framework does not replace end-to-end MLOps infrastructure, continuous integration, or production monitoring; it supplies a prespecified model-selection audit layer that can be embedded within such lifecycle processes.

The remaining gap is therefore not the absence of forecasting architectures or individual statistical tools. Existing work provides time-aware evaluation methods, model-selection diagnostics, MCS uncertainty procedures, and warnings about overfitting, but these elements are seldom organized around the validation-selected action itself as the object of audit. PC-Audit addresses this gap by integrating them into a reproducible evidence architecture that can support, qualify, or reject a deployment interpretation and can carry completed audit outputs forward to the next review cycle without retroactively tuning reported test results.

3. Materials and Methods

Figure 1 summarizes the order in which information enters the audit. It separates model fitting and validation selection from the evidence used to judge that selection, and it shows that completed test outcomes are carried forward only as governance inputs for the next review cycle. Algorithm 1 states the case-level protocol used for each ETF-window split. The 2025 split repeats the same fixed procedure once as a temporal stress test.

Algorithm 1 PC-Audit for one ETF-window case

1: Assign train, validation, and test membership by target date. 2: Train the fixed CPU-only candidate pool on the training window. 3: Generate validation and test prediction caches without test-year tuning. 4: Divide the validation year into chronological blocks. 5: Estimate non-negative calibration slopes only from earlier validation blocks. 6: Compute prequential validation losses for scored validation blocks. 7: From the same prequential loss matrix, independently select PC-Select and estimate the moving-block-bootstrap MCS uncertainty set. 8: Refit the selected model’s non-negative scalar on the full validation year. 9: Evaluate the selected model on the next test year. 10: Report transfer, regret, MCS coverage, paired inference, and stress checks. 11: After the test year ends, log audit-state variables for the next review.

3.1. Audit Design, Data Provenance, and Target-Date Partitioning

PC-Audit is organized as a fixed pre-deployment audit of validation-based model selection. The dataset is treated as an immutable benchmark input; the ETF universe, coverage, and row counts are reported with the experimental results in Table 2 so that the Methods section can focus on the audit logic. Although the public data vendor facilitates reproduction, it is not the exchange’s official historical database [21]. To prevent later vendor revisions from changing the experiment, the package stores the exact snapshots and file hashes. This supports computational replication while leaving one-vendor dependence and vendor-revision risk as limitations. We do not claim that the dataset is survivorship-free.

Throughout the paper, t indexes the date on which a target is realized. Let $P_t$ denote the ETF closing price on trading date t. The close-to-close return realized on that date is

$$r_t={\displaystyle \frac{P_t-P_{t-1}}{P_{t-1}}}.$$

(1)

At the preceding forecast origin $t-1$, the future price $P_t$ and hence $r_t$ are not yet observed; they become available only when the target is labeled for evaluation. Let ${\mathbf{x}}_{t-1}$ denote the feature vector constructed solely from information observed no later than date $t-1$. It contains lagged returns, rolling moments, momentum, price ranges, volume changes, moving-average ratios, and the technical indicators listed in the configuration file. Missing values introduced by rolling windows are removed deterministically.

Let $y_t$ denote the target realized on date t. For the primary signed-return audit, $y_t=r_t$. Candidate model m maps the lagged feature vector to a one-step-ahead prediction,

$${\widehat{y}}_{m,t}=f_m\left({\mathbf{x}}_{t-1}\right),$$

(2)

where $f_m$ is the prediction function fitted for candidate m and ${\widehat{y}}_{m,t}$ is its raw prediction of $y_t$. Thus, no quantity dated t is used as an input when predicting the target at date t.

The signed-return target remains the primary audit case because it corresponds to the common deployment question of whether a validation-selected model supports a directional interpretation. As a risk-surveillance extension, the same fixed pipeline is also run on three volatility-like targets:

$$v_t=\left|r_t\right|,s_t=r_t^2,q_t=log\left({\displaystyle \frac{H_t}{L_t}}\right),$$

(3)

where $v_t$ is absolute return, $s_t$ is squared return, $q_t$ is a high–low range proxy, and $H_t$ and $L_t$ are respectively the high and low prices on target date t. For each extension, $y_t$ is replaced by the corresponding target. These targets are evaluated with prediction-scale losses, rank transfer, MCS size, and selection regret. They are not used for directional accuracy, transaction-cost signal returns, or trading-performance claims. In the robustness scripts, target-derived columns are explicitly excluded from ${\mathbf{x}}_{t-1}$ before model fitting, so the alternative targets remain prediction targets rather than predictors.

Each observation stores both its feature date $t-1$ and target date t. Split membership is assigned by the target date, preventing a final training feature row from carrying a validation-period outcome and a final validation row from carrying a test-period outcome. Formally,

$$({\mathbf{x}}_{t-1},y_t)\in \left\{\begin{array}{cc}{\mathcal{D}}_{\mathrm{train}},\hfill & t\le T_{\mathrm{train}},\hfill \\ {\mathcal{D}}_{\mathrm{val}},\hfill & T_{\mathrm{train}}<t\le T_{\mathrm{val}},\hfill \\ {\mathcal{D}}_{\mathrm{test}},\hfill & t>T_{\mathrm{val}},\hfill \end{array}\right.$$

(4)

where ${\mathcal{D}}_{\mathrm{train}}$, ${\mathcal{D}}_{\mathrm{val}}$, and ${\mathcal{D}}_{\mathrm{test}}$ are the training, validation, and test sets, while $T_{\mathrm{train}}$ and $T_{\mathrm{val}}$ are the target-date cutoffs ending the training and validation windows. This target-date convention is used in all subsequent equations.

The outer design uses five years for training, one year for validation, and one year for testing. Test years 2020–2024 produce $9\times 5=45$ ETF-window cases. A sixth fixed split trains on 2019–2023, validates on 2024, and tests on 2025. The 2025 split is a temporal stress test, not an independent market or data-provider sample.

3.2. Candidate Models and Prequential Calibration

The candidate pool contains ElasticNet, Ridge, LightGBM, Random Forest, and a small GRU [22,23,24,25,26]. Hyperparameters are fixed in model_configs.json and are not retuned on test years. The GRU receives causal 10-trading-day sequences; validation and test sequences use only preceding context. All models run on CPU.

The pool represents common low-cost prediction models with different inductive biases: regularized linear models, tree ensembles, boosting, and a compact recurrent network. It is not an exhaustive architecture benchmark. The object of study is the audited selection protocol, not a new model architecture or a search for the globally strongest predictor. To avoid attributing weak separation only to the use of individual models, the extended analysis also includes simple historical-prediction and prediction-aggregation references. Larger alternatives such as transformer-style sequence models, additional boosting variants, or broad hyperparameter searches are left outside the default pool because they would change the study from a CPU-only deployment audit into a model-zoo search and would increase selection and multiple-testing risk.

Let $\mathcal{M}=\{1,\dots ,M\}$ denote the fixed candidate-model pool, where $M=5$. For target date t, ${\widehat{y}}_{m,t}$ is the raw prediction of candidate $m\in \mathcal{M}$. To address prediction-scale mismatch, calibration uses a non-negative scalar estimated from an explicit information set. For a calibration index set $\mathcal{C}$, define

$${\widehat{a}}_m\left(\mathcal{C}\right)=arg\underset{a\ge 0}{min}\sum _{t\in \mathcal{C}}{(y_t-a{\widehat{y}}_{m,t})}^2=max\left\{0,{\displaystyle \frac{{\sum }_{t\in \mathcal{C}}{\widehat{y}}_{m,t}{y}_t}{{\sum }_{t\in \mathcal{C}}{\widehat{y}}_{m,t}^2}}\right\}.$$

(5)

The set $\mathcal{C}$ is the slope-estimation information set. Non-negativity permits shrinkage to zero, but no arbitrary upper cap; a zero slope means that the available sample supports the zero forecast among non-negative scalar multiples. The numerical tolerance ${10}^{-15}$ matches the supplied implementation.

Using the same validation outcomes both to estimate ${\widehat{a}}_m$ and score calibrated validation predictions would be optimistic. PC-Audit therefore follows a prequential evaluation logic in which predictions are scored only after the information used for calibration is available [27]. It divides the validation year into four contiguous blocks ${\mathcal{V}}_1,{\mathcal{V}}_2,{\mathcal{V}}_3,{\mathcal{V}}_4$. Block 1 is a warm-up. For each scored block $b\in \{2,3,4\}$, the available calibration history is

$${\mathcal{H}}_b={\displaystyle \bigcup _{j=1}^{b-1}}{\mathcal{V}}_j,$$

(6)

and the block-specific prequential slope is

$${\widehat{a}}_{m,b}={\widehat{a}}_m\left({\mathcal{H}}_b\right).$$

(7)

Thus, ${\widehat{a}}_{m,b}$ is estimated only from validation labels that occur before block b, and the corresponding validation prediction is ${\tilde{y}}_{m,t}^{\left(b\right)}={\widehat{a}}_{m,b}{\widehat{y}}_{m,t}$. The prequential absolute loss for model m at scored validation date t is

$${\ell }_{m,t}^{PC}=\left|y_t-{\widehat{a}}_{m,b\left(t\right)}{\widehat{y}}_{m,t}\right|,t\in {\mathcal{V}}_2\cup {\mathcal{V}}_3\cup {\mathcal{V}}_4,$$

(8)

where $b\left(t\right)$ is the validation block containing date t. PC-Select is the audit-defined deployment recommendation: it names the candidate with the lowest mean ${\ell }_{m,t}^{PC}$ but does not by itself imply that the selected candidate is statistically separated from the alternatives. Let

$${\overline{\ell }}_m^{PC}={\displaystyle \frac{1}{|{\mathcal{V}}^{PC}|}}\sum _{t\in {\mathcal{V}}^{PC}}{\ell }_{m,t}^{PC},{\mathcal{V}}^{PC}={\mathcal{V}}_2\cup {\mathcal{V}}_3\cup {\mathcal{V}}_4,$$

(9)

where ${\overline{\ell }}_m^{PC}$ is the mean prequential validation MAE for model m over the scored validation blocks. The first, selection-stage decision is

$$m^{*}=arg\underset{m\in \mathcal{M}}{min}{\overline{\ell }}_m^{PC}.$$

(10)

Exact ties are resolved by the fixed candidate order recorded in the supplied implementation. After the validation year has fully elapsed, the selected model receives one final deployment-time calibration on the complete validation set $\mathcal{V}={\bigcup }_{b=1}^4{\mathcal{V}}_b$:

$${\tilde{y}}_{m^{*},t}^{\mathrm{test}}={\widehat{a}}_{m^{*}}\left(\mathcal{V}\right){\widehat{y}}_{m^{*},t},t\in \mathcal{T}.$$

(11)

The first stage selects without same-label calibration bias; the second estimates the selected candidate’s deployment scalar after validation is complete, before the test year begins. It neither changes the selected model nor reuses test outcomes; the remaining audit diagnostics govern how strongly the choice may be interpreted.

3.3. Confidence Sets, Transfer Diagnostics, and Deployment Checks

For each case, a moving-block-bootstrap MCS is computed from the candidate loss matrix

$$\ell ={\left[{\ell }_{m,t}\right]}_{m\in \mathcal{M},t\in {\mathcal{V}}^{PC}},$$

(12)

where each row corresponds to a candidate model and each column corresponds to a scored validation date. The default prequential-loss MCS uncertainty audit uses ${\ell }_{m,t}={\ell }_{m,t}^{PC}$, the prequential absolute loss defined above. The MCS is estimated using the maximum-t statistic, $\alpha =0.10$, 5000 bootstrap repetitions, and block length approximately equal to the square root of the evaluated validation length [16,17,28]. The 10% level is used as a screening-oriented confidence-set threshold rather than as a declaration of model separation; 5% and 20% alternatives and block-length multipliers of 0.5, 1, and 2 are included in the sensitivity audit. The lowest prequential-MAE model remains the deployment rule; MCS membership quantifies uncertainty. Pairwise MCS comparisons are based on loss differences

$$d_{ij,t}={\ell }_{i,t}-{\ell }_{j,t},$$

(13)

where $d_{ij,t}<0$ means that model i has lower loss than model j on date t. The audit therefore asks whether observed differences among candidates are statistically distinguishable under the specified dependent-data bootstrap.

Two controls isolate calibration timing. The raw-loss MCS uncertainty audit uses uncalibrated errors on the same post-warm-up validation observations. The same-label calibrated MCS fits one coefficient on the complete validation year and evaluates it on those same labels. The MCS equal-weight comparator averages the individually final-calibrated test predictions of the prequential-loss MCS members:

$${\tilde{y}}_{c,t}^{\mathrm{MCS}-\mathrm{EW}}={\displaystyle \frac{1}{|{\mathcal{S}}_c|}}\sum _{m\in {\mathcal{S}}_c}{\tilde{y}}_{m,c,t}^{\mathrm{test}}.$$

(14)

No test outcome enters either member calibration or the average. The raw-loss selector, same-label selector, individually calibrated candidates, zero-return predictor, historical-mean predictor, and full five-model ensemble average are comparators. The last two are low-cost reference checks derived from available historical targets and saved candidate predictions; they are not tuned on the test period.

For any test index set $\mathcal{T}$, prediction-scale errors are computed from the final-calibrated prediction ${\tilde{y}}_{m,t}^{\mathrm{test}}={\widehat{a}}_m\left(\mathcal{V}\right){\widehat{y}}_{m,t}$. The root-mean-square error and mean absolute error for model m are

$${RMSE}_m\left(\mathcal{T}\right)={\left({\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}{(y_t-{\tilde{y}}_{m,t}^{\mathrm{test}})}^2\right)}^{1/2},$$

(15)

and

$${MAE}_m\left(\mathcal{T}\right)={\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}|y_t-{\tilde{y}}_{m,t}^{\mathrm{test}}|,$$

(16)

where $\mathcal{T}$ is the test-period date set for one ETF-window case. Direction is evaluated using directional accuracy (DA), balanced accuracy (BA), and Matthews correlation coefficient (MCC). Define the realized sign label and predicted sign label as

$$z_t=\mathbf{1}(y_t>0),{\widehat{z}}_{m,t}=\mathbf{1}({\tilde{y}}_{m,t}^{\mathrm{test}}>0),$$

(17)

where $\mathbf{1}(·)$ equals one when the condition is true and zero otherwise. Then

$${DA}_m\left(\mathcal{T}\right)={\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}\mathbf{1}\left(z_t={\widehat{z}}_{m,t}\right).$$

(18)

BA and MCC are computed from true positives ($TP$), true negatives ($TN$), false positives ($FP$), and false negatives ($FN$):

$${BA}_m\left(\mathcal{T}\right)={\displaystyle \frac{1}{2}}\left({\displaystyle \frac{TP}{TP+FN}}+{\displaystyle \frac{TN}{TN+FP}}\right),$$

(19)

and

$${MCC}_m\left(\mathcal{T}\right)={\displaystyle \frac{TP·TN-FP·FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}}.$$

(20)

These directional metrics prevent an always-zero or always-down forecast from appearing skillful solely because of class imbalance. The supplied implementation uses the deterministic conventions of scikit-learn: if MCC has a zero denominator, it is returned as zero; if only one observed class is present, BA is the recall of that observed class. Both conventions are recorded rather than silently replacing an undefined value.

For each ETF-window case c, validation candidate ranks are compared with test-MAE ranks. Let $V_{m,c}={\overline{\ell }}_{m,c}^{PC}$ be the prequential validation loss of model m, and let $Q_{m,c}={MAE}_{m,c}\left({\mathcal{T}}_c\right)$ be its test MAE. Let $R_{m,c}^V$ be the rank of $V_{m,c}$ among $\{V_{j,c}:j\in \mathcal{M}\}$, and let $R_{m,c}^Q$ be the rank of $Q_{m,c}$ among $\{Q_{j,c}:j\in \mathcal{M}\}$, with rank 1 assigned to the smallest loss. The case-level validation-to-test rank correlation is then the ordinary Pearson correlation applied to these two rank vectors:

$${\rho }_c={\displaystyle \frac{{\sum }_{m\in \mathcal{M}}(R_{m,c}^V-{\overline{R}}_c^V)(R_{m,c}^Q-{\overline{R}}_c^Q)}{\sqrt{{\sum }_{m\in \mathcal{M}}{(R_{m,c}^V-{\overline{R}}_c^V)}^2}\sqrt{{\sum }_{m\in \mathcal{M}}{(R_{m,c}^Q-{\overline{R}}_c^Q)}^2}}},$$

(21)

where ${\overline{R}}_c^V=M^{-1}{\sum }_{m\in \mathcal{M}}{R}_{m,c}^V$ and ${\overline{R}}_c^Q=M^{-1}{\sum }_{m\in \mathcal{M}}{R}_{m,c}^Q$. This is the Spearman rank correlation written explicitly rather than treated as a named black-box function. Annual summaries average ${\rho }_c$ over the ETF-window cases completed in year k.

The top-one hit indicator for case c is

$${\delta }_{\mathrm{hit},c}=\mathbf{1}\left(m_c^{*}\in arg\underset{m\in \mathcal{M}}{min}{Q}_{m,c}\right),$$

(22)

so an exact tie for the lowest test MAE counts as a hit. The reported top-one hit rate is the average of ${\delta }_{\mathrm{hit},c}$ over the evaluated cases. A uniform five-model reference is 20%; a label-permutation audit additionally preserves the observed frequencies of selected and lowest-test-loss model labels, with a within-ETF permutation as a dependence-oriented control. Selection regret is the selected model’s test loss minus the best attainable test loss within the candidate pool:

$${\mathcal{R}}_c=Q_{m_c^{*},c}-Q_{min,c},Q_{min,c}=\underset{m\in \mathcal{M}}{min}{Q}_{m,c},$$

(23)

where $Q_{min,c}$ is an ex post candidate-pool minimum, not an additional candidate model. Regret is non-negative and equals zero only when PC-Select chooses a lowest-test-MAE candidate. MCS coverage records whether a lowest-test-MAE candidate was retained by the validation MCS.

Paired comparisons use two-sided Wilcoxon signed-rank tests. Holm correction is applied over the prespecified predictive comparison family. Since the 45 cases share ETFs, years, and overlapping training intervals, mean-difference intervals are also obtained by resampling ETFs as clusters and test years as blocks. The interpretation rules are applied conservatively and are allowed to return a negative or inconclusive audit outcome.

Signal diagnostics use the sign of the prediction to form an unlagged long-short proxy and one-day-lagged long-cash and long-short controls. Turnover costs of 0, 5, 10, and 20 bps are deducted. These are stress diagnostics, not implementable trading claims. Because the study is motivated by resource-constrained deployment, predictive and signal metrics are reported together with computational cost rather than collapsed into a single optimized score. No utility-cost score is tuned on the test period, and no favorable cost summary is allowed to override weak predictive, dependence-aware, or temporal-stress evidence.

Sensitivity analysis varies validation blocks (3–6), MCS significance (0.05, 0.10, 0.20), loss (absolute or squared), block-length multiplier (0.5, 1, 2), and calibration rule (non-negative, unconstrained, or the legacy clipped rule). Saved prediction caches are used without retraining so that sensitivity differences reflect the audit protocol rather than changes in fitted candidate models. The default four-block absolute-error non-negative protocol was fixed before sensitivity analysis and is retained even when another variant gives a more attractive economic proxy.

To make the audit falsifiable, each positive interpretation requires evidence beyond a favorable average. A lower-error claim requires a multiplicity-adjusted comparison and dependence-aware intervals pointing in the same direction. Reliable selection requires both nontrivial top-one evidence and validation-to-test rank-transfer intervals excluding zero. Economic deployability requires consistent signs under transaction costs, one-day execution delay, protocol sensitivity variants, and the 2025 stress split. When these conditions are not met, the manuscript reports the result as weak, mixed, or unsupported rather than treating a favorable single metric as a discovery.

3.4. Deployment-Cycle Audit-State Construction and Reproducibility

PC-Audit is not implemented as an online learner in this benchmark. No model is retrained, no hyperparameter is retuned, and no reported test-year result is revised using realized test outcomes. To distinguish the annual review-cycle index from the target variable $y_t$, let k index a completed test year. After year k has ended, its realized audit outputs are logged as monitoring variables for the next scheduled review in year $k+1$. This construction represents a feedback mechanism at the audit-governance level rather than at the model-fitting level.

Signed-return and volatility-proxy audits are separate. Let $u\in \{\mathrm{ret},\mathrm{vol}\}$ identify the target and define the completed-case state

$$A_c^{\left(u\right)}=\left(m_c^{*,\left(u\right)},K_c^{\left(u\right)},I_c^{\left(u\right)},{\rho }_c^{\left(u\right)},{\mathcal{R}}_c^{\left(u\right)},C_c^{\left(u\right)}\right),$$

(24)

where $K_c^{\left(u\right)}$ is MCS size, $I_c^{\left(u\right)}=\mathbf{1}(m_c^{*,\left(u\right)}\in {\mathcal{S}}_c^{\left(u\right)})$ records selection–MCS membership, ${\rho }_c^{\left(u\right)}$ is rank transfer, and ${\mathcal{R}}_c^{\left(u\right)}$ is regret. The stress components use opposite favorable directions: larger is favorable for the unlagged 20-bps signed-return proxy $C_c^{\left(\mathrm{ret}\right)}$, whereas smaller is favorable for $C_c^{\left(\mathrm{vol}\right)}$, PC-Select MAE minus EWMA(0.97) MAE. For completed year k, let ${\mathcal{E}}_k$ denote its nine ETF-window cases. Its annual components are

$$\begin{array}{cccc}\hfill {\overline{K}}_k^{\left(u\right)}& ={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{K}_c^{\left(u\right)},\hfill & \hfill {\overline{I}}_k^{\left(u\right)}& ={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{I}_c^{\left(u\right)},\hfill \\ \hfill {\overline{\rho }}_k^{\left(u\right)}& ={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{\rho }_c^{\left(u\right)},\hfill & \hfill {\overline{\mathcal{R}}}_k^{\left(u\right)}& ={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{\mathcal{R}}_c^{\left(u\right)},\hfill \\ \hfill {\overline{C}}_k^{\left(u\right)}& ={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{C}_c^{\left(u\right)}.\hfill \end{array}$$

(25)

The annual audit-state summary is

$${\overline{A}}_k^{\left(u\right)}=({\overline{K}}_k^{\left(u\right)},{\overline{I}}_k^{\left(u\right)},{\overline{\rho }}_k^{\left(u\right)},{\overline{\mathcal{R}}}_k^{\left(u\right)},{\overline{C}}_k^{\left(u\right)}).$$

(26)

The selected-model identity is retained in each case record but is categorical and therefore not numerically averaged into ${\overline{A}}_k^{\left(u\right)}$. The next-cycle deployment interpretation is then a target-specific deterministic function of the completed-year audit state:

$$\begin{array}{cc}\hfill D_{k+1}^{\left(u\right)}& =g^{\left(u\right)}\left({\overline{A}}_k^{\left(u\right)}\right),\hfill \\ \hfill D_{k+1}^{\left(u\right)}& \in \{\mathrm{Green},\mathrm{Yellow},\mathrm{Red}\}.\hfill \end{array}$$

(27)

The reported simulation uses four warning indicators for each target:

$$\begin{array}{cccc}\hfill W_{1,k}^{\left(u\right)}& =\mathbf{1}({\overline{K}}_k^{\left(u\right)}\ge 4.5),\hfill & \hfill W_{2,k}^{\left(u\right)}& =\mathbf{1}({\overline{\rho }}_k^{\left(u\right)}\le 0),\hfill \\ \hfill W_{3,k}^{\left(u\right)}& =\mathbf{1}({\overline{\mathcal{R}}}_k^{\left(u\right)}>5\times {10}^{-5}),\hfill & \hfill W_{4,k}^{\left(u\right)}& =\left\{\begin{array}{cc}\mathbf{1}({\overline{C}}_k^{\left(\mathrm{ret}\right)}\le 0),\hfill & u=\mathrm{ret},\hfill \\ \mathbf{1}({\overline{C}}_k^{\left(\mathrm{vol}\right)}>0),\hfill & u=\mathrm{vol}.\hfill \end{array}\right.\hfill \end{array}$$

(28)

The four warnings flag weak separation, nonpositive transfer, regret above a fixed tolerance, and target-specific adverse stress. ${\overline{I}}_k^{\left(u\right)}$ is a descriptive selection–MCS coherence measure, intentionally excluded from the traffic-light count because no threshold was prespecified. The warning count is

$$N_k^{\left(u\right)}={\displaystyle \sum _{j=1}^4}{W}_{j,k}^{\left(u\right)}.$$

(29)

The traffic-light rule is then

$$g^{\left(u\right)}\left({\overline{A}}_k^{\left(u\right)}\right)=\left\{\begin{array}{cc}\mathrm{Green},\hfill & N_k^{\left(u\right)}=0,\hfill \\ \mathrm{Yellow},\hfill & N_k^{\left(u\right)}\in \{1,2\},\hfill \\ \mathrm{Red},\hfill & N_k^{\left(u\right)}\ge 3.\hfill \end{array}\right.$$

(30)

Green maintains a cautious selected-model interpretation, Yellow calls for review, and Red recommends fallback or re-audit. These conservative thresholds are fixed, not fitted, retrospective audit rules: ${\overline{K}}_k^{\left(u\right)}\ge 4.5$ means that nearly all candidates remain indistinguishable; ${\overline{\rho }}_k^{\left(u\right)}\le 0$ means no positive ordering evidence; and ${\overline{\mathcal{R}}}_k^{\left(u\right)}>5\times {10}^{-5}$ marks loss relative to the candidate-pool minimum. For signed returns, ${\overline{C}}_k^{\left(\mathrm{ret}\right)}\le 0$ flags a nonpositive high-cost proxy; for volatility, ${\overline{C}}_k^{\left(\mathrm{vol}\right)}>0$ means PC-Select is worse than EWMA(0.97). Requiring three warnings avoids a full re-audit after one noisy diagnostic. All variables are carried forward only after the corresponding test year ends.

The fallback simulation converts this interpretation state into a fixed next-year action within the reported retrospective exercise. Let ${\pi }_{k+1}^{\mathrm{ret}}$ and ${\pi }_{k+1}^{\mathrm{vol}}$ denote the actions applied in year $k+1$ for the signed-return and volatility-proxy audits, respectively. For signed returns,

$${\pi }_{k+1}^{\mathrm{ret}}=\left\{\begin{array}{cc}\mathrm{PC}-\mathrm{Select},\hfill & D_{k+1}^{\left(\mathrm{ret}\right)}=\mathrm{Green},\hfill \\ \mathrm{MCS}\mathrm{EqualWeight},\hfill & D_{k+1}^{\left(\mathrm{ret}\right)}=\mathrm{Yellow},\hfill \\ \mathrm{Zero}\mathrm{return},\hfill & D_{k+1}^{\left(\mathrm{ret}\right)}=\mathrm{Red}.\hfill \end{array}\right.$$

(31)

For the volatility-proxy audit, where strong naive volatility references are available,

$${\pi }_{k+1}^{\mathrm{vol}}=\left\{\begin{array}{cc}\mathrm{PC}-\mathrm{Select},\hfill & D_{k+1}^{\left(\mathrm{vol}\right)}=\mathrm{Green},\hfill \\ \mathrm{Naive}-\mathrm{Select},\hfill & D_{k+1}^{\left(\mathrm{vol}\right)}=\mathrm{Yellow},\hfill \\ \mathrm{EWMA}\left(0.97\right),\hfill & D_{k+1}^{\left(\mathrm{vol}\right)}=\mathrm{Red},\hfill \end{array}\right.$$

(32)

where Naive-Select chooses among Roll-10, Roll-21, Roll-63, EWMA(0.94), and EWMA(0.97) using only the preceding validation year. These fallback rules are not optimized on the test years; they test whether audit states can guide conservative next-cycle governance without claiming to improve current-year accuracy.

To test whether the audit could instead support retaining or replacing the current-year PC-Select action, a separate falsification analysis uses only information available at deployment: prequential validation MCS size, the relative validation margin, and the frequency with which the final selected model wins the three scored chronological validation blocks. The validation-only score components are defined in Supplementary Equation S1; the information boundary and risk–coverage evidence are reported in Supplementary Table S5, Supplementary Figure S2, and Supplementary Table S6. Test regret, validation-to-test transfer, candidate-pool minimum coverage, and current test outcomes are excluded from score construction and are used only for evaluation. The analysis compares the three individual signals with the equal-weight score over their full risk–coverage paths. This is a falsification of hypothetical prospective filtering, not an additional model-selection stage.

The experiment used Linux x86_64, Python 3.13.5, NumPy 2.3.5, pandas 2.2.3, scikit-learn 1.8.0, LightGBM 4.6.0, and CPU-only PyTorch 2.10.0+cpu. The container exposed 56 logical CPU interfaces and 4.0 GB RAM. Exact raw files, feature tables, prediction caches, scripts, configurations, results, audit-state tables, and hashes are included. The supplied checks verify the current headline values against the CSV result files, validate raw-data inventories, test the deterministic MCS implementation, and recompute file hashes.

4. Results

The results are organized to answer the three audit questions rather than to select the most attractive metric. Table 2 fixes the case-study boundary; Table 3, Table 4 and Table 5 evaluate whether signed-return validation selection deserves a strong deployment interpretation; Table 6 and Table 7 examine how validation reliability changes across prediction targets; and Table 9 evaluates the feedback and fallback governance mechanism. Supplementary Tables S1, S2, S3, S4, S5, and S6; Supplementary Figures S1 and S2; and Supplementary Equation S1 supply the detailed tables, window-level records, robustness outputs, and validation-only score definition.

4.1. Data Universe and Predictive Baselines

Table 2 reports the fixed ETF universe used in the audit. The benchmark uses Yahoo Finance daily OHLCV snapshots for nine mainland Chinese index ETFs from 2 January 2014 through 31 December 2025. The objective is not to claim universal applicability across all financial markets, but to provide a controlled case study for deployment-oriented audit methodology. Each raw file contains 2916 unique trading dates. The universe was defined before the PC-Audit analysis using three operational requirements: continuous coverage from the beginning of 2014, no duplicate dates, and representation of broad-market, size, style, growth, and sector exposures. It is a controlled audit benchmark rather than an exhaustive or market-wide sample of Chinese ETFs.

The equal row counts in Table 2 matter for interpretation: differences among models are not driven by unequal sample coverage or ticker-specific missing-history rules. At the same time, the table also defines the external-validity boundary. The benchmark is deliberately narrow enough to support a reproducible audit, but it is not a claim about all Chinese ETFs or all market regimes.

Table 3 reports 2020–2024 averages at zero transaction cost. In all 45 cases, the model named by PC-Select is also retained by the prequential-loss MCS, so the deployable recommendation is not rejected by the confidence-set audit even though the set is usually large. PC-Select records RMSE 0.015821, MAE 0.010593, balanced accuracy 0.49998, and MCC $-0.00061$. The directional metrics are effectively at chance.

The naive references are strong on prediction-scale error. The historical-mean reference has the lowest average MAE (0.010546), and the zero-return predictor has slightly lower RMSE (0.015798) and MAE (0.010563) than PC-Select. The paired PC-Select versus zero-return MAE comparison has unadjusted $p=0.0748$ and Holm-adjusted $p=1.000$; ETF-cluster and year-block mean-difference intervals favor the zero reference. Therefore, PC-Select does not establish lower prediction error than naive benchmarks under the present sample. The historical-mean reference also produces a nearly constant sign exposure, so its return proxy is interpreted only as a stress diagnostic rather than as timing skill.

No PC-Select comparison is significant after Holm adjustment. Relative to the same-label calibrated selector, the unadjusted MAE test favors PC-Select ($p=0.0080$), and both dependence-aware intervals have the same orientation, but the Holm-adjusted value is 0.431. The auxiliary ensemble-average row is deliberately included because it is a common robustness device; here it performs worse than the calibrated single-model references on RMSE and MAE, so the mixed conclusion cannot be dismissed as merely a failure to average candidates. This is suggestive evidence that protocol choices alter selection, not evidence of a universal lower-error advantage. Table 3 therefore functions as a guardrail for the rest of the Results section: subsequent transfer and feedback analyses are interpreted as audit diagnostics, not as proof that PC-Select beats naive prediction.

4.2. Validation Transfer and Statistical Separation

Figure 2 and Table 4 quantify validation-to-test transfer. Under prequential calibration, mean rank correlation is 0.090 and both dependence-aware 95% intervals include zero. PC-Select nevertheless chooses the lowest-test-MAE candidate in 15 of 45 cases (33.3%). The exact binomial diagnostic against a uniform 20% rate gives $p=0.0379$, but the more conservative global label-permutation audit gives $p=0.0521$; the within-ETF permutation gives $p=0.0433$. We therefore treat the hit rate as limited, borderline transfer evidence rather than a confirmatory discovery. It does not extend to a stable ranking of all candidates and falls to 22.2% in 2025.

The same-label control has a 35.6% hit rate in 2020–2024 but negative mean rank correlation and larger regret. Its apparent top-one success therefore does not imply better global ranking. Across protocols, no method consistently dominates both the main period and 2025.

The weak transfer result is also visible before testing through small validation margins. The audit records the validation margin between the first- and second-ranked candidates under the default prequential loss. Let $V_{\left(1\right)}$ and $V_{\left(2\right)}$ denote the smallest and second-smallest values in the set $\{{\overline{\ell }}_m^{PC}:m\in \mathcal{M}\}$ for one ETF-window case. The selection-margin statistic is

$${\Delta }_{\mathrm{sel}}=V_{\left(2\right)}-V_{\left(1\right)}.$$

(33)

In 2020–2024, the median ${\Delta }_{\mathrm{sel}}$ is $8.0\times {10}^{-6}$, the interquartile range is 0 to $2.74\times {10}^{-5}$, and 93.3% of cases have ${\Delta }_{\mathrm{sel}}<{10}^{-4}$. These small margins explain why naming a single validation winner is operationally possible but statistically fragile. Figure 3 shows the same issue visually: a deployment recommendation can exist even when the scientific claim remains weak, because the selector is forced to name the smallest validation loss but the distance between the first and second candidates is often too small to justify strong language about superiority.

Calibration timing changes the apparent statistical admissibility of candidates. The default prequential MCS contains 4.80 of five candidates on average and retains all five in 38 of 45 cases (84.4%). The raw-loss MCS contains 3.67 models on average and never retains all five. Same-label calibration retains 4.98 and includes all five in 44 cases. Raw and prequential memberships differ in 43 of 45 cases (95.6%); same-label and prequential memberships differ in 17.8%.

Table 5 shows that the more selective raw-loss protocol is not necessarily the more reliable protocol. It removes more candidates, but it also has lower coverage of a candidate that attains the minimum test MAE and almost always changes the confidence set relative to the prequential design. The omitted plot gives the same trade-off visually; the table is kept in the main text because it carries the numerical audit evidence. This is why PC-Audit reports confidence-set composition as uncertainty evidence rather than treating exclusion as automatic evidence of transferable inferiority.

The raw-loss MCS excludes GRU-small in every main-period case, whereas prequential and same-label calibration retain it in 95.6% and 100% of cases. This contrast shows that apparent model elimination can reflect amplitude mismatch rather than transferable inferiority. PC-Select chooses LightGBM 14 times, ElasticNet and GRU-small 10 times each, Random Forest six times, and Ridge five times; no candidate dominates the validation periods.

Full-validation non-negative slopes equal zero in 41.8% of the 225 model-case combinations. The rate is highest for GRU-small (53.3%) and lowest for Ridge (28.9%). Zero slopes are not treated as failed fits: they state that no positive multiple of the raw forecast lowers validation squared error relative to the zero forecast.

4.3. Validation Reliability Across Prediction Targets

The same audit was also run on volatility-like risk-surveillance targets rather than directional signals. Table 6 reports the target-level comparison, with additional target-robustness details in Supplementary Table S2. The more favorable evidence concerns target choice rather than superiority of any individual model. Signed-return selection has weak main-period rank transfer (0.090) and a nearly uninformative MCS size (4.80 of five models). The absolute-return proxy $|r_t|$, squared-return proxy $r_t^2$, and range proxy $log(H_t/L_t)$ all show stronger main-period validation-to-test rank transfer and smaller confidence sets. Under this CPU-only workflow, the validation evidence for these risk-surveillance targets is therefore more stable than that for signed daily returns.

The volatility-like evidence remains bounded. The absolute-return proxy has a lower average MAE than the historical-mean reference in 2020–2024 (0.007201 versus 0.007392), a descriptive 3.75% improvement. Squared return and range proxy also produce stronger main-period rank transfer than signed returns. However, all three target families weaken under 2025 temporal stress, and the range proxy’s strong rank transfer does not translate into a high top-one rate. The result is therefore not a stable volatility-modeling breakthrough. Its role is narrower: PC-Audit distinguishes an unsupported signed-return deployment claim from more auditable, but still stress-sensitive, risk-surveillance targets.

Table 7 prevents overinterpretation of the volatility-proxy result; Supplementary Table S3 gives the corresponding detailed reference checks. PC-Select improves on zero and historical references, and several paired signed-rank comparisons are favorable. Strong rolling and EWMA volatility references remain competitive, however, and EWMA(0.97) has a slightly lower mean MAE than PC-Select. Therefore, the volatility-proxy result is not reported as universal model-level superiority. The defensible conclusion is that volatility-like targets produce more transferable validation rankings and more selective confidence sets than signed returns, while simple volatility smoothers remain important references.

4.4. Sensitivity, Costs, and Temporal Stress

Across 11 prespecified sensitivity configurations, agreement with the default selection ranges from 55.6% to 100%, and mean MCS size ranges from 4.53 to 4.91. RMSE ranges from 0.015821 to 0.015878, MAE from 0.010593 to 0.010647, balanced accuracy from 0.4964 to 0.5054, and MCC from $-0.0082$ to 0.0123. Thus, the identity of the deployed model is specification-sensitive, while directional evidence remains weak.

Economic stress diagnostics are more sensitive. Across the prespecified sensitivity configurations, gross signal diagnostics vary materially and the 5 bps diagnostics include both positive and negative outcomes. The full values are retained in the reproduction package listed in Supplementary Table S1 to document sign and cost sensitivity, not economic profitability. Unconstrained calibration gives the strongest gross diagnostic but permits sign reversal; the legacy clipped rule gives a negative 5 bps result. The default non-negative rule is retained because it has a clear interpretation and was fixed before comparing these alternatives, not because it maximizes any economic proxy.

Under the default unlagged diagnostic, the PC-Select signal proxy turns negative once the transaction-cost assumption is increased to the highest stress level. One-day-lagged controls remain positive under some specifications, whereas the MCS equal-weight control is negative under others. In 2025, PC-Select is positive under lagged long-cash but negative under the unlagged 5 bps specification. This split is treated exclusively as an external temporal stress check and is not included in statistical inference. Since sign and magnitude depend on calibration, execution, cost, and period, these stress diagnostics are not interpreted as trading strategies or investable evidence. Table 8 reports the compact 2025 stress signs; detailed execution and Sharpe tables are provided in the accompanying reproduction package.

The historical-mean reference has the lowest 2025 mean MAE, while PC-Select remains close to the zero-return reference. The historical-mean signal proxy mainly reflects a nearly constant exposure determined by the preceding validation year; it is not evidence of timing ability. The 2025 sample contains only nine ETF cases, so its statistics are descriptive stress evidence rather than a confirmatory study.

4.5. Deployment-Cycle Feedback and Computational Boundary

Table 9 converts completed annual audit outputs into next-cycle monitoring states. The table is not an online-learning result: the realized test-year outcomes are used only after each year is complete and only to define review information for the following cycle. Under the warning-count rule, 2020, 2022, and 2024 trigger Red next-review states, while 2021 and 2023 trigger Yellow states. This avoids treating any single warning as sufficient for re-audit and instead requires several weak audit signals to align. The rule-based retrospective simulation records when the next deployment review would weaken the selected-model interpretation or require re-audit; it does not constitute prospective validation of the traffic-light thresholds.

The audit-state table is the empirical feedback mechanism rather than a conceptual add-on. Red years combine several warnings, such as large confidence sets with weak rank transfer, regret above tolerance, or nonpositive high-cost stress. Yellow years still require manual review, but they do not trigger the stronger fallback recommendation because fewer warnings align. This distinction keeps the feedback mechanism conservative and prevents the article from claiming that the audit rule improves the already completed test-year results.

Panel B of Table 9 applies each target’s completed-year traffic-light state as a next-year fallback rule, with detailed fallback values in Supplementary Table S4. The comparison starts in 2021 because the 2020 action has no previous completed audit state. For signed returns, Green would continue PC-Select, Yellow falls back to the MCS equal-weight control, and Red falls back to the zero-return reference. For the volatility-proxy target, the separately computed volatility audit state maps Yellow to validation-year Naive-Select and Red to EWMA(0.97). The simulation does not improve average MAE: signed-return mean MAE changes by only 0.024%, and volatility-proxy mean MAE worsens by 0.52%. Its governance value is narrower. The worst-year mean MAE is reduced by 0.35% for signed returns and by 0.90% for the volatility proxy, without using current-year test outcomes to tune the current-year action.

The deployment-time falsification does not support promoting the validation signals to a current-cycle filtering rule. For signed returns, the equal-weight score has essentially no association with subsequent candidate-pool regret ($\rho =0.040$, $p=0.792$), while the margin-only score fails to provide the intended monotone assurance ($\rho =0.498$, $p<0.001$); the latter association is descriptive, not causal. For the absolute-return proxy, MCS selectivity is weakly favorable ($\rho =-0.251$, $p=0.096$), but the full score is weaker ($\rho =-0.164$, $p=0.281$). Full-score associations are also small in the separate 2025 cases ($-0.075$ and $-0.083$, respectively). Prespecified threshold, cluster, leave-one-group, and risk–coverage checks are reported in Supplementary Table S6 and Supplementary Figure S2. The evidence supports next-cycle use of completed audits, not same-cycle filtering.

Finally, PC-Select averages 0.525 s of candidate training time and 0.0018 s of deployed-model inference per case in the stated environment. The MCS equal-weight control retains multiple models and averages 2.37 s training and 0.0090 s inference. The full five-model ensemble average is reported as an additional low-cost diagnostic in statistical terms but would require retaining all five candidate predictions at deployment time. These timings are therefore relative indicators under one controlled CPU environment rather than universal edge-device latency claims; a single-core or hardware-constrained deployment would require separate timing.

5. Discussion

5.1. Why PC-Audit Is a Decision-Reliability Framework

PC-Audit should be read as a decision-reliability framework rather than a return-prediction model. Its input is a fixed candidate pool and a chronological validation decision; its output is a selected model name plus an uncertainty and robustness record explaining whether that selected action deserves a strong interpretation. This distinction matters because the candidate with the lowest validation MAE can always be named, even when the validation window provides little evidence that the choice will transfer. PC-Audit makes that gap visible through confidence-set size, candidate-pool minimum coverage, rank transfer, regret, multiplicity-adjusted paired tests, dependence-aware intervals, cost and execution checks, sensitivity variants, and the 2025 temporal stress evaluation.

This framing also explains why PC-Select is not evaluated as a new predictive architecture. Large confidence sets, weak rank transfer, or unstable economic signs are informative audit outcomes because they limit the claim that can be attached to the selected candidate. A weak audit outcome is therefore not a failure of PC-Audit; it is decision-relevant evidence that the audit is designed to expose. The relevant comparison is between an unaudited selection workflow and one that reports the uncertainty and practical constraints surrounding the same deployment action.

5.2. What the Audit Establishes

The evidence is deliberately mixed. PC-Select chooses the test-year MAE winner more often than a uniform five-model reference in 2020–2024, but the result is borderline under label permutation and is interpreted only as limited transfer in the top-ranked candidate. Yet the average rank correlation is small and uncertain, the full ordering does not transfer, the selected model changes under reasonable protocol variants, and the 2025 top-one rate returns to 22.2%. The defensible conclusion is therefore not that validation is useless, but that one year supplies insufficient evidence for a stable winner.

Table 10 summarizes the change in interpretation caused by the audit layer. It is placed here rather than among the numerical result tables because its role is interpretive: it shows how a conventional validation-selection workflow could overstate the evidence and how PC-Audit weakens or rejects that interpretation.

Calibration timing is consequential. Raw-loss MCS appears more decisive but has lower candidate-pool minimum coverage and almost never agrees with the prequential set. Same-label calibration retains nearly everything and reuses outcomes for fitting and scoring. The prequential design avoids label reuse and exposes uncertainty more transparently, even though it cannot create predictive information that is absent from the data.

The result also clarifies the role of MCS. PC-Select always names one deployable candidate, while the prequential-loss MCS usually retains all five. Reporting only the selected name would conceal the lack of statistical separation. Conversely, averaging candidates does not automatically provide a more useful deployment signal: both the MCS equal-weight control and the full five-model ensemble average have weaker direction or prediction-scale diagnostics than several single-model references.

The comparison across prediction targets provides the main favorable empirical result. Under the same target-date partitioning and prequential-selection logic, $|r_t|$, $r_t^2$, and $log(H_t/L_t)$ produce stronger main-period rank transfer and smaller confidence sets than signed returns. This does not change the paper into a volatility-modeling benchmark, because rolling and EWMA volatility references remain competitive and the 2025 stress split weakens the target-level evidence. It does show that PC-Audit can compare the deployability of different prediction targets: a target may be more useful for risk surveillance even when signed-return selection remains unreliable.

5.3. Implications for Applied Deployment

First, studies that screen multiple models on short financial validation periods should report selection hit rates, regret, and confidence-set size in addition to the winning score. Second, calibration belongs inside the comparison protocol. Applying it only after model selection can penalize candidates with amplitude mismatch and produce misleading elimination. Third, naive zero and historical-mean references remain essential when the target is close to unpredictable; they should be reported even when their sign exposure is not interpreted as timing skill. Fourth, economic metrics should be treated as stress outcomes rather than used to choose among many protocol variants. In applied automated-decision settings, the practical value of such a protocol is not only to find a candidate for deployment, but also to document when deployment evidence is too weak for a strong claim.

The deployment-cycle audit-state simulation extends this implication from one-off reporting to scheduled model governance. A completed test year can be converted into monitoring variables for the next review: confidence-set size, rank transfer, regret, and stress outcomes. The warning-count rule is intentionally less mechanical than a one-warning rejection rule; it weakens interpretation when one or two warnings appear and recommends fallback or re-audit only when several warnings align. The fallback simulation confirms the appropriate scope of this mechanism. It does not improve average MAE and should not be marketed as an accuracy booster. Its value is conservative governance: it reduces the worst-year MAE modestly while preventing the next deployment cycle from blindly inheriting the previous validation winner when the realized audit state is weak. The deployment-time falsification further shows why this distinction matters: validation-only MCS, margin, and block-stability signals do not form a stable same-cycle authorization score in this sample. PC-Audit therefore functions as a reliability layer around model selection rather than as an adaptive trading engine.

The non-negative calibration rule is intentionally simple. It prevents sign reversal, allows evidence-based shrinkage to zero, and avoids an arbitrary upper cap. The frequent zero slopes are substantively informative: the validation data often do not support carrying a model’s raw directional signal forward at any positive scale.

The value of PC-Audit is not to identify a universally superior forecasting model, but to quantify when a validation-selected deployment decision should be trusted, weakened, or rejected. In this sense, the framework provides an auditable decision layer for model-selection governance rather than another prediction model.

5.4. Scope, Limitations, and Future Work

The study uses nine ETFs from one market and one public data vendor. The inclusion rule favors instruments with continuous historical coverage and introduces availability and survivorship bias; this design supports a controlled benchmark but not a market-wide claim about all Chinese ETFs. The study demonstrates protocol feasibility and evidence diagnosis in this panel; it does not establish cross-market generalizability. Because the vendor is not the official exchange record, replication on exchange-licensed data would be a useful external validity check. Future work should extend the audit to multi-market ETF panels, individual stocks, intraday horizons, and alternative vendors.

The candidate pool is small and fixed by design, and additional baseline comparisons are introduced to test whether the observed instability is attributable to model-pool weakness or to the underlying signal environment. In particular, a leakage-controlled historical-mean reference and an equal-weight ensemble of the five candidate predictions are evaluated without additional model training. More powerful architectures, fundamental data, cross-sectional learning, or additional hyperparameter searches could produce different results, although broader searches would also increase selection and multiple-testing risk. The five-model pool should therefore be interpreted as a controlled CPU-only audit panel spanning different computational regimes, not as an exhaustive model universe. Future work can apply the same audit layer to larger architecture sets, including foundation or transformer-style time-series models, provided that the expanded search space is accompanied by stronger multiplicity and selection-risk controls. PC-Audit evaluates selection among the stated CPU-only models; it is not an impossibility result for ETF prediction and should not be read as evidence that Chinese ETF returns are intrinsically unpredictable.

The calibration corrects only global amplitude. It does not address nonlinear conditional bias, volatility-dependent calibration, or probability calibration. Long-term regime changes may also require adaptive monitoring rather than a fixed annual validation rule. The audit-state carry-over and fallback rules reported here are lightweight governance simulations; a fully adaptive implementation that changes the model pool, validation length, fallback policy, or hyperparameter schedule would require a separately prespecified online protocol and additional statistical checks. The volatility-like target extension is also limited: it uses the same five-model CPU-only pool and tests whether risk-oriented targets give more auditable validation transfer, not whether PC-Select dominates all volatility references. The robustness scripts explicitly remove target-derived columns from the feature matrix, but the targets still share the same ETF panel and OHLCV data source. The sensitivity variants reuse the same dataset, and the exact top-one test treats cases as Bernoulli trials even though dependence remains; the cluster and year-block summaries are therefore reported alongside it. Finally, signal calculations omit bid-ask variation, market impact, taxes, short-sale constraints, and fund-specific implementation details. They are not investable backtests.

6. Conclusions

This paper presents PC-Audit, a reproducible decision-reliability protocol for auditing validation-based machine-learning model selection under temporal dependence. It is evaluated here in weak-signal financial time series. The framework is tested in a controlled CPU-only benchmark of nine Chinese index ETFs. It enforces target-date partitioning, evaluates non-negative scale calibration prequentially, separates deployment ranking from MCS uncertainty, measures validation-to-test transfer and regret, audits dependence, costs, execution, sensitivity, and temporal stress behavior, and records completed-year audit states for the next deployment review.

The three audit questions posed in the Introduction are answered as follows. First, the leakage-controlled validation rule does not justify a strong signed-return deployment claim: PC-Select has only borderline top-one evidence and no Holm-adjusted lower-error advantage over naive or individual-model references. Directional metrics remain near chance, while weak 2025 transfer, large confidence sets, small validation margins, weak average rank transfer, and sensitivity to calibration timing further limit the interpretation. Second, volatility-like risk-surveillance targets are more auditable than signed returns in the main period because their rank transfer is stronger and their confidence sets are smaller, but this conclusion remains bounded by competitive rolling/EWMA references and weaker 2025 stress results. Third, completed audit outputs can be carried forward as monitoring variables and fixed fallback actions in a rule-based retrospective simulation without converting the benchmark into an online learner; the simulation modestly reduces worst-year MAE but does not improve average MAE.

These answers lead to a conservative practical conclusion. In this setting, ETF signed-return deployment remains a weak-signal problem, validation ranking does not guarantee future stability, and a one-year validation interval provides insufficient evidence for declaring a stable winning return model. The main empirical finding is not that PC-Select dominates naive forecasts, but that ordinary validation-based model selection would be overinterpreted without an audit layer. The favorable methodological finding is that PC-Audit can distinguish deployment targets: signed returns do not support a strong deployment interpretation, whereas volatility-like risk-surveillance targets yield more stable validation evidence but remain sensitive to temporal stress. Rather than arguing that financial prediction has become reliable, PC-Audit argues that deployment decisions themselves should be audited before they are interpreted. This distinction is the principal contribution of the present work. Future research may extend the framework to broader asset universes, alternative data sources, and adaptive audit schedules while preserving strict out-of-sample evaluation.