Reproducible Multi-Metric Stress Testing for Diagnosing Binary Classifier Failure Modes

Reliable deployment of binary classifiers requires more than reporting a single performance metric, because calibration errors, decision-threshold shifts, class-specific collapse, and ranking inversion may produce different operational consequences while yielding partially similar summary scores. This study develops a reproducible multi-metric stress-testing and diagnostic profiling framework that treats standard evaluation metrics as a joint representation of controlled classifier degradation. The primary simulation combines a calibrated baseline, seven controlled degradation paths, four positive-class prevalence levels, 1000 Monte Carlo replicates, and 11 severity levels (λ ∈ [0.0, 1.0]), producing 308,000 evaluation scenarios (280,000 nonzero degradation profiles across 4000 independent replicate groups). Each scenario is represented by an 11-dimensional profile spanning threshold-dependent, correlation-based, ranking-based, and probability-based metrics. Diagnostic separability is evaluated using 5-fold GroupKFold cross-validation, paired Wilcoxon signed-rank tests with 95% bootstrap confidence intervals, metric-family and redundancy ablations, principal component analysis (PCA), linear and non-linear statistical probes, leave-one-prevalence-out (LOPO) validation, and zero-shot transfer to non-Gaussian latent score distributions (37,800 additional scenarios). Under the conservative multinomial logistic regression probe, the full 11D profile achieved Balanced Accuracy (BA) = 0.7841 (95% CI: [0.7821, 0.7860], Macro-F1 = 0.7851), whereas non-linear probes reached BA = 0.8460 (Random Forest), 0.9273 (HistGradientBoosting), and 0.9333 (MLP). The full profile significantly outperformed the strongest single family (threshold/confusion BA = 0.6148, p < 10^(-100)), a minimal cross-family triplet (MCC + ROC AUC + Brier Score BA = 0.4370, p < 10^(-100)), and single-metric baselines (MCC BA = 0.2602). PCA showed that the first five components capture 96.81% of variance, and removing algebraically derived metrics (F1 and BA) changed linear probe accuracy by only 0.0050 (BA = 0.7791). Zero-shot transfer of Gaussian-trained probes to unseen non-Gaussian distributions yielded BA = 0.8318 (HistGB) on bimodal Gaussian mixtures, 0.5989 on heavy-tailed Student-t (ν = 3) scores, and 0.5483 under heteroscedastic class variances, while LOPO validation yielded 0.660 ± 0.100. To connect the controlled experiment with operational reporting, seven pretrained deepfake classifiers were evaluated on the hardfakevsrealfaces subset (N = 1289, prevalence = 0.5431) in VisionMetrics, where complete 11D profiles exposed severe metric discordance and triggered structured rule-based alerts (CLASS_ASYMMETRY, PROBABILITIES_WORSE_THAN_CONSTANT) relative to a neutral p = 0.5 reference. The results establish multi-metric response profiling as an interpretable controlled simulation and diagnostic profiling methodology rather than a standalone general-purpose failure detector.