Negative Stratum and Score Shift, Not Threshold Choice, Dominate Reported Operating Characteristics of a Chest-Radiograph Pneumonia Classifier

Background. Chest-radiograph classifiers are usually evaluated against healthy controls, and the resulting AUROC is often read as evidence of clinical viability. Two properties of such benchmarks remain invisible in that number: which patients supply the negatives, and how the score distribution aligns with the population the model is applied to. Methods. One EfficientNet-B0 pneumonia classifier was trained on 46,238 ChestX-ray14 images (788 positive; positive-class weight 57.7; no resampling) and prior-corrected to population prevalence. The operating threshold was fixed at 90% sensitivity on a patient-disjoint validation slice (88 positives) and then transferred unchanged to a held-out set of 17,494 images. Held-out negatives were healthy controls (9,861), co-prevalent disease without pneumonia (effusion, oedema or atelectasis; 7,078; withheld from training), or externally sourced normal films (10,192). Wilson and bootstrap 95% confidence intervals are reported throughout, together with an untrained-network negative control and a site-separability probe. Results. Validation AUROC was 0.813. Transferring the validation threshold to held-out healthy controls gave sensitivity 0.962 (95% CI 0.943–0.975) and specificity 0.262 (0.253–0.271); specificity at a matched 90% sensitivity, obtained by placing the threshold on the evaluation positives, was 0.459. Replacing healthy negatives with co-prevalent disease, holding the same 555 positive patients, reduced AUROC from 0.759 (0.740–0.777) to 0.589 (0.563–0.615) and specificity at matched sensitivity from 0.459 to 0.129. Mean predicted probability was 0.0443 for pneumonia, 0.0362 for co-prevalent disease and 0.0206 for healthy controls, so pneumonia was separated from other abnormality by a factor of 1.22, against 2.15 for health. The gap persisted within both view positions separately (+0.119 AP, +0.224 PA), so view composition cannot account for it. Prior correction reduced expected calibration error from 0.195 to 0.008 without moving the operating point. Reaching 90% specificity at 90% sensitivity would require an AUROC of approximately 0.965. Conclusions. For this model, specificity at a clinically defensible sensitivity was determined by the ranking and by the choice of negative stratum, not by calibration or thresholding; a well-calibrated classifier can remain unusable at the operating point a screening programme needs. Benchmarks that report a single aggregate against healthy controls do not measure disease discrimination. Whether label quality bounds achievable AUROC is not tested here.