Autonomous Hypothesis Testing Against Public Biomedical Data with Literature-Grounded Calibration

Public biomedical repositories contain petabytes of data, yet testing hypotheses against those resources still requires expert effort. I present a four-phase blinded workflow for autonomous computational hypothesis testing: an agent receives a natural-language hypothesis, discovers public datasets with academic publisher domains blocked, executes an analysis pipeline and validates findings against the literature. Published studies serve as calibration rulers, measuring how closely an autonomous workflow approaches known answers before it is trusted on unknown ones. Across five biomedical tasks spanning bacterial genomics, single-cell multiomics, metagenomics, cancer genomics and GWAS functional genomics, the agent scored 5–9 out of 10 on pre-specified rubrics in 20–188 minutes (the single-cell multiomics score was corrected from an initial 8 to 5 after blinded judge review identified over-credited criteria; see Methods). An exploratory comparison across three successive model generations found scores directionally higher for the first generation (claude-opus-4-6) on all five tasks (one-sided Wilcoxon p = 0.031, rank-biserial r = 1.0; n = 5 studies), with newer models completing tasks faster; the benchmark also exposed generation-specific safety policy and schema differences that silently confound cross-generation comparisons if not corrected. Failures fell into three categories, tool-version sensitivity, data-scope limitation and analytical completeness, all self-diagnosed by the validation phase in all five cases examined here without human prompting. These results introduce a calibration framework and failure taxonomy for characterising when autonomous computational science workflows should and should not be trusted.