Distance-Dependent Miscalibration of AlphaGenome Regulatory Effect Scores Against Experimental Ground Truth

1. Introduction

Roughly 98% of the human genome does not encode protein; it encodes regulation. Most trait-associated variants fall in this fraction, and their interpretation is the central bottleneck of statistical and clinical genetics. Sequence-to-function models have narrowed the gap — Enformer¹, Borzoi² — and AlphaGenome now predicts multi-modal regulatory output at base resolution from 1 Mb of input, matching or exceeding the best external model on 24 of 26 variant-effect evaluations^3,4.

A predictor becomes an instrument only when its output is calibrated. The missense field made this transition explicitly: computational predictors were calibrated against ClinVar to likelihood ratios and mapped to ACMG/AMP evidence strengths^5–7, because a missense score has a fixed referent — the protein. No equivalent exists for AlphaGenome, and its developers have said so. Asked on the community forum for “a recommended cutoff ... to classify a variant as ‘pathogenic,’ or to use it as supporting evidence such as PP3”, a team member replied:

“We do not have a recommended cutoff. The calibration approach might also differ for different classes of variants / predicted modalities.”⁸

Absent a cutoff, every current analysis applies an ad hoc global threshold to the raw score. That is only meaningful if a raw score means the same thing everywhere in the genome. We test whether it does, using the strongest available external reference: saturation-mutagenesis reporter assays that measured the effect of every possible single-nucleotide change in a panel of regulatory elements⁹. This converts the calibration question from model-internal introspection into a direct empirical comparison: does the model’s spread of predicted effects match the spread that is actually measured?

2. Results

2.1. The Model’s Null Dispersion Does Not Match the Measured Null

For each of 15 regulatory elements with saturation-mutagenesis measurements (Kircher et al. 2019⁹, GRCh38), we scored every single-nucleotide variant with AlphaGenome and compared the dispersion (median absolute deviation, MAD) of predicted effects to the dispersion of the laboratory-measured effects at the same locus. Under perfect calibration the ratio MAD(model)/MAD(measured) would be 1 everywhere.

It is not. The ratio ranges from 0.03 to 5.79 across loci — a 177-fold span (Figure 1, Figure 2; Table 1). The model over-disperses at some loci and collapses at others: it is too wide (ratio > 1) at 6 of 15 loci and too narrow at 9. No global rescaling can fix a discrepancy that changes direction between loci.

2.2. The Miscalibration Is Distance-Dependent

The ratio is not random across loci: it is a strong, monotone function of the element’s distance to the transcription start site (Figure 1). Spearman correlation between log-distance and the MAD ratio is ρ = −0.80 (p = 3.4 × 10⁻⁴, 15 loci). The pattern is mechanistically legible:

At promoters (<500 bp from the TSS) the model over-disperses — median MAD ratio 2.26, reaching 5.79 at the HBB promoter. It predicts a wider spread of consequences than the reporter assay measures.

At distal enhancers (≥10 kb) the model collapses — median MAD ratio 0.10, falling to 0.03 at the SORT1 enhancer 123 kb from its target. Here the model predicts almost no spread of effects at all, even though the assay measures a broad one.

This mirrors, against external truth, a decay we independently measured in the model’s own null: sampling 4,800 matched control variants across 12 genes and 8 distance bins, the internal null dispersion falls as a gene-specific power law in distance (MAD ∝ d^−k, k = 0.27–0.62), and the decay is modality-specific — RNA-seq dispersion falls ~25-fold from promoter to 100 kb, whereas chromatin-accessibility dispersion falls only ~2-fold, consistent with a longer causal chain from a distal variant to a change in gene expression than to a local change in accessibility (Supplementary analysis). The saturation-mutagenesis data show that this decay is too steep: the model’s effect scale shrinks with distance faster than real regulatory effects do.

2.3. Raw Scores Are Not Comparable Across Loci

The direct consequence is that a raw AlphaGenome score cannot be thresholded genome-wide. A score of 0.1 is unremarkable at a promoter, where the model’s spread is large, but extreme at a distal enhancer, where the model’s spread is near zero. When we pool all 20,549 variants across the 15 loci and rank them by |raw score| against the experimental truth (whether the assay called the variant functional at p < 0.05), the ranking is barely better than chance: AUC ≈ 0.56. The information the model carries within each locus is largely destroyed when raw scores from different loci are placed on one axis.

Local calibration — expressing each score relative to its own locus’s null — is the natural remedy and points in the right direction (pooled AUC 0.59 vs 0.56), but on 15 independent loci this improvement is not statistically robust (95% bootstrap CI on the gain crosses zero: [−0.08, +0.14]). We therefore make the weaker, defensible claim: local calibration is necessary for cross-locus comparability in principle, and the data are consistent with a detection benefit, but this panel is underpowered to establish one. The strong, external result is the miscalibration itself (§2.1–2.2), not a specific correction for it.

2.4. The Best-Characterised Causal Variant Is Demoted

TERT C228T (rs1242535815) creates a de novo ETS/TCF binding site and is the most recurrent non-coding driver mutation in human cancer^10,11. In the saturation-mutagenesis assay it is the strongest single variant in the entire TERT promoter: measured rank 1 of 777 in glioblastoma-derived cells, where TERT reactivation is a genuine driver (measured effect +2.86; p ≈ 0).

AlphaGenome predicts C228T in the correct direction (a gain of function) but demotes it to rank 14 of 777 (Figure 3). Thirteen variants the model scores at least as highly are, by measurement, weaker. A pipeline that took AlphaGenome’s ranking at face value would not place the single most important non-coding cancer mutation at the top of its own promoter. The model’s within-locus correlation with measured effects is moderate and cell-type-dependent (Spearman ρ from +0.31 in HEK to +0.52 in glioblastoma across the four TERT assays), which is a useful ranking signal — but not a substitute for measurement, and not a calibrated magnitude.

2.5. A Coordinate-Verification Guardrail, and a Caution It Exposed

Building the comparison pipeline surfaced a failure mode worth reporting because it is easy to hit and silent. Published coordinates for famous variants are frequently in GRCh37/hg19; C228T is chr5:1,295,228 in hg19 but chr5:1,295,113 in hg38, a 115 bp shift that moves the variant between distance bins. When we supplied the hg19 coordinate with its reference allele to the AlphaGenome API, the API accepted a variant whose reference base does not exist at that hg38 position and returned a score, with no error. We therefore verify every reference allele against the genome before scoring; the released pipeline rejects mismatches. All results here use GRCh38 coordinates with verified reference alleles (0 mismatches across 20,549 variants after verification).

3. Discussion

3.1. What Is Established

AlphaGenome’s effect-score scale is not physically calibrated across the genome. Measured against saturation-mutagenesis ground truth, its spread of predicted effects is too large at promoters and too small at distal enhancers, varying 177-fold across loci as a monotone function of distance to the TSS. This is not an artefact of the model’s internal null (which could be dismissed as “the biology really is more variable at promoters”): it is a mismatch between the model and directly measured effects, in both directions.

The practical corollary is immediate and consequential. Because raw scores are not cross-locus comparable, any genome-wide threshold — the operation every current user performs, since none is provided — pools incomparable quantities and ranks variants near chance. This is why the search for “the AlphaGenome cutoff” has not converged: the quantity a cutoff would act on is not on a stable scale.

3.2. Why the Missense Strategy Does Not Transfer

Missense calibration succeeded globally because a missense score refers to a fixed object, the protein. A regulatory effect has no position-invariant referent: the same nucleotide change means something different 50 bp and 50 kb from a promoter, and the model’s own scale reflects that — but reflects it wrongly, over-correcting with distance. Porting the ACMG-style single-threshold approach to regulatory scores will therefore assign evidence strengths that do not match the strength delivered. Calibration must be local.

3.3. Limitations

We state these plainly. Fifteen loci. The saturation-mutagenesis panel is small and promoter-heavy; the distance correlation is strong but rests on few distal points, and the enhancer arm would benefit from more elements. Underpowered on the fix. We show miscalibration robustly but cannot prove that local calibration improves detection on this panel (§2.3); that requires a larger element set. One modality, one variant class. RNA-seq / expression, SNVs, model FOLD_0. Reporter assays are not the endogenous locus: saturation-mutagenesis measures episomal or integrated reporter activity, which need not equal effect on the native gene; this bounds “ground truth” but is the best available external reference. Cell-type matching is imperfect: the assay cell line and the AlphaGenome track set are not identical, contributing to the moderate within-locus correlations. Provenance is time-bounded: all data were generated on 17 July 2026, after the 14 July 2026 indel-scoring fix¹² and the 18 June 2026 quantile recalibration¹³; neither affects SNV raw scores, but scores from other dates are not comparable.

3.4. Outlook

The finding is not a criticism of AlphaGenome, which is a genuine advance on a problem that was intractable. It is a statement about what kind of instrument it is: within a locus it ranks variants informatively; across loci its scale is not a ruler. The constructive path is a published, genome-wide local-calibration resource — per-locus, per-distance, per-modality reference distributions, computed once, in the way gnomAD¹⁴ made allele frequencies a shared reference. Saturation-mutagenesis and lentiMPRA panels can anchor such a resource to measured effects where they exist. Our procedure, the per-locus factors reported here, and the coordinate-verification guardrail are released toward that end.

4. Methods

4.1. Model and Scoring

AlphaGenome was queried through the public API (client v0.6.1), model version FOLD_0, with the full 1,048,576 bp context window centred on the target gene’s canonical TSS. Signal is raw_score from the RNA_SEQ scorer (GeneMaskLFCScorer), a natural-log fold change of predicted expression for the target gene, aggregated across the ~400 returned tracks by abs-max (not mean, per developer guidance that averaging dilutes signal¹⁵).

4.2. Experimental Ground Truth

Saturation-mutagenesis measurements are from Kircher et al. 2019⁹ (OSF 10.17605/OSF.IO/75B2M), using the GRCh38 release to avoid liftover error. We used all 22 element×cell-line assays with a resolvable target gene inside the 1 Mb window; for cross-locus pooling we reduced to 15 distinct loci (one assay per gene; for TERT, the glioblastoma line as the most physiological). A variant was called “functional” at the assay’s own significance threshold (p < 0.05). Only single-nucleotide variants were used (indels excluded).

4.3. Coordinate and Allele Verification

Gene coordinates and canonical TSS were taken from GENCODE v46 (the annotation AlphaGenome itself uses), reading the TSS from the Ensembl_canonical transcript rather than the gene boundary — the two differ by >100 bp for 32% of protein-coding genes and by 797 bp for MYC, enough to change distance bins. GENCODE Start is 0-based and was converted to 1-based (verified against Ensembl: TERT and MYC match to the base). Every variant’s reference allele was checked against the GRCh38 sequence before scoring (§2.5); mismatches were rejected.

4.4. Statistics

Dispersion is MAD scaled to Gaussian σ (1.4826 × MAD), chosen over the SD because both measured and predicted effect distributions are heavy-tailed. The distance–miscalibration relation was tested by Spearman correlation between log₁₀(median element–TSS distance) and the per-locus MAD ratio. Cross-locus detection was assessed by the AUC (Mann–Whitney) of |score| against the binary functional call, pooled over the 15 distinct loci; the local-vs-global gain and its 95% CI were estimated by 2,000-replicate bootstrap resampling over loci, so that the interval reflects between-locus variability rather than pseudo-replication from correlated variants within an element. The model’s internal null (Supplementary) used 4,800 matched control SNVs across 12 genes × 8 log-spaced distance bins, sampled bilaterally around the TSS to control the model’s strand-dependent positional artefact¹⁶. All tests two-sided.