Exposure Before Success in Indirect Prompt Injection Measurement

Indirect prompt injection is now measured the way accuracy is measured, by running an attack corpus against a model and reporting the fraction of trials that succeed. This paper argues that the resulting number is not interpretable on its own, and shows why on a purpose-built harness covering three agent environments, six user tasks, seven attack patterns and eleven candidate models. The central claim is that success cannot be read without exposure, meaning a record of whether the model’s own tool calls ever placed the injected text in front of it. A pre-registered manipulation moved one payload from past a search snippet cut to inside it, on the same page, with no other change. Exposure for two models rose from 0 of 126 trials to 126 of 126 while neither model altered a single tool call. Ranking the seven usable models by raw attack success rate and by exposure conditional success rate produces two orderings that are mildly anti-correlated, with a Spearman rho of -0.25. Writing the measures down formally shows why: the raw rate splits into success among exposed trials plus credit earned in trials the attack never reached, and a valid measurement needs that second term to be zero. It is zero on four of the seven models. It supplies 73 and 84 percent of the raw rate on the two that top the raw leaderboard. Three further checks behave the same way. A target-state checker recorded 126 attack successes on one model and 42 on another in trials where the payload was never read, while the same checker was correct on 310 of 310 such trials for three other models. A capability floor excluded 4 of 11 candidates, and a twelve-trial screen admitted one model that later measured 0 of 8 on the central task. Identical inputs re-scored between 7 and 60 percent of prompts differently at temperature zero. The corpus also carries English, Urdu and Roman Urdu arms, and the language effect is reported as a null that failed replication. None of the four failure modes depends on the corpus being multilingual.