All We Also Need Is ABSTAIN: Reducing Hallucinations via an Explicit Abstention Token

Large language models (LLMs) suffer from hallucinations—confidently generating false information when uncertain. Conventional token prediction provides no dedicated output action for abstention: uncertainty can be represented only indirectly through the output distribution, while the model is nevertheless trained to select ordinary vocabulary tokens. We investigate whether an explicit abstention state, combined with scalable supervision for when abstention is appropriate, improves selective prediction. We add a single ABSTAIN token to the vocabulary and train models to predict it using corruption augmentation—a scalable data augmentation technique where corrupted inputs are mapped back to the abstain token. In a simple feedforward network tasked with single-token prediction, this approach reduced the hallucination rate on unseen data from 95% to 0% while maintaining perfect accuracy on known examples. The same principle also scaled to a real question-answering (QA) model: a distilled Bidirectional Encoder Representations from Transformers (BERT) model fine-tuned on SQuAD abstained on 89% (±7.8) of nonsense questions at corruption level p = 0.10, rising to 94.5% (±7.8) at p = 0.20, while retaining approximately 95% of its baseline accuracy at both levels; this abstention transferred to natural text corruption (OCR errors, encoding artifacts, character damage) without any noise-specific training, and mining the model's own confident errors provided complementary supervision, extending abstention to fluent-but-unanswerable questions that mechanical corruption misses; combining the two supervision sources yielded broader abstention coverage across all evaluated failure modes.