Look Back, Look Ahead: Coreference Signals in Pretrained Language Model Attention

1. Introduction

Coreference resolution decides which mentions refer to the same entity, a prerequisite for merging mentions into a single node when building knowledge graphs. Strong systems fine-tune encoders (Martinelli et al., 2024) or large decoders (Gan et al., 2025). Small decoder-only language models, by contrast, are often deployed as frozen general-purpose backbones whose internals are directly accessible. This raises the question of which representations make coreference most accessible to a lightweight probe on a frozen model: pooled span states, pairwise attention features, or their combination?

Prior work uses attention heads and contextual representations for pronoun resolution and referential probing (Clark et al., 2019; Klein & Nabi, 2019; Sorodoc et al., 2020; Tenney et al., 2019; Tikhonov & Ryabinin, 2021; Vig & Belinkov, 2019; Yordanov et al., 2020). We evaluate this question on document-level benchmarks across model families, using capacity and layer-access controls.

We train mention-ranking probes on hidden states, attention features, or both, and evaluate them with gold and predicted mentions. Our findings are:

• Combining attention with hidden states gives the best OntoNotes scores in every model we test, and the gain depends on the pair-specific alignment of attention (Section 5).
• In three models, mean-ablating the 16 heads ranked highest by the attention probe lowers the hidden-state readout even after the probe is retrained, with much smaller effects for 16 heads sampled outside the top 64 (Section 6).
• Without target-domain calibration, attention-only probes transfer best to pronoun benchmarks; threshold calibration on GAP reduces their advantage over combined probes.
• Attention that a span receives from later tokens helps detect mentions of non-singleton coreference chains, making end-to-end resolution from frozen features possible (up to 72.5 CoNLL F1 with Llama-3.2-3B).

Combining the two representations also helps relation extraction with given entities (Section 7).

2. Related Work

Coreference resolution.

Span-based mention ranking (Lee et al., 2017) with fine-tuned encoders remains strong and efficient; Maverick (Martinelli et al., 2024) reaches 83.6 CoNLL F1 on OntoNotes. Fine-tuned decoders reach high scores with constrained generation (Gan et al., 2025), and coreference resolvers have also used frozen contextual embeddings (Kantor & Globerson, 2019). We keep the language model frozen and compare which of its internal representations a lightweight resolver can use.

Attention and probing for coreference.

Attention-based pronoun resolution has been studied using individual heads and pooled multi-head features (Klein & Nabi, 2019; Tikhonov & Ryabinin, 2021; Yordanov et al., 2020), and BERT heads link mentions to their antecedents on OntoNotes (Clark et al., 2019). Attention weights from selected BERT layers have also served as features for an SVM resolving conversational references (Kumar & Callan, 2019). Earlier work probes referential information in autoregressive language models (Sorodoc et al., 2020), edge probing includes coreference between given spans (Tenney et al., 2019), and recent experiments include coreference in modern model families (Li & Subramani, 2025, v4). Analyses of decoder attention find heads that attend from pronouns to their antecedents (Vig & Belinkov, 2019), and mechanistic work traces circuits for plural reference on synthetic data (Dang et al., 2026). Coreference has also been probed in attention value projections of QLoRA adapters (Shore et al., 2026); we instead study native attention weights and pooled states of untuned backbones. We focus on the incremental benefit of pairwise attention over pooled span representations in frozen decoders, using document-level clustering and capacity controls.

Attention features as classifier inputs.

Pooled attention features have been used for discourse classification in frozen decoders (Eichin et al., 2025) and for relation classification in frozen BERT (Chizhikova et al., 2022), while attention-based readouts have been compared with hidden-state classifiers for hallucination detection (Chuang et al., 2024), and attention has been combined with hidden states for hallucination and answer-correctness detection (Liu et al., 2026; Volodichev et al., 2026). Attention received from later tokens has been used to classify hallucinated spans (Ogasa & Arase, 2025); we evaluate such features for detecting coreferential mentions, complementing mention detection from frozen language models (Morand et al., 2026; Popovic & Färber, 2024). Relation probes on decoder internals (Popovič & Färber, 2026) are closest to our relation-extraction extension, which evaluates given entity pairs, including a no-relation decision.

3. Method

Input.

A document is rendered as plain text (detokenized; speaker turns in OntoNotes are prefixed with the speaker name) and tokenized; a sink token (BOS, or EOS if the tokenizer has no BOS) is prepended. Mentions are word spans mapped to token spans. For each mention j we consider the $K=100$ preceding mentions as candidate antecedents, which covers 99.75% of nearest gold antecedents on OntoNotes dev. Documents longer than the context window are processed in half-overlapping windows; each mention and its pairs are read in the earliest window that contains the mention, and pairs whose antecedent lies partly outside that window receive a flag feature. For given mentions, all features of a window come from one forward pass with per-layer hooks that pool attention maps immediately, so no full attention tensor is stored.

Hidden-state features (H).

From the residual stream after eight evenly spaced layers we take, for every mention, the mean over its tokens and its last token. Features are standardized per layer and combined by a learned scalar mix.

Attention features (A).

Let ${\alpha }_{s,t}^{\ell ,h}$ be the attention of query token s to key token t in layer ℓ, head h (softmax in fp32, eager implementation). For a pair (antecedent i, anaphor j) with token sets $T_i,T_j$ we compute, for every layer and head,

$$\begin{array}{cc}\hfill a_{\ell h}^{\max }& =\underset{s\in T_j}{\max }\underset{t\in T_i}{\max }{\alpha }_{s,t}^{\ell ,h},\hfill \\ \hfill a_{\ell h}^{\mathrm{mean}}& =\frac{1}{|T_j\left|\right|T_i|}{\sum }_{s\in T_j}{\sum }_{t\in T_i}{\alpha }_{s,t}^{\ell ,h},\hfill \\ \hfill v_{\ell h}^{\max }& =\underset{s\in T_j}{\max }\underset{t\in T_i}{\max }{\alpha }_{s,t}^{\ell ,h}\parallel W_O^h{v}_t^h\parallel ,\hfill \end{array}$$

where $W_O^h{v}_t^h$ is the contribution of token t through head h to the residual stream (Kobayashi et al., 2020); for grouped-query attention, $v^h$ is the value of the key-value group of query head h. For spans longer than 64 tokens, T contains the first and last 32 tokens. This gives $3LH$ features per pair (e.g., 1,008 for Qwen2.5-0.5B), which are log-transformed and standardized. Because the anaphor follows its antecedent, these features are available in causal models without modification.

Probes.

We use mention ranking: for anaphor j, a score $s(i,j)$ is computed for each candidate i and for a dummy antecedent with score 0; training maximizes the marginal likelihood of gold antecedents. All probes include distance features (mention, token, and sentence distance buckets; nesting flags) through a linear term.

• H: an MLP over $[u_i,u_j,u_i\odot u_j]$ and the distance features, with $u=Pg$ a 256-dimensional projection of the mixed span representation g.
• A: the attention features are projected to 256 dimensions through a nonlinear layer before the scoring MLP.
• HA: one MLP over the hidden-state and projected attention features.

Linear variants (A_lin: a linear score on attention features; H_bil: a low-rank bilinear score; HA_lin: their sum) appear in the analyses. Clusters are decoded greedily by linking every mention to its best-scoring antecedent if that score exceeds the dummy.

Capacity controls.

We train (i) HA and A probes whose attention features are permuted across pairs within each minibatch, while hidden-state and distance features stay aligned, (ii) an H probe with rank 512 and hidden size 1024 instead of 256 and 512, and (iii) a layer-matched variant in which H and A read the same eight layers or all layers.

End-to-end: mention detection with received attention.

OntoNotes annotates only mentions that belong to non-singleton coreference chains. Whether a span belongs to such a chain can depend on text that follows it, which hidden states computed at the span itself cannot reflect in a causal model; the attention that later tokens pay to the span can. Received-attention features have previously been used for hallucinated-span detection (Ogasa & Arase, 2025). For every token s we pool, per layer and head, the attention it receives from tokens at least 16 positions later (maximum and mean) and from the 15 following tokens (maximum):

$$\begin{array}{cc}\hfill r_{\ell h}^{\mathrm{far},\max }\left(s\right)& =\underset{t\ge s+16}{\max }{\alpha }_{t,s}^{\ell ,h},\hfill \\ \hfill r_{\ell h}^{\mathrm{far},\mathrm{mean}}\left(s\right)& =\underset{t\ge s+16}{mean}{\alpha }_{t,s}^{\ell ,h},\hfill \\ \hfill r_{\ell h}^{\mathrm{near},\max }\left(s\right)& =\underset{s<t<s+16}{\max }{\alpha }_{t,s}^{\ell ,h}.\hfill \end{array}$$

A candidate span (up to 30 words within a sentence) is represented by these features of its first and last token and their mean over the span (log-transformed and standardized), by the same three statistics of hidden states from layers $L/2$, $3L/4$ and L, and by a width embedding; an MLP scorer is trained with binary cross-entropy on gold mentions. We compare detectors using hidden states only (H), received attention only (A), or both (HA), and choose the decision threshold on dev for a target recall. Because spurious mentions lower the precision of the whole resolver (Zhu et al., 2024), the coreference probe is trained and evaluated on predicted mentions: the detector logits of both mentions enter the pair features, and spurious mentions are trained to select the dummy antecedent. Detection and pair extraction are separate forward passes.

4. Experimental Setup

Data.

OntoNotes 5.0 / CoNLL-2012 English (Pradhan et al., 2012) (2,802/343/348 documents) is the main benchmark. For transfer we evaluate OntoNotes-trained probes on LitBank (Bamman et al., 2020) (split 0), PreCo (H. Chen et al., 2018) (official dev as test), English GUM from CorefUD 1.4 (Nedoluzhko et al., 2022; Novák et al., 2026), GAP (Webster et al., 2018) and WinoBias (Zhao et al., 2018). Probes are additionally trained in domain on LitBank, GUM and Polish PCC (CorefUD); since the public CorefUD release has no test sets, we use the official dev sets as test and hold out 10% of train for development.

Evaluation.

We report the CoNLL F1 (average of MUC, B³ and CEAF_e) computed with coval; we verified that gold clusters score 100, that our pipeline reproduces the official scorer on the original OntoNotes files, and that it reproduces Maverick’s published OntoNotes result. Singletons are excluded for OntoNotes and CorefUD and included for LitBank, PreCo, GAP and WinoBias. GAP is evaluated with its F1 over pronoun–name decisions and the female/male F1 ratio; WinoBias with the accuracy of linking the pronoun to the correct occupation on the pro- and anti-stereotypical test sets. Unless stated otherwise, results use gold mentions and are the mean over three probe seeds; learning rate and epoch are selected on dev (Appendix A); control, layer-control and component probes reuse the learning rate selected for the corresponding main probe. For OntoNotes we report paired bootstrap confidence intervals over documents (1,000 resamples).

Models.

The main gold-mention comparison covers 15 decoder models from the SmolLM2, Qwen2.5, Qwen3, Llama, Gemma-3, OLMo-2 and Pythia families (160M–7B parameters) and 3 encoders with the same probes; decoders read documents in windows of 4,096 tokens (2,048 for Pythia, its context length) and encoders in windows of 512 tokens; the models used for mention detection, end-to-end resolution and relation extraction are listed in the corresponding tables. Encoders and Pythia run in float32 and the other decoders in bfloat16; for Pythia-160M, bfloat16 with eager attention raises the mean negative log-likelihood of 3 OntoNotes dev documents from 3.59 to 4.46 nats per token. Each mention detector is trained once (seed 0); end-to-end linkers use three seeds.

Baselines.

(i) Maverick (Martinelli et al., 2024) with the official OntoNotes checkpoint, evaluated by us with gold and predicted mentions, and (ii) LoRA fine-tuning (Hu et al., 2022) ($r=16$ on the attention projections) of the same model with a jointly trained mention-ranking head of the H type; unlike the frozen probes, it reads the final layer only and truncates documents to 4,096 tokens instead of using windows.

Implementation.

All experiments run on NVIDIA RTX 3090 GPUs (24GB); coreference feature extraction and probes use PyTorch 2.6.0 and Transformers 5.19.0 with eager attention, as recorded for every probe run, and the Maverick baseline runs in its own environment. We validated the feature extractor against attention maps and value projections computed without hooks (maximum relative error $4.8·{10}^{-4}$ for the attention features).

5. Results

Combining attention with hidden states is best in every model.

Table 1 reports OntoNotes test results with gold mentions. HA is the best configuration for all 15 decoders and all 3 encoders; for decoders it improves over H by +1.5 to +2.5 CoNLL F1, and the paired bootstrap confidence intervals of HA−H are positive for all 18 models (lowest lower bound 0.8). A alone is weaker than H in domain. On Qwen2.5-0.5B and 1.5B, frozen HA probes score 87.8 and 88.8 F1, compared with 88.9 and 90.7 for our LoRA baselines, respectively. These are gold-mention scores; Maverick reaches 94.2 in the same setting.

Pair-specific attention accounts for most of the gain.

Most of the improvement is lost when attention features are shuffled across pairs (Table 2): shuffled HA retains a median of 3% of the HA−H gain, with the largest share for OLMo-2-1B (36%). Shuffled A falls to 28.3–36.4 F1, below a probe using distance features only (44.1 for Qwen2.5-0.5B), and increasing the dimensions of the hidden-state probe does not match HA.

The gain holds with matched layer access.

The main H probe reads eight layers, while A uses all layers. We therefore re-extract OntoNotes with hidden states from all layers and give H and A access to either the same eight layers or all layers (Table 3). Reading all layers changes H by -0.1 to 0.7 F1, while HA−H remains 0.2 to 2.2 with eight layers and 0.8 to 2.6 with all layers across 18 models.

Attention-only probes transfer well without target-domain calibration.

Table 4 summarizes the transfer of OntoNotes-trained probes (full results in Appendix B). With the threshold of standard decoding, the A probe outperforms H on GAP for 18/18 models and on WinoBias for 17/18, and it also outperforms HA on GAP (18/18) and WinoBias (16/18). When the GAP decision threshold is calibrated on GAP dev, A remains better than H (16/16 models with calibration) but better than HA only for 7/16. The uncalibrated scores thus suggest greater sensitivity to domain shift when hidden-state features are included, part of which is a shift of the score scale. On the full-document transfer benchmarks LitBank, PreCo and GUM, HA has the highest mean score across models; OLMo-2-1B is the only model for which A is worse than H on WinoBias.

Larger gains on literary and multi-genre data.

Trained in domain on the smaller LitBank, GUM and Polish PCC datasets (Table 5), HA−H ranges from 1.9 to 4.1 F1 on LitBank, 2.2 to 7.1 on GUM and -0.2 to 4.1 on PCC; the gains on LitBank and GUM exceed those on OntoNotes in all available model–dataset comparisons.

End-to-end resolution with received attention.

At the threshold selected on dev for a 90% recall target, the Qwen2.5-0.5B detector reaches a test precision of 48.1 with hidden states and 63.5 with hidden states and received attention, at test recalls of 90.4 and 90.8 (Table 6); received attention alone (57.5) is also more precise than hidden states, with the same pattern for the other models. These results show that received-attention features help identify membership in non-singleton coreference chains. With mentions from the HA detector, end-to-end CoNLL F1 ranges from 69.9 for Qwen2.5-0.5B to 72.5 for Llama-3.2-3B (Table 7), and combining attention with hidden states again gives the best linker for all 4 models; fine-tuned Maverick reaches 83.6. Under identical settings for Qwen2.5-0.5B (recall target 93%, 100 candidate antecedents), we separate the two uses of attention: with the hidden-state detector, the H and HA linkers reach 64.4 and 65.3; with the received-attention detector, 68.7 and 69.9. Most of the end-to-end gain thus comes from mention detection, and the linker adds a smaller, consistent improvement.

6. Reading Coreference from Heads and Layers

A few heads recover most of the attention signal.

We rank heads by the group norm of their weights in the linear attention probe (trained on train, selected on dev) and retrain probes on the top-k heads only (Figure 1). A handful of heads recovers most of the performance of all heads, and performance grows roughly log-linearly in k, in line with earlier findings that a small set of heads carries pronoun-resolution signal (Tikhonov & Ryabinin, 2021). The ranking locates signal accessible to the probe; the ablation below tests its contribution to hidden-state readout and pronoun prediction.

Ablating the top-ranked heads.

We mean-ablate heads (Wang et al., 2023): the input of the output projection for a head is replaced by its mean over 50 OntoNotes training documents. We compare the 16 heads ranked highest by the linear attention probe with 16 random heads sampled outside the top 64 for 3 models (Table 8). Probes trained on the intact model lose 2.5 to 13.7 F1 with hidden states when the top heads are ablated, and at most 0.4 with random heads. A frozen probe may fail only because its input distribution shifts, so we also retrain the H and HA probes on the ablated models. After mean-ablating the selected heads, retrained H probes remain 1.8 to 6.4 F1 below their intact-model baselines, while the sampled control heads change them by at most 0.3 F1. This shows that the intervention reduces coreference accessibility under the tested readout. The ablation also raises the model’s loss on anaphoric pronouns, but its specificity differs across models: for Qwen2.5-1.5B the increase is concentrated on pronouns (+0.94 nats, against +0.11 on other tokens), for SmolLM2-360M it is partly specific (+0.45 against +0.31), and for Qwen3-1.7B the same heads matter for language modeling in general (+2.00 against +2.30). The attention-only probe falls further partly by construction, because the features of the ablated heads are set to their mean. The retraining control uses one random draw per model, and three models do not show how the effect varies with family and size.

Intermediate layers.

Probes restricted to a single layer (Figure 2) give the strongest readouts in intermediate layers for both attention and hidden states in the tested configurations; the final layers yield weaker coreference readouts under these probes.

Feature components.

The effect of value-weighted attention is model dependent: adding it to the max- and mean-pooled weights lowers the mean score of the A probe for 13 of 18 models and improves it for Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-3B, Gemma-3-1B and RoBERTa-large. We keep the predefined three-statistic feature set for all models rather than selecting it after evaluation; the component probes reuse the learning rate selected for the full A probe.

7. Extension: Relation Extraction with Given Entities

For an entity pair $(h,t)$ with given mentions, we max-pool the attention features of all their mention pairs separately for the two directions (a mention of h after vs. before a mention of t), add presence flags, and represent each entity by the mean hidden state of its mentions. Probes are trained with adaptive thresholding (Zhou et al., 2021), which includes a no-relation decision, and evaluated with micro F1 on CoNLL04 (Roth & Yih, 2004) and Re-DocRED (Tan et al., 2022). As in earlier attention readouts with a query placed after the context (S. Chen et al., 2025), we also append the query “Relation between [h] and [t]:” after the document, reusing its key-value cache, and read at the last query token the per-head attention mass on the tokens of h and t, its value-weighted variant and the hidden state (S features; HAS combines them with HA).

Combining the representations again gives the best scores (Table 9): HA improves over H by +3.5 to +6.1 F1 on CoNLL04 and by +1.2 to +2.9 on Re-DocRED. On Re-DocRED, hidden states are the stronger single source, while combining them with attention yields the highest scores; the suffix query improves over HA in all 7 available comparisons.

8. Conclusions

In frozen language models with up to 7B parameters, pairwise attention features consistently improve a pooled-span coreference readout, and the improvement depends on the pair-specific alignment of attention. In the three tested models, heads selected by the attention probe contribute to representations useful for coreference readout, with much smaller effects for the sampled control heads. Attention-only probes transfer well without target-domain calibration, and attention received from later tokens gives a mention detector access to right-context evidence about membership in coreference chains.

9. Limitations

Our probes measure coreference accessibility under the tested readouts. The ablations establish effects on these readouts in three models, using one random control set per model for retraining. They do not identify a complete coreference mechanism or characterize all information in the residual stream. The gold-mention comparison covers decoders from 160M to 7B parameters; we do not test larger models. End-to-end results use one detector per model and remain well below fine-tuned systems; the detector results do not establish which later tokens supply the relevant signal. Polish results exclude zero mentions (dropped subjects) and use exact span matching, so they are not comparable to CRAC shared-task scores; CorefUD test sets are not public, so we report results on the official dev sets. All models see the same prompt-free rendering of the input, and speaker formatting may affect conversational genres. OntoNotes is distributed by the LDC under a license; we use the CoNLL-2012 release for research only.

Checkpoints.

The main comparison uses the Hugging Face checkpoints HuggingFaceTB/SmolLM2-360M, Qwen/Qwen2.5-0.5B, Qwen/Qwen2.5-1.5B, Qwen/Qwen2.5-3B, Qwen/Qwen2.5-7B, Qwen/Qwen3-0.6B-Base, Qwen/Qwen3-1.7B-Base, meta-llama/Llama-3.2-1B, meta-llama/Llama-3.2-3B, google/gemma-3-1b-pt, allenai/OLMo-2-0425-1B, EleutherAI/pythia-160m, EleutherAI/pythia-410m, EleutherAI/pythia-1b, EleutherAI/pythia-2.8b, FacebookAI/roberta-large, microsoft/deberta-v3-large, answerdotai/ModernBERT-large (main branch, downloaded in October 2026; revisions were not pinned).

Hidden states are standardized per layer and dimension with statistics from a training sample before the scalar mix; the mixed vector is not normalized again.