Explainable NLP in the Era of Large Language Models: A Unified Taxonomy, Evaluation Frameworks, and Decision Guidance

1. Introduction

Large language models (LLMs) now draft clinical documentation, review legal and financial documents, and moderate content at platform scale, where an incorrect or biased output can affect a diagnosis, a credit decision, or a person’s ability to take part in online discourse. Regulators have begun to respond. The EU AI Act (Regulation (EU) 2024/1689) is now in force and is phasing in its obligations, several of which bear directly on transparency and human oversight [80]; Section 7 sets out the staging and the amendments that have since revised it.

People use explanations to make decisions with a model’s help [138] and to calibrate how far they trust it [120], and developers use them to find bugs and data artifacts [102]. For an LLM, an explanation can take two forms: an attribution of the output to parts of the input, or the reasoning the model writes before it answers.

1.1. Two Literatures

Explanation methods for NLP grew up in two literatures with their own vocabularies and, to a large extent, their own venues. Saphra and Wiegreffe [220] trace how a separate mechanistic-interpretability community formed alongside the established NLP one; we take the divide as given and ask what translates across it. The first literature explains task models such as fine-tuned classifiers with feature attribution [214,234], probing classifiers [32,112], and extractive rationales [143]. Its evaluation methodology assumes that an explanation is a subset of the input: ERASER scores a rationale by erasing it and measuring how the prediction changes [68]. The second literature studies LLMs, from the faithfulness of chain-of-thought reasoning [142,243] to sparse autoencoder features and circuit tracing [8,45], and much of it appears first in industry reports that are not peer-reviewed. Most surveys sit on one side of the divide. Those that cross it do so as a technical primer [85], by training paradigm [279], or by stakeholder [48], and none works out which faithfulness tests carry across (Section 1.5). A reader who wants to compare a SHAP attribution with a chain of thought, or to know whether ERASER-style tests apply to the latter, therefore has to translate between them. This survey asks four questions. Can one taxonomy place both literatures? Which faithfulness tests carry over to an explanation the model writes itself? Given the access a deployment allows, which methods remain usable? How much of the LLM-era toolbox is new?

1.2. Scope and Contributions

We restrict the survey to NLP, where inputs are sequences of discrete tokens whose meaning depends on context, and we cover methods from other modalities only where they were later imported into NLP (Section 1.3). We make three contributions:

1.

A unified taxonomy: We organize explanation methods for NLP along four dimensions: scope (local vs. global), mechanism (how the explanation is computed), model access (model-agnostic vs. model-specific), and output form (importance scores, rules, examples, counterfactuals, rationale spans, concepts, natural language). The taxonomy places classical attribution methods such as LIME [214] and SHAP [153], attention and representation analysis, and current LLM-oriented techniques in a single framework (Section 3).

2.

Practical decision frameworks: We turn the taxonomy into a decision tree for method selection that starts from model access and then considers scope and audience, with an access table for LLMs and reasoning models, task-specific recommendations for classification, sequence labeling, question answering, and generation, guidance on evaluation protocols and tooling, and three worked examples that trace the framework end to end on concrete deployments (Section 6).

3.

LLM-era interpretability: We set twelve LLM-era techniques against their closest classical counterparts in a lineage table that records what changed between the two and whether each technique rebuilds an older idea, extends it, or has no counterpart (Table 6, Section 4.7). Around the table we synthesize the evidence on chain-of-thought (CoT) faithfulness in reasoning models, self-explanation and introspection, and mechanistic interpretability from sparse autoencoders to circuit tracing (Section 4).

Main findings:

1.

Faithfulness and plausibility come apart, and the erasure tests do not transfer to a chain of thought. Comprehensiveness and sufficiency [68] erase part of the input, and they carry over only where a self-explanation cites input words that can be redacted [162]. A chain of thought is an output, so the intervention moves to the trace itself, truncating it or inserting a mistake into it [142], and asks whether the stated reasoning drives the answer rather than which input tokens do. Chains of thought can rationalize an answer driven by a cue they never mention [54,243], and whether such omissions count as unfaithfulness or as incompleteness is itself disputed [276]. Zhao et al. [279] note that erasure metrics cannot be applied directly to natural language explanations; we separate the cases where erasure still applies from those where it does not, and build both into a minimum evaluation protocol (Section 6.3).

2.

Access settles which mechanisms are usable before scope or audience matter. With open weights every mechanism is available. An API that returns log-probabilities or embeddings supports perturbation and surrogate attribution over those outputs, and a text-only API leaves perturbations that read only the predicted label, such as Anchors [215], and prompting. Reasoning traces narrow the options further: the OpenAI and Anthropic APIs return at most a summary of the trace (Section 2), while an open-weight model makes the trace available for intervention tests. The decision framework therefore starts from access (Figure 2 and Table 10), and the three worked deployments show where access is the binding constraint and where, at platform scale, throughput is (Section 6).

3.

Most LLM-era techniques rebuild or extend older ideas. Of the twelve techniques in the lineage table (Table 6), ten have a classical counterpart: five rebuild an older idea and five extend one. Activation patching applies the intervention of causal mediation analysis [172,252], a lineage that Mueller et al. [189] trace in detail; Faithful CoT [156] repeats the faithful-by-construction design of select-then-predict rationales [143]; and the dispute over chain-of-thought as explanation [24,243,276] has the structure of the attention debate [36,121,261]. Two problems have no classical counterpart: whether a reasoning trace stays monitorable under optimization pressure [21,134], and whether a model has introspective access to its own states [39,149]. Both problems rest partly on industry reports that were not peer-reviewed [21,149], and peer-reviewed work has since qualified each: training only against a monitor of the output can still obfuscate the trace [70], and evidence that models detect tampering with their internal states may reflect generic anomaly detection [229].

4.

Faithfulness metrics disagree, and dictionary features do not yet beat simple baselines. Faithfulness metrics rank methods inconsistently [51], and tests advertised as measuring chain-of-thought faithfulness may measure only output-level self-consistency [202]. On standardized benchmarks (Section 4.4), sparse-autoencoder methods are matched or beaten by linear probes for concept detection [123] and by prompting or finetuning for steering [267], even when features are selected by their effect on the output [10], and sparse autoencoders trained on randomly initialized transformers can earn automated interpretability scores similar to those from trained transformers [109]. Both halves warn against the same shortcut, treating one proxy (a single faithfulness metric, or a feature that looks interpretable) as evidence for the property it stands in for. An evaluation should therefore name the tests it ran and combine at least two that fail in different ways (Section 5).

5.

Explanations raise acceptance of model outputs more reliably than they raise decision quality. In a controlled comparison across several tasks, explanations made people more likely to accept the model’s recommendation, correct or not, without improving team accuracy over showing model confidence alone [22]. The pattern holds for LLM explanations, which raise reliance on incorrect answers as well as correct ones [129] and, when longer, raise users’ confidence without making the answers more accurate [233]. The target is warranted trust, which lets users anticipate when the model is right and when it is wrong [120], so extrinsic evaluation should measure the decisions people make with an explanation (Section 5).

Section 8.1 turns these findings into six open problems.

1.3. Survey Scope and Methodology

This survey is a structured narrative review. Because narrative reviews involve judgment about what to include, we describe our selection process here so that readers can assess its coverage and its limitations.

Corpus construction: We seeded the corpus with prior surveys of XAI and explainable NLP [25,64,101,157,161,279] together with foundational method and evaluation papers [68,118,121,153,214,234]. We then expanded the seed set with searches that crossed three keyword families: explainability terms (explainab*, interpretab*, faithful*), NLP terms (NLP, language model, LLM), and the names of specific methods and constructs (for example attention, saliency, SHAP, probing, chain-of-thought, sparse autoencoder). Searches were run over the ACL Anthology, Semantic Scholar, and the arXiv cs.CL and cs.LG listings. Finally, we performed bidirectional citation snowballing, following both the references of included papers and the later papers citing them, until additional rounds stopped surfacing work that met the inclusion criteria below. The last search was run in September 2026.

Time window: We cover foundational work from roughly 2015 onward, the period in which the shift to deep neural architectures made opacity an acute problem for NLP, and we cite earlier classics where they are needed for context. Coverage of LLM-era interpretability extends through September 2026. The reference list holds 282 works: 182 papers in conference or workshop proceedings, 36 journal articles, one book, and 63 preprints, technical reports, legal texts, software releases and documentation pages; 82 of the 282 date from 2025 or 2026.

Inclusion criteria: A work was included if it fell into at least one of four categories: (i) explanation methods applicable to text and NLP models; (ii) evaluation of explanation quality, including faithfulness, plausibility, and human utility; (iii) applications of NLP with explicit explainability requirements; or (iv) LLM interpretability, including widely cited industry technical reports. We admit this last form of gray literature deliberately: a substantial share of current mechanistic interpretability research first appears, and sometimes only appears, in industry reports, and excluding it would misrepresent the state of the field. Wherever we rely on such reports, we flag that they are not peer-reviewed.

Exclusion criteria: We exclude vision-only XAI work, except for foundational methods that were later imported into NLP, such as layer-wise relevance propagation [20]. We also exclude fairness and bias research that lacks an explicit explanation component, since fairness is a large field in its own right with dedicated surveys.

Coverage claim: We prioritize influential and representative work and do not claim the exhaustive coverage of a PRISMA-style systematic review. Where the literature is too large to cover fully, for instance on probing or attention analysis, we select representative works and point to specialized surveys. How generative AI tools were used in this work, including for literature retrieval, is stated at the end of the article.

1.4. Paper Organization

Section 2, Section 3, Section 4, Section 5 and Section 6 give background, the taxonomy, the LLM-era synthesis with the lineage table, evaluation, and the decision frameworks. Section 7 covers the application domains (healthcare, legal and financial services, social science research, content moderation, and education), Section 8 the six open problems, Section 9 the limitations of this survey, and Section 10 concludes.

1.5. Related Surveys

Several surveys address explainability in machine learning broadly and in NLP specifically. Table 1 compares the most closely related ones with the present survey along the dimensions that structure this paper. Several earlier surveys already classify methods along named dimensions or give evaluation a dedicated treatment, and two mechanistic reviews give explicit procedures for choosing a method. What sets this survey apart is a selection procedure that spans both literatures and is keyed to the access a deployment allows, together with an analysis of which faithfulness tests carry over to explanations the model writes itself.

General XAI surveys: The general surveys in Table 1 classify methods by the problem they solve [101], by complexity, scope and dependence on the model [3], or within a taxonomy oriented toward responsible AI [25]. They give little attention to the discrete, sequential nature of text or to methods specific to language models.

NLP-specific surveys: Early surveys covered explainability and post-hoc methods for neural NLP [64,161]. Lyu et al. [157] then organize the field around faithfulness, asking whether explanations reflect the model’s actual reasoning, and offer the most thorough treatment of faithfulness evaluation among existing surveys. Their property table already scores five method families on time, model accessibility, scope, unit of explanation, and form of explanation, and Madsen et al. [161] cross the form an explanation takes with the information a method needs, so scope, model access, and output form are not new as dimensions. What our taxonomy adds is the extension of these axes to LLM-era methods, with prompting as a mechanism and sparse-autoencoder and circuit methods placed alongside classical attribution (Section 3).

LLM-era surveys: These fall into two groups. Breadth surveys of LLM explainability classify techniques by training paradigm and scope [279], or by local and global analysis together with uses such as model editing [155], and a position paper argues that LLMs shift interpretability from individual predictions toward dataset-level and capability-level understanding [228]. Mechanistic reviews organize inner-interpretability methods by network component [212], by causal nature, learning phase, locality and comprehensiveness from an AI-safety perspective [34], by research task [209], or by the causal mediator a method intervenes on and the search over mediators [189]; Ferrando et al. [85] give a technical primer on decoder-only transformers that runs from input attribution and probing to sparse autoencoders, and Shu et al. [225] cover sparse autoencoders alone. Two of the mechanistic reviews already tell a researcher which method to use. Rai et al. [209] give step-by-step workflows that map a type of study to techniques, and Mueller et al. [189] say which mediators and search methods suit which research goal. Both procedures start from a research question and assume access to the model’s internals; ours starts from the access a deployment allows and also covers perturbation, surrogate and prompting methods. On evaluation, Zhao et al. [279] observe that erasure metrics such as comprehensiveness and sufficiency cannot be applied directly to natural language explanations, the point our first finding develops. Calderon and Reichart [48] come closest to our framing. They classify interpretability paradigms by properties that overlap with our dimensions, among them scope, model access, and how insights are presented, and they use an LLM to annotate thousands of retrieved papers, finding that natural language explanations have spread quickly to fields outside NLP while explanations of internal model components are rarely used there. Their organizing question is which stakeholders each paradigm serves; ours is which faithfulness tests apply to a given explanation and which methods remain usable under the access a deployment allows.

Relation to our prior work: This survey is a companion to our earlier survey of explainable NLP across application domains [178] (preprint arXiv:2502.00837, currently under journal review). That work is organized by application domain and the requirements each imposes. The present survey is organized by method, and it treats evaluation and LLM-era interpretability in greater technical depth, with coverage through September 2026. The two also differ in method: the earlier survey follows a systematic bibliometric review pipeline, whereas this one is a structured narrative synthesis, and the two share no text, tables, or figures. Where the domain discussion in Section 7 overlaps with the earlier survey, we summarize and refer readers there.

2. Background

This section fixes terms and the two distinctions from general XAI, scope and model access, that the taxonomy in Section 3 extends, and then traces what an explanation attaches to as NLP models changed.

2.1. Terms and Classic Distinctions

An explanation is faithful to the extent that it accurately represents the reasoning process behind a model’s prediction, and plausible to the extent that people find it convincing [118]. The two can come apart: a model trained on human-written explanations can produce plausible ones with no guarantee that they are faithful [118], and much of this survey concerns that gap. The term interpretability has no settled meaning: the literature offers diverse motivations for it and many notions of what makes a model interpretable [152]. We use it interchangeably with explainability.

Explanations also differ in when they are produced relative to training [25]. A directly interpretable model, such as a linear model or a small decision tree, needs no separate explanation: the computation is the explanation, so it is faithful by construction. Post-hoc methods are applied after training and can treat the model as a black box, for instance by attributing a prediction to input features [153] or by fitting a simpler surrogate model [26,236]. Rudin [219] argues that high-stakes decisions should use an interpretable model in place of a post-hoc explanation of a black box, since on structured data carefully designed interpretable models often rival black-box accuracy. NLP has a counterpart: a select-then-predict model whose classifier sees only the rationale its extractor selected came close to a full-text model on several classification tasks while reading 10–30% of the input [122]. Our taxonomy has no separate timing dimension; direct interpretability appears there as the intrinsic value of the mechanism dimension (Table 2).

Explanation methods are commonly classified by scope and by model access [3]. A local explanation accounts for one prediction (“Why did the model make this specific prediction?”) and a global explanation for the model’s behavior across inputs (“How does the model generally work?”); local explanations can be aggregated into a global picture [154]. Model-agnostic methods such as LIME [214] and KernelSHAP [153] need only query access to the model, while model-specific methods use its internals, as DeepLIFT [224] and Layer-wise Relevance Propagation (LRP) [20] do for neural networks and attention visualization does for transformers. In deployment, access comes in three levels: open weights expose gradients and activations, some APIs return log-probabilities or embeddings along with the generated text, and a text-only API returns the output alone (Section 6). Section 3 keeps scope and model access and adds two dimensions, the mechanism by which an explanation is computed and the output form it takes.

2.2. From Linear Classifiers to Reasoning Models

Each shift in NLP architecture changed what an explanation most naturally attaches to. Linear classifiers over engineered features [114,278] could be read through their feature weights. Distributed representations, from word embeddings [174] and recurrent networks [75] to pre-trained transformers [67,250], took that reading away, and explanation moved in two directions: outward to the input tokens, which attribution scores weight and rationales select, and inward to the hidden states, which probing classifiers and later dictionary features decode. Attention weights seemed to offer a shortcut between the two, but reading them as importance scores remains contested (Section 4.6). LLMs, which learn new tasks from examples in the prompt [46], added a fourth unit: text the model writes about its own answer. None of these units displaced the earlier ones, and the output forms of Section 3 cover all four.

Reasoning models such as o1 [198] and DeepSeek-R1 [65], trained with large-scale reinforcement learning to write long chains of thought before answering, make the trace an obvious interface for interpretation, but whether anyone outside the developer can read it depends on the deployment. Open-weight reasoning models such as DeepSeek-R1 and gpt-oss return the full trace [199]. The OpenAI, Anthropic and Google APIs return at most a summary of the reasoning, in one case written by a separate model, and unless asked for the summary some return only the answer [9,98,200]. Whether a trace reflects the computation that produced the answer is the question Section 4 takes up, and the tests that intervene on the trace (Section 5.2) need the raw trace.

3. A Unified Taxonomy of Explainable NLP

This section presents a unified taxonomy of explanation methods for NLP, organized along four dimensions: scope, mechanism, model access, and output form. Figure 1 places the methods on the first two.

3.1. Taxonomy Framework

We classify explanation methods along four dimensions. The first dimension is scope, distinguishing between local explanations for single predictions and global explanations that characterize model-wide behavior. The second dimension is mechanism, which describes how the explanation is computed and takes seven values. Perturbation methods query the model with modified inputs (through occlusion, erasure, sampling-based feature removal, or counterfactual search) and read importance off the resulting output changes. Gradient/propagation methods backpropagate signals from the output through the model’s weights, as in vanilla gradients, Integrated Gradients, or Layer-wise Relevance Propagation. Surrogate methods fit an interpretable proxy model, either locally around one prediction (as in LIME) or globally via distillation. Internal inspection methods read internal states such as attention weights, hidden representations, probe outputs, or learned dictionary features without modifying the model. Intervention methods ablate or patch model components and measure the behavioral consequences, as in head pruning, layer ablation, and activation patching. Prompting-based methods elicit an explanation from the model itself in natural language, as in chain-of-thought prompting and self-explanation. Finally, intrinsic methods build the explanation into the model itself, so that the forward computation produces it, as in select-then-predict rationale architectures, concept bottleneck models, and weight-sparse transformers. The third dimension is model access, separating model-agnostic methods that work with any model from model-specific methods designed for particular architectures. The fourth dimension is output form, categorizing methods by whether they produce importance scores, rules, examples, counterfactuals, rationale spans, concepts, or natural language. Rationale spans differ from importance scores in being a hard selection: a subset of the input is designated as the evidence and the rest is discarded, where an importance score weights every token.

The dimensions are not independent. Gradient/propagation, internal inspection and intervention need weights or activations and are model-specific, as are most intrinsic methods, which need a model trained for the purpose (Faithful CoT, built by prompting, is the exception). Perturbation, surrogate and prompting need only queries, and SHAP is agnostic or specific depending on the estimator [153]. Access is thus largely implied by mechanism, which is why the decision frameworks of Section 6 ask about it first. Query access comes in grades: an API that returns scores or token log-probabilities lets a perturbation method measure a change in probability, and one that returns labels only, a change in label.

Table 2 classifies prominent explanation methods by scope and mechanism. Most methods occupy a single cell, although some span categories: SHAP values can be aggregated from local to global scope, and activation patching supports both single-prediction and model-wide analyses.

3.2. Local Explanation Methods

Attribution by perturbation, gradient/propagation and surrogate. These mechanisms all score input tokens and differ mainly in how they realize a token’s absence. Occlusion deletes or masks the token [145]; LIME fits a linear surrogate to predictions on copies of the input with words masked [214]; SHAP averages a token’s effect over coalitions of the other tokens, and its common KernelSHAP estimator is itself a weighted linear surrogate fitted to such samples [153]. Integrated Gradients [234] and DeepLIFT [224] measure change against a baseline input or reference activations. Integrated Gradients accumulates gradients along the path from the baseline, so a token keeps its credit where the output has flattened, and its attributions sum to the output difference between input and baseline. Vanilla gradients [227] and SmoothGrad [230] use no reference and measure local sensitivity, and LRP redistributes the output backward under a conservation rule [16,20]. Methods that define absence differently need not agree. On an LSTM news classifier six attribution methods disagreed on every metric tested, and data scientists often resolved such disagreements with ad hoc heuristics [136]. For moderately rich model classes, neural networks among them, complete and linear methods such as Integrated Gradients and SHAP can provably fail to beat random guessing at inferring model behavior [38]. Deletion and masking also create inputs unlike the training data (Section 5).

Attribution for generation. For a generative model the target comes first: one next-token probability, the difference between a token and a named alternative, or a whole generated statement. Contrastive targets verify grammatical phenomena better than plain attribution and help people simulate the model’s contrastive decisions [273]. Classical mechanisms have been rebuilt for this setting. AttnLRP extends relevance propagation to attention layers and attributes LLaMa 2 and Mixtral 8x7b outputs at about the cost of one backward pass [2]. ContextCite fits a sparse linear surrogate to context ablations and so attributes a generated statement to the sources the model relied on [57], which matters because a cited source can support a statement without having been used [256].

Perturbation with discrete outputs. An anchor is a set of words that keeps the prediction fixed with high precision under the method’s perturbation distribution, chosen for coverage, the probability that it applies to a sample from that distribution; in a user study, anchors let people predict the model on unseen instances with higher precision than linear explanations did [215]. A counterfactual is a minimal edit that changes the prediction [270], the contrastive form human explanations take [176]. MiCE searches for minimal, fluent edits guided by the model’s gradients and is model-specific [218], while Polyjuice needs only queries to the explained model [263]. Generation is now often delegated to an LLM prompted to change one concept while holding the others fixed [92], the device Matton et al. [168] use to test self-explanations (Section 5.2). An LLM asked for a counterfactual of its own prediction often writes one on which its prediction does not change as claimed [66]. Human-authored counterfactuals also serve as training data that reduces reliance on spurious cues [127].

Attribution to training data. Influence functions estimate the effect of removing one training example without retraining. The derivation assumes a convex loss, though Koh and Liang [130] found the approximation informative where that theory breaks down. Scaling to models of up to 52 billion parameters with an EK-FAC approximation, Grosse et al. [99] note that for neural networks the estimates match poorly the counterfactual that motivated them, and treat them as approximating a more local quantity, the proximal Bregman response function. Representer points [272] are a cheaper alternative.

Internal inspection and intervention. Whether raw attention weights track importance is disputed [121,261]; context-mixing measures such as ALTI [84], which account for value vectors and the residual stream, are the better default (Section 4.6), and value zeroing makes the question an intervention by ablating a token’s value vector [185]. Other local readings decode each layer’s hidden state into the vocabulary [33], treat feed-forward layers as key-value memories [94], or track hidden units and gates in recurrent networks [124]. Prompting-based explanations are local by nature and are treated in Section 4.1 and Section 4.2.

Intrinsic. Select-then-predict architectures choose a short subset of the input and classify from it alone, so the selected tokens are the only evidence the classifier sees [143]. Differentiable relaxations later replaced the reinforcement-learning selector and allow control over rationale length [27], and a fully discrete two-stage pipeline retains competitive accuracy [122]. Concept bottleneck models predict human-specified concepts as an intermediate layer and the label from those alone, so a concept can be intervened on at test time [131]. In both designs the guarantee covers only the second stage. The selector or concept predictor is itself unexplained, and a selector can encode the predicted label in which tokens it selects, a Trojan explanation that the usual definition of faithfulness does not rule out [119], so its rationales still need the checks of Section 5. Weight-sparse transformers instead train most weights to zero so that the circuit for a task can be pruned out and read, trading capability for interpretability [90].

3.3. Global Explanation Methods

Several global methods aggregate local ones. Local attributions can be combined into a picture of global structure [154], but the aggregate inherits the disagreement among the methods it is built from, and corpus-level attention statistics describe what heads attend to [56] without measuring importance.

Surrogate. Model distillation trains an interpretable student, such as a decision tree or rule set, to mimic the black-box teacher [26,236]. It explains the teacher only as far as the two agree, so that fidelity has to be measured [141].

Concepts. Concept methods differ in who names the concept and whether its use is tested. TCAV defines a concept by examples a researcher chooses and measures a prediction’s directional sensitivity to it [128]. A probe defines it by an annotated label set and tests only whether it can be decoded [32,112]. A concept bottleneck trains it as a supervised layer, so intervening on it changes the prediction [131]. Sparse autoencoders learn a dictionary without labels, which removes the researcher’s choice but leaves naming to post hoc description (Section 4.4). For detection, supervised baselines still match or beat SAE features [123,267].

Probing. Probe accuracy conflates what a representation encodes with what the probe can learn, which control tasks calibrate against a random-label baseline [111]. Results are most informative as a profile across depth, as when the stages of the classical NLP pipeline emerge in rough order across the layers of BERT [217,238]. Amnesic probing adds a test of use: it removes a property from the representation and measures the change in predictions, and finds that probe accuracy does not track how much the task relies on the property [72]. Belinkov [31] reviews these critiques.

Intervention. These methods remove or replace a component and observe the effect, from head pruning [173,253] and causal mediation analysis [252] to activation patching [172,277] and circuit discovery [60,257] (Section 4.3). Their evidence is causal for the component intervened on. Attribution graphs are a partial exception: they are computed on a replacement model built from cross-layer transcoders, whose mechanisms need not match the original’s [8] (Section 4.4).

One gap recurs across these families: finding information in a component does not show that the model uses it there. Probes decode properties, such as phrase boundaries in BERT’s last layer, that its word predictions do not use from that layer [72]; where causal tracing localizes a fact gives no insight into which layer is best to edit [108]; the knowledge-neuron account of factual recall [63] is at best an oversimplification [194]; and very different attention distributions can yield the same prediction [121]. Intervention closes the gap only for the component it acts on.

3.4. What Each Mechanism Can Establish

Table 3 compares the seven mechanisms on access, cost, the claim each supports, and a documented failure. Perturbation, gradient/propagation and surrogate methods establish how the output responds to the input. Internal inspection establishes what a representation contains and intervention that a component matters; only these two support claims about the model’s internal states, and both need activations. A prompted explanation is a self-report whose relation to the computation must be tested separately (Section 5.2), and intrinsic methods pay a training run for an explanation that is part of the computation. Because the failures differ from row to row, tests drawn from two rows can check each other, the reasoning behind the minimum of two tests that fail in different ways (Section 6.3).

3.5. Summary of Methods

Table 4 places key methods, classical and LLM-era, along all four dimensions, with a practical note on each.

Output form decides which faithfulness test applies at all (Table 5). Erasure presupposes an explanation that picks out part of the input, as importance scores, rationale spans and anchors do; examples, concepts and natural-language text each need a test of their own, which is why the erasure tests do not transfer to a chain of thought (Section 5.2).

4. LLM-Era Interpretability

Large language models present interpretability challenges and opportunities that differ from earlier neural paradigms. Their scale and, for many, their API-only availability close off methods that presuppose gradients or activations, while their fluency opens a route no earlier model offered: the model can be asked to explain itself in language.

This section follows that split. Section 4.1 and Section 4.2 cover what the model says about its own reasoning, through chain-of-thought and self-explanation, and what is known about whether those accounts are faithful. Section 4.3 and Section 4.4 cover what can be read off the weights and activations, from circuits and superposition through sparse dictionary learning and circuit tracing, and Section 4.5 checks the first kind of evidence against the second. Section 4.6 asks whether the attention-explanation debate, conducted on encoders, carries over to LLMs, and Section 4.7 sets each technique against its classical counterpart. In the taxonomy of Section 3, the first group is prompting-based. The second spans internal inspection, intervention, surrogate replacement models and intrinsically sparse architectures, all of which need weights or activations, so the access a deployment allows usually decides which group is available at all.

4.1. Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting has a model write intermediate steps before its answer [132,259]. The trace is at once an explanation and part of the computation, so its faithfulness is the question of whether the answer depends on the trace in the way the text says. Reasoning models trained with reinforcement learning to write long traces [65,198] made the trace the default interface, though not always a visible one. Whoever runs an open-weight model such as DeepSeek-R1 sees every token of its trace. OpenAI’s o1 instead shows ChatGPT users a summary written by a separate summarizer model, and the external evaluator Apollo Research worked without its hidden trace [198]. A summary is itself a generated account of the trace, and the tests that truncate or edit a trace (Section 5.2) need one that can be edited and fed back, so behind such an interface trace-level faithfulness tests are out of reach. Access thus decides which faithfulness tests are possible, as it decides which explanation methods are (Section 6).

Measured CoT faithfulness is not a fixed property of chain of thought. It moves with the test, the training regime, the scale of the model and the token budget. Early evidence came from planted cues: when the few-shot answers in a prompt were reordered so that the correct option was always the same letter, models followed the bias and wrote explanations that justified the biased answer without mentioning the reordering [243]. On tests of this kind, reasoning models reveal a hint they used in at least 1% of cases for most settings, but often in fewer than 20%, and outcome-based reinforcement learning raises the rate at first and then plateaus [54]. Reasoning training still helps: for one such cue, DeepSeek-R1 describes the cue’s influence 59% of the time against 7% for the non-reasoning DeepSeek model, in a workshop paper [55]. Without any planted cue, asking a model whether X is bigger than Y and then whether Y is bigger than X exposes implicit post-hoc rationalization at rates of up to 13% for production models, 0.37% for DeepSeek-R1 and 0.04% for Claude 3.7 Sonnet with thinking [11]. On soft-reasoning tasks, models distilled from a reasoning model often revise their initial answer as the trace unfolds, while instruction-tuned and RL-trained reasoning models do so far less often; the instruction-tuned models’ traces often justify an answer already reached, and the reasoning models show mixed dynamics [144]. Scale cuts the other way: in the intervention tests of Section 5.2, larger and more capable models produce less faithful reasoning on most tasks [142]. When the token budget is raised, hint verbalization reaches 90% in some settings, which suggests that much apparent unfaithfulness comes from tight token limits [276].

Part of the disagreement is over what the tests measure. A planted-cue test asks whether the trace mentions a cause of the answer, and not whether that cause acts through the trace. Zaman and Srivastava [276] argue that the first criterion confuses unfaithfulness with the incompleteness of any linear account of a distributed computation. In some models more than half of the chains it flags are judged faithful by other metrics, and causal mediation analysis shows that hints the trace never names can still change the prediction through it. Resampling studies agree: a hint that affects the answer without being mentioned exerts a subtle, cumulative influence on the trace that persists even after the hint is removed [158]. A trace can therefore carry an influence while leaving it unstated, and so read as a complete account when it is not. Faithful CoT closes the gap for tasks that can be written as a program: the model translates the query into a symbolic chain that a deterministic solver executes, so the chain fixes the answer [156].

Monitorability, whether a reader of the trace can catch misbehavior, is a different property from faithfulness. The evidence so far turns on two conditions, whether the task needs the trace [54,76] and whether training optimizes against a monitor that reads it [21,100]. All four studies come from frontier laboratories, and only the monitorability evaluation suite [100] has been peer-reviewed; Problem 4 (Section 8.1) treats the evidence in full.

Analysis of the trace itself has begun to borrow from attribution. Thought anchors are found by resampling each sentence of a trace and measuring how the distribution of final answers shifts, which is occlusion moved from input tokens to generated sentences; the sentences with outsized influence are mostly planning and uncertainty management [42]. A model defines a distribution over traces, which self-consistency decoding already exploits by sampling several [258]. A single trace therefore supports few causal claims (Section 5.2). All of these tests assume that the reasoning is in the text. Models can be trained to reason in continuous latent states that are never decoded into words [106], and transformers given dense supervision can use meaningless filler tokens in place of a chain of thought to solve two hard algorithmic tasks they cannot solve without intermediate tokens [206]. Where reasoning moves into such channels, the trace stops being evidence of how the answer was reached.

4.2. Self-Explanation and Introspection

A model can also be asked after the fact why it answered as it did. Such self-explanations may be post-hoc rationalizations [271], and a preprint finds that they can read as convincing while diverging from the features that drive the prediction [5]. The tests that carry over from erasure feed the explanation back to the model. Madsen et al. [162] redact the words a self-explanation calls important, or apply the counterfactual it proposes, and find that faithfulness depends on the explanation type, the model and the task. Self-generated counterfactuals fail in a more basic way: models sometimes cannot produce one, and when they do, their prediction on it often disagrees with their own counterfactual reasoning [66]. Self-critique has the same limit. Models improve their code by explaining it and reading its execution results [52], but without such an external signal they often fail to correct their reasoning and can revise correct answers into wrong ones [115]. In both cases the evidence supports trusting the model’s account of itself only where something outside the model checks it.

Whether models have privileged access to their own states is now tested directly, and the answer depends on the design. Positive results come from setups that train or perturb the model. Models finetuned to predict properties of their own behavior outperform a stronger model trained on the same behavior, though only on simple tasks [39], and models finetuned on examples of a policy that are never described as such can state the policy [35]. When concepts are injected into its activations, Claude Opus 4.1 reports the injected thought on about 20% of trials at the best layer and strength, according to an industry report [149]. A peer-reviewed follow-up on open-weight models finds that detection holds across prompts and dialogue formats, at moderate rates with no false positives, traces it to a circuit that emerges in post-training, and finds the capacity under-elicited: ablating refusal directions in Gemma 3 27B raises detection from 10.8% to 63.8%, while false positives rise from 0% to 7.3% [159]. Designs with a stricter control find no evidence of privileged access. Across 21 open models, prompted grammaticality judgments and word predictions track a model’s own string probabilities no better than the prompted responses of a model with nearly identical knowledge do [232]. Singh et al. [229] require that an introspection test not be solvable from the input and that it separate second-order from first-order accounts. On that standard, input-only classifiers match a model’s predictions of labels derived from its own hidden states, and models cannot reliably tell an intervention on their states from a manipulation of their input. Both accounts of the injection result thus place it in a detector of anomalous internal states: Macar et al. [159] call that detector mechanistically nontrivial, and Singh et al. [229] call it generic anomaly detection. For linguistic work, the consequence is that a model’s prompted judgments should not be conflated with the generalizations its string probabilities show [232].

4.3. Mechanistic Interpretability

Mechanistic interpretability aims to understand models at the level of individual components, including neurons, attention heads, and circuits. The label is used loosely in current work, having broadened from its original sense of reverse-engineering algorithms out of weights into a general banner for internal analysis; readers comparing claims across papers should check which sense is intended [220].

The circuits agenda [196] takes features and the weighted connections between them as the unit of analysis. In transformer language models its best-known results are induction heads, which implement a match-and-copy pattern and for which there is preliminary evidence that they underlie most in-context learning [73,197], and hand-analyzed circuits in GPT-2 small for indirect object identification (IOI) [257] and numerical greater-than comparison [103]. ACDC automates the search by pruning a computational graph to the subgraph responsible for a behavior [60], and Boundless Distributed Alignment Search works top-down, aligning the representations of the instruction-tuned Alpaca model with the variables of an interpretable causal model [265]. The causal machinery predates the circuits vocabulary: causal mediation analysis was already measuring how much specific neurons and heads mediate gender bias [252].

A circuit claim is validated much as a rationale is. Wang et al. [257] require a circuit to be faithful (on its own it performs the task about as well as the full model), complete (it and the full model stay similar when any subset of its components is knocked out) and minimal. Faithfulness and completeness are the counterparts of sufficiency and comprehensiveness, moved from input tokens to model components, and the IOI circuit shows why both are needed: it recovers 87% of the full model’s logit difference, yet a greedy search found knockout sets under which circuit and model diverge by as much as 87% of the original logit difference. Three failure modes recur across circuit studies. The choice of corruption and metric can decide the result [277], which is the out-of-distribution problem of erasure moved inside the model. Overlap with a reference circuit does not establish faithfulness, since two discovery methods can both overlap heavily with a circuit found by intervention and still differ in faithfulness [104]. Nor need the explanation be unique: on small networks where every candidate can be enumerated, several circuits reproduce the same behavior and one circuit admits several interpretations [171], the circuit-level form of the finding that different attention distributions can yield the same prediction [121]. Knowledge localization shows a related gap. Causal tracing locates some factual associations in transformer weights where they can be edited [172], and their recall can be followed from subject representations to the answer [95], but where tracing localizes a fact does not reliably predict where editing it succeeds [108]. Section 5.3 discusses how such internal explanations can be evaluated.

A central obstacle for this agenda is superposition: models may represent more features than they have dimensions, with features sharing neurons through approximately orthogonal directions, so that one neuron responds to several unrelated features [74].

A parallel line of work asks what the residual stream represents. Sequence models trained on board games develop representations of board state that can be read out and causally edited [146], truth-like directions can be recovered from hidden states without supervision [47], and many high-level concepts are encoded as linear directions [203]. Such directions can also be acted on. Adding a contrastively derived vector at inference time steers behavior along a concept [216], function vectors carry a compact task representation across contexts [241], representation engineering reads and steers concepts in population-level activations [282], and a single direction mediates refusal across chat models, so that ablating it suppresses refusals and adding it induces them on harmless prompts [12]. Removing a concept by projecting out a direction is an older idea: debiasing identified a gender direction in word embeddings and removed it [43], and iterative nullspace projection repeatedly trains a linear classifier for a property and projects the representations onto its null space [213]. Two caveats limit the linear picture. The objective behind unsupervised truth directions is satisfied by arbitrary features, so such methods tend to find whatever feature is most prominent [82], a result reported in a preprint, and some features are irreducibly multi-dimensional, such as the circular representations of days and months used in modular arithmetic [77].

Language models can also write the descriptions. Bills et al. [37], in an industry report, had GPT-4 describe what individual neurons of GPT-2 XL respond to, at a scale hand inspection cannot reach. A score for such a description measures how plausible the label is. Tested both observationally and by intervention, even the most confident of those descriptions had high error rates and little or no causal efficacy [116], and the same kind of automated score applied to dictionary features fails a random-model baseline (Section 4.4).

4.4. Sparse Autoencoders and Circuit Tracing

Sparse autoencoders (SAEs) address superposition by rewriting each activation vector as a sparse combination of features from a learned, overcomplete dictionary. The method was first shown on the MLP of a one-layer transformer [45] and, by Huben et al. [117],1 on the residual stream of Pythia models, where its features scored as more interpretable under automated scoring than PCA or ICA directions. It was then scaled to millions of features from Claude 3 Sonnet [237] and, with k-sparse (TopK) autoencoders, to a 16-million-latent dictionary on GPT-4 activations [89]. JumpReLU SAEs improve reconstruction at a given sparsity [210], the Gemma Scope release made hundreds of them public for the Gemma 2 models [147], and crosscoders learn one dictionary across layers or across models [150]. Several of these results appeared as industry reports without peer review [45,150,237].

Whether a dictionary feature explains the model is the faithfulness-plausibility question again. An automated interpretability score measures how plausible a feature’s label is, while reconstruction of the model’s behavior, stability across training runs and comparison with a random-model baseline test whether the decomposition belongs to the model. The two kinds of evidence come apart. SAEs trained on randomly initialized transformers often reach automated-interpretability and reconstruction scores similar to those of SAEs trained on the real models [109], and SAEs that differ only in their random seed share only 30% of their features in a 131K-latent dictionary on a Llama 3 8B layer, with TopK SAEs more seed-dependent than ReLU ones [204]. Paulo and Belrose [204] therefore read an SAE as one useful decomposition of activation space among several.

Where a supervised baseline exists, dictionary features have not beaten it. SAE-based probes do not consistently outperform probes on the raw activations under data scarcity, class imbalance, label noise or covariate shift [123]. On AxBench, prompting and then finetuning lead for steering and representation-based methods such as difference-in-means lead for concept detection, with SAEs competitive on neither [267], and SAEBench finds that gains on proxy metrics do not reliably carry over to practical tasks [125]. Choosing features by their effect on the output, instead of by the inputs that activate them, improves SAE steering two- to threefold and makes it competitive with the representation-based methods, though prompting still scores about twice as high on the same benchmark [10]. Where no labels exist the case is stronger but thinner. Sparse feature circuits let a human ablate the features judged irrelevant to a task and so improve a classifier’s generalization [165]. In a blind auditing game, three of four teams found an objective deliberately trained into a model, using SAEs alongside behavioral attacks and training-data analysis [166]. When the game was rerun with automated auditing agents, SAEs trained on generic data did not reliably improve win rates in any configuration, while a natural-language autoencoder that maps activations to text raised them, both for agents without training-data access and for agents with full access [88]; both reports come from one industry laboratory. Benchmarking now reaches beyond SAEs to circuit and causal-variable localization [188]. On this evidence, dictionaries currently generate hypotheses about a model and do not yet measure it; Problem 5 (Section 8.1) asks on which tasks that could change.

Dictionary features become explanations of behavior when they are connected into computations. Transcoders replace an MLP layer with a sparse approximation of its input-output function [71], sparse feature circuits connect SAE features into causal graphs for a behavior [165], and attribution graphs built on cross-layer transcoders trace the steps by which a model produces one output [8]. Applied to Claude 3.5 Haiku, attribution graphs exposed mechanisms for multi-step reasoning, planning ahead in poetry, multilingual processing, hallucination and refusal [151]. In the terms of Section 3, an attribution graph explains a surrogate: a replacement model trained to mimic the MLPs, read by linear attribution and checked by intervention. Its faithfulness is therefore a fidelity question of the kind LIME raises, and its error nodes measure the part of each MLP output the replacement leaves unexplained. Its authors list the limits. Attention patterns are taken as given, so the graphs do not show why information moves between positions; the replacement may use different mechanisms from the model; and the graphs gave satisfying insight on about a quarter of the prompts tried [8,151]. Both are industry reports, and an open-source library for building such graphs on open-weight models has been tested on models of up to 14B parameters [105]. The neuron basis is a baseline here as well: MLP neurons turn out to be as sparse a basis as SAE features, and gradient-based tracing on neurons finds a circuit of about a hundred neurons that controls subject-verb agreement [14].

4.5. Checking What a Model Says Against What It Computes

The two halves of this section meet where a self-report can be checked against the computation, and only methods with access to internals can make that check. Their verdicts so far are mixed. Asked how it added 36 and 59, Claude 3.5 Haiku described the carry algorithm taught in school, while its attribution graph showed a low-precision pathway estimating the sum running in parallel with one computing the ones digit [151]. Given a hint to a problem it could not compute, the model worked backwards from the suggested answer to the intermediate value that would produce it and tailored its written steps to match; on a problem it could compute, the graph confirmed the computation the trace described [151]. The causal mediation result of Section 4.1 shows hints acting through a trace that never names them; unlearning the content of a reasoning step from the parameters often changes the prediction [244], a test described in Section 5.2; and concept injection shows some self-reports tracking an internal state (Section 4.2). A trace or a self-report is thus a claim about the computation that internal methods can test, and the test requires weights, which most deployments of frontier models do not give.

4.6. The Attention-Explanation Debate

The debate over attention as explanation was conducted on encoders and recurrent models. Against attention, alternative attention distributions can yield identical predictions [121], attention may not agree with gradient-based importance [223], and models can be trained to push attention away from impermissible tokens while still relying on them, so that attention looks clean while the prediction is not [208]. On the other side, attention can be a valid explanation under appropriate conditions [261], and constrained attention can be made more faithful [184]. Bibal et al. [36] bring together the research areas involved, which seldom cite one another, and set out the terms of the debate, including the distinction between faithful and plausible explanations. The question for LLMs is whether these conclusions survive in deep decoders with many heads.

Some encoder findings carry over. In BERT, many heads attend to easily described positional or syntactic targets and much of the attention mass falls on delimiter tokens that carry little task content [56]. Decoder LLMs show the same pattern as attention sinks, strong attention to the initial tokens even when they carry no meaning; evicting those tokens from a sliding-window cache makes generation on long texts fail [268]. The responses take more of the computation into account. Rollout and flow trace attention across layers [1], and ALTI [84] and value zeroing [185] measure how much context actually mixes into a representation, correcting for the residual stream and the value transformation; on encoder models both score as more faithful than the gradient-based or attention-based scores they were compared with. For decoder LLMs, AttnLRP propagates relevance through the attention layers of models such as LLaMA 2 and Mixtral and reports higher faithfulness than the attribution methods it was compared with [2], and information-flow routes extract the important paths through Llama 2 from a single forward pass, without the prompt templates that activation patching needs [83]. Single heads can still carry an identifiable function. Retrieval heads, fewer than 5% of the heads in the long-context models studied, fetch information from the context, and pruning them makes retrieval fail while pruning other heads at random does not [264]. Whether context-mixing scores also improve on raw attention in decoder LLMs, measured against importance established by intervention, is part of Problem 3 (Section 8.1).

4.7. What Is New and What Is Rebuilt

Many techniques in this section have an ancestor in interpretability work that predates LLMs, much of it covered in Section 3. Table 6 pairs each LLM-era technique or question with its closest classical counterpart and records what changed between the two: the access the method needs, the scale it runs at, the object it intervenes on or decodes, or the form of its output. Every work cited in the counterpart column was published in 2020 or earlier.

Ten of the twelve rows have a classical counterpart. Five rebuild a classical idea and five extend one, which is the sense in which we say that most LLM-era techniques rebuild or extend older ideas under new constraints of scale and access. For activation patching the Rebuilt verdict is shared by two recent syntheses, one that organizes mechanistic interpretability by the causal mediator a method intervenes on [189] and one that restates patching, mediation analysis, circuit analysis and sparse autoencoders in the single language of causal abstraction [93]. Two patterns run across rows. Faithful CoT makes the move of select-then-predict models, routing the answer through an explanation that determines it, and the two designs share a limit: the guarantee covers the second stage only, so neither explains why the selector picked those tokens or why the model wrote that program. The dispute over chain-of-thought as explanation has the structure of the attention debate. In both, a quantity that practitioners read as an explanation is shown to diverge from what drives the prediction, and the reply questions the criterion behind that finding (Section 4.1 and Section 4.6).

The counterparts matter because their failure modes and tests carry over. A saliency map that barely changes when the model’s weights are randomized cannot explain the trained model [4], and the same control shows SAEs on random transformers often scoring like SAEs on trained ones [109]. Remove-and-retrain answered the distribution shift of erasure by retraining without the removed features [113], and unlearning a reasoning step from the parameters likewise changes the model and not only its input, since a step erased from the trace alone can still be recovered from the parameters [244]. Causal mediation analysis [252] now runs through the trace and finds hints acting through it unnamed [276]. The finding that different attention distributions yield the same prediction [121] returns as the non-identifiability of circuits [171]. Control tasks asked whether a probe’s accuracy reflected the representation or the probe [111], and baseline probes on raw activations ask the same of SAE probes [123]. Influence estimates proved a poor match to retraining in deep networks [29], so the LLM-scale estimator is validated against a local proximal objective, a comparison run only on models of up to 810M parameters, and how well that objective captures what retraining would show is left open [99].

The two new rows each have a near precursor, and the difference is what makes them new. In the nearest precursor for monitorability, deceptive attention was itself the training objective [208]. In a reasoning model the trace can be part of the computation, so a model that needs its trace can hide it only with substantial help [76], and obfuscation arises as a side effect of optimizing too hard against a safety monitor [21]. Models trained on human-written explanations learn to produce explanations that resemble them [49]; introspection asks instead whether a model’s report draws on its own states beyond what another model can predict, one trained on the same behavior for Binder et al. [39] and one with nearly identical internal knowledge for Song et al. [232]. Neither question has a classical test literature, and monitorability returns as an open problem in Section 8.1.

5. Evaluation Frameworks

Explanations are hard to evaluate because there is rarely a ground truth for what the “correct” explanation of a prediction should be, and because interpretability itself bundles several distinct desiderata that call for different measurements [152]. A useful organizing distinction is due to Doshi-Velez and Kim [69]: evaluations can be functionally grounded (proxy metrics with no humans), human-grounded (simplified tasks with lay users), or application-grounded (real tasks with domain experts), with cost and validity rising in that order. Three results organize what follows. Every faithfulness test carries a confound of its own, so the method that looks best depends on the test chosen, for input attributions, chains of thought and circuits alike. The erasure tests built for extracted rationales reach a generated explanation only by changing what they erase. Explanations also raise acceptance of model outputs more reliably than they raise the quality of the decisions people make with them.

5.1. Intrinsic Evaluation

5.1.1. Faithfulness

Faithfulness measures how accurately an explanation reflects the model’s actual reasoning process [118,119]. An explanation is faithful if it changes when the model’s reasoning changes. Erasure tests ask whether the highlighted input drives the prediction. Surrogate fidelity, used for explanations such as LIME’s, asks only how often an interpretable surrogate agrees with the model’s predictions in the region being explained [141]; agreement on outputs does not show that the surrogate’s features are the ones the model uses.

The most widely used faithfulness metrics for extractive rationales are the erasure-based measures of the ERASER benchmark [68]. Let $m{\left(x_i\right)}_j$ denote the probability that model m assigns to its originally predicted class j for input $x_i$, and let $r_i$ be the rationale, that is, the highlighted portion of the input. Comprehensiveness asks whether the rationale captures all of the evidence the model relied on, by measuring how much the predicted-class probability drops when the rationale is removed from the input:

$$\mathrm{comprehensiveness}=m{\left(x_i\right)}_j-m{(x_i\setminus r_i)}_j,$$

(1)

where $x_i\setminus r_i$ is the input with the rationale tokens deleted; higher is more faithful. Sufficiency asks the converse question of whether the rationale alone is adequate to support the prediction:

$$\mathrm{sufficiency}=m{\left(x_i\right)}_j-m{\left(r_i\right)}_j,$$

(2)

where $m{\left(r_i\right)}_j$ is the predicted-class probability when the model is given only the rationale; lower is better. Both metrics are signed differences of probabilities, computed for the class the model originally predicted, which need not be the gold label. Because a single rationale threshold yields only a point estimate, ERASER also defines an aggregate “Area Over the Perturbation Curve” (AOPC) variant that averages Equation 1 over bins of the top 1%, 5%, 10%, 20%, and 50% of tokens ranked by importance score:

$$\mathrm{AOPC}\text{-}\mathrm{comprehensiveness}={\displaystyle \frac{1}{\left|\mathcal{B}\right|+1}}{\displaystyle \sum _{k=0}^{\left|\mathcal{B}\right|}}\left(m{\left(x_i\right)}_j-m{(x_i\setminus r_{ik})}_j\right),$$

(3)

where $r_{ik}$ denotes the tokens up to and including bin k; AOPC sufficiency is defined analogously from Equation 2 [68].

5.1.2. Limitations of Faithfulness Metrics

Erasure-based metrics share a structural weakness: deleting tokens produces inputs unlike anything the model encountered during training, so a drop in confidence may reflect the out-of-distribution character of the corrupted input as much as the importance of the removed tokens. ERASER’s metrics accept this confound in exchange for scoring the deployed model cheaply. Two remedies remove it, each at a price. ROAR (RemOve And Retrain) retrains the model on inputs from which the allegedly important features have been removed; on image classifiers it found many popular saliency methods no better than a random assignment of importance [113]. Its NLP adaptation, Recursive ROAR, masks and retrains recursively [160]. Both score a retrained model in place of the deployed one. Fine-tuning the model so that masked inputs stay in distribution makes erasure measurable without retraining, but only for the fine-tuned model [163]. Sanity checks, also from image classification, answer a different question. A saliency map that barely changes when the model’s weights or its training labels are randomized cannot be explaining the trained model, and two widely used methods, Guided Backpropagation and Guided GradCAM, fail this test [4]; such checks rule methods out without ranking the ones that pass.

Which method looks most faithful depends on the test, and studies that share no method agree on this. Against a list of diagnostic properties, gradient-based methods led on most properties across three text classification tasks and three architectures, but perturbation methods best predicted the model’s confidence and no single method led on rationale consistency [17]. Recursive ROAR finds the faithfulness of importance measures to be both model-dependent and task-dependent [160]. Chan et al. [51] start from the observation that faithfulness metrics prefer different interpretations and find sufficiency and comprehensiveness the most diagnostic of the metrics they compare, and among the cheapest to compute. Human judgment does not settle the question either: in an early comparison for text classification, crowdsourced judgments of local explanations correlated only moderately with word-deletion measures [193]. None of these comparisons included a decoder-only LLM: the models were linear classifiers, small feedforward, convolutional or recurrent networks, and BERT-style encoders. The same dependence recurs for internal explanations (Section 5.3) and for the free-text tests of Section 5.2, where a study that scores faithfulness metrics against known answers finds their diagnostic power varying across tasks and models [275].

The recurring answer to the missing ground truth has been to construct one. Hybrid documents, built by concatenating fragments of documents with known labels, give a reference that holds on average: a method scores a hit when its most relevant word falls in a fragment whose label matches the prediction [207]. Later designs plant the answer in the model, through partially synthetic training data with planted shortcuts, on which some of the most popular salience configurations give poor results even for the simplest shortcut [28], or through model editing, which yields faithful and unfaithful pairs of natural-language explanations [275]. Each design checks a method only on the kind of mechanism it plants, which is why Problem 2 in Section 8.1 asks for a suite of them. We therefore follow Lyu et al. [157] in treating faithfulness as a graded property probed by several complementary tests. A method that passes independent tests warrants more confidence than one validated against a single metric, so an evaluation should name each test it ran and report its result separately.

5.1.3. Additional Quality Criteria

The remaining criteria divide by the relation they test. Soundness, which requires every part of an explanation to be true of the system, and completeness, which requires it to cover the system’s relevant behavior [137], relate the explanation to the model, as faithfulness does; so does generalizability, whether an explanation holds beyond the explained instance [231]. Plausibility, how convincing an explanation is to people [118], usually measured against human rationales such as those of e-SNLI [49] and CoS-E [211], and comprehensibility, for which the number of features, rule complexity and explanation length serve as proxies [140], relate it to a person. Only the first group can be checked without people, and only the second is visible to the people who use the explanation, which is one reason plausibility is the easier property to obtain. The two groups need not agree: in a diagnostic study of saliency methods, models whose explanations scored high on faithfulness did not necessarily score high on agreement with human rationales [17], and Section 5.4.1 reports the same gap for LLM explanations.

5.2. Evaluating Free-Text and LLM Explanations

The metrics above presuppose extractive rationales, that is, subsets of input tokens. Explanations generated by LLMs, whether CoT reasoning traces produced before the answer or self-explanations offered after it, are free-form text. Such an explanation is generated by the model, so there is no input span to delete, and erasure survives only by changing its target. A chain of thought is part of the context on which the answer is conditioned, so truncating or corrupting it tests whether the answer depends on the stated reasoning; Table 6 records this move as an extension of erasure. For a post-hoc self-explanation, erasure applies only where the explanation names input words that can be redacted [162]. The remaining tests intervene on the prompt, the input or the parameters and check whether the explanation tracks the change. Table 7 groups the tests by what they intervene on.

Taken together, the tests show that faithfulness of a free-text explanation is a property of a model on a task. Models condition on their chain of thought strongly for some tasks and barely for others, and larger, more capable models produce less faithful reasoning on most tasks studied [142]. Self-consistency checks on post-hoc self-explanations reach the same verdict from a different design, with faithfulness depending on explanation type, model and task [162]. How a trace is edited also matters. On decision-making tasks, off-policy edits to a chain had small and unstable effects compared with resampling the continuation from the model itself [158]. This bears on the adding-mistakes test, since an inserted error is text the model did not write.

The tests also disagree on what counts as unfaithful. The biasing-feature test counts a chain unfaithful when a cue planted in the prompt changes the answer and the chain never mentions it [243]. Zaman and Srivastava [276] reply that this confuses unfaithfulness with incompleteness, and both their mediation analysis and a resampling study [158] find unmentioned hints still acting through the chain (Section 4.1). Omitting a cue therefore does not show that the chain played no part in the answer, and Parcalabescu and Frank [202] argue that most tests in this family measure self-consistency at the output level. Label-rationale association is a cheaper and weaker check: on the two measures Wiegreffe et al. [262] introduce, robustness equivalence and feature-importance agreement, jointly trained T5 models that predict and rationalize satisfy both on commonsense question answering and natural language inference, while pipelines that are faithful for extractive rationales do not extend as reliably to these tasks. Association is required for faithfulness and does not establish it.

Monitorability is scored differently from faithfulness. The quantity is how often a monitor reading the trace catches misbehavior known to have occurred, such as a reward-hacking exploit [21], or detects that a planted cue influenced the answer, a design adapted from hint-verbalization tests of faithfulness [54,243]; one evaluation suite separates such intervention designs from process and outcome-property designs [100]. Section 4.1 and Problem 4 in Section 8.1 discuss the results.

The use of an LLM as a judge of explanation quality needs its own check. Such judges show position and verbosity effects and may favor their own answers [280], and they are swayed by the arguments they read, as people are [201]. Their agreement with people has now been measured for explanation quality. On ACORN, a dataset of commonsense explanations rated aspect by aspect, LLM ratings correlated with the human majority unevenly across quality aspects, and an additional human rater improved agreement more than an added LLM did in every case [44]. LLM-judged scores are therefore best reported together with the behavioral tests above and a human-rated subsample. In moral-alignment evaluation, EvalMORAAL adds a model-as-judge peer-review stage that flags conflicting judgments, most of which a majority vote among the judging models resolves, leaving the rest as candidates for human validation [183].

5.3. Evaluating Internal Explanations

A mechanistic explanation claims that a component, feature or circuit carries out part of the computation. Causal abstraction gives the criterion: a network is explained by a high-level algorithm if there is a map between the two under which interventions on one match interventions on the other, and the theory formalizes how far a partial match counts as graded faithfulness [93]. The criterion needs constraints to mean anything. If the map may be arbitrarily non-linear, then under reasonable assumptions any network can be mapped to any algorithm, and alignment maps reach perfect interchange-intervention accuracy on indirect object identification even in randomly initialized language models [235]. In many settings, random models expose the proxy metrics of sparse autoencoders as well [109] (Section 4.4). The scores also move with the experimenter’s choices. Circuit-faithfulness measures are highly sensitive to the ablation methodology [175], and activation-patching conclusions to the corruption scheme and metric [277], which is the internal counterpart of the out-of-distribution confound in erasure.

Benchmarks with a known answer are the response. MIB finds attribution and mask-optimization methods best at circuit localization, and for causal-variable localization finds supervised distributed alignment search best and SAE features no better than neurons [188]; an auditing game supplies a hidden objective planted by training as the answer to recover [166]. Together with the probing and steering comparisons of Section 4.4 [123,267], every comparison reviewed here that sets dictionary features against simpler methods finds them matched or beaten, and Problem 5 in Section 8.1 asks where that could change. Each failure above has a matching control, and an internal explanation is best reported with all four: a randomized-model baseline, a stated constraint on the alignment map, the ablation scheme used, and the strongest simple baseline.

5.4. Extrinsic Evaluation

Extrinsic evaluation asks what an explanation does for the people who use it. Miller [176] synthesizes the social-science literature and finds that human explanations are contrastive, selective and social, and among the output forms of Section 3, counterfactuals answer the contrastive question most directly. Studies of human-AI decision making differ widely in task, assistance and outcome measure, and a review of the empirical literature calls for common frameworks so that their results can build on one another [139]. The outcome that matters is the accuracy of the decisions people make with an explanation. In deception detection, explanations alone improved human accuracy slightly, while showing the model’s predicted label improved it far more [138]. A stricter criterion is complementary team performance, in which the human-AI team outperforms both the human and the model working alone [22].

5.4.1. Trust and Decision Support

The target is warranted trust, which lets users anticipate when the model is right and when it is wrong [120]. Table 8 collects user studies that measure reliance or decision accuracy with and without explanations, and most point the same way: explanations raise acceptance of the model’s output whether or not it is correct. Bansal et al. [22] found that explanations made people more likely to accept the AI’s recommendation regardless of its correctness, without adding to the complementary gains that AI assistance itself produced. LLM-era studies repeat the result with free-text explanations in question answering and fact-checking [129,226], and the form of the text does part of the work. Longer explanations raised users’ confidence without making the answers more accurate [233], and in hate-speech moderation fluent natural-language explanations drew more over-reliance than feature-based ones [242]. LLM-written arguments for or against an answer also moved people’s plausibility judgments in the direction argued [201].

Where explanations made reliance more appropriate or decisions better, they gave the user a way to check the model. Sources, and explanations that contained inconsistencies, lowered reliance on incorrect answers [129]; showing the case for and against a claim reduced over-reliance in fact-checking [226]; and physicians, who are better placed to check a diagnosis, stayed resilient when the model erred, while lay users gained when it was right and lost when it erred [269]. The pattern fits the argument of Fok and Weld [87] that explanations help only to the extent that they let the decision maker verify the prediction, and that most tasks do not allow easy verification. In one study, three interventions aimed at reliance itself reduced over-reliance without generally improving appropriate reliance [41], and showing two explanation types together increased over-reliance [23]. The outcome that separates the two effects is reliance split by whether the model was right, which Problem 6 in Section 8.1 asks future studies to report.

Simulatability, whether users can predict model behavior after studying explanations, is a second behavioral outcome. In the first tests to isolate the effect of explanations on it, clear gains appeared for very few combinations of method and data domain, and users’ ratings of an explanation did not predict how much it helped them [107]. The LLM-era version asks whether an explanation lets a reader infer the model’s answers on counterfactual inputs; for GPT-3.5 and GPT-4 this precision was low and did not correlate with how plausible the explanation was judged [53].

5.5. Benchmarks and Datasets

A survey of the datasets available for explainable NLP, covering roughly sixty-five resources and the design decisions behind their collection, is a useful starting point for choosing among them and for understanding the annotation artifacts each introduces [260]. The ERASER benchmark [68] remains the standard resource for evaluating extractive rationales, unifying seven datasets with human rationale annotations under the common metrics of Equations 1–3. e-SNLI [49] and CoS-E [211] add crowd-written free-text explanations for natural language inference and commonsense question answering and are widely used to train and evaluate models that generate explanations; ERASER includes extractive versions of both. Domain datasets for hate speech and fact verification, among them HateXplain [167], FEVER [240] and LIAR-PLUS [7], are discussed with their applications in Section 7. Table 9 records what each resource can test. Agreement with human annotations measures plausibility, so the faithfulness of free-text and CoT explanations rests on the intervention tests of Section 5.2, and the benchmarks for internal explanations replace annotations with a mechanism or concept that a method must recover (Section 5.3).

6. Practical Guidelines

This section turns the taxonomy into practical decision frameworks for choosing and deploying an explanation method. The frameworks start from model access, because the access a deployment allows settles which mechanisms are usable before scope or audience matter, and three worked deployments test them. Where a recommendation rests on a published comparison we cite it, where it rests on our judgment we say so, and Table 11 marks each task accordingly.

6.1. Method Selection Decision Framework

6.1.1. Key Decision Factors

Figure 2 gives the decision tree. Three of the taxonomy’s four dimensions (Section 3) enter it as inputs: the access the deployment allows, the scope required, and, through the audience, the output form. The fourth, mechanism, is what the procedure returns. Deployment access is graded by what the deployment exposes, which differs from whether a method is model-agnostic, and Table 10 distinguishes three levels. Open weights make every mechanism family usable, from Integrated Gradients [234] and context-mixing measures such as ALTI [84] to patching model components. An API that returns output scores (class probabilities, log-probabilities or embeddings) supports perturbation and surrogate methods such as SHAP [153] and LIME [214], scored on those values; one that returns only labels or text leaves label-flip perturbation, Anchors [215], counterfactuals and prompting. For reasoning models a further question cuts across these levels: whether the trace comes back raw and editable, as open-weight models allow, or as a summary or not at all, as on the OpenAI and Anthropic APIs [9,200]. The faithfulness tests a deployment can run follow from the same exposure. The mapping follows from what each method and test takes as input; it does not rank the methods.

Scope then narrows the choice within those access constraints. Local methods such as LIME, Integrated Gradients, or a counterfactual edit explain a single prediction; aggregated SHAP values, probing classifiers, and distillation into an interpretable surrogate characterize overall behavior. Many deployments need both, local explanations for contested decisions and global summaries for audits.

Different audiences need different output forms. Human explanations are contrastive [176], which favors a counterfactual or a highlighted span for end users. Fluent prose is the riskier form for non-experts [242,269], and showing two explanation types together raised over-reliance [23] (Section 5.4.1). Explanations of model internals are rarely used outside NLP [48], so layer-wise context-mixing maps and other views of internals suit practitioners and auditors more than domain users. Regulators need the documented methodology that the reporting standards of Section 6.3 produce (Section 7). Whether a form serves its audience has to be measured, because users’ own ratings of an explanation do not predict how well it helps them simulate the model [107].

6.1.2. Decision Framework for LLM-Era Systems

For systems built on large language models, additional branches extend the framework, and the first question remains access. When a model is reachable only through an API, as is typical for frontier model families such as GPT, Claude, and Gemini, perturbation and surrogate attribution over the API’s outputs supplies the evidence. For generated text the attribution needs a scalar target: the log-probability of the original output given the perturbed input where the API returns it, or a similarity score between perturbed and original outputs where it returns text only [186]. Chain-of-thought and self-explanation are added on top as a presentation layer. Passing the tests of Section 5.2 that intervene on the trace rules out a trace the answer ignores; whether it shows the trace to be faithful is disputed (Problem 1, Section 8.1), so a trace that passes is reported together with the tests it passed. With open weights every mechanism is available, and the binding constraint becomes compute, as in the sexism-detection example below.

A second branch concerns reasoning models such as o1 [198] and DeepSeek-R1 [65], which are trained with reinforcement learning to reason at length before answering. Whether the trace can be tested at all depends on what the deployment returns (Table 10). Where it can, the evidence reviewed in Section 4.1 counsels caution, since reasoning models often leave unmentioned a hint they used [54] and post-hoc rationalization occurs even without a planted bias [11]. Three studies bear on a second question for this branch, whether the task needs multi-step computation. How much the answer depends on the chain varies by task [142] (Section 5.2). Chen et al. [54] suggest that in settings like theirs, where the task does not require the chain, monitoring it at test time is unlikely to catch rare, catastrophic behaviors reliably. When a task does require it, models evaded a monitor only with substantial help [76] (Problem 4, Section 8.1). We therefore give a trace more weight on a task that needs it, such as a multi-step calculation, than on a one-pass judgment such as scoring a credit application from structured features. This weighting is our reading of the three studies; none tests it directly.

The final branch sets how much testing the stakes require. Every deployment runs the two-test minimum of Section 6.3 where its access allows. For a chain of thought, the two cheapest interventions, early answering and adding mistakes [142], share what we read as a weakness: both feed the model a trace it would not have produced, and such off-policy edits gave unstable effects in decision-making tasks [158] (Section 5.2). The second test therefore comes from outside the trace, a biasing feature planted in the prompt [243] or a concept-level counterfactual on the input [168]. In high-stakes settings we also check the trace against an attribution computed over the inputs, as in the credit example below. A prompting-based explanation that has passed no faithfulness test is presented as the model’s own account of its answer.

6.1.3. Three Worked Examples

The three walkthroughs below record the decisions the framework forces and the evaluation each decision implies. They are illustrative and none is a benchmarked comparison of methods; each falls in a different row of Table 10.

Clinical coding assistant, API-only access. A hospital assigns ICD codes to discharge summaries using a commercial LLM reached through an API. Access is the binding constraint, leaving perturbation, surrogate, and prompting (Table 10). The audience is clinicians who must verify each assigned code against the note, which argues for highlighted evidence spans as the output form. The framework therefore lands on SHAP or LIME computed over sentence-level units of the note, presenting the top-ranked sentences as the justification for each code; any prompted natural-language rationale is a presentation layer over that attribution. Where the API returns token log-probabilities, the attribution is computed over the probability the model assigns to the code, and where it returns labels only, label-flip perturbation or Anchors [215] work from the assigned code alone. Sentence-level attribution needs many API calls for each assigned code, so it is computed for the codes a clinician queries, the same split by need that the next example makes by appeal. Evaluation combines comprehensiveness and sufficiency over the highlighted sentences (with labels only, whether removing or isolating those sentences changes the assigned code) with an application-grounded clinician study measuring whether verification time falls without a rise in accepted miscodes. Where the model can be trained in house, a per-label attention architecture such as CAML [190] yields highlighted spans with no query cost. Its authors’ evaluation by a physician found those spans meaningful, which is evidence of plausibility; the attention is internal inspection and has to pass the same comprehensiveness and sufficiency checks (Section 4.6 and Section 5) before it replaces the attribution.

Sexism detection at platform scale, full access. A platform fine-tunes an open-weight encoder and needs per-decision explanations for appeals as well as periodic global audits. Full access opens every mechanism, but throughput closes most of them again, since billions of items make per-item Shapley estimation infeasible. The workable split is by scope. Locally, plain gradient attribution over tokens runs inline at the cost of one backward pass, and SHAP, which Mohammadi et al. [177] used to surface the terms behind each sexism score, is reserved for items that enter the appeals queue. The split is our design. Gradient-based explanations performed best on most properties across tasks and architectures in the diagnostic comparison of Atanasova et al. [17], which justifies trying the cheap method first, and the faithfulness tests below decide whether it stays. Globally, probing classifiers and aggregated attribution run offline to test whether the model has come to rely on dialect or identity terms as proxies. Evaluation pairs plausibility against the word-level rationales of HateXplain [167], which cover English posts only, with two faithfulness tests of the inline attribution that fail in different ways: comprehensiveness and sufficiency on a sample of items, and a Recursive ROAR check [160].

Credit adverse-action notices, open-weight reasoning model. A lender runs an open-weight reasoning model such as DeepSeek-R1 [65] on its own servers to decide applications from structured features, and must state the specific principal reasons for each adverse decision under Regulation B (Section 7). Open weights make the trace available for testing, whereas the OpenAI and Anthropic APIs return a summary of the reasoning or nothing [9,200]. The principal reasons come from a model-agnostic attribution over the application features. The trace is then tested against that attribution with concept-level counterfactuals [168], varying income, debt ratio and each protected attribute in turn and checking whether the trace cites the features whose change flips the decision. Early answering and adding mistakes [142] test whether the decision depends on the trace at all. We expect that it largely does not. A decision over a dozen structured features needs no multi-step reasoning, the kind of setting in which Chen et al. [54] suggest trace monitoring is unlikely to catch rare failures reliably, and Emmons et al. [76] single out bias auditing as an application where post-hoc rationalization in the trace is an important failure mode. The notice therefore rests on the attribution, and the trace serves as an internal audit record. The attribution method, its hyperparameters and the tests run are logged so that the notice can be reconstructed at audit.

6.2. Task-Specific Recommendations

Table 11 lists starting-point methods, evaluation focus, and metrics by NLP task. Its last column gives the basis for each row: the studies that compared methods or tests for that task, or our judgment where no comparison has been run. Text classification is the only task with several comparative studies behind it.

6.2.1. Text Classification

For document and sentence classification, begin with gradient-based attribution such as Integrated Gradients [234] where the weights are available, and with LIME or SHAP where the model can only be queried. Where a view of how context flows through the model is wanted, a context-mixing measure such as ALTI [84] or value zeroing [185] is a better choice than raw attention weights (Section 4.6). Anchors adds rule-based explanations that practitioners can read and check directly. The comparative evidence behind this order is mixed. Gradient-based explanations performed best across tasks and architectures on most of the diagnostic properties of Atanasova et al. [17], yet ALTI outperformed gradient methods on comprehensiveness and sufficiency for three sentiment datasets and subject-verb agreement [84], and Recursive ROAR found the faithfulness of importance measures to depend on both model and task [160]. Which method ranks first thus depends on the test and the model, so the starting point is chosen for cost and the protocol of Section 6.3 decides whether it stays. An attribution validated in distribution is retested under domain shift, since it may not hold on data unlike the training set.

6.2.2. Sequence Labeling Tasks

For token-level tasks such as named entity recognition, attribution is computed per labeled position and evaluated per entity span with the erasure tests of Section 5. Probing locates the layers at which entity information emerges [238]; transition weights are informative only for taggers that keep a CRF layer.

6.2.3. Question Answering

For extractive question answering, where models select answer spans from context, a context-mixing measure such as ALTI [84], or attention rollout [1] where only attention weights are exposed, traces how information from the question and context reaches the answer tokens, evidence sentence extraction with confidence scores gives justifications that users can verify, and counterfactual edits to the context show what information the model relies on. For generative question answering with LLMs, chain-of-thought prompting elicits a reasoning trace, which is reported together with the intervention tests of Section 5.2 it passed. Self-consistency across multiple reasoning paths indicates how reliable an answer is, and source attribution is treated as in the next subsection.

6.2.4. Text Generation Tasks

For summarization, translation, and open-ended generation, source attribution links each generated span to the input passages that support it, which shows whether the output is grounded in the source. In translation, word alignment between source and target gives a word-level view of the same link. The attribution needs checking in its turn. Even the best models tested on the ALCE benchmark lacked complete citation support about half of the time on ELI5 [91]. A cited passage that does support a statement need not be the one the model relied on: in one experiment up to 57% of citations lacked that faithfulness [256]. Where the model is reachable only through an API, perturbation attribution over the source, scored by text similarity when no log-probabilities are returned, gave more faithful explanations for summarization than the model’s self-explanations [186]. A chain of thought or self-explanation is tested with the interventions of Section 5.2. Where weights are available, patching the component an attribution points to [277] serves as an intervention check on that attribution. Whether sparse-autoencoder features repay their cost for a given task is open: on standardized benchmarks they have not beaten simpler baselines for concept detection or steering [123,267] (Problem 5, Section 8.1).

6.3. Evaluation Protocol Recommendations

6.3.1. Evaluation and Reporting Standards

We recommend the following minimum protocol for any explainability deployment. Faithfulness metrics disagree with one another [51], and how well a metric separates faithful from unfaithful free-text explanations varies across tasks and models [275], so faithfulness is checked with at least two tests that fail in different ways. Erasure tests such as comprehensiveness and sufficiency [68] can mistake the effect of an out-of-distribution input for the importance of the removed tokens; retraining [113,160] and in-distribution masking [163] avoid that confound at the price of retraining or fine-tuning (Section 5). Pairing an erasure test with one of the other two is a sensible default. For an API-only deployment such as the clinical example only erasure remains, so the minimum cannot be met; the report says so and states the out-of-distribution confound alongside the result. For an explanation the model writes in natural language the tests become interventions, on the reasoning trace for a chain of thought and on the words a self-explanation cites (Section 5.2). When the two tests disagree, the explanation counts as unvalidated for that deployment: the report gives both results, and users see the explanation only with that status. Stability under small input perturbations is a robustness property and is reported separately from faithfulness. Where human rationales exist, plausibility is measured on its own, by token-level F1 against the annotations and by Intersection-over-Union (IoU) for span-level explanations. For user-facing applications, the primary human-evaluation outcome is the quality of the decisions people make with the explanation, because explanations have raised acceptance of model outputs without raising team accuracy [22]. The strict version of that outcome, a team that outperforms both the human and the model working alone, is a demanding target. A meta-analysis of 106 experiments found human-AI combinations on average worse than the better of human or model alone (Hedges’ $g=-0.23$), with losses on decision tasks and gains mainly where the human alone outperformed the model [248]. A study should therefore report which of the two is stronger alone before interpreting the team result. Comprehensibility ratings and trust calibration are secondary measures.

Reports should document the dataset and its size, the model architecture and training details that may affect explanation quality, and the explanation method’s hyperparameters, such as the baseline input for Integrated Gradients or the number of perturbation samples for LIME and KernelSHAP. Each faithfulness test is named and its result given with a confidence interval or significance test; a single combined faithfulness score would hide the disagreements described above (Section 5). Reports also give the computational cost in time and memory and describe failure cases.

6.4. Tools and Libraries

Table 12 lists open-source libraries that implement the methods discussed in this survey, with the latest release of each as a rough guide to whether it is maintained. The ferret library implements the erasure tests of the protocol above as built-in metrics, and Captum implements infidelity and sensitivity measures.

The LLM era has produced a further layer of tooling aimed at models with billions of parameters. For open-weight models, TransformerLens [191] loads GPT-style architectures with standardized internal components and lets researchers cache, edit, and patch any activation during a forward pass, and has become common infrastructure for activation patching. NNsight offers a complementary interface built on deferred execution graphs, and together with the NDIF service it makes the internals of models too large to host locally accessible for the same interventions [86]; pyvene expresses such interventions, static or trainable, as configurations that can be shared [266]. Neuronpedia [148] hosts feature dashboards, automated feature descriptions, steering, and circuit tracing through a web interface and API. Automated interpretation, in which a language model describes what a neuron or feature responds to [37], has become a reusable resource, as have public releases of trained dictionaries, which spare practitioners the cost of training them: Gemma Scope for Gemma 2 [147], and Gemma Scope 2, an industry release that covers every layer of the Gemma 3 models from 270M to 27B parameters with sparse autoencoders and transcoders and adds, for the 270M and 1B models, the cross-layer transcoders that attribution graphs use [170]. SAELens trains and loads such autoencoders [40], and the circuit-tracer library builds attribution graphs [8]; its authors tested it on open-weight models of up to 14B parameters [105]. For API-served LLMs, perturbation attribution over generated text needs only query access, by scoring each perturbed output against the original [186]; explanations the model writes itself inherit the caveats of Section 5.2.

6.5. Computational Cost

At scale, cost decides which methods run where, and the costs differ by orders of magnitude. Plain gradient attribution needs one backward pass. Integrated Gradients needs one per interpolation step, and its authors found 20 to 300 steps enough to approximate the integral [234]. Perturbation methods need one model call per sample: the reference LIME implementation defaults to 5,000 perturbed texts [214], and KernelSHAP in the SHAP library defaults to $2M+2,048$ coalitions for M features [153]. Per-item perturbation is therefore expensive at high volume or when an API bills each call, which is why the clinical and sexism-detection examples reserve it for queried or appealed items. Retraining tests retrain the model once per masking ratio, and trace interventions regenerate the answer once per cut point or edit. Global explanations can be computed offline, so that one thorough analysis replaces repeated ones. Where latency is tight, a select-then-predict model whose rationale is faithful by construction [122] avoids post-hoc attribution at serving time.

7. Applications

Our companion domain-organized survey [178] examines the deployment contexts of explainable NLP in depth. Here each of the five domains is kept brief and read through four questions: who acts on the explanation, what model access is typical, where the requirement for an explanation comes from, and what evidence shows that explanations help. Table 13 in Section 7.6 sets the answers side by side.

7.1. Healthcare NLP

In the United States, clinical decision support software falls outside the definition of a medical device only if, among other criteria, it enables a clinician “to independently review the basis” for its recommendations (Criterion 4, section 520(o)(1)(E)(iii) of the FD&C Act). The FDA’s January 2026 guidance holds that software for critical, time-sensitive decisions does not meet the criterion, because the clinician is unlikely to have time for that review, and grounds this reading in automation bias [247]. Read as a test of verifiability, the criterion is met only if the basis shown is the basis the software used, which is a faithfulness condition. CAML offers clinicians a checkable output form: its per-label attention highlights the snippets of a clinical note that justify each assigned ICD code [190]. Those highlights are evidence of plausibility until they pass erasure tests (Section 4.6 and Section 5). Ghassemi et al. [97] argue further that current explanation methods are unlikely to give assurance about the prediction for an individual patient, and they call for rigorous internal and external validation of the models.

A chain of thought in which an LLM states its diagnostic reasoning is the model’s own account and needs the trace tests of Section 5.2 before a clinician treats it as the basis (Problem 1, Section 8.1). The clearest user evidence is from image diagnosis: in dermatology, physicians given LLM-written explanations were not misled when the model erred, while lay users were [269] (Section 5.4.1). What transfers to clinical NLP is that the reader’s expertise influenced whether the explanation supported review or replaced it.

For literature review and evidence synthesis the explanation is a citation, with the limits set out in Section 7.6. Device regulation asks mainly for documentation. Building on its 2021 action plan [245], the FDA issued draft guidance on AI-enabled device software functions in January 2025 that recommends total-product-lifecycle documentation covering transparency, bias evaluation and post-market performance monitoring [246]; as of October 2026 it is still a draft.

7.2. Legal and Financial NLP

In legal research the person acting on an explanation is trained to check it against a source: a cited case either supports the proposition or does not. Chalkidis et al. [50] released a dataset of European Court of Human Rights cases annotated with paragraph-level rationales, and found that some constraints proposed for word-level rationales (sparsity, continuity, comprehensiveness) do not help at the paragraph level, while others must be reformulated for the multi-label task. Deployed tools explain through citations instead. In a preregistered evaluation, commercial retrieval-augmented legal research tools hallucinated on 17–33% of queries, where a response counted as hallucinated if it was incorrect or misgrounded, that is, if it asserted that a source supports a statement it does not support [164]. Since a correct citation need not be the one the answer relied on [256], even a well-grounded citation is only evidence of plausibility (Section 7.6).

In credit the duty to explain is statutory. Under the Equal Credit Opportunity Act and Regulation B (12 C.F.R. § 1002.9(b)(2)), a creditor must disclose the specific principal reasons for an adverse action. CFPB Circular 2022-03 stated that this duty is not excused when decisions rely on complex or opaque models; the CFPB withdrew the circular, with other Bureau guidance, effective May 2025 (90 Fed. Reg. 20084), but the statutory and regulatory requirement remains in force [61]. Creditworthiness assessment is also a high-risk use under Annex III of the EU AI Act [80]. The applicant who receives the reasons cannot check them against the model, so the reasons have to be faithful on their own account, the case the credit worked example of Section 6 traces. None of the works reviewed tests whether the reasons lenders state, or the attributions they derive them from, match the operative ones.

7.3. Social Science Research

In social science research the person acting on an explanation is the researcher, who wants to know whether an automated coding measures the intended construct; no statute requires the explanation. Across 25 computational social science benchmarks, zero-shot LLMs did not beat the best fine-tuned models at classification but reached fair agreement with human annotators, and on free-form coding tasks their explanations often exceeded the quality of crowdworkers’ reference explanations [281]. That is a plausibility result. A preprint that replicates 37 annotation tasks from 21 published social science studies, testing 2,361 hypotheses with 18 LLMs, finds that configuration choices such as the model and the prompt lead to incorrect conclusions for about 31% of hypotheses with state-of-the-art LLMs, even under standard research practice [30]. In sexism annotation, annotator demographics explained about 8% of label variance against the dominant effect of the text, persona prompting often did not improve LLM agreement with human judgments and sometimes lowered it, and attribution showed predictions resting on content tokens related to sexism, not on correlates of demographics [182]. Comparisons of model outputs with international survey data show that LLMs often compress cross-cultural variation in moral attitudes and align poorly with survey patterns [181], so agreement on aggregate scores can hide the absence of the variation a social scientist wants to measure.

Read together, these results put the main threat to validity in sensitivity to configuration more than in opacity: an explanation serves as a measurement check only when it holds across prompts and models. Explanation can also generate hypotheses. HypotheSAEs trains a sparse autoencoder on text embeddings, keeps the features that predict an outcome, and has an LLM describe each one, which yielded findings on partisan differences in Congressional speeches and on drivers of engagement with online headlines [187].

7.4. Content Moderation

Moderation explanations have two readers: the moderator who reviews a flag and the user whose content is restricted. For the user, the EU Digital Services Act requires a hosting service to give a statement of reasons that includes the facts and circumstances relied on and, where applicable, the use made of automated means in detecting the content or deciding on it (Article 17) [79]. Practice falls short of the form. In a sample of 131 million statements of reasons submitted to the DSA Transparency Database in November 2023, several large platforms filled the free-text fields with one or two standard sentences [126].

7.4.1. Hate Speech and Toxicity Detection

HateXplain [167] is the only moderation resource in Table 9 with word-level rationales, and its authors report that classifiers with high accuracy can still score poorly on plausibility and faithfulness. For sexism detection, a pipeline that combines several BERT variants with a CNN uses SHAP values to identify the terms that contribute most to each prediction and combines them with the model’s predictions into graded sexism scores [177]. User evidence favors this kind of output for moderators, for whom fluent natural-language explanations increased over-reliance while feature-based ones encouraged scrutiny [242] (Section 5.4.1). Attributions can also guide evasion. Replacing the tokens that SHAP and LIME rank highest significantly reduced a single classifier’s ability to detect machine-generated text, although an ensemble detector kept strong performance across languages and domains [179]; a platform that showed its attributions to the users it moderates could hand them the same map.

7.4.2. Misinformation Detection

In fact-checking the explanation is the evidence. FEVER [240] and LIAR-PLUS [7] supply sentence-level evidence and justification paragraphs (Table 9, where FEVER is covered through ERASER). Kotonya and Toni [135] survey explanation generation for automated fact-checking and identify desirable properties for its explanations, and Atanasova et al. [18] show that training veracity prediction jointly with explanation generation improves the verdicts and, by human judgment of coverage and overall quality though not by ROUGE, the explanations. In LLM-based systems the evidence arrives as inline citations from retrieval-augmented generation, and benchmarks that score citation precision and recall alongside answer correctness find that fluent, well-cited answers often cite passages that do not support the sentence they are attached to [91], a problem Section 7.6 follows across domains.

7.5. Education and Learning

The EU AI Act lists AI systems that determine admission to education or evaluate learning outcomes as high-risk under Annex III [80], and their obligations now apply from 2 December 2027 (Section 7.6.1). The explanation a learner needs is feedback tied to a rubric, which in the terms of the taxonomy combines rationale spans with natural language. Among the works reviewed there is no explanation-annotated educational benchmark and no test of whether such feedback is faithful to the scoring model, so recommendations for this domain rest on the general framework of Section 6.

7.6. Cross-Domain Considerations

Where the law asks for reasons or for a reviewable basis, as for adverse credit actions, statements of reasons under the DSA, the basis of a clinical recommendation and the logic of an automated decision under the GDPR (Section 7.6.1), it asks for the reasons actually used, so an unfaithful explanation is a legal exposure as well as a scientific one; the credit worked example of Section 6 traces such a case. Where the person acting can check the output against a source, as a clinician checks a code against the note or a lawyer checks a cited case, a citation becomes a testable explanation. Its precision can be scored [91], deployed legal tools fail at measurable rates [164], and in an experiment on LLM question answering, showing sources lowered reliance on incorrect answers [129] (Section 5). A correct citation still does not show use: in a test on a RAG model, up to 57% of citations lacked faithfulness, meaning that the model had not relied on the cited document [256].

Where the person cannot check, as with a credit applicant, a moderated user or a learner, the explanation carries the weight alone. The two user studies in the table, in dermatology [269] and hate speech moderation [242], both find that the reader’s expertise or the explanation’s form shapes how people rely on the model, in line with the fifth main finding (Section 1). Social science research and education have no worked example and the thinnest evidence. Conversational assistants and business analytics, covered for CRM and churn prediction in the companion survey [178], inherit the citation problem of retrieval-augmented generation.

7.6.1. Regulatory Landscape

The EU AI Act (Regulation (EU) 2024/1689) [80] entered into force on 1 August 2024 and applies in stages: prohibitions since 2 February 2025, obligations for general-purpose AI models since 2 August 2025, and most other provisions since 2 August 2026, among them the Article 50 duties to disclose that a person is interacting with an AI system and to mark synthetic content, which require disclosure but no explanation. Generative systems already on the market before that date have until 2 December 2026 to meet the marking duty [81]. The Digital Omnibus on AI, Regulation (EU) 2026/1744, published on 24 July 2026 and in force since 27 July 2026, postponed the obligations for high-risk systems to 2 December 2027 for the stand-alone uses listed in Annex III, among them creditworthiness assessment and education, and to 2 August 2028 for AI in products covered by Annex I, among them medical devices [81]. Article 86 of the Act gives a person significantly and adversely affected by a decision based on an Annex III high-risk system the right to “clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken” [80]. Whether the GDPR [78] contains a right to explanation was long debated [222,254]. In Dun & Bradstreet Austria the Court of Justice of the EU held that Article 15(1)(h) gives a “genuine right to an explanation” of the “procedure and principles actually applied”, that neither communicating an algorithm nor describing every step of the decision satisfies it, that complexity does not relieve the controller of the duty, and that telling the person how far a change in their data would have changed the result may suffice [62]. The first holding comes close to a faithfulness requirement, the second suggests that handing over a raw reasoning trace would not count as an explanation, and the last matches the counterfactual output form of the taxonomy (Section 3). In the United States, Colorado repealed and reenacted its 2024 AI statute [58] as SB 26-189, signed on 14 May 2026. For consequential decisions made from 1 January 2027, in areas that include lending, health care and education, a deployer whose automated system materially influenced a decision with an adverse outcome must give the consumer, within 30 days, a plain-language description of the decision and of the role the system played in it [59]. The duty is to describe the system’s role, not to state its reasons.

8. Challenges and Future Directions

This section states six open problems, each with the evidence that it is still open and the test or result that would count as progress. Shorter notes on research directions and on responsible deployment follow.

8.1. Open Problems

1. A faithfulness criterion for free-text explanations. Comprehensiveness and sufficiency score an explanation by erasing it from the input, but a chain of thought or a self-explanation is an output of the model, so there is no input span to erase. The substitutes reviewed in Section 5.2 intervene on the trace [142], on the parameters that produced it [244], or on the prompt, by planting a biasing feature and checking whether the explanation mentions it [243]; others compare the concepts that change a prediction with those the explanation cites [168], or ask whether a rationale is associated with the label at all [262]. Each test presupposes a different causal property, and several of these presuppositions are disputed: the biasing-feature criterion may confuse unfaithfulness with incompleteness [276], edits to a trace are off-policy and give unstable effects in decision tasks [158], and many of the tests may measure only output-level self-consistency [202]. Progress would be a criterion stating which causal property of the model a free-text explanation must match, scored on testbeds that plant the property by training as well as by model editing [275], and that extend to reasoning models.

2. Ground truth for faithfulness metrics. What a trained model relied on for a given input is not observed, so faithfulness metrics are validated against other proxies, and Section 5 shows that the proxies disagree. Retraining [113] and in-distribution fine-tuning [163] remove the out-of-distribution confound of erasure without supplying the missing reference, sanity checks [4] rule methods out without ranking the ones that pass, and LLM judges bring position and verbosity biases of their own, and may favor their own answers [280]. A reference can be built where the mechanism is planted. An auditing game in which a hidden objective is trained into a model gives auditors a known answer [166], weight-sparse transformers yield circuits small enough to check by hand [90], MIB scores circuit and causal-variable localization on shared models and tasks [188], and Causal Diagnosticity plants faithful and unfaithful free-text explanations by model editing [275]. Planted references still need randomly initialized controls, on which both automated-interpretability scores and unconstrained causal-abstraction maps can look as good as on trained models [109,235] (Section 5.3). Progress would be a suite of models with planted mechanisms and randomly initialized controls, on which each faithfulness metric, and each LLM judge, is ranked by agreement with the planted answer, not by agreement with other metrics.

3. Attention-style explanations in deep, many-headed models.Section 4.6 sets out the evidence on both sides of the attention-explanation debate [36,121,261]. Three questions remain open. Wiegreffe and Pinter [261] proposed tests of when attention can serve as an explanation in recurrent models, but no checkable criterion has been established for many-headed transformers, so practice still oscillates between using attention uncritically and discarding it outright. How attention should be combined with gradient-based attribution, with rollout across layers [1], and with the context-mixing measures that partly supersede it [84,185] is also unsettled. The debate was also conducted on single-task encoders with a handful of heads, so its conclusions need not hold for decoder LLMs. There, attention concentrates on the initial tokens whatever their content [268], the decoder counterpart of the delimiter-token finding of Section 4.6, and attention-aware relevance propagation reports more faithful attributions than competing methods on LLaMA 2 and Mixtral [2]. We found no study, however, that compares raw attention, rollout and context-mixing scores against importance established by intervention across the depth of current LLMs. That comparison is the most tractable of the three questions, and it would show which attention-derived view, if any, survives at scale.

4. Keeping reasoning traces monitorable through training. Reasoning models such as o1 [198] and DeepSeek-R1 [65] write long chains of thought before answering, and for whoever can read the trace (the OpenAI and Anthropic APIs return at most a summary; Section 2) it is the main window on their behavior. Where Problem 1 asks whether one explanation is faithful, this problem asks whether a property of the trace survives training, and classical attribution had no analogue: a saliency map is computed by the analyst, while a trace is generated by the model and shaped by whatever training rewards. The evidence divides by the pressure applied. Training against a monitor that reads the trace, beyond a low-pressure regime, teaches the model to hide its intent while it keeps reward hacking [21]. Penalizing specific strings in the trace teaches models an encoding that generalizes to held-out strings of the same class [169], and even training against a monitor that sees only the output can make traces look safe [70]. Without such pressure the picture is better. An evaluation suite finds most frontier models fairly but not perfectly monitorable, and finds that RL optimization did not materially reduce monitorability in the two frontier training runs it tracked [100]. Reasoning models control what their traces contain far less well than what their outputs contain [274], and when the misbehavior requires multi-step reasoning, models evade a monitor only with substantial help such as human-written strategies [76]. Monitoring stays imperfect where the trace is not needed for the task: traces often omit hints the model used [54], and misleading rationalizations in the trace can lower a monitor’s detection of obvious sabotage, while a hybrid monitor that scores reasoning and actions separately beats both [13]. Other training choices matter too, with GRPO yielding more faithful chains than DPO in larger models [180], and a cross-laboratory position paper asks developers to weigh monitorability in training decisions [134]. Because the frontier-scale measurements come from the developers’ own laboratories [21,54,100], progress would be a monitorability metric of this kind tracked across the checkpoints of an open-weight reasoning model under each kind of pressure (monitor reward, process supervision, output supervision), with a planted exploit as the fixed target.

5. When dictionary features beat simple baselines. Sparse autoencoders and the methods built on them (Section 4.4) offer a basis of interpretable features, and the open question is on which tasks that basis buys something a simpler method does not. On the benchmarks reviewed there, simpler baselines match or beat SAE-based methods for probing [123] and for steering and concept detection [267]; selecting features by their effect on the output makes SAE steering competitive with the representation-based methods, though prompting still scores higher [10]. All of these tasks act on a concept someone has already named. Peng et al. [205] argue that SAEs are especially promising for the other kind of task, discovering concepts nobody thought to label, which a probe cannot do without labels. Open-ended auditing is such a task, and the evidence there is mixed (Section 4.4): SAEs were one of several techniques that led human teams to a hidden objective [166], but did not reliably help automated auditing agents in a report outside peer review, where natural-language autoencoders helped under some data-access conditions [88]. For circuit tracing a simple baseline is the neuron basis itself, which is as sparse as an SAE basis and yields a circuit for subject-verb agreement, a behavior earlier traced with dictionary features [14]. Any claimed advantage also needs two controls, an SAE trained on a randomly initialized transformer [109] and one trained with a different random seed [204]. Progress would be a named class of discovery tasks, evaluated on held-out instances with both controls, on which dictionary features beat the strongest simple baseline given the same supervision.

6. Explanations that improve decisions, not only trust. Across the pre-LLM [22] and LLM-era [23,129,242] user studies of Section 5.4.1, explanations raised acceptance of the model’s output, correct or not, more reliably than they raised the accuracy of the decisions people made. The exceptions to this pattern have two sources. As Bansal et al. [22] observe of earlier studies, gains came where the AI alone outperformed both the human and the team [138], so that more acceptance meant more correct answers. In image-based diagnosis, physicians stayed resilient to model errors [269], which fits the argument of Fok and Weld [87] that an explanation helps only insofar as it lets the person verify the AI’s answer; in a maze task whether explanations reduced over-reliance depended on the cost of doing the task and of checking the explanation [249]. LLMs may widen the gap, since a fluent free-text rationale reads as plausible whether or not it is faithful (Problem 1) and sways people’s plausibility judgments [201]. The target is warranted trust [120], and in one comparison of three interventions on LLM advice, reducing over-reliance did not generally make reliance appropriate [41]. Progress would be studies that compare an explanation condition against a confidence-only baseline, report team accuracy separately on the instances where the model is wrong, state how costly verification was for the participants, and treat complementary performance, a team that beats both the human and the model working alone, as the target result.

8.2. Research Directions

Mechanistic interpretability is moving in two directions. One automates the hand analysis behind early circuit results, from circuit discovery [60] to LLM-written descriptions of units (Section 4.3), and the same analysis can follow a model through training, as when progress measures derived from a reverse-engineered algorithm tracked grokking in a small transformer [192], the kind of checkpoint tracking Problem 4 asks for. The other changes the model and leaves the analysis as it is: transformers trained with mostly zero weights yield circuits for simple tasks that can be pruned out and read [90], though keeping such models interpretable beyond tens of millions of nonzero parameters is unsolved.

For reasoning, the firmest guarantee comes from constraining the format. Faithful CoT has the model write a symbolic chain that a deterministic solver executes, so the trace determines the answer [156]. Free-form traces carry no such guarantee, and they are the subject of Problems 1 and 4. Conversational explanation, where a user asks follow-up questions about an answer, inherits Problem 1 at every turn and adds consistency across turns. Multimodal models add to each problem the question of which modality an answer drew on.

8.3. Responsible Deployment

The regulation that bears on explanation is set out in Section 7. Several of its duties ask for the reasons that actually produced a decision: the Court of Justice of the EU reads the GDPR right of access as a right to an explanation of “the procedure and principles actually applied” [62], and US creditors must state the specific principal reasons for an adverse credit action (Section 7). The faithfulness tests of Problems 1 and 2 can therefore serve as compliance evidence as well as research instruments. Hendrycks et al. [110] place interpretability among the defenses against deceptive alignment and unexpected harmful behavior in their overview of catastrophic AI risks, a defense that depends on Problems 4 and 5.

9. Limitations

Three limitations qualify the conclusions of this survey. First, on selection: this is a structured narrative synthesis (Section 1.3), without the reproducible coverage of a PRISMA-style review. Relevant contributions may be missing, and the corpus carries the biases of its construction, skewing toward English-language work at major venues, so explainability research on other languages, at regional venues, and in unpublished industry practice is underrepresented. Coverage is also uneven across application domains. Section 7 keeps each domain brief. Social science research and education have no worked example in Section 6, and the evidence that explanations help their users is thinner there than in healthcare or content moderation. Several of the social-science and moderation examples also come from our own studies.

Second, regarding currency, parts of the surveyed literature are moving quickly, and mechanistic interpretability is moving fastest of all. Several results we discuss are industry technical reports that have not undergone peer review, including the sparse-autoencoder, circuit-tracing, introspection and natural-language-autoencoder findings from the Transformer Circuits series [45,88,149,151] and the system-card and alignment analyses of reasoning-trace faithfulness [54,198]. We flag these as non-peer-reviewed where they are discussed. Some now have peer-reviewed follow-ups: circuit tracing is available as an open library tested on models of up to 14B parameters [105], and the detection of injected concepts has been traced to a two-stage circuit in open-weight models [159]. Our statements about specific model families, available tooling, and the regulatory picture describe the state of affairs in October 2026 and will age faster than the survey’s conceptual framework.

Third, the decision frameworks of Section 6 are a reasoned synthesis, not an empirical ranking. For most pairs of methods the head-to-head comparison under a shared protocol that would settle an ordering has not been run, and Section 5 shows that the faithfulness metrics disagree with one another and that the benchmarks able to adjudicate them are only now appearing. We expose the reasoning at each decision point and trace it end to end on three deployments, so that a reader can see which recommendations rest on published results and which on our judgment, and can revise them as the benchmarks arrive.

10. Conclusions

10.1. Summary of Contributions

We have made three main contributions:

1. Unified Taxonomy: We presented a framework that locates any explanation method along four dimensions: scope (local vs. global), mechanism (how the explanation is computed), model access (model-agnostic vs. model-specific), and output form (importance scores, rules, examples, counterfactuals, rationale spans, concepts, natural language). Its seven mechanism values are perturbation, gradient/propagation, surrogate, internal inspection, intervention, prompting-based, and intrinsic, the last covering models designed to be interpretable. Classical attribution, attention-based analysis, and the current LLM toolbox all sit inside this one frame.

2. Practical Decision Frameworks: Section 6 turns the taxonomy into method choices. Its decision tree branches on model access first and on scope and audience after (Figure 2), and an access table extends it to LLMs and reasoning models (Table 10). Task-specific recommendations cover classification, sequence labeling, question answering, and generation, and the evaluation protocol comes with an overview of available tooling. Three worked examples trace the framework end to end on an API-only clinical deployment, a full-access moderation system, and a reasoning model under a legal duty to state reasons.

3. LLM-Era Interpretability: Section 4 brings the interpretability literature of the current model generation into the same frame: reasoning traces and what is known about their faithfulness and their monitorability, self-explanation and the open question of introspective access, dictionary learning and the circuit tracing built on top of it, and the long-running dispute over attention. The lineage table (Table 6) sets each technique against its closest classical counterpart. Ten of its twelve rows rebuild or extend an older idea; the chain-of-thought faithfulness tests, for instance, carry erasure over from the input to the reasoning trace. Only the monitorability of reasoning traces and introspective access have no classical counterpart.

10.2. Findings and Recommendations

Applied to the literature, the frame supports five findings. Table 14 sets out, for each, the main supporting evidence, the settings it comes from, the work that qualifies it, and where the evidence is thin.

1.

Faithfulness and plausibility come apart, and the erasure tests do not transfer to a chain of thought. Chains of thought and self-explanations can rationalize answers driven by cues they never mention, and people find such text persuasive. Testing them means intervening on the trace, on the parameters that produced it, or on the prompt and input (Section 5.2), with plausibility reported as a separate quantity.

2.

Access settles which mechanisms are usable before scope or audience matter. Behind an API, gradient/propagation, internal inspection and intervention are out of reach, so the decision framework asks about access first and selects by scope and audience among what remains (Section 6).

3.

Most LLM-era techniques rebuild or extend older ideas. For the ten rows of Table 6 with a classical counterpart, the counterpart’s failure modes and tests carry over. The two without one, the monitorability of reasoning traces under optimization pressure and introspective access, have no inherited methods, and monitorability has to be measured through training (Problem 4 in Section 8.1).

4.

Faithfulness metrics disagree, and dictionary features do not yet beat simple baselines. A single score therefore says little. An evaluation should name the tests it ran and on which model, and run at least two that fail in different ways, such as an erasure-based score checked against a retraining-based one (Section 5).

5.

Explanations raise acceptance of model outputs more reliably than they raise decision quality. Extrinsic evaluation should measure the decisions people make with an explanation, report accuracy separately on the instances where the model is wrong, and state how costly it was for people to verify the model’s answer.

10.3. Looking Forward

For researchers, Section 8.1 states six open problems, from a faithfulness criterion for free-text explanations to explanations that improve decisions and not only trust. Reasoning models make the first and the fourth of these pressing: their chains of thought can omit the very cues that drive their answers [54], so neither the faithfulness of a trace nor its value to a monitor can be assumed.

For practitioners in healthcare, legal and financial services, social science research, content moderation, and education, the output form should match the audience that will act on the explanation (Table 13), and the user studies behind Finding 5 suggest that an explanation helps most when it lets that audience check the model’s answer. Where the law requires reasons, as for automated decisions under the GDPR [62] and for adverse credit decisions, the stated reasons have to be the ones actually applied, so the faithfulness tests of Section 5 can serve as compliance evidence (Section 6 and Section 7).

10.4. Final Remarks

Explainable NLP now spans two literatures that grew up apart, one built around attributing predictions to inputs and one around reverse-engineering computations from weights. Placing them in a single frame is what lets a practitioner see that the choice between them is usually made for them by access, and lets a researcher see which LLM-era techniques carry an older idea forward and which problems, such as keeping a reasoning trace monitorable, arrived with the models themselves (Table 6).