Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

1. Introduction

Large Language Models (LLMs) have recently achieved remarkable success, demonstrating outstanding performance across a wide spectrum of applications, ranging from complex reasoning and multilingualism, to highly specialized domains [264,265,266,267,268,269,270,271,272,273,274,275]. Despite these advancements, a critical challenge remains: the internal decision-making processes of these models are largely opaque, often operating as “black boxes.” This lack of transparency poses significant risks, particularly in safety-critical applications, and severely limits our ability to efficiently debug, control, and optimize model behaviors [276,277,278]. Consequently, Mechanistic Interpretability (MI) has emerged as a pivotal research direction. Unlike traditional behavioral analysis, MI aims to “reverse-engineer” these complex neural networks, decomposing their intricate computations into understandable components and causal mechanisms [47,279,280].

Current research in this field generally falls into two categories. A significant body of work focuses on the theoretical and foundational aspects of MI [47,279,280,281,282,283,284,285,286]. These studies provide technical roadmaps for dissecting Transformer architectures and identifying fundamental units. However, they primarily prioritize scientific discovery—aiming to elucidate the model’s inner working mechanisms for the sake of understanding itself. They typically treat MI as an observational science, leaving the question of how to translate these microscopic insights into practical model improvements underexplored.

Recognizing the applied potential of interpretability, a second line of work has begun to bridge the gap between theoretical understanding and practical utilization. These surveys discuss how MI techniques can be leveraged to aid downstream tasks or assist in specific domains [287,288,289,290,291,292,293]. However, despite their contributions, these existing reviews face two primary limitations that hinder broader adoption. First, they often lack a sufficient categorization and clear definition of MI methods within practical application contexts. The distinction between diagnostic tools and intervention techniques is frequently blurred. Second, their coverage of applications is often incomprehensive, and the illustration of methods is typically too general. This high-level abstraction makes it difficult for researchers to translate theoretical mechanistic insights into actionable interventions for specific problems. Consequently, there is a distinct lack of a unified guide that systematically categorizes these methods and clearly presents a concrete pipeline for active model improvement.

To fill this gap, we propose the “Locate, Steer, and Improve” pipeline. This conceptual framework is designed to systematically transform MI from a passive observational science into an actionable intervention discipline. Our work makes the following key contributions:

• 1) A Rigorous Pipeline-Driven Framework: We establish a structured framework for applying MI to real-world model optimization. We begin by defining the core Interpretable Objects within LLMs (e.g., neurons, attention heads, residual streams). Based on the application workflow, we clearly categorize methodologies into two distinct stages: Localizing (Diagnosis), which identifies the causal components responsible for specific behaviors, and Steering (Intervention), which actively manipulates these components to alter model outputs. Crucially, for each technique, we provide a detailed Methodological Formulation along with its Applicable Objects and Scope, helping readers quickly understand the technical implementation and appropriate use cases.
• 2) Comprehensive Paradigms for Application: We provide an extensive survey of MI applications organized around three major themes: Improve Alignment, Improve Capability, and Improve Efficiency. These themes cover eight specific scenarios, ranging from safety and multilingualism to efficient training. Instead of merely listing relevant papers, we summarize representative MI application paradigms for each scenario. This approach allows readers to quickly capture the distinct usage patterns of MI techniques across different application contexts, facilitating the transfer of methods to new problems.
• 3) Insights, Resources, and Future Directions: We critically discuss the current challenges in actionable MI research and outline promising future directions. To facilitate further progress and lower the barrier to entry, we curate a comprehensive collection of over 200 papers, which are listed in Table 2. These papers are systematically tagged according to their corresponding localizing and steering methods, providing a practical and navigable reference for the community.

2. Core Interpretable Objects of LLMs

In this section, we establish a unified mathematical formulation for the core interpretable objects within LLMs. We focus specifically on the decoder-only Transformer architecture [294], which serves as the predominant framework for contemporary state-of-the-art models [266,267,275]. We present the core interpretable objects and their corresponding mathematical notations in Table 1.

2.1. Token Embedding

The entry point of the model maps discrete tokens from a vocabulary $\mathcal{V}$ to continuous vector representations. We define the Embedding Matrix as ${\mathbf{W}}_E\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times d_{\mathrm{model}}}$, where $\left|\mathcal{V}\right|$ denotes the vocabulary size and $d_{\mathrm{model}}$ represents the hidden dimension of the model. For a given input token $t_i$ at position i, its Token Embedding—which also serves as the initial state of the residual stream, denoted as ${\mathbf{x}}_i^0$—is obtained by retrieving the corresponding vector from ${\mathbf{W}}_E$ and adding positional information:

$${\mathbf{x}}_i^0={\mathbf{W}}_E\left[t_i\right]+{\mathbf{p}}_i$$

(1)

where ${\mathbf{p}}_i$ is the positional embedding vector. It is worth noting that while earlier architectures used absolute positional embeddings added at the input, modern LLMs [266,267,275] typically employ Rotary Positional Embeddings (RoPE) [295]. In these architectures, positional information is applied directly to the query and key vectors within the attention mechanism rather than to the residual stream at the embedding layer.

2.2. Transformer Block and Residual Stream

Typically, an LLM is composed of L stacked layers. Each layer l consists of two primary blocks: a Multi-Head Attention (MHA) block and a Feed-Forward Network (FFN) block. The fundamental communication channel connecting these blocks is the residual stream.

As illustrated in Figure 2, the residual stream acts as the central “highway” for information propagation [43,66,116,144,296]. It preserves a shared memory state that is iteratively updated by the blocks. The update dynamics for the residual stream state ${\mathbf{x}}^l$1 at layer l are defined as follows:

$$\begin{array}{cc}\hfill {\mathbf{x}}^{l,\mathrm{mid}}& ={\mathbf{x}}^l+{\mathbf{h}}_{\mathrm{attn}}^l\left({\mathbf{x}}^l\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\mathbf{x}}^{l+1}& ={\mathbf{x}}^{l,\mathrm{mid}}+{\mathbf{h}}_{\mathrm{ffn}}^l\left({\mathbf{x}}^{l,\mathrm{mid}}\right)\hfill \end{array}$$

(3)

where ${\mathbf{x}}^{l,\mathrm{mid}}$ represents the intermediate state after the MHA block but before the FFN block.2

This additive structure—where ${\mathbf{x}}^{l+1}={\mathbf{x}}^l+\mathrm{MHA}\left({\mathbf{x}}^l\right)+\mathrm{FFN}\left({\mathbf{x}}^l\right)$—is critical for MI analysis. It implies that features in the residual stream can be viewed as linear combinations of outputs from all previous components. This property enables the decomposition of the model’s final prediction into individual component contributions, facilitating methods like “Logit Lens” [93,107,297] and causal mediation analysis [43,49,120,122,144].

2.3. Multi-Head Attention (MHA)

The Multi-Head Attention mechanism allows tokens to contextualize information by attending to other positions in the sequence. It consists of H independent heads, which primarily manage information routing and the resolution of contextual dependencies [116,117,298,299,300].

Standard Formulation

For a specific head h at layer l, we define the learnable weight matrices as ${\mathbf{W}}_Q^{l,h},{\mathbf{W}}_K^{l,h},{\mathbf{W}}_V^{l,h}\in {\mathbb{R}}^{d_{\mathrm{model}}\times d_{\mathrm{head}}}$ and the output projection matrix as ${\mathbf{W}}_O^{l,h}\in {\mathbb{R}}^{d_{\mathrm{head}}\times d_{\mathrm{model}}}$. Here, T denotes the sequence length, the attention mechanism first computes the attention score matrix ${\mathbf{A}}^{l,h}\in {\mathbb{R}}^{T\times T}$, which represents the relevance of each token to every other token:

$${\mathbf{A}}^{l,h}=\mathrm{softmax}\left({\displaystyle \frac{\left({\mathbf{x}}^l{\mathbf{W}}_Q^{l,h}\right){\left({\mathbf{x}}^l{\mathbf{W}}_K^{l,h}\right)}^{\top }}{\sqrt{d_{\mathrm{head}}}}}+\mathbf{M}\right),$$

(4)

where $\mathbf{M}\in {\mathbb{R}}^{T\times T}$ denotes the attention mask that prevents attention to invalid positions (e.g., future tokens in causal attention or padding tokens).

Functionally, attention heads “read” information from the residual stream of previous tokens via the query–key subspace projections, and then “write” the attended information back to the current position via the value and output projections. The output for a single head h, denoted as ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$, is computed as:

$${\mathbf{h}}_{\mathrm{attn}}^{l,h}=\left[{\mathbf{A}}^{l,h}\left({\mathbf{x}}^l{\mathbf{W}}_V^{l,h}\right)\right]{\mathbf{W}}_O^{l,h}.$$

(5)

The total output of the MHA block is the sum of the outputs from all H heads: ${\mathbf{h}}_{\mathrm{attn}}^l={\sum }_{h=1}^H{\mathbf{h}}_{\mathrm{attn}}^{l,h}$.

Mechanistic View: QK and OV Units

While the standard formulation describes how attention is computed, the unit perspective [116] offers deeper insight into what task each head performs. As illustrated in the detailed view of Figure 2, each head can be decomposed into two functionally distinct units:

1) The QK Unit (${\mathbf{W}}_{QK}$): This unit determines where to attend. By merging the query and key matrices into a single low-rank matrix ${\mathbf{W}}_{QK}^{l,h}={\mathbf{W}}_Q^{l,h}{\left({\mathbf{W}}_K^{l,h}\right)}^{\top }$, the attention pattern depends directly on the interaction between residual stream states. The attention score $a_{i,j}^{l,h}$ (e.g., $a_{3,1}$ in Figure 2) is derived from the bilinear form ${\left({\mathbf{x}}_i^l\right)}^{\top }{\mathbf{W}}_{QK}^{l,h}{\mathbf{x}}_j^l$.

2) The OV Unit (${\mathbf{W}}_{OV}$): This unit determines what information is transmitted. By merging the value and output matrices into ${\mathbf{W}}_{OV}^{l,h}={\mathbf{W}}_V^{l,h}{\mathbf{W}}_O^{l,h}$, we can view the head’s operation as reading a vector from the source token j, transforming it linearly via ${\mathbf{W}}_{OV}^{l,h}$, and adding it to the destination token i, weighted by the attention score. This separation allows researchers to classify heads into distinct roles, such as “Induction Heads” (which copy previous tokens) or “Previous Token Heads” [33,34,117].

2.4. Feed-Forward Network (FFN)

Standard Formulation

The Feed-Forward Network block acts as a position-wise feature transformer. Unlike attention heads, FFNs operate independently on each token position, applying non-linear transformations to the input. They are often conceptualized as “Key-Value” memories, where the first layer projects the stream into a high-dimensional state (detecting patterns or “Knowledge Keys”) and the second layer writes the retrieved knowledge back to the stream [64,94,301].

Mathematically, the output of the FFN block ${\mathbf{h}}_{\mathrm{ffn}}^l$ is given by :3

$${\mathbf{h}}_{\mathrm{ffn}}^l=\sigma \left({\mathbf{x}}^{l,\mathrm{mid}}{\mathbf{W}}_{\mathrm{in}}^l\right){\mathbf{W}}_{\mathrm{out}}^l$$

(6)

where ${\mathbf{x}}^{l,\mathrm{mid}}$ is the input to the FFN, and $\sigma$ is a non-linear activation function. The weight matrices are defined as ${\mathbf{W}}_{\mathrm{in}}^l\in {\mathbb{R}}^{d_{\mathrm{model}}\times d_{\mathrm{ffn}}}$ and ${\mathbf{W}}_{\mathrm{out}}^l\in {\mathbb{R}}^{d_{\mathrm{ffn}}\times d_{\mathrm{model}}}$.

Mechanistic View: Neurons

In this context, the neuron j is defined as an atomic unit comprised of a pair of weights: the key weight ${\mathbf{k}}_j^l$ (the j-th row of ${\mathbf{W}}_{\mathrm{in}}^l$) and the value weight${\mathbf{v}}_j^l$ (the j-th column of ${\mathbf{W}}_{\mathrm{out}}^l$). The intermediate state ${\mathbf{s}}^l=\sigma \left({\mathbf{x}}^{l,\mathrm{mid}}{\mathbf{W}}_{\mathrm{in}}^l\right)$ represents the vector of neuron activation.

2.5. Sparse Autoencoder (SAE) Feature

While the internal objects described above (e.g., neuron activation ${\mathbf{s}}^l$ or residual stream state ${\mathbf{x}}^l$) are fundamental to the model’s operation, they are often polysemantic. This is due to the phenomenon of superposition, where neural networks represent more features than they have physical neurons by encoding them as nearly orthogonal directions in the high-dimensional activation space [303]. Consequently, a single neuron may activate for multiple unrelated concepts, making direct interpretation difficult.

Sparse Autoencoders (SAEs) provide a principled method to resolve this by disentangling dense, polysemantic representations into monosemantic features [127]. As illustrated in Figure 3, an SAE acts as a “microscope” for the LLM. It projects low-dimensional dense activations into a higher-dimensional sparse latent space, effectively “unpacking” the superposition.

Mathematical Formulation

SAEs are trained in a layer-wise manner as independent modules attached to a specific object of a frozen LLM. They can be applied to nearly all internal objects, including neuron activation ${\mathbf{s}}^l$, residual stream state ${\mathbf{x}}^l$, MHA output ${\mathbf{h}}_{\mathrm{attn}}^l$, and FFN output ${\mathbf{h}}_{\mathrm{ffn}}^l$ [304,305]. For instance, when applying an SAE to reconstruct a residual stream state ${\mathbf{x}}^l$, the forward pass is defined as:

$$\begin{array}{cc}\hfill \mathbf{a}& =\sigma ({\mathbf{x}}^l{\mathbf{W}}_{\mathrm{enc}}+{\mathbf{b}}_{\mathrm{enc}})\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\widehat{\mathbf{x}}}^l& =\mathbf{a}{\mathbf{W}}_{\mathrm{dec}}+{\mathbf{b}}_{\mathrm{dec}}\hfill \end{array}$$

(8)

where ${\mathbf{W}}_{\mathrm{enc}}\in {\mathbb{R}}^{d_{\mathrm{model}}\times d_{\mathrm{SAE}}}$ and ${\mathbf{W}}_{\mathrm{dec}}\in {\mathbb{R}}^{d_{\mathrm{SAE}}\times d_{\mathrm{model}}}$ are learnable weights. A critical hyperparameter here is the Expansion Factor—the ratio of $d_{\mathrm{SAE}}$ to $d_{\mathrm{model}}$. To capture the vast number of features hidden in superposition, $d_{\mathrm{SAE}}$ is typically set to be $16\times$ to $128\times$ larger than the model dimension [177,304,305,306,307,308,309].

The training objective is to minimize the reconstruction error while enforcing sparsity on the latent activations $\mathbf{a}$:

$$\mathcal{L}=\parallel {\mathbf{x}}^l-{\widehat{\mathbf{x}}}^l{\parallel }_2^2+\lambda {\parallel \mathbf{a}\parallel }_1$$

(9)

In this framework, the SAE feature ${\mathbf{f}}_j$ (the j-th row of ${\mathbf{W}}_{\mathrm{dec}}$) represents a distinct semantic direction in the activation space. The SAE feature activation$a_j$ (the j-th element of $\mathbf{a}$) quantifies the strength of this feature in the current input. Crucially, this decomposition transforms opaque vectors into an actionable vocabulary, allowing researchers to steer model behavior by targeting these granular, interpretable features [171,177,189,304,310,311].

Training Challenges and Resources

Training high-quality SAEs presents unique challenges. One major issue is Dead Latents, where many feature neurons never activate during training, effectively wasting capacity. Techniques such as ghost gradients or periodic resampling are commonly employed to mitigate this [115,127]. Another challenge is Feature Absorption, where broad, high-frequency features suppress specific, low-frequency ones. Advanced architectures like Gated SAEs, Top-K SAEs, and BatchTopK SAEs have been proposed to improve feature quality and reconstruction fidelity [312,313,314,315,316].

To facilitate research and reduce computational barriers, several high-quality pre-trained SAE suites have been released. Notable examples include Gemma Scope [304], Llama Scope [305], and “Golden Gate Claude” features [177]. These resources enable the community to focus on localizing and steering without incurring the cost of training SAEs from scratch.

3. Localizing Methods

Localizing Methods aim to identify interpretable objects that are responsible for a particular behavior or encode specific information. These techniques serve as a diagnostic step to narrow down the search space to manageable functional units. By pinpointing key components such as specific neurons, attention heads, or SAE features, they provide the necessary foundation for subsequent detailed mechanism analysis and targeted model steering.

3.1. Magnitude Analysis

Methodological Formulation

Magnitude Analysis methods serve as a fundamental heuristic in interpretability, operating on the premise that internal elements with larger numerical values often exert greater influence on the model’s computation. It scores internal objects via a scalar function to identify salient components [1,2,3].

Formally, consider a set of internal objects $\mathcal{O}=\{o_1,o_2,\dots ,o_N\}$, where each $o_j$ represents a candidate element (e.g., a specific weight parameter row, a neuron, an SAE feature, or an attention head). We define an Importance Score $s_j$ for each object using a magnitude function $f\left(·\right)$:

$$s_j=f\left(o_j\right),\mathrm{e}.\mathrm{g}.,s_j={\parallel o_j\parallel }_p\mathrm{or}{s}_j=\underset{k}{max}\left|{\left(o_j\right)}_k\right|$$

(10)

Common choices for $f\left(·\right)$ include the $L_2$-norm (${\parallel ·\parallel }_2$) to measure the aggregate energy, the $L_{\infty }$-norm (max-value) to capture peak activation, or frequency-based metrics. Based on these scores, a subset of salient objects ${\mathcal{O}}_{\mathrm{salient}}$ is selected for further inspection or intervention, typically via a thresholding mechanism or a top-k ranking:

$${\mathcal{O}}_{\mathrm{salient}}=\{o_j\mid s_j\ge \tau \}\mathrm{or}\underset{j\in \{1,\dots ,N\}}{arg\; topk}{s}_j$$

(11)

Applicable Objects

This method applies broadly to both static structure and dynamic computation. We categorize the applicable objects as follows:

1) Static Parameters: In the context of model weights, Magnitude Analysis is often used to identify outliers or “heavy hitters” without running inference. Researchers typically compute per-weight or per-row norms of weight matrices (e.g., $\parallel {\mathbf{W}}_{\mathrm{in}}[j,:]\parallel$) to highlight parameters that dominate the inner product computations. These high-magnitude weights are often associated with critical knowledge storage or outlier features [1,4,5,6,9,15,16,17,18,19,20,21,22,23].

2) Dynamic Components (Neurons, SAE Features, or Attention Heads): For functional units whose activity varies with input, ranking them by their activation statistics helps localize specialized capabilities [2,3,10,24,25,26,27,28,29,30].

• Specialized Neurons and SAE Features: By feeding domain-specific datasets into the model and monitoring activations (e,g., neuron activation state ${\mathbf{s}}^l$ or SAE feature activation state $\mathbf{a}$), researchers can isolate components dedicated to specific concepts. For instance, in the context of higher-level reasoning, Galichin et al. [3] utilized SAEs to disentangle the residual stream state ${\mathbf{x}}^l$. As shown in Figure 4 (a), they proposed a metric called ReasonScore, which aggregates the activation frequency and magnitude of SAE features $a_j$ specifically during “reasoning moments” (e.g., when the model meets tokens like “Wait or “Therefore”). By ranking features based on this score, they successfully localized Reasoning-Relevant SAE features that encode abstract concepts like uncertainty or exploratory thinking. Similarly, for style transfer, Lai et al. [24] employed Magnitude Analysis to identify Style-Specific Neurons. As illustrated in Figure 4 (b), they calculated the average activation magnitude of FFN neurons across corpora with distinct styles (e.g., positive vs. negative). Neurons that exhibited significantly higher average activation for the source style compared to the target style were identified as “Source-Style Neurons,” serving as candidates for subsequent deactivation.
• Attention Heads: The magnitude and distribution of attention scores (${\mathbf{A}}^{l,h}$) serve as a direct indicator of a head’s functional role [10,31,32,33,34,35,36]. For instance, Zhou et al. [35] introduced the Safety Head ImPortant Score (Ships), which aggregates attention weights on refusal-related tokens to localize “Safety Heads” critical for model alignment. In the multimodal domain, Sergeev and Kotelnikov [36] and Bi et al. [10] measured the concentration of attention mass on image tokens versus text tokens, successfully pinpointing heads responsible for visual perception and cross-modal processing. Similarly, Singh et al. [33] measured “induction strength”—derived from the attention probability assigned to token repetition patterns—to track the formation and importance of Induction Heads.

3) Layer-wise Representations: Furthermore, measuring the magnitude of layer-wise distances reveals structural roles. Comparing representations across contrastive inputs (e.g., $\parallel {\mathbf{x}}^l-{\mathbf{x}}^{\prime l}\parallel$) localizes layers where task-specific information diverges most strongly [11,12,37,38], whereas comparing consecutive layers (e.g., $\parallel {\mathbf{x}}^l-{\mathbf{x}}^{l+1}\parallel$) identifies layers with minimal state updates, pointing to redundant computation [13,14,39,40,41].

Characteristics and Scope

The scope of Magnitude Analysis for dynamic quantities is characterized as training-free but data-dependent.

• Advantages: It does not require training auxiliary classifiers or performing computationally expensive backward passes. This makes it highly scalable and suitable for analyzing large models in real-time.
• Limitations: It serves primarily as a lightweight heuristic. High activation magnitude implies high presence but does not guarantee causal necessity (e.g., a high-magnitude feature might be cancelled out by a subsequent layer). Furthermore, its success relies heavily on the quality of the input data; if the dataset fails to elicit the specific behavior, the relevant components will remain dormant. Therefore, Magnitude Analysis is typically used as a “first-pass" screening tool to filter candidate objects for more rigorous verification methods.

3.2. Causal Attribution

Methodological Formulation

Causal Attribution methods constitute the gold standard for localization in MI. Unlike correlation-based analyses, these techniques identify which internal objects are causally responsible for a specific model behavior by systematically measuring the effect of controlled interventions [42,43,44,45,46,47,48,49].

Formally, let $F\left(·\right)$ denote a scalar model output of interest, such as the logit or probability of a target token. Let o be an internal object (e.g., a neuron activation $s_j^l$ or a head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$) defined in §2. To evaluate the causal effect of o, we compare the model’s output under a counterfactual intervention against the baseline state:

$$do(o\leftarrow \tilde{o}):\Delta F\left(o\right)=F\left(do(o\leftarrow \tilde{o})\right)-F\left(o\right)$$

(12)

where $F\left(o\right)$ represents the model’s behavior in the standard “clean” run, and $do(o\leftarrow \tilde{o})$ represents the intervention where the object o is forced to take on a modified value $\tilde{o}$ while all other causal factors are held constant (ceteris paribus). The intervention typically takes two forms: Patching (where $\tilde{o}$ is an activation computed from a counterfactual input) or Ablation (where $\tilde{o}$ is set to zero or a mean vector). A large magnitude $|\Delta F(o\left)\right|$ indicates that the object o acts as a critical mediator or information node for the behavior encoded by F.

Applicable Objects

This analysis primarily targets dynamic objects involved in the inference process, including the residual stream state ${\mathbf{x}}^l$, the output of FFN ${\mathbf{h}}_{ffn}^l$, and the output of specific attention head ${\mathbf{h}}_{attn}^{l,h}$.

1) Patching (Interchange Intervention): This approach replaces an object computed from the original input with one computed from a counterfactual input to isolate specific information pathways. By systematically patching across layers and positions, one can localize exactly where task-specific information (e.g., factual knowledge) is introduced or transformed [43,44,49,50,51].

We exemplify this mechanism using Causal Tracing [43], a representative technique designed to localize factual associations (e.g., “The Space Needle” → “Seattle”). As illustrated in Figure 5, this process involves three key steps:

• Corrupted Run (Intervention): First, the specific knowledge is erased from the model’s computation. A corrupted input is created by adding Gaussian noise to the embeddings of the subject tokens (e.g., “Space Needle”), causing the probability of the correct prediction (“Seattle”) to drop significantly.
• Patched Run (Restoration): The core operation systematically restores specific internal states. For a specific layer l and token position i, the method copies the hidden activation from a separate original clean run and pastes (restores) it into the corrupted computation graph.
• Effect Measurement: The causal effect is quantified by the Indirect Effect (IE), which measures how much of the original target probability is recovered by this restoration. A high IE score implies that the patched state at $(l,i)$ carries critical information.

Through this rigorous process, Meng et al. [43] revealed that factual recall relies on two distinct localized mechanisms: an early retrieval phase in the FFN blocks at subject tokens, and a late information transport phase in the MHA blocks at the final token.

2) Ablation (Knockout): Alternatively, ablation-based attribution explicitly “zeros out” or removes objects, such as masking specific attention heads ${\mathbf{h}}_{attn}^{l,h}$ or neurons, and measures the resulting performance drop to determine their causal necessity [2,46,52,53,54]. This rigorous verification has been applied across various domains: Wang et al. [52] and Yu and Ananiadou [46] employed ablation to isolate minimal heads responsible for indirect object identification and in-context learning, respectively. In the context of specialized capabilities, Yu and Ananiadou [54] utilized pruning (permanent ablation) to identify heads critical for arithmetic reasoning, while Tang et al. [2] masked specific neurons to demonstrate the existence of language-specific functional regions. Furthermore, Geva et al. [53] applied blocking interventions to dissect the precise roles of FFN value vectors in factual recall mechanisms.

Characteristics and Scope

The scope of Causal Attribution is characterized as rigorously causal but computationally intensive.

• Advantages: Unlike Magnitude Analysis (§3.1), which only establish correlation, Causal Attribution provides definitive evidence that a component is a functional driver of the model’s output. This allows researchers to distinguish essential mechanisms from features that are highly activated but causally irrelevant to the specific behavior.
• Limitations: This rigor incurs a significant computational overhead. Verifying causality typically requires intervening on objects individually and performing a separate forward pass for each intervention. Consequently, the cost scales linearly with the number of objects analyzed, making it prohibitively expensive for dense, sweeping searches over large models. This inefficiency often necessitates the use of Gradient Detection (§3.3), which utilizes gradients to rapidly approximate these causal effects, enabling efficient screening before performing expensive, fine-grained interventions.

3.3. Gradient Detection

Methodological Formulation

Gradient Detection methods localize influential internal objects by scoring them with the sensitivity of a scalar target $F\left(x\right)$ (e.g., a logit, margin, or loss) with respect to an object $o_j$: $s_j\left(x\right)=\varphi ({\nabla }_{o_j}F\left(x\right),o_j)$, where common instantiations include the gradient norm $s_j=\parallel {\nabla }_{o_j}F\left(x\right)\parallel$ and the gradient–input score $s_j={\nabla }_{o_j}F{\left(x\right)}^{\top }{o}_j$ [55,56]. These scores serve as fast, first-order proxies for intervention effects. Specifically, under an additive modification $o_j\mapsto o_j+\Delta o_j$, a first-order Taylor expansion yields

$$F(o_j+\Delta o_j)-F\left(o_j\right)={\nabla }_{o_j}F{\left(x\right)}^{\top }\Delta o_j+\mathcal{O}(\parallel \Delta o_j{\parallel }^2),$$

(13)

indicating that the dot product ${\nabla }_{o_j}F{\left(x\right)}^{\top }\Delta o_j$ represents the directional derivative of F along $\Delta o_j$. A common local “removal” surrogate sets $\Delta o_j=-o_j$, giving $F(o_j-o_j)-F\left(o_j\right)\approx -{\nabla }_{o_j}F{\left(x\right)}^{\top }{o}_j$, which motivates using ${\nabla }_{o_j}F{\left(x\right)}^{\top }{o}_j$ (or its magnitude) as a signed influence score.

To mitigate saturation and explicitly model the notion of “absence,” Integrated Gradients (IG) attribute the change from a baseline ${\tilde{o}}_j$ to the input $o_j$ by integrating gradients along the straight-line path $\gamma \left(\alpha \right)={\tilde{o}}_j+\alpha (o_j-{\tilde{o}}_j)$:

$${\mathrm{IG}}_k(o_j;{\tilde{o}}_j)={(o_j-{\tilde{o}}_j)}_k{\int }_0^1{\displaystyle \frac{\partial F\left(\gamma \right(\alpha \left)\right)}{\partial {\gamma }_k}}d\alpha ,\sum _k{\mathrm{IG}}_k(o_j;{\tilde{o}}_j)=F\left(o_j\right)-F\left({\tilde{o}}_j\right),$$

(14)

where k indexes the components of $o_j$, and each ${\mathrm{IG}}_k$ quantifies the contribution of the k-th component to the output difference $F\left(o_j\right)-F\left({\tilde{o}}_j\right)$. In practice, the integral is approximated by an m-step Riemann sum over $\alpha =t/m$ [56]. Scores are typically computed over a dataset $\mathcal{D}$ and aggregated to stabilize rankings (e.g., ${\mathbb{E}}_{x\sim \mathcal{D}}\left[s_j\left(x\right)\right]$ or $\mathbb{E}\left[\right|s_j\left(x\right)\left|\right]$), without explicitly applying perturbations during scoring.

Applicable Objects

Because F is differentiable with respect to any internal object $o_j$ (Table 1), Gradient Detection applies uniformly across inputs, activations, and parameters. Below we expand the object categories and make explicit the correspondence between symbols and the underlying model components.

1) Inputs and Layer-wise States (${\mathbf{x}}_i^0,{\mathbf{x}}^l$): For input embeddings ${\mathbf{x}}_i^0$ and the residual stream state ${\mathbf{x}}^l$, gradients directly quantify how sensitive $F\left(x\right)$ is to changes in specific prompt components and their propagated representations. In practice, one computes ${\nabla }_{{\mathbf{x}}_i^0}F\left(x\right)$ or ${\nabla }_{{\mathbf{x}}^l}F\left(x\right)$ and derives token-level influence, such as the gradient norm $\parallel {\nabla }_{{\mathbf{x}}_i^0}F\left(x\right)\parallel$, the gradient–input score ${\nabla }_{{\mathbf{x}}_i^0}F{\left(x\right)}^{\top }{\mathbf{x}}_i^0$, or integrated gradients [59]. Aggregating these scores across positions i (optionally across layers l) yields a ranked view of which tokens or contextual spans are most responsible for a target output, as used to analyze CoT prompting [60] and which depth regions contribute most strongly to the formation of that output [61], with closely related layer-/token-saliency signals also supporting dynamic token pruning [62] and inference-time steering [63].

2) Intermediate Outputs: Beyond inputs, Gradient Detection can score internal computational units whose activations vary with the input.

• Neurons (${\mathbf{s}}^l$): A standard neuron-level object is the FFN activation vector ${\mathbf{s}}^l$ at layer l. Gradients ${\nabla }_{{\mathbf{s}}^l}F\left(x\right)$ can be converted into per-neuron scores to rank neurons by their local influence on F. This has been used to localize knowledge- or context-sensitive neurons and analyze their dependencies [64,65,66,67,68,69]. Figure 6 illustrates a concrete LLM-specific instance: Shi et al. [65] computes Integrated Gradients scores to identify neurons most responsible for processing contextual cues under knowledge conflicts (via a context-aware attribution and a high-score intersection criterion), and then reweights the identified neurons to promote context-consistent generation.
• Attention Head Outputs (${\mathbf{h}}_{\mathrm{attn}}^{l,h}$):Gradient Detection also applies to attention-related activations such as the attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$. Computing ${\nabla }_{{\mathbf{h}}_{\mathrm{attn}}^{l,h}}F\left(x\right)$ and scalarizing it with $s_j\left(x\right)=\varphi ({\nabla }_{{\mathbf{h}}_{\mathrm{attn}}^{l,h}}F\left(x\right),{\mathbf{h}}_{\mathrm{attn}}^{l,h})$ yields head-level rankings that can highlight salient heads or attention submodules for further analysis and subsequent intervention [70,71,72].

3) Parameters (${\mathbf{W}}_{Q/K/V/O}^{l,h},{\mathbf{W}}_{\mathrm{in}/\mathrm{out}}^l$): Because F is differentiable with respect to model weights, Gradient Detection can score parameters at multiple granularities. At the block level, common targets include attention projection matrices ${\mathbf{W}}_{Q/K/V/O}^{l,h}$ and FFN matrices ${\mathbf{W}}_{\mathrm{in}/\mathrm{out}}^l$. Gradients such as ${\nabla }_{{\mathbf{W}}_Q^{l,h}}F\left(x\right)$ can be turned into scalar salience measures (e.g., $\parallel {\nabla }_{\mathbf{W}}F\left(x\right)\parallel$) to rank influential attention/FFN modules [70,71,72]. At finer granularity, the same principle is used to select influential individual weights [65,73] or structured blocks [74,75].

Characteristics and Scope

The scope of Gradient Detection is data-dependent and defined relative to the analyst’s target F, so rankings can shift under alternative objectives (e.g., $-logp\left(y^{★}\right|x)$[73,74], logit margins ${\mathrm{logit}}_y-{\mathrm{logit}}_{y^{\mathrm{foil}}}$[44,77], or contrastive/counterfactual gaps $|{\mathrm{logit}}_y\left(x\right)-{\mathrm{logit}}_y\left(x^{cf}\right)|$ [65,78]). It incurs extra compute from backpropagation, but remains substantially cheaper than exhaustive intervention search; consequently, it is commonly used as a scalable ranking/filtering stage that proposes candidate objects for more expensive causal validation [76].

• Advantages:Gradient Detection is applicable to a broad class of objects without requiring additional training. Compared with exhaustive interventions, it can produce rankings with a relatively small number of backward passes, making it practical as an initial localization step when the candidate set is large.
• Limitations: Gradients provide a local proxy, not causal necessity: salience can be offset by downstream computation, and finite interventions may depart from first-order effects in non-linear regimes. For these reasons, gradient-ranked objects are typically paired with Causal Attribution (§3.2) to validate whether the identified objects are genuinely responsible for the target behavior.

3.4. Probing

Methodological Formulation

Probing methods interpret model signals by training an auxiliary predictor $g_{\psi }$ (often linear) to decode a labeled property y from an internal vector $\mathbf{z}\in {\mathbb{R}}^{d_{\mathrm{model}}}$ (e.g., the residual stream state ${\mathbf{x}}^l$ at layer l); in sequence models with token-indexed states ${\mathbf{z}}_t$, one first defines a single probe input either token-wise (choosing $\mathbf{z}={\mathbf{z}}_{t^{*}}$ at a designated position such as the last token) or via pooled aggregation across positions (e.g., mean pooling), while the probe formulation itself is unchanged, e.g.,

$$\widehat{y}=g_{\psi }\left(\mathbf{z}\right)=\mathrm{softmax}\left({\mathbf{W}}_P\mathbf{z}\right),$$

(15)

using a supervised dataset $\mathcal{D}=\left\{\right(\mathbf{z},y\left)\right\}$ [79,80].

Operationally, probing treats the model as a frozen feature extractor and assesses decodability: whether y is recoverable from $\mathbf{z}$ by a restricted hypothesis class (commonly linear), which supports localization by comparison across candidate objects (layers/heads/FFNs) via decoding performance or information-theoretic surrogates [81,82], typically followed by Causal Attribution (§3.2) to test functional necessity. Methodologically, it is standard to interpret probe results with care: high probe accuracy alone does not imply the model uses that information, motivating controls (e.g., selectivity / control tasks) and complementary causal tests [80,83,84].

Applicable Objects

Probing is defined on internal vectors, and is most naturally applied to any intermediate quantity that can be represented as a vector in ${\mathbb{R}}^{d_{\mathrm{model}}}$. In LLMs, a typical workflow mirrors the pipeline in Figure 7: (i) constructs labeled probing evidence (including factual and counterfactual variants), (ii) runs the evidence through the frozen LLM and logs candidate internal objects across layers and submodules, and (iii) trains a fixed probe family on each object to compare decodability and localize where the target signal is most recoverable.

1) Residual Stream States (${\mathbf{x}}^l$, ${\mathbf{x}}^{l,\mathrm{mid}}$): The most common probing target is the residual stream state ${\mathbf{x}}^l\in {\mathbb{R}}^{d_{\mathrm{model}}}$, as well as intermediate residual states ${\mathbf{x}}^{l,\mathrm{mid}}$. Layer-wise probes trained on ${\mathbf{x}}^l$ directly instantiate the “extract residual stream state across layers → train probing classifiers” step depicted in Figure 7, and have been used to track where context knowledge, knowledge conflicts, and truthfulness-related signals become most decodable along depth [12,84,85,86,87].

2) Block Outputs (${\mathbf{h}}_{attn}^{l,h},{\mathbf{h}}_{ffn}^l$): Probing can target intermediate block outputs by extracting $\mathbf{z}$ from either an attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$ or the FFN output ${\mathbf{h}}_{\mathrm{ffn}}^l$ (optionally token-wise, e.g., ${\mathbf{h}}_{\mathrm{attn},t}^{l,h}$ or ${\mathbf{h}}_{\mathrm{ffn},t}^l$), and training a matched probe family across layers (and heads for attention). Comparing decodability across $(l,h)$ and l supports fine-grained “localization by comparison,” ranking where a target property is most linearly accessible and contrasting attention- vs. FFN-based localization under a consistent protocol [88,89,90].

3) SAE Feature Activation State ($\mathbf{a}$): Probing also integrates with SAE features. Given sparse SAE feature activation states $\mathbf{a}$, one can define $\mathbf{z}$ as the feature activation vector $\mathbf{a}=(a_1,\dots ,a_m)$ (or a selected subset) and train classifiers on these sparse coordinates. This yields concept-aligned decoding axes that can be inspected at the feature level and cross-referenced with feature-level interpretations [91,92].

Characteristics and Scope

Probing focuses on supervised decoding: it trains an auxiliary predictor $g_{\psi }$ on $\mathcal{D}=\left\{\right(\mathbf{z},y\left)\right\}$ to measure how well a labeled property y is predictable from an internal vector $\mathbf{z}$. Treating LLM as a frozen feature extractor, probing evaluates decodability under a restricted hypothesis class, making it primarily a tool for representational localization rather than causal responsibility. In practice, probe-based rankings are commonly used to shortlist candidate layers/heads/FFNs for subsequent intervention-based analyses (e.g., Causal Attribution in §3.2).

• Advantages: With a fixed probe family, Probing enables standardized comparisons across objects, supporting efficient layer-wise tracking and large-scale ranking of candidate modules. Simple probes (e.g., linear) are lightweight and interpretable, allowing broad sweeps while keeping the LLM frozen.
• Limitations: Decodability is not causality: high probe accuracy does not imply the model uses y, nor that the probed object is necessary or sufficient. Results are sensitive to dataset and design choices (e.g., labeling, token positions), so controls and follow-up causal tests are typically required for functional claims.

3.5. Vocabulary Projection

Methodological Formulation

The most prominent technique in this category is the Logit Lens [93]. It operates on the premise that the pre-trained unembedding matrix ${\mathbf{W}}_U\in {\mathbb{R}}^{d_{\mathrm{model}}\times \left|\mathcal{V}\right|}$, which maps the final layer’s hidden state to vocabulary logits, can serve as a universal decoder for intermediate states throughout the model. Formally, let $\mathbf{z}\in {\mathbb{R}}^{d_{\mathrm{model}}}$ denote a generic internal object (e.g., the residual stream state ${\mathbf{x}}^l$ or an attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$). Vocab Projection computes a distribution $\mathbf{p}$ over the vocabulary $\mathcal{V}$ by projecting $\mathbf{z}$ through the unembedding matrix:

$$\mathbf{p}=\mathrm{softmax}\left(\mathbf{z}{\mathbf{W}}_U\right)$$

(16)

By inspecting the tokens with the highest probabilities in $\mathbf{p}$, researchers can directly interpret the semantic content encoded in $\mathbf{z}$ in terms of the model’s output vocabulary.

Applicable Objects

Vocab Projection is a versatile tool that applies to various objects defined in §2, ranging from global residual streams to specific attention heads, neurons, and SAE features.

1) Residual Stream State (${\mathbf{x}}^l$): Projecting the residual stream state ${\mathbf{x}}^l$ allows researchers to trace the layer-wise evolution of predictions and identify the crucial layers where specific concepts emerge [95,96,97,98,99,100,101,102]. For instance, Wendler et al. [98] applied this to multilingual models, revealing distinct processing phases as shown in Figure 8 (a): initial layers focus on the surface form of the input language; middle layers process semantics in an abstract, “English-centric” concept space; and final layers rotate back to the target language. This confirms that English serves as an internal pivot for reasoning even in non-English tasks.

2) Attention Head Output (${\mathbf{h}}_{\mathrm{attn}}^{l,h}$): Applying projection to the output of individual heads reveals the specific information (e.g., copied names or next-token candidates) that a head transmits to the residual stream. This has been instrumental in identifying functional heads in mechanistic studies [52,103,104,105,106,107]. For example, in reverse-engineering the Indirect Object Identification (IOI) task, Wang et al. [52] identified “Name Mover Heads” (which explicitly project to the correct name, e.g., “Mary”) and “Negative Name Mover Heads” (which suppress the correct name).

3) Neuron Value Weight (${\mathbf{v}}_j^l$): Geva et al. [94] demonstrated that FFNs operate as key-value memories. By projecting the value weight vector ${\mathbf{v}}_j^l$ (a column of ${\mathbf{W}}_{\mathrm{out}}^l$) into the vocabulary, one can see which tokens are promoted by a specific neuron [94,108,109,110]. Individual neurons often boost semantically related clusters (e.g., “press”, “news”, “media”), suggesting that FFNs refine predictions by composing these pre-learned semantic distributions.

4) SAE Feature (${\mathbf{f}}_j$): For SAEs, output-based explanations leverage the decoder weights to interpret monosemantic features. By computing the logits contribution ${\mathbf{l}}_j={\mathbf{f}}_j{\mathbf{W}}_U$ for a feature vector ${\mathbf{f}}_j$, one can identify top-ranked tokens [111,112,113,114,115]. As shown in Figure 8 (b), a feature whose projection yields high positive logits for tokens like “Food” and “food” is interpreted as encoding a “food” concept, directly grounding the sparse feature in human-understandable semantics.

Characteristics and Scope

The scope of Vocab Projection is characterized by direct semantic mapping. It offers an intrinsic view of internal representations without requiring auxiliary training.

• Advantages: It provides a zero-shot interpretation method that is computationally efficient and intuitive. Unlike Probing (§3.4), it does not require collecting a labeled dataset or training a separate classifier, allowing for immediate inspection of any model state.
• Limitations: The primary limitation is the assumption that intermediate states exist in the same vector space as the output vocabulary (basis alignment). While this often holds for the residual stream due to the residual connection structure, it may be less accurate for components inside sub-layers (like FFN and MHA) or in models where the representation space rotates significantly across layers. Consequently, results should be interpreted as an approximation of the information that is linearly decodable by the final layer.

3.6. Circuit Discovery

Methodological Formulation

Circuit Discovery methods aim to uncover mechanistic pathways: structured, directed dependencies among internal objects that mediate computation for a target behavior [116,117,118,119]. Formally, let $(\mathcal{O},\mathcal{E})$ be the model’s computational graph over internal objects $\mathcal{O}$ and directed edges $\mathcal{E}$, where an edge $e_{ij}\in \mathcal{E}$ denotes signal flow from object $o_i$ to $o_j$. A circuit $\mathcal{C}\subseteq \mathcal{E}$ is faithful if restricting computation to $\mathcal{C}$ (e.g., by patching/ablating all other edges) preserves the target output $F\left(x\right)$ or task performance.

Under the residual–rewrite view, heads and MLPs read from and write to the residual stream, inducing a directed graph whose edges represent additive residual updates. Circuit Discovery can be cast as edge-level causal subgraph selection: edges are retained if intervening on the corresponding information flow degrades a target metric $\mathcal{R}$ [120]. Automatic Circuit DisCovery (ACDC) instantiates this by iteratively testing and pruning edges via patching-based interventions, avoiding brute-force $O\left(\right|\mathcal{E}\left|\right)$ enumeration while recovering circuits such as GPT-2’s greater-than mechanism [118,121].

Attribution-based methods such as Edge Attribution Patching (EAP) approximate patching with a first-order expansion, producing an edge score from two forward passes (clean/corrupted) and one backward pass [122,123]. Here, clean input ${\mathbf{x}}_{clean}$ elicits the target behavior, while corrupted input ${\mathbf{x}}_{corr}$ is a minimally modified version designed to break it (e.g., by perturbing relevant evidence or adding a counterfactual distractor), so the difference isolates the causal signal. For a sender object u, let ${\mathbf{a}}_u\left(\mathbf{x}\right)$ denote its output activation vector (e.g., head/FFN output written into the residual stream) on input $\mathbf{x}$; the sender delta $\Delta {\mathbf{a}}_u={\mathbf{a}}_u\left({\mathbf{x}}_{clean}\right)-{\mathbf{a}}_u\left({\mathbf{x}}_{corr}\right)$ captures how the sender’s contribution changes between the clean and corrupted runs. EAP then scores an edge via the dot product between the sender delta and the receiver sensitivity ${\nabla }_{{\mathbf{z}}_v}\mathcal{R}$ (computed on the clean run):

$$S_{\mathrm{EAP}}(u\to v)\approx \underset{\Delta {\mathbf{a}}_u}{\underbrace{\left({\mathbf{a}}_u\left({\mathbf{x}}_{clean}\right)-{\mathbf{a}}_u\left({\mathbf{x}}_{corr}\right)\right)}}·\underset{{\nabla }_{{\mathbf{z}}_v}\mathcal{R}}{\underbrace{{\displaystyle \frac{\partial \mathcal{R}}{\partial {\mathbf{z}}_v}}{|}_{{\mathbf{x}}_{clean}}}}.$$

(17)

To mitigate non-linearity/saturation, EAP with Integrated Gradients (EAP-IG) replaces the local gradient with a path-averaged gradient along ${\mathbf{x}}_{\alpha }={\mathbf{x}}_{corr}+\alpha ({\mathbf{x}}_{clean}-{\mathbf{x}}_{corr})$ [56,123,124]:

$$S_{\mathrm{EAP}-\mathrm{IG}}(u\to v)=\Delta {\mathbf{a}}_u·{\int }_0^1{\displaystyle \frac{\partial \mathcal{R}}{\partial {\mathbf{z}}_v}}{|}_{{\mathbf{x}}_{\alpha }}d\alpha \approx \Delta {\mathbf{a}}_u·{\displaystyle \frac{1}{n}}\sum _{k=1}^n{\displaystyle \frac{\partial \mathcal{R}}{\partial {\mathbf{z}}_v}}{|}_{{\mathbf{x}}_{k/n}}.$$

(18)

A standard workflow is: (i) collect sender deltas $\Delta {\mathbf{a}}_u$ from ${\mathbf{x}}_{clean}$ vs. ${\mathbf{x}}_{corr}$, (ii) compute receiver gradients (single-point for EAP; path-averaged for EAP-IG with n backward passes), (iii) score and rank edges by $\left|S\right|$, and (iv) prune/threshold to obtain a sparse circuit, optionally validating via targeted interventions on retained edges.

Applicable Objects

Circuit Discovery targets edges between objects (Tab. 1), ranging over directed dependencies among any interpretable objects. In LLMs, it is commonly instantiated under the residual–rewrite view, so edges correspond to additive signal transmission across layers. Figure 9 illustrates a sparse cross-layer knowledge circuit supporting the completion “The official language of France is French” in GPT-2-Medium, with attention/logit analyses clarifying how selected edges route and transform information [119].

Practically, Circuit Discovery is operationalized in three broad ways:

1) Intervention-based edge search (patching/ablation): One can directly test causal necessity at the edge level by patching or ablating a candidate dependency $e_{u\to v}$ (e.g., blocking contributions from a sender module such as an attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$ or FFN output ${\mathbf{h}}_{\mathrm{ffn}}^l$ into a downstream receiver input ${\mathbf{z}}_v$) and measuring the change in a task metric $\mathcal{R}$. Because exhaustive edge testing scales as $O\left(\right|\mathcal{E}\left|\right)$, practical workflows rely on structured search or automated procedures to reduce interventions [45,52,121].

2) Attribution-based edge scoring: Attribution methods rank edges by efficiently approximating their patching effect. EAP combines sender activation differences $\Delta {\mathbf{a}}_u$ (clean vs. corrupted) with receiver sensitivity ${\nabla }_{{\mathbf{z}}_v}\mathcal{R}$ to produce an edge ranking from two forward passes and one backward pass, while EAP-IG uses a path-averaged gradient to reduce saturation/non-linearity issues at the cost of additional backward passes [122,123,124]. Position-aware refinements follow the same edge-scoring principle while better aligning sender/receiver accounting with token-wise computation [125,126].

3) Feature-based replacement models:Circuit Discovery can be lifted to sparse feature spaces via replacement models such as SAE/transcoder variants. Here, the relevant objects are SAE features (sparse feature activations and decoder directions), and circuit edges represent directed dependencies in feature space, enabling attribution graphs and prompt-specific circuit tracing that are often more interpretable than raw residual coordinates [127,128,129].

Characteristics and Scope

Circuit Discovery identifies a sparse, directed cross-layer causal subgraph whose edges jointly mediate a target behavior and remain approximately faithful under interventions. Unlike single-component localization, it targets structured pathways of information routing and transformation, returning a minimally (or strongly) sufficient directed subnetwork. Practically, edges are often pre-ranked by scalable attribution-style scores and then confirmed with targeted interventions (e.g., patching/ablation).

• Advantages:Circuit Discovery yields mechanistically structured explanations: selecting edges reveals how multiple objects compose a computation and exposes cross-layer routing patterns that node-wise rankings can miss. This aligns with transformers’ residual-update structure, where heads and FFNs contribute additive edits that can be tracked as directed dependencies. Attribution-based edge scoring also enables scalable screening of large edge sets when exhaustive interventions are infeasible.
• Limitations: Circuits are defined relative to a specific behavior, metric $\mathcal{R}$, and contrast (clean vs. corrupted), so results are often objective- and dataset-dependent. Because attribution scores approximate intervention effects, they may miss non-linear interactions, so rankings are best treated as proposals and typically require intervention-based validation on the retained subgraph.

4. Steering Methods

While localization methods (§3) identify the specific objects responsible for model behaviors, this section focuses on a distinct class of techniques: those that manipulate these localized components to steer model outputs, thereby enabling controlled intervention into LLM’s generation process.

4.1. Amplitude Manipulation

Methodological Formulation

Amplitude Manipulation steers model behavior by directly modifying the activation magnitude of a targeted internal object o during the forward pass. Unlike optimization-based methods that update weights, this approach acts as a transient intervention on the runtime state. Formally, let o be the original activation (e.g., a neuron activation $s_j^l$, an SAE feature activation $a_j$ or an attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$) and $\tilde{o}$ be the modified state. The intervention is defined as:

$$\tilde{o}=\mathcal{T}(o,\alpha )$$

(19)

where $\mathcal{T}$ represents the transformation function. This typically takes two forms:

• Ablation or Patching: Here, the object is suppressed or replaced, i.e., $\tilde{o}\in \{0,\mathbb{E}\left[o\right],o_{\mathrm{tgt}}\}$. Setting $\tilde{o}=0$ (Zeroing) or $\mathbb{E}\left[o\right]$ (Mean centering) removes the component’s influence, while $\tilde{o}=o_{\mathrm{tgt}}$ (Patching) injects information from a different context.
• Scaling: Here, the activation strength is adjusted via a scalar coefficient $\alpha$, such that $\tilde{o}=\alpha ·o$. This allows for continuous amplification ($\alpha >1$) or attenuation ($0<\alpha <1$) of a specific feature’s downstream impact.

While these operations are mechanically similar to those in Causal Attribution (§3.2), the objective differs fundamentally: attribution employs them to diagnose causality, whereas Amplitude Manipulation employs them to actively intervene and control model behavior.

Applicable Objects

This method is applied across a wide range of dynamic objects, including residual stream state ${\mathbf{x}}^l$, attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$, neuron activation state ${\mathbf{s}}^l$, and SAE feature activation state $\mathbf{a}$.

1) Ablation (Zeroing) and Removal: Ablation is extensively used to mitigate unwanted behaviors by suppressing the components responsible for them. Tang et al. [2] utilized this to control the output language of multilingual LLMs. They identified “language-specific neurons” that selectively activate for the particular language (e.g., Chinese). As illustrated in Figure 10 (a), by setting the activation of these Chinese-specific neurons to zero, they suppressed the model’s ability to generate Chinese, thereby forcing the model to switch its output to English even when the prompt might suggest otherwise. Distinct from general steering, Nie et al. [130] applied ablation to address Language Confusion—a phenomenon where models erroneously switch to a non-target language. They identified interfering neurons that activate for the wrong language (e.g., German neurons firing during an English task) and demonstrated that ablating these specific noisy components restores the correct target language generation. In the domain of Safety and Bias, Goyal et al. [131] and Yeo et al. [132] zeroed out specific SAE features associated with toxicity or refusal, effectively detoxifying the model’s output. Liu et al. [133] and Chandna et al. [134] applied zero-ablation to neurons and circuit edges encoding social bias, while Huang et al. [124] masked specific circuit edges to alleviate “knowledge overshadowing” where strong knowledge suppresses relevant but weaker information. Furthermore, ablation is used for Efficiency: Liu et al. [25] and Men et al. [14] demonstrated that removing redundant layers or components can accelerate inference without significant performance loss. Zhou et al. [135] and Niu et al. [136] also utilized attention head ablation to study and improve safety and contextual entrainment.

2) Patching (Replacement): Patching allows for precise injection of attributes. Ahsan et al. [137] and Raimondi et al. [138] utilized activation patching to steer demographic and moral characteristics. As shown in Figure 10 (b), Ahsan et al. [137] performed a “Male Patch” by replacing the internal representation of a patient with a male-associated vector. This intervention not only altered the pronouns in the generated vignette (from “Ms.” to “Mr.”) but also causally changed the downstream clinical prediction (shifting the depression risk from “Yes” to “No”), highlighting the causal link between demographic representations and model decisions.

3) Scaling (Amplification/Attenuation): Scaling offers fine-grained control by adjusting the intensity of features. Tang et al. [2] also employed scaling to amplify target-language neurons to further stabilize multilingual generation. Gao et al. [139] scaled the activation of “Hallucination Neurons” to modulate the model’s factual reliability. In the context of SAE features, Pach et al. [140] demonstrated that scaling specific feature activations allows for continuous steering of model outputs. Meanwhile, Galichin et al. [3] showed that amplifying the activations of reflection-related features can increase the length of generated output, thereby enhancing the model’s reasoning performance. Finally, scaling is integral to Vector Arithmetic (§4.3) in the context of model merging: Stoehr et al. [141], Liu et al. [72], and Yao et al. [142] optimized the scaling coefficients of steering vectors or task vectors to balance different model capabilities, while Wang et al. [143] scaled the activations of expert modules to enhance mathematical reasoning.

Characteristics and Scope

The scope of Amplitude Manipulation is characterized by inference-time activation control. It provides a mechanism to transiently modulate model behavior without permanent weight updates.

• Advantages: It is an optimization-free and reversible intervention. It allows for “surgical” edits to model behavior (e.g., removing specific biases) by simply masking or scaling activations during inference. This makes it highly flexible and suitable for real-time control.
• Limitations: It relies heavily on the accurate localization of the target components. If the features responsible for a behavior are not perfectly disentangled (i.e., polysemantic), ablating or scaling them may cause unintended side effects or degrade general performance. Furthermore, finding the optimal scaling factor $\alpha$ often requires empirical tuning.

4.2. Targeted Optimization

Methodological Formulation

Targeted Optimization (under Localizing Methods) frames model optimizing as a small, localized update that enforces a desired behavioral change while minimizing unintended side effects. Let $f_{\theta }$ be the base model and ${\theta }^{\prime }$ the targeted model. We restrict updates to a selected subset of objects via a (hard or soft) mask M, and optimize a simple trade-off between a target objective and a preservation objective:

$${\theta }^{\prime }\leftarrow \theta +(M\odot \Delta \theta ),\Delta {\theta }^{★}=arg\underset{\Delta \theta }{min}{\mathcal{L}}_{\mathrm{tgt}}(f_{{\theta }^{\prime }};{\mathcal{D}}_{\mathrm{tgt}})+\lambda {\mathcal{L}}_{\mathrm{pres}}(f_{{\theta }^{\prime }},f_{\theta };{\mathcal{D}}_{\mathrm{pres}}).$$

(20)

Here, ${\mathcal{D}}_{\mathrm{tgt}}$ specifies the target behavior (e.g., rewriting a fact or enforcing refusals), while ${\mathcal{D}}_{\mathrm{pres}}$ anchors the model to its original capabilities. The localization mask M operationalizes “where the change is allowed to happen” (layers, modules, neurons/heads, or other structured subsets).

Applicable Objects

In practice, “what is optimized” can be grouped into two representative localized objects:

1) Localized Parameters for Knowledge Editing: This line performs direct parameter-space updates that are intentionally constrained (e.g., low-rank or small support) to rewrite specific behaviors with minimal spillover. Representative examples include rank-one / layer-local knowledge editing extensions [43,144], cross-model knowledge transfer via localized adapters [145], and constraining adaptation to low-dimensional task subspaces or coarse-to-fine masked tuning for better retention [66,146].

2) Fine-grained Subsets for Specialization: Here, localization is enforced at neuron/head/region granularity to isolate the functional unit relevant to a capability or a safety property. Concretely, rather than updating the full model, Targeted Optimization learns a targeted update within a small object subset (implicitly corresponding to a mask M in Eq. 20), thereby limiting unnecessary parameter drift and reducing interference across tasks or languages. Related lines of work localize adaptation to compact trainable units at different granularities. This includes neuron-level fine-tuning [147] and methods that identify core parameter regions or language-agnostic factual neurons [148,149,150], and in safety-preserving or security-aware partial tuning that freezes or restricts sensitive objects [88,151,152]. Relatedly, head-level analyses further motivate localizing optimization to essential computational pathways (e.g., arithmetic-relevant heads) [153].

A representative example is shown in Figure 11: LANDeRMT [154] performs selective fine-tuning for multilingual machine translation by (i) first localizing the update to language-pair-relevant layers, (ii) quantifying neuron-level language awareness, and (iii) routing gradients only through the most relevant neurons, which concretely illustrates how fine-grained locality reduces cross-lingual interference and limits parameter drift.

Characteristics and Scope

The scope of this method is characterized by persistence and surgical precision. Unlike Amplitude Manipulation (§4.1), Targeted Optimization performs parameter optimization on ${\mathcal{D}}_{\mathrm{tgt}}$ to produce a targeted model whose behavior durably satisfies a specified objective, while constraining the update to a localized subset of objects (e.g., layers, modules, neurons/heads). This objective-driven and localized training enables not only precise rewrites to particular memories or facts, but also focused capability enhancement, with reduced collateral impact on unrelated traits.

• Advantages: It offers strong precision, controllability, and persistence. The desired behavioral change is directly encoded in a target objective, and localization helps minimize interference with unrelated competencies. Consequently, it is well-suited for targeted factual rewrites, controlled specialization, and safety-preserving adaptation where lasting changes are required.
• Limitations: Its reliability hinges on correct localization and well-specified supervision. If the chosen subset does not capture the causal mechanism, optimization may underachieve the intended target behavior, shift the behavior to other objects, or yield brittle side effects. In practice, success often requires carefully constructed target/preservation data and robust criteria for selecting the localized update region.

4.3. Vector Arithmetic

Methodological Formulation

Positing that high-level concepts or skills are encoded linearly within the model’s representation space, Vector Arithmetic steers a generic target object $\mathbf{z}$ (e.g., a residual stream state or a parameter vector) by injecting a specific steering vector$\mathbf{v}$. This approach assumes that adding a vector representing a concept effectively “moves” the model’s internal state towards that concept in the high-dimensional space. Formally, the update rule for the intervention is defined as:

$$\widehat{\mathbf{z}}\leftarrow \mathbf{z}+\alpha ·\mathbf{v}$$

(21)

where $\mathbf{v}$ represents the directional encoding of a target attribute (such as “honesty” or “sycophancy”) and $\alpha$ is a scalar coefficient that controls the intervention strength (or steering intensity).

Applicable Objects

The target object $\mathbf{z}$ typically falls into two categories: dynamic hidden states during inference or static model parameters.

1) Dynamic Hidden States: The primary targets for runtime steering are the residual stream states ${\mathbf{x}}^l$ and the outputs of attention heads ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$. For these dynamic objects, the steering vector $\mathbf{v}$ is typically derived using one of two methods:

• Contrastive Activation Means: This method, often referred to as “Activation Addition” or “Mass-Mean Shift,” assumes that a concept can be isolated by comparing the model’s internal states across opposing contexts [162,163,164,165,166,167]. Formally, let ${\mathcal{D}}^{+}$ be a set of prompts eliciting the target behavior and ${\mathcal{D}}^{-}$ be a set eliciting the opposing behavior. The steering vector $\mathbf{v}$ is calculated as the difference between the centroids of the residual stream states ${\mathbf{x}}^l$ for these two sets:

  $$\mathbf{v}={\mu }^{+}-{\mu }^{-}={\displaystyle \frac{1}{|{\mathcal{D}}^{+}|}}\sum _{{\mathbf{x}}_i\in {\mathcal{D}}^{+}}{\mathbf{x}}_i^l-{\displaystyle \frac{1}{|{\mathcal{D}}^{-}|}}\sum _{{\mathbf{x}}_j\in {\mathcal{D}}^{-}}{\mathbf{x}}_j^l$$

  (22)
  By adding $\alpha ·\mathbf{v}$ to the residual stream, we shift the model’s current state towards the centroid of the positive behavior.
• SAE Features: SAEs offer a more precise way to derive $\mathbf{v}$ by utilizing monosemantic features [131,168,169,170,171,172]. As illustrated in Figure 12, the process involves two steps:
  • Feature Identification: First, we collect residual stream states from a positive dataset ${\mathcal{D}}^{+}$ (eliciting the target concept, e.g., “Happiness”) and a negative/neutral dataset ${\mathcal{D}}^{-}$. By passing these states through the SAE encoder, we calculate the differential activation score${\delta }_j$ for each feature j:

    $${\delta }_j={\mathbb{E}}_{\mathbf{x}\in {\mathcal{D}}^{+}}\left[a_j\left(\mathbf{x}\right)\right]-{\mathbb{E}}_{\mathbf{x}\in {\mathcal{D}}^{-}}\left[a_j\left(\mathbf{x}\right)\right]$$

    (23)

    where $a_j\left(\mathbf{x}\right)$ denotes the j-th feature activation for input $\mathbf{x}$. Features with high positive ${\delta }_j$ constitute the set of “Target Features” $\mathcal{J}$ that specifically encode the desired trait.
  • Vector Construction: The steering vector $\mathbf{v}$ is then synthesized as the weighted sum of these identified feature. Let ${\mathbf{f}}_j$ denote the j-th feature (the j-th column of the SAE decoder weights ${\mathbf{W}}_{\mathrm{dec}}$). The steering vector is computed as:

    $$\mathbf{v}=\sum _{j\in \mathcal{J}}{\delta }_j·{\mathbf{f}}_j$$

    (24)
  Finally, this obtained steering vector is injected into the model’s residual stream during inference ($\widehat{\mathbf{x}}\leftarrow \mathbf{x}+\alpha ·\mathbf{v}$). As shown in Figure 12 (c), this enables precise manipulation of specific semantic traits like “Happiness” or “Confusion” to drastically alter generation styles while minimizing interference with unrelated concepts.

2) Static Parameters: For static weights, the steering vector $\mathbf{v}$ is explicitly defined as a Task Vector in Model Merging [72,142,173,174]. This vector is computed as the element-wise difference between the weights of a fine-tuned model and its pre-trained base ($\mathbf{v}={\mathbf{W}}_{\mathrm{ft}}-{\mathbf{W}}_{\mathrm{base}}$), effectively encapsulating a transferable skill or behavior. Recent advancements have evolved beyond simple element-wise addition by employing localization techniques to determine adaptive merging coefficients. For instance, Liu et al. [72] proposed Sens-Merging, which utilizes Gradient Detection-based sensitivity analysis to evaluate parameter importance, allowing for the precise balancing of weights based on their impact on task performance. Complementarily, Yao et al. [142] introduced Activation-Guided Consensus Merging, which leverages Magnitude Analysis of internal representations. By calculating the mutual information between activations of the base and fine-tuned models, they derive layer-specific scaling coefficients to optimally integrate the task vector.

Characteristics and Scope

The scope of this method is characterized by additive directionality. Unlike the precise rewriting in Targeted Optimization (§4.2), Vector Arithmetic acts as a steering force, dynamically pushing the model towards a target attribute without permanently altering weights.

• Advantages: It is a lightweight and reversible intervention. Since it typically operates at inference time (for hidden states) or via simple weight addition, it does not require complex optimization or gradient descent during deployment. It allows for flexible control over model behavior by simply adjusting the steering coefficient $\alpha$.
• Limitations: The effectiveness relies on the “Linear Representation Hypothesis.” If the target concept is not encoded linearly or if the steering vector $\mathbf{v}$ is entangled with other concepts (which is common with “Contrastive Activation Means”), the intervention might introduce unintended side effects.

5. Applications

Building on localizing methods (§3) that identify internal objects associated with specific behaviors and steering methods (§4) that intervene on these objects to modulate model outputs, this section summarizes how these lines of work translate into practical use cases. We organize the literature around three overarching objectives: alignment, capability, and efficiency.

5.1. Improve Alignment

5.1.1. Safety and Reliability

1) Safety-Critical Component Manipulation

Unsafe or unreliable behaviors in LLMs have been shown to be mediated by relatively localized internal components. Accordingly, a body of work first localized safety-relevant objects and then intervened via targeted mechanistic techniques. At the attention level, Zhou et al. [135] showed that a small subset of attention heads played a disproportionate role in safety-related behaviors, particularly refusal and rejection of harmful queries. Using Causal Attribution to localize safety-critical heads and Amplitude Manipulation to intervene, they demonstrated that suppressing these heads substantially weakened safety capability while modifying only a negligible fraction of parameters. At the neuron level, several studies applied Magnitude Analysis to identify neurons whose activations were strongly associated with unsafe or misaligned behaviors. Zhao et al. [178] introduced safety neurons and showed that a very small subset — predominantly located in early self-attention layers — collectively governed safety behavior; they then performed Targeted Optimization by selectively tuning these neurons during training, significantly improving safety without degrading general performance. Complementarily, Suau et al. [176] used magnitude-based criteria to pinpoint toxicity-related neurons and applied Amplitude Manipulation by scaling down their activations at inference time to mitigate toxic generations. Similarly, Gao et al. [139] identified hallucination-associated neurons (H-neurons) via Magnitude Analysis and validated their causal impact through Amplitude Manipulation, showing that suppressing these neurons reduced hallucinations without broadly affecting other capabilities. Beyond individual neurons, recent work leveraged SAEs to disentangle safety-related representations into interpretable features. Using Magnitude Analysis over SAE feature activation states, Templeton et al. [177] showed that SAE features extracted from LLMs exhibited strong monosemanticity, including features associated with harmful or toxic content. Building on this insight, Goyal et al. [131] applied Amplitude Manipulation to suppress selected SAE features and thereby detoxify model outputs. Likewise, Yeo et al. [132] performed SAE-based Magnitude Analysis to identify harm- and refusal-related feature sets and validated their roles through targeted Amplitude Manipulation, enabling fine-grained control and mechanistic insight into refusal behavior.

While Amplitude Manipulation-based interventions typically operate at inference time, several works pursued more persistent safety improvements through Targeted Optimization. Huang et al. [124] identified safety-relevant circuits and updated only parameters within these circuits to mitigate harmful behaviors. At finer granularity, Zhao et al. [178], Chen et al. [175], and Li et al. [152] showed that selectively updating neuron-associated weights enabled precise safety edits with minimal side effects. At a coarser level, Li et al. [151] demonstrated that safety behavior could be localized at the layer level, while Lee et al. [161] analyzed how alignment objectives reshaped internal representations during optimization.

2) Latent Safety and Reliablity Representation Steering

A complementary line of research shows that many safety-relevant behaviors are encoded as approximately linear directions in LLM’s latent space, motivating safety interventions based on Vector Arithmetic in the residual stream.

Arditi et al. [179] and Zhao et al. [180] showed that refusal was encoded as a compact low-dimensional subspace identified via Causal Attribution, and that some jailbreaks succeeded by suppressing this refusal signal via Vector Arithmetic without changing the model’s harmfulness belief. Extending these findings to reasoning models, Yin et al. [181] identified a refusal-cliff phenomenon using Probing, where refusal intent was maintained during intermediate reasoning but was abruptly suppressed at the final generation stage, a failure mode attributed to a small set of refusal-suppressing attention heads. Building on these analyses, multiple studies identified actionable safety directions and applied steering interventions. Wang et al. [183] proposed a training-free, single-vector ablation method (a form of Vector Arithmetic) that selectively removed false refusal while preserving true refusal and general capabilities, enabling fine-grained safety calibration. Wang et al. [184] further demonstrated that refusal directions were approximately universal across safety-aligned languages, helping to explain the effectiveness of cross-lingual jailbreaks as well as vector-based interventions.

Vector Arithmetic-based steering was also applied to hallucination reduction and factuality improvement. Chuang et al. [11] introduced contrastive layer decoding (a form of Vector Arithmetic) during generation to amplify factual signals identified via Vocabulary Projection. Similarly, Zhang et al. [12] identified a truthfulness direction in the residual space using Probing and then edited it via Vector Arithmetic, enabling controllable enhancement of truthful behavior. Complementarily, Orgad et al. [86] showed that hallucination-related representations could be detected internally via Probing even when they were not expressed at the output level, highlighting the diagnostic value of latent safety signals. Finally, recent work applied Vector Arithmetic to improve instruction-following reliability. He et al. [171] leveraged SAE-derived directions to steer instruction adherence, while Stolfo et al. [187], Jiang et al. [188], and Li et al. [189] demonstrated that instruction-following behavior could be improved through Vector Arithmetic steering without full retraining.

5.1.2. Fairness and Bias

1) Gender Bias Localization and Selective Debiasing

Mechanistic studies of gender bias established a canonical fairness pipeline: first localizing bias mediation with Causal Attribution, then steering the identified carriers via either transient inference-time control or persistent parameter updates. Vig et al. [42] provided an early template using causal mediation analysis in GPT-2 to quantify which internal components mediated gendered associations, and demonstrated mitigation by replacing bias-inducing activations with counterfactual ones, a direct instance of Amplitude Manipulation. To achieve persistent mitigation, subsequent work increasingly shifted to selective updates of localized components via Targeted Optimization. Chintam et al. [192] showed that responsibility for gender bias could concentrate in specific late-layer attention heads and reduced bias by fine-tuning only these components. Cai et al. [190] characterized a division of labor where lower FFN blocks encoded bias-relevant information while upper attention modules exploited it, proposing an editing-style method to update the responsible subset. Finally, Yu and Ananiadou [109] refined the intervention granularity to the neuron level, identifying distinct “gender neurons” versus “general neurons” and introducing an interpretable neuron-editing procedure to reduce bias while preserving general performance.

2) Distributed Attribute and Cultural Bias Signals

Beyond gender, mechanistic evidence suggested that demographic, social, and cultural biases are often encoded more diffusely. This motivated localization strategies that avoid assuming a single “bias module,” alongside mitigation strategies combining targeted suppression with global representational steering. In domain-conditioned settings like healthcare, Ahsan et al. [137] used activation patching, a form of Causal Attribution, to localize racial information across multiple LLMs, reporting that racial signals are more scattered across early and middle FFN layers compared to gender. Similarly, Yu et al. [51] adopted Patchscope-style interventions to “read out” cultural knowledge from internal representations. Rather than proposing a mitigation, their results focused on diagnosis, revealing how cultural salience and resource imbalance manifest as systematic representational asymmetries. Addressing broader societal biases, Liu et al. [133] employed Gradient Detection to identify neurons associated with multiple social attributes and demonstrated mitigation by suppressing their activations. To scale localization beyond hand-picked modules, Chandna et al. [134] combined Magnitude Analysis over internal structures with causal validation to create a reusable recipe for bias analysis across attributes. Finally, acknowledging that values can be represented linearly, Kim et al. [317] used Probing to identify attention heads predicting political ideology, and then steered generations via Vector Arithmetic.

3) Evaluation Bias Engines in Judgment and Framing

A complementary thread targeted cognitive biases arising from judgment heuristics, prompt formats, or decision framing, rather than demographic correlations. These works mitigated such biases through inference-time controls guided by importance signals via Magnitude Analysis or validated by Causal Attribution. For positional anchoring in multiple-choice questions (MCQs), Li and Gao [297] identified higher-layer mechanisms in GPT-2 that preferentially routed evidence toward anchored option tokens, providing concrete intervention loci. Generalizing beyond MCQs, Wang et al. [318] formulated position bias across judge-style evaluation and retrieval-augmented QA, introducing a mechanism to re-assign positions based on attention-derived importance signals. Complementarily, Yu et al. [319] traced “lost-in-the-middle” failures to a positional hidden-state channel and proposed a search-and-scale procedure to rescale this channel, improving robustness on long-context benchmarks. Extending to domain-specific decision-making, Dimino et al. [320] localized mid-to-late transformer layers as core “bias engines” driving positional skew in financial advisory tasks. Finally, regarding moral judgment, Raimondi et al. [138] analyzed the Knobe effect, localized its mediation to residual activations, and reduced the intentionality attribution gap by patching fine-tuned states with their pretrained counterparts, selectively reverting value shifts introduced during alignment.

5.1.3. Persona and Role

1) Global Persona Modulation via Vectors

A growing body of work suggests that complex persona-specific behavioral traits can be manipulated by intervening in the global activation state of the model. Rimsky et al. [162] utilized Contrastive Activation Addition, a form of Vector Arithmetic, to steer models away from sycophantic and hallucinatory behaviors. By extracting steering vectors from the residual stream differences between positive and negative examples, they demonstrated that high-level alignment properties can be precisely modulated during inference without fine-tuning. Expanding on this, Chen et al. [194] developed an automated pipeline to extract “Persona Vectors” for arbitrary traits (e.g., “evil” or “sycophantic”) using natural language descriptions. They found that these vectors not only allow for post-hoc steering but can also be used to predict and mitigate unintended persona shifts (e.g., emergent misalignment) that occur during fine-tuning by monitoring the projection of training data onto these vectors. Poterti et al. [193] applied this concept to professional domains, constructing “Role Vectors” (e.g., Chemist, Doctor) from model activations. Their analysis revealed that reinforcing these role-specific directions significantly improves performance on domain-specific tasks and even yields cross-domain benefits, suggesting that role-playing is mechanistically grounded. Furthermore, Pai et al. [321] proposed BILLY, a training-free framework that blends multiple persona vectors (e.g., Creative Professional + Environmentalist) to simulate collective intelligence within a single model. This approach steers the model with a composite vector, enhancing creativity and diversity in generation without the computational cost of multi-agent systems. Similarly, Sun et al. [37] explored the task vector (i.e., the steering vector in the context of model merging), extracting personality vectors by subtracting pre-trained weights from fine-tuned ones. They showed that these vectors can be linearly composed to continuously modulate trait intensity (e.g., Extraversion) across different models. Finally, Handa et al. [195] conducted a rigorous comparative study of personality manipulation methods using the Big Five traits. Their results showed that Vector Arithmetic provides a lightweight yet effective approach for controlling model personas at inference time.

2) Persona-Specific Component Editing

Rather than steering the global state, this paradigm seeks to identify and edit the specific neural components responsible for personality expression. Deng et al. [197] proposed NPTI, a method that identifies “personality-specific neurons” by applying Magnitude Analysis on the activation differences between opposing trait descriptions (e.g., Extraversion vs. Introversion). By selectively activating or deactivating these neurons via Amplitude Manipulation, they achieved fine-grained control over the model’s personality without model training. Su et al. [196] extended this to ethical values, introducing ValueLocate. They constructed a dataset based on the Schwartz Values Survey [322], a well-established framework that classifies values into four dimensions: Openness to Change, Self-transcendence, Conservation, and Self-enhancement. Using this dataset, they located value-critical neurons and demonstrated that controlling them via Amplitude Manipulation can effectively alter the model’s value orientation. Addressing the specific issue of sycophancy, Chen et al. [198] identified a sparse set of attention heads (∼4%) that significantly contribute to “yes-man” behavior. They proposed Supervised Pinpoint Tuning, a form of Targeted Optimization, which fine-tunes only these specific heads while freezing the rest of the model, successfully mitigating sycophancy while preserving general reasoning abilities better than standard instruction tuning.

3) Psychological Profiling and Diagnosis

MI techniques are also extensively used to understand how psychological constructs are represented internally by applying Probing. Tak et al. [199] investigated emotion inference, finding that emotion processing is functionally localized in MHA units within middle layers. They validated this by showing that interventions on latent “appraisal concepts” (e.g., pleasantness) predictably shift the generated emotional tone. Yuan et al. [200] explored how language identity affects psycholinguistic traits like sound symbolism and word valence. Their Probing analysis revealed that these signals become decodable in deeper layers and that language conditioning (e.g., bilingual persona) significantly modulates internal representations. Ju et al. [201] introduced a layer-wise Probing framework to analyze the Big Five personality traits, discovering that personality information is predominantly encoded in the middle and upper layers. They further proposed a method to edit response personality by applying Amplitude Manipulation to perturb hidden states orthogonal to the probing boundaries. In the realm of truthfulness, Joshi et al. [323] proposed the “persona hypothesis,” suggesting LLMs model truthfulness by inferring a “truthful persona” from the context. They provided evidence that Probing for this persona can predict the truthfulness of generated answers. Ghandeharioun et al. [324] utilized techniques like Patchscopes to reveal “latent misalignment,” showing that user personas (e.g., “altruistic” vs. “selfish”) significantly affect a model’s willingness to answer harmful queries, mediated by internal interpretations of the user’s intent. For user-facing transparency, Karny et al. [202] developed an interface that visualizes “Persona Scores” derived from neural activations. Their user study highlighted that users often miscalibrate their expectations of model behavior, and neural transparency tools can bridge this gap. Finally, Banayeeanzade et al. [203] introduced PsySET, their evaluation results showcase that although Vector Arithmetic steering is effective for modulating persona traits, it can introduce unintended side effects, such as “joy” steering reducing privacy awareness or “anger” steering increasing toxicity, necessitating rigorous safety evaluations. Bas and Novak [38] further differentiated between steering “internal dispositions” versus “external knowledge”, finding that steering method such as Vector Arithmetic is highly effective for latent traits (e.g., personality) but struggles with knowledge-heavy personas (e.g., specific public figures), where it often degrades coherence.

5.2. Improve Capability

5.2.1. Multilingualism

1) Language-Specific Component Manipulation

A central line of multilingual MI research shows that multilingual capabilities in LLMs are supported by a relatively small subset of internal components exhibiting strong language specificity. Accordingly, existing work has focused on localizing these components and manipulating their activations to control output language or enhance multilingual performance. Zhao et al. [204] formalized this observation through a layered Multilingual Workflow (MWork), showing that representations became English-centric in intermediate layers and were mapped back to the query language in later layers. They employed PLND to localize language-specific neurons and showed that intervening on only a tiny fraction could sharply disrupt multilingual performance, while selectively updating these neurons via Target Optimization enabled data-efficient language-specific adaptation. Along similar lines, Tang et al. [2] introduced Language Activation Probability Entropy (LAPE) as a Magnitude Analysis tool to quantify cross-language activation selectivity. Their results showed that language-specific neurons concentrated in the bottom and top layers, and that applying Amplitude Manipulation to these neurons provided direct control over output language, effectively reducing off-target generation. Complementing these localization-driven studies, Kojima et al. [325] analyzed neuron activation patterns across languages using Magnitude Analysis and confirmed the functional importance of language-specific neurons through Amplitude Manipulation, including targeted ablation and scaling. Together, these results reinforced component-level activation control as a practical mechanism for multilingual intervention.

Beyond identifying individual language-specific neurons, Gurgurov et al. [30] systematized neuron-level Amplitude Manipulation through Language Arithmetic, demonstrating that language-specific neurons exhibited additive properties that enabled controlled language switching and interpolation via linear operations on activations. In parallel, related work extended this paradigm beyond language identity to other forms of linguistic specialization. Liu et al. [205] identified relation-specific neurons whose activation patterns generalized across languages in multilingual factual probing tasks, demonstrating that neuron-level specialization could transfer cross-lingually beyond language identity. At a finer representational granularity, Jing et al. [206] employed SAEs to extract and analyze a wide range of interpretable linguistic features whose activation patterns varied systematically across languages, and showed that Amplitude Manipulation of these features could causally affect corresponding linguistic behaviors. Similarly, Andrylie et al. [29] identified language-specific SAE features via Magnitude Analysis and demonstrated that Amplitude Manipulation on these features enabled fine-grained control over multilingual behavior. Finally, Brinkmann et al. [207] showed that SAE-based representations captured shared, cross-lingual grammatical abstractions, with targeted feature-level analyses providing supporting evidence.

2) Cross-Lingual Representation Steering in Residual Space

A second major paradigm improves multilingual behavior by intervening on internal representations in the residual stream, where language representations are progressively transformed and aligned across layers. Chi et al. [210] showed that cross-lingual transfer could be activated without task-specific supervision by restructuring and aligning multilingual representations across model components, suggesting that pretrained models encoded latent cross-lingual structure that could be activated without end-task data. To localize where multilingual representations diverged or aligned, several studies relied on Vocabulary Projection. In particular, Wendler et al. [98] revealed that multilingual models often operated in an English-centric latent space during intermediate layers, even for non-English inputs, motivating interventions in later layers to restore language-faithful generation. Complementary representation-space analyses further supported this view: Philippy et al. [208] analyzed the relationship between language distance and representation divergence, while Mousi et al. [209] studied alignment dynamics in shared multilingual spaces using clustering-based metrics, together characterizing how cross-lingual alignment evolved across layers.

Building on these localization insights, subsequent work intervened more directly on internal representations to influence multilingual behavior. Hinck et al. [211] analyzed English-dominant responses in vision-language models and showed that targeted Vector Arithmetic on internal attention and hidden states could mitigate this bias. More recent studies moved from localization to failure-mode diagnosis with MI tools. Using layer-wise Vocabulary Projection and representation analysis, Wang et al. [213] attributed cross-lingual factual inconsistency to late-layer transitions into language-related subspaces, while Liu et al. [215] traced when multilingual factual knowledge emerged across pretraining checkpoints, providing a developmental view of cross-lingual consistency rather than a post-hoc intervention account. Complementarily, Wang et al. [214] analyzed language mixing in reasoning by characterizing when and how internal states drift between languages during generation. Nie et al. [130] further combined late-layer lens-style analysis with targeted neuron-level interventions (Amplitude Manipulation) to mitigate language confusion.

5.2.2. Knowledge Management

1) Precise Knowledge Updating

MI-based knowledge updating shares a common workflow: First localizes carriers responsible for a target association, then intervenes either at the parameter level (persistent) or activation level (reversible), with careful measurement of locality and collateral effects.

• Localized Parameter Rewriting: A core result is that many factual associations are mediated by localized pathways, often concentrated in mid-layer FFN output ${\mathbf{h}}_{\mathrm{ffn}}^l$ and neuron activation state ${\mathbf{s}}^l$. Meng et al. [43] used Causal Attribution to identify carriers responsible for factual recall and applied structured weight edits (on FFN matrices such as ${\mathbf{W}}_{\mathrm{out}}^l$) to rewrite specific associations, a process referred to as Targeted Optimization, providing a mechanistic alternative to diffuse fine-tuning. Scaling beyond single edits, Meng et al. [144] extended this paradigm to large edit batches by coordinating updates across multiple layers, demonstrating that persistent rewriting could remain localized while handling substantial edit volume. Subsequent work refined the localization premise: Chen et al. [216] argued that editability was frequently query-conditioned, motivating consistency-aware localization under a broader Query Localization assumption, rather than a fixed set of knowledge neurons. For long-form QA, Chen et al. [27] introduced QRNCA (a form of Causal Attribution), which yielded actionable neuron groups that better tracked query semantics. In multilingual settings, Zhang et al. [150] identified language-agnostic factual neurons via Magnitude Anaylysis and applied Targeted Optimization on these shared neurons to improve cross-lingual edit consistency. In backward propagation, Katz et al. [217] complemented forward analyses with Vocabulary Projection of backward-pass gradients, offering an orthogonal diagnostic on where learning signals concentrated during updates.
• Activation-Space Editing and Unlearning: When persistent rewrites are undesirable (e.g., reversible control or safety-motivated removal), activation-level interventions on residual stream states ${\mathbf{x}}^l$ at layer l, or on head/feature activations, provide a practical alternative. Lai et al. [218] jointly localized and edited attention-head computations (intervening on attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$) through gated activation control, instantiating a targeted form of Targeted Optimization. SAE-based approaches decomposed residual stream states ${\mathbf{x}}^l$ into sparse features with activations $\mathbf{a}$, enabling feature-level interventions: Muhamed et al. [219] proposed dynamic SAE guardrails that selected and scaled relevant features via Magnitude Analysis to achieve precision unlearning with improved forget–utility trade-offs, while Goyal et al. [131] applied Amplitude Manipulation to steer toxicity-related SAE features (scaling selected ${\mathbf{f}}_j$ via their activations $a_j$) to reduce harmful generations with controlled fluency impact.

2) Knowledge Retention and Stability

Retention work traced failures induced by repeated updates or context injection to identifiable carriers, and stabilized behavior via inference-time suppression or training-time adaptation, guided by MI diagnostics. Residual stream states ${\mathbf{x}}^l$ and attention head outputs ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$ are often key objects of intervention.

• Conflict Suppression and Mitigation: Failures under retrieval or context injection often arose from attention heads that mediated the integration of parametric memory and external evidence in the residual stream. Jin et al. [220] performed Causal Attribution to localize conflict-mediating heads and applied test-time head suppression/patching, i.e., Amplitude Manipulation over attention head output ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$, to rebalance memory vs. context usage. Li et al. [221] further used Magnitude Anaylysis to identify heads exhibiting superposition effects and applied targeted gating via Targeted Optimization to stabilize behavior under conflicts. Long-context distraction was traced to entrainment-related heads: Niu et al. [136] localized such heads using Causal Attribution and ablated or modulated their outputs (${\mathbf{h}}_{\mathrm{attn}}^{l,h}$), reducing echoing of irrelevant context tokens. Jin et al. [9] further characterized concentrated massive values in computations mediated by the Q/K weight matrices ${\mathbf{W}}_Q^{l,h}$ and ${\mathbf{W}}_K^{l,h}$ (reflected in attention scores ${\mathbf{A}}^{l,h}$ via Magnitude Analysis), then guided Amplitude Manipulation over corresponding head outputs ${\mathbf{h}}_{\mathrm{attn}}^{l,h}$ to maintain contextual reading without disrupting magnitude-structured signals.
• Constraining Continual Adaptation: To reduce catastrophic forgetting, MI localized stability-critical carriers and restricted learning via Targeted Optimization. Zhang et al. [74] applied Gradient Detection to identify a “core linguistic” parameter region and froze it, mitigating forgetting. Zhang et al. [66] further constrained adaptation through coarse-to-fine module selection and soft masking, balancing specialty and versatility. Representation-level interventions were also employed: Wu et al. [222] localized residual stream states ${\mathbf{x}}^l$ and applied lightweight edits on ${\mathbf{x}}^l$ with a frozen backbone (a form of Targeted Optimization), improving stability relative to weight-centric updates. Monitoring side effects, Du et al. [88] used Probing over residual stream states and attention heads to detect security-relevant drift and selected safer module update schedules, enabling controlled adaptation.

3) Knowledge Consolidation

Consolidation composes multiple specialized models by combining internal carriers while controlling interference. A common approach represents each fine-tuned model as a parameter “task vector” (a delta from a shared base) and merges these deltas via Vector Arithmetic.

Yadav et al. [224] improved multi-model composition over naive averaging by first trimming task vectors (a form of Magnitude Analysis), resolving sign conflicts, and then merging consistent update directions. Sens-Merging [72] further computed layer-wise sensitivity scores via Gradient Detection to weight deltas during merging, yielding stronger merged performance across diverse capability suites. Differently, Yao et al. [142] used Magnitude Analysis over layer-specific task vectors to derive importance scores that modulate merge weights, improving alignment of merged models with dominant capability directions. Beyond parameter deltas, Chen et al. [223] showed that affine mappings between residual stream states (a form of Probing) could transfer linear features across models, enabling consolidation at the level of feature bases and amortizing training cost across model sizes.

5.2.3. Logic and Reasoning

1) Specific Refinement of Numerical and Logical Components

LLMs often struggle with precise arithmetic operations. Rather than treating the model as a monolith, MI research has demonstrated that mathematical abilities are often localized within specific sub-modules. Quirke and Barez [225] conducted a granular circuit analysis of modular addition, identifying that specific attention heads and MLP layers form a dedicated algorithm for numerical processing. By characterizing these circuits, they demonstrated that targeted interventions on these specific components could predictably alter the model’s output distribution. Yang et al. [226] analyzed the activation dynamics of CoT processes, revealing that reasoning tasks predominantly activate a broader set of neurons in the final layers compared to standard prompting. Leveraging such insights, Zhang et al. [153] proposed an “identify-analyze-finetune” pipeline. This method first identified “reasoning-critical” attention heads and FFNs via Causal Attribution, then froze most model parameters and performed Targeted Optimization exclusively on these identified components to boost computational performance. Similarly, Tan et al. [160] decomposed the language model policy into “Internal Layer Policies.” Identifying that early layers maintain high entropy to facilitate exploration, they proposed Bottom-up Policy Optimization (BuPO), a method that selectively optimizes these foundational layers to refine the model’s internal reasoning policy efficiently.

2) Inference Trajectory Steering

Beyond basic arithmetic, complex reasoning requires the adoption of effective strategies. MI methods enable the extraction and injection of these high-level cognitive patterns by manipulating the model’s internal representations via Vector Arithmetic and Amplitude Manipulation. Researchers have extensively utilized steering vectors to modulate reasoning behaviors. Venhoff et al. [227] utilized contrastive activation means to extract “Backtracking Steering Vectors,” demonstrating that injecting this vector increases the model’s tendency to self-correct. Similarly, Højer et al. [233] and Tang et al. [234] derived control vectors from residual streams to elicit reasoning capabilities; notably, Tang et al. [234] showed that “Long Chain-of-Thought” capabilities can be unlocked via representation engineering without extensive fine-tuning. Hong et al. [235] identified a single linear feature direction that mediates the trade-off between reasoning and memorization, allowing for causal control over the model’s problem-solving mode. Zhang and Viteri [236] discovered “Latent CoT” vectors that, when injected, induce reasoning patterns without explicit natural language prompting. For more granular control, Liu et al. [237] introduced “Fractional Reasoning,” which enables continuous adjustment of reasoning intensity at inference time by scaling latent steering vectors. Efficiency is also a key benefit; Sinii et al. [238] demonstrated that training a single steering vector (bias-only adaptation) matches the reasoning performance of fully RL-tuned models. Taking a different approach, Wang et al. [230] proposed an optimization-based framework: instead of training weights, they optimized hidden representations directly to maximize the likelihood of reasoning paths, utilizing these optimized states to guide the model’s trajectory. Li et al. [239] proposed a dual framework, utilizing SAEs to extract interpretable reasoning features while also introducing an SAE-free algorithm to compute steering directions directly from residual activations. Galichin et al. [3] employed SAEs and introduced “ReasonScore” (a form of Magnitude Analysis) to identify sparse features associated with uncertainty and exploratory thinking. By amplifying these features via Amplitude Manipulation, they successfully guided the model toward more robust reasoning. Troitskii et al. [229] focused on latent states preceding “wait” tokens. They located specific features that promote or suppress these tokens and showed that modulating them fundamentally alters the subsequent reasoning process. Regarding latent states, Cywiński et al. [232] demonstrated the feasibility of transplanting reasoning patterns. By employing Causal Attribution to localize critical latent vectors and subsequently applying patching (a form of Amplitude Manipulation), they effectively forced the model to adopt specific latent reasoning paths.

3) Stepwise Diagnosis and Correction

A major challenge in multi-step reasoning is error propagation. MI provides tools for real-time internal diagnosis. Sun et al. [231] introduced a Probing framework to detect reasoning failures. They trained lightweight classifiers on the model’s internal activations to distinguish between correct and hallucinatory reasoning steps, acting as an internal monitor to flag errors before the final output is generated. Taking a probabilistic perspective, You et al. [87] introduced ARES, a framework that employs Magnitude Analysis on the entailment probability of internal states. They found that distinct uncertainty patterns emerge when the model deviates from logic. Based on this, they proposed a self-correction mechanism: when the internal monitor detects high “reasoning uncertainty,” the model is triggered to backtrack and regenerate the current step, significantly improving the reliability of long-chain deductions. Finally, Wu et al. [60] employed Gradient Detection to compute feature attribution scores for tokens within CoT traces. While primarily analytic, this method serves as a potential diagnostic tool by quantifying the semantic influence of each reasoning step, allowing researchers to verify whether the model is attending to relevant logic or spurious correlations during generation.

5.3. Improve Efficiency

5.3.1. Efficient Training

1) Sparse Fine-tuning

Unlike PEFT methods that introduce external modules [326,327,328], achieve efficiency by fine-tuning intrinsic subsets, often matching or exceeding the performance of full fine-tuning. At the neuron granularity, researchers utilize diagnostic tools to pinpoint task-specific units. Zhu et al. [154] proposed the LANDeRMT framework, which employs Taylor expansion to evaluate the “awareness score” of FFN neurons for machine translation, enabling Gradient Detection-based selective update of language-general and language-specific neurons to mitigate parameter interference. Song et al. [240] introduced SIFT, which exploits the “quasi-sparsity” of pre-trained gradients—where the top 1% of components can account for 99% of the total gradient norm—using hook functions to perform memory-efficient in-place sparse updates via Gradient Detection. Similarly, Xu et al. [147] developed NeFT, which identifies sensitive neurons through Magnitude Analysis by calculating the cosine similarity between weights before and after a brief full-parameter fine-tuning run. Furthermore, Mondal et al. [241] and Gurgurov et al. [242] leveraged Language Activation Probability Entropy [2] to identify language-sensitive neurons via Magnitude Analysis, achieving significant gains by updating less than 1% of the model.

More granular approaches achieve massive efficiency by isolating extremely sparse mechanistic components. Zhao et al. [204] proposed Parallel Language-specific Neuron Detection, identifying consistently activated neurons for specific languages without labeled data via Causal Attribution; they found that deactivating just 0.13% of these neurons causes a total loss in multilingual generation. Sergeev and Kotelnikov [36] introduced Head Impact scores based on Magnitude Analysis to identify attention heads, demonstrating that fine-tuning only 0.01% of parameters in the highest-impact layers significantly improves model understanding capability. Lai et al. [218] proposed JOLA, a framework that employs HardConcrete gates with expected-$L_0$ regularization to jointly learn which attention heads to edit and whether to apply additive or multiplicative interventions. Furthermore, Li et al. [243] reframed fine-tuning as a “subgraph search” process, introducing a circuit-tuning algorithm that iteratively builds and optimizes a task-relevant circuit via Circuit Discovery within the computational graph to preserve general model capabilities.

2) Training Dynamic Monitoring

The second paradigm predominantly leverages Magnitude Analysis and other quantitative diagnostics to monitor the state evolution of internal objects, addressing the limitations of traditional validation loss in capturing critical phase transitions.

In the context of Grokking—where generalization emerges long after overfitting—MI metrics provide crucial signals that enable practitioners to confidently continue training despite zero progress in validation loss. Understanding that grokking arises from the competition between fast-learning memorization circuits and slow-learning but efficient generalization circuits [329,330], researchers have developed specific indicators to track the latter’s formation. Nanda et al. [246] proposed Restricted Loss, a metric derived by projecting weights onto the Fourier basis, which reveals that structured mechanisms form gradually during the apparent loss plateau. Similarly, Furuta et al. [248] introduced Fourier Frequency Density (FFD) to characterize the sparsity of internal representations; tracking FFD allows for real-time assessment of generalizability, serving as a reliable proxy for circuit maturation. Moving to early detection, Notsawo Jr et al. [331] analyzed the spectral signature of the training loss curve itself, demonstrating that specific low-frequency oscillations in early epochs can effectively predict whether grokking will eventually occur, thus saving computational resources on unpromising runs. In the realm of Mixture-of-Experts (MoE), Li et al. [250] applied Magnitude Analysis on router activations and proposed two pathway metrics—similarity and consistency. These metrics monitor how routing patterns evolve from random fluctuations to stable structures, serving as a precise indicator to determine the onset of grokking and enabling optimal early stopping.

Beyond grokking, similar monitoring strategies are applied to the emergence of In-Context Learning (ICL). Hoogland et al. [244] utilized the Local Learning Coefficient (LLC) from singular learning theory to quantify the geometry of the loss landscape. They observed that plateaus in the LLC curve distinctively mark developmental stages (e.g., from bigram statistics to induction heads), allowing researchers to determine when a model has completed a specific structural transformation. Furthermore, Minegishi et al. [245] extended this to In-Context Meta-Learning, developing circuit-specific metrics such as label attention scores. By monitoring the shift in these metrics, they identified that models progress through multiple distinct phases (Non-Context → Semi-Context → Full-Context), providing a granular “progress bar” for the model’s acquisition of meta-learning capabilities that is invisible to standard loss evaluation.

5.3.2. Efficient Inference

1) Selective Computation via Saliency Detection

The core premise of selective computation is that not all architectural or data components contribute equally to the final output. MI provides a principled tool to quantify such contributions and to prune redundant components accordingly.

• Data Level: Researchers have developed advanced token- and KV-cache-level pruning strategies that leverage Magnitude Analysis and Gradient Detection to effectively identify and remove unimportant tokens. By leveraging Magnitude Analysis to identify tokens with minimal contribution to the reasoning process in CoT sequences, TokenSkip [159] selectively skips these tokens, achieving substantial compression with negligible performance degradation. Lei et al. [251] explored explanation-driven token compression for multimodal LLMs, where Gradient Detection is used to map attention patterns to explanation outcomes, enabling the effective pruning of visual tokens during the input stage. For KV cache-level pruning, FitPrune [253] and ZipCache [21] employed Magnitude Analysis saliency metrics to identify and retain critical KV states. Guo et al. [252] introduced Value-Aware Token Pruning (VATP), which applied Magnitude Analysis to attention scores and the L1 norm of value attention vectors to identify crucial tokens. Moving beyond token-wise pruning, Circuit Discovery techniques have been applied to identify “Retrieval Heads” that are essential for long-context tasks, enabling non-critical heads to operate with a fixed-length KV cache [155,254,332,333].
• Model Level: MI-guided metrics enable the skipping of entire architectural blocks, such as redundant layers, MoE experts, or neurons, thereby facilitating inference acceleration with minimal impact on model performance. Men et al. [14] introduced “Block Influence” (BI), a similarity metric based on Magnitude Analysis that compares the input and output of each layer. This technique effectively removes layers with minimal contribution to the representation space. Dynamic bypassing methods, such as GateSkip [255] and LayerSkip [13], employ learnable residual gates to skip layers during inference, also based on Magnitude Analysis. Similarly, HadSkip [257] and SBERT [258] models leverage Magnitude Analysis to facilitate effective layer skipping. In MoE architectures, Lu et al. [259] skipped unimportant experts during inference based on the Magnitude Analysis of router scores. Su et al. [8] further identified Super Experts by analyzing the Magnitude Analysis of experts’ output activations, showing that these experts are essential for logical reasoning and that pruning them leads to catastrophic performance degradation. Finally, by localizing specialized multilingual neurons [25] and language-specific sub-networks [260] through Magnitude Analysis on their activations, LLMs can activate only the sub-circuits necessary for the specific task at hand.

2) Layer-Specific Adaptive Quantization

While standard quantization applies a uniform bit-width across all parameters, MI-driven research promotes mixed-precision quantization based on layer-wise “functional saliency.” Many of these metrics are based on Magnitude Analysis to identify sensitive layers. Dumitru et al. [39] proposed a pragmatic approach to measure layer importance by examining shifts in the embedding space or the presence of weight outliers, assigning higher bit-precision to layers that caused larger representational shifts. Similarly, Zhang et al. [4] introduced SensiBoost and KurtBoost, which used activation sensitivity and weight distribution kurtosis to identify layers that were "hard-to-quantize," allocating them more memory budget. LieQ [261] further uncovered a strong correlation between training-induced energy concentration and representational compactness, providing a geometry-driven sensitivity proxy for automatic bit-width allocation. Beyond static analysis, Mix-QViT [262] employed Layer-wise Relevance Propagation (LRP)—a form of Gradient Detection—to assess the contribution of each layer to the final classification, thereby guiding mixed-precision quantization in vision transformers. LSAQ [263] adaptively adjusted quantization strategies in real-time by applying Vocab Projection to obtain the vocab distribution for each layer. It then calculated the Jaccard similarity between these distributions to identify sensitive layers, ensuring that they maintained high precision while more robust layers were aggressively compressed to meet the resource constraints of edge devices.

6. Challenges and Future Directions

Challenges

Despite substantial progress and growing methodological sophistication, it remains unclear whether MI is indispensable for any downstream task, rather than serving as an alternative or complementary analysis tool. This uncertainty amplifies the importance of the fundamental challenges discussed below, which continue to limit the scalability, reliability, and practical impact of MI.

First, MI remains difficult to scale beyond low-level components [334,335]. While individual neurons or learned features are increasingly well-characterized [127,336], identifying higher-level computational structures, such as multi-layer interactions, cross-module pathways, or distributed mechanisms, still relies heavily on manual inspection [119,337,338,339,340]. Although recent work has made progress toward automation [121,123], current methods often require substantial human intervention and do not robustly generalize across prompts, tasks, or models [129,250,341]. As a result, many MI analyses remain artisanal rather than systematic. In addition to these methodological limitations, computational scalability poses a major bottleneck. Prominent approaches such as SAEs or transcoders rely on training replacement or surrogate models to obtain more interpretable representations, introducing additional training costs that grow with model size and feature dimensionality. This often restricts their application to a limited subset of layers or models. A related challenge arises in fine-grained causal localization. Precisely attributing behavior to individual neurons or SAE features would in principle require exhaustive interventions, but the scale of modern LLMs renders such causal tracing computationally infeasible [44,123]. As a result, most analyses [122,128,342,343] operate at coarser granularities or rely on heuristic approximations, limiting the resolution at which mechanisms can be reliably identified.

Second, the field lacks robust and widely accepted evaluation frameworks to assess the faithfulness of localization and explanation methods [344]. Although some benchmarks [126,345,346,347,348] have been proposed, there remains no consensus on metrics that can determine whether an identified component truly corresponds to the underlying causal mechanism. This issue is particularly acute for methods that rely on surrogate or replacement models, where output-level agreement does not guarantee mechanistic fidelity. Importantly, the scalability constraints discussed above further exacerbate this problem. Because exact fine-grained causal interventions are computationally impractical, researchers must rely on approximate localization methods designed for tractability rather than optimal causal identification. In the absence of reliable ground truth at the mechanism level, it becomes difficult to distinguish true causal components from computationally convenient proxies, making rigorous validation and comparison of MI methods inherently challenging.

Third, current mechanistic analyses often face a fundamental trade-off between sparsity and completeness of representation [140,349]. Many interpretability methods, including SAEs and other sparse decomposition techniques, aim to force the model’s internal representations into a small set of monosemantic, easily interpretable components. By promoting sparsity, these methods can disentangle polysemantic neurons and highlight feature directions that correspond to specific concepts, making interpretation more tractable. However, aggressively enforcing sparsity may prune or obscure components that are genuinely part of the true mechanism but do not fit a sparse pattern. This leads to a tension: methods that induce sparsity can improve interpretability but risk overlooking distributed or “inactive” subcomponents of genuine mechanisms, while approaches that preserve dense, distributed representations may be harder to interpret systematically. Accounting for this trade-off, and developing evaluation metrics that balance sparsity, fidelity, and mechanistic completeness, remains an open challenge for MI.

Finally, interventions informed by MI, such as model editing or steering, often lack robustness and predictability [168,350]. Changes intended to modify a specific behavior can introduce unintended side effects on other tasks or domains, raising concerns about generalization and reliability [66,351,352,353,354,355,356,357]. For instance, Yu and Ananiadou [109] demonstrate that modifying a very small number of neurons can lead to substantial degradation in overall language performance. The need for accurate target localization and steering methods that avoid collateral behavioral disruption remains a central technical challenge as MI increasingly informs targeted intervention design.

Future Directions

Looking forward, several directions appear particularly promising for advancing MI. A key priority for mechanistic interpretability is to move from isolated, low-level analyses toward integrated, system-level explanations. Most existing MI work focuses on task-specific and localized mechanisms, such as knowledge neurons, safety-related neurons, arithmetic heads, or specific task circuits for in-context learning or arithmetic [30,151,153,186,358,359,360,361]. While informative, these approaches are inherently low-level and offer limited insight into how models organize computation more broadly [280]. In contrast, cognitive science characterizes cognition in terms of higher-level systems, such as System 1 vs. System 2 reasoning [264], as well as attention, memory, language, and executive control systems [362,363,364,365]. Comparable system-level accounts in MI remain scarce. Developing such accounts requires frameworks that connect low-level components to higher-order organization, enabling more coherent system-level explanations of LLM computation [47].

In parallel, stronger theoretical foundations are needed. Connecting internal representations to principles from cognitive science [366,367,368] or information theory [369] may help unify disparate MI findings and reduce reliance on ad-hoc interpretations. A principled framework could also clarify what kinds of internal structures should be expected in large-scale models and why [370].

Finally, an emerging direction is the progression from interpretation to intervention and, ultimately, model design. Insights from MI are increasingly used not only to explain behavior, but also to edit, steer, or modularize models. This direction connects naturally to earlier work on intrinsically interpretable models, such as Concept Bottleneck Models [371,372,373,374,375,376,377] and Weight-sparse transformers [349], which enforce transparency through architectural constraints. However, despite their interpretability benefits, such models typically underperform black-box architectures on large-scale, complex tasks [378]. Looking forward, a key challenge is to bridge this gap by designing interpretable backbone architectures that can serve as viable alternatives to transformers, achieving interpretability by construction while maintaining performance comparable to state-of-the-art black-box models. In this sense, interpretability-informed design may move beyond post-hoc analysis toward fundamentally more controllable, customizable, and transparent model architectures.

7. Conclusions

In this survey, we systematically reframe MI from a predominantly observational endeavor into a practical, actionable paradigm. By organizing existing methods around the unified pipeline of “Locate, Steer, and Improve”, we clarify how interpretable objects can be precisely localized, causally manipulated, and ultimately leveraged to enhance alignment, capability, and efficiency in LLMs. Our analysis highlights that many recent advances—ranging from safety and persona alignment, to knowledge editing, and further to sparse fine-tuning—are most effective when grounded in explicit mechanistic intervention. We further discuss key challenges and future directions in §6, with the goal of providing a coherent foundation for future research that tightly integrates interpretability, intervention, and model design. Ultimately, we hope this perspective will accelerate the transition toward more powerful, transparent, and reliable LLMs.