Understanding LLM Agents Requires a Chain of Attribution: On the Interplay Between Mechanistic Interpretability and Behavioral Approaches

Mechanistic Interpretability (MI) is a subfield of interpretability seeking to explain the internal workings of a model. It has become a thriving subfield for interpreting large language models (LLMs). In this position paper, we argue that while MI can provide insight about LLMs in controlled and/or simplified task settings, it is not sufficient for interpreting LLMs in complex settings such as LLM agents. This is because understanding LLM agents, which consist of multiple interconnected components, operate over long trajectories, and interact with external environments, requires attributions beyond only the backbone model internals. Thus, we argue that interpreting LLM agents requires pairing MI with the traditional behavioral approaches, which examine the external input-output attributions of a model. As such, a chain of attributions tracing an agent’s action back to the output of the responsible component and timestep, then to the input that drove it, and finally to the internals of that component’s LLM backbone, can form a complete, traceable, causal, and actionable interpretation.