Theory of Agent: The Science of Internalization and Externalization for LLM-Based Agents

1. Introduction

An LLM-based agent is mainly understood as two coupled parts [1,2]: 1) model: the parametric weights that store factual and procedural knowledge and supply the ability to reason over them; and 2) harness: the external apparatus of context windows, memory stores, tool and skill libraries, retrieval interfaces, and orchestration logic through which the model observes the world and acts upon it. Nearly every advance of the past years can be located on one side or the other: larger and powerful models on one hand [3,4], and on the other a rapidly thickening harness of retrieval [5,6], long context [7], tool protocols [8,9], memory [10,11], and multi-agent scaffolds [12,13]. This shift has turned them into general-purpose problem solvers across different domains in the real-world, including web and computer use [14,15], deep research [16,17], and scientific discovery [18], where success depends less on answering a single query than on sustaining a long trajectory of decisions in a changing environment. Together, the distinction between model and harness reflects a single design decision at the center of building agents: where competence should live, internalized into the model’s parameters or externalized onto the surrounding harness.

Neither direction is a safe default, because each does something the other cannot, and the choice shapes not only immediate performance but the long-term trajectory of agentic intelligence.

• Internalization builds the model’s own competence. Compiling stable, recurrent knowledge and procedures into the weights lets the agent resolve them by reasoning, without reloading long context or paying for repeated external calls; exercising that internal competence is also how it grows over time, whereas offloading it wholesale leaves the model’s reasoning underdeveloped [19].
• Externalization keeps the agent correct and current. Fresh facts, exact computation, verifiable execution, and the live state of an environment are things no fixed set of weights can hold. Keeping them in the harness and retrieving or acting on them when needed grounds the agent’s decisions in the world rather than in what it happened to memorize [1].

Moreover, the possible ultimate goal is to internalize part or even whole world into the parametric space, and reducing the reliance on the external environments by simulating the consequences of an action without taking it in the real-world [20,21]. This architectural distinction runs deeper than it first appears, rooted in a fundamental difference between the two ways agents acquire knowledge or information: reasoning and acting. To reason is to draw on what has been internalized; to act is to reach for what remains external. This perspective unifies debates that grew in separate communities. Long-context versus RAG [22], parametric versus external memory [23], and tool-integrated reasoning (TIR) [24] versus tool-internalized reasoning (TInR) [25,26] all ask where competence should live for a different resource. Naming the axis makes methods once thought incommensurable directly comparable and shows that much of agent research converges on a single design problem: placing the internal–external boundary.

However, drawing this boundary well is difficult, because it does not behave like a fixed line and its placement is consequential, as shown in Figure 1. We consider there are several important properties make its placement demanding: 1) Probabilistic and implicit. An agent cannot read off which tasks lie inside its own competence, it can only estimate, as a probability rather than a binary, whether the knowledge a step needs is already internalized, then act on that estimate against a threshold encoding its risk and cost tolerance [27]. A miscalibrated estimate produces the most common failures: overthinking and underacting when the agent trusts knowledge it lacks [28], and overacting and underthinking when it distrusts knowledge it holds [29]. These four are not separate defects but one effort spent on the wrong side of the boundary; 2) Context-dependent. Where competence should sit depends on the model, the task, and the environment, none of which holds still. A step that a weak model must delegate, a stronger one can resolve internally [19]; a fact current today is stale tomorrow; and within a single run each retrieved result shifts what the model can now settle on its own. The boundary drifts inward as models absorb what the harness once supplied, and outward as new interfaces expose what they cannot yet hold; 3) Asymmetrically costly. Reaching outward buys grounding at the price of latency, money, irreversible side effects, and wider permission exposure [30]. An external action is worth taking only when internal reasoning cannot resolve the remaining uncertainty at acceptable cost and risk.

Most existing papers approach this landscape from the engineering side, cataloguing how individual harness components are built and how they connect with the model side [1]. We instead combine theory and practice, organizing the literature center around the trade-off between internalization and externalization. To this end, we draw on the Theory of Agent (ToA) [19], which recasts the agent as a decision-maker managing uncertainty at each step: it can reduce uncertainty either by reasoning over what it already holds (internalization) or by acting to obtain what it does not (externalization). ToA formalizes the structure of this decision and supplies a normative principle for boundary placement. This decision-level account has direct architectural consequences: during training, which competencies should be built into the model and which should be supplied by the harness; at deployment, how the agent should calibrate its reasoning-versus-acting choices under real-world constraints. Both levels shape reliability, cost, safety, and scalability, affecting the performance and intelligence level of agent.

Key Questions. We organize the remainder of the paper around four questions.

Q1. Learning. What capabilities should be internalized into the model, what should remain externalized in the harness, and how do we reason about this trade-off?

Q2. Aligning. How do we calibrate the trade-off between internalization and externalization to align with model capability, efficiency constraints, human preferences, environmental constraints, and other agents?

Q3. Evolving. How do agents revise their epistemic boundaries over time, both on the model side (internal evolution) and the harness side (external evolution), and how do agent–environment systems co-evolve?

Q4. Evaluation. How do we measure epistemic intelligence and boundary decisions rather than just task success rate?

Build on top of this foundation, we study several key research questions in this field and map them to specific part of the paper. The remainder of the paper is organized into five technical chapters and a discussion of open problems. Section 2 (Foundations) develops the ToA formalism: the agent as an epistemic decision-maker, reasoning and acting as complementary mechanisms of knowledge acquisition, and the notions of epistemic necessity, boundary, and effort allocation. Section 3 (Learning) examines how training reshapes the internal–external trade-off, covering internalization through pre-training, supervised fine-tuning, and reinforcement learning, and externalization through context, memory, tools, and workflow harnesses [6,31,32]. Section 4 (Aligning) treats calibration as the connecting thread across alignment with knowledge boundaries, human preferences, environments, and external collaborators [33,34,35]. Section 5 (Evolving) studies how agents scale reasoning and acting ability over time, through internal evolution of models, external evolution of harness, and co-evolution between agent and harness [2,36]. Section 6 (Evaluation) reinterprets benchmarks by the type of epistemic effort they expose rather than the domain they target, explains why success-rate scoring is structurally unable to surface over- and under-internalization, and outlines trade-off-oriented benchmark principles based on controllable information conditions, controlled access settings, full trajectory observation, multi-axis reporting, and longitudinal measurement. Section 7 closes with open problems for the second half of agent research. Throughout, our aim is to offer the internalization–externalization boundary as both a descriptive tool for reading the recent literature and a normative design principle for the agents that come next.

Figure 2. Overview of the paper structure. We first introduce the foundations of epistemic agents (Section 2), then organize the discussion around learning (Section 3), aligning (Section 4), and evolving (Section 5, and finally examine evaluation principles and benchmarks (Section 6).

Figure 2. Overview of the paper structure. We first introduce the foundations of epistemic agents (Section 2), then organize the discussion around learning (Section 3), aligning (Section 4), and evolving (Section 5, and finally examine evaluation principles and benchmarks (Section 6).

2. Foundations: Epistemic Agents

This section establishes the conceptual foundation for the rest of the paper following WANG et al. [19]. First, we review the standard Markov formulation commonly used to analyze agentic systems, where an agent is modeled as a policy that maps interaction histories to actions for task completion (Section 2.1). We then reinterpret the same framework from an epistemic perspective, viewing the agent as an epistemic decision-maker. Under this view, the agent’s actions are divided into two provenance-distinct operators: internal reasoning and external acting, both serving as mechanisms for knowledge acquisition (Section 2.2). Building on this formulation, we introduce the core concepts that underpin the remainder of the paper, including the internal and world task sets, the knowledge boundary between them (Section 2.3), and the normative notions of epistemic necessity, epistemic effort, and epistemic intelligence, which provide the unifying perspective of this paper. These definitions are used consistently throughout the subsequent chapters, allowing Section 3–Section 6 to build directly upon the foundations established in Section 2.

Figure 3. A taxonomy of AI agent research organized around learning, aligning, evolving, and evaluation. The structure follows the organization of Section 3, Section 4, Section 5 and Section 6; each row lists representative work cited in the corresponding part of the paper.

Figure 3. A taxonomy of AI agent research organized around learning, aligning, evolving, and evaluation. The structure follows the organization of Section 3, Section 4, Section 5 and Section 6; each row lists representative work cited in the corresponding part of the paper.

2.1. Preliminaries: The Markov Decision Process View of Agents

We begin with the formalism under which most existing agentic frameworks are described, so that the perspective developed in the rest of this section can be stated as a reinterpretation of familiar machinery rather than a new formalism.

The sequential decision problem.

A conventional agent is modeled as a policy interacting with an environment over discrete steps. When the environment state is fully observed, this is a Markov decision process (MDP) $\langle \mathcal{S},\mathcal{A},T,R,\gamma \rangle$: at step t the agent occupies a state $s_t\in \mathcal{S}$, selects an action $a_t\in \mathcal{A}$ from a policy $\pi (a_t\mid s_t)$, receives a reward $r_t=R(s_t,a_t)$, and transitions to $s_{t+1}\sim T(·\mid s_t,a_t)$. The objective is a policy that maximizes the expected discounted return ${\mathbb{E}}_{\pi }\left[{\sum }_t{\gamma }^t{r}_t\right]$.

Partial observability.

LLM agents almost never observe the environment state directly. A web agent sees a rendered page, not the server’s database; a coding agent sees test output, not the program’s full semantics; a research agent sees retrieved snippets, not the corpus. The appropriate model is therefore a partially observable MDP (POMDP) $\langle \mathcal{S},\mathcal{A},\mathcal{O},T,Z,b_0,c\rangle$, where $\mathcal{O}$ is an observation space, $Z(o_{t+1}\mid s_{t+1},a_t)$ an observation kernel, $b_0$ a prior over initial states, and c a cost function. Because the agent cannot condition on $s_t$, it conditions on its interaction history

$${\tau }_t\triangleq \left(q,a_1,o_1,\dots ,a_{t-1},o_{t-1}\right),$$

(1)

where q specifies the task. A policy is then a map $\pi (a_t\mid {\tau }_t)$ from histories to actions, and the agent maintains a belief$b_t=p(s_t\mid {\tau }_t)$, the posterior over the latent state given everything observed so far. This is standard POMDP machinery; we adopt it unchanged and build on top of it in Section 2.2.

Why this view is insufficient.

The task-success formulation is formally complete but analytically blunt for the systems we now build, for three reasons that the rest of this section addresses in turn. First, it conflates having an answer with knowing it to be correct: a binary success signal cannot distinguish a trajectory that resolved the task through grounded reasoning from one that arrived at the same answer by a fluent but unfounded guess, and a growing body of evidence shows that high success rates coexist with poor calibration, brittle generalization, and substantial wasted compute, especially on open-ended tasks where the correct procedure is itself uncertain. Second, it treats the action space as homogeneous: a reasoning step, a retrieval call, and an environment-altering action are scored only by their contribution to the terminal predicate, even though they differ sharply in cost, latency, reversibility, and the kind of information they supply. Third, and most consequentially, it provides no principled account of when an agent should act on the world at all rather than continue to reason, a decision that, we will argue, most directly governs whether an agent is intelligent and able to keep improving. The next subsection reframes the same POMDP around this decision.

2.2. Agents as Epistemic Decision-Makers

The shift we propose is to stop reading the agent as a function from a task to a terminal answer and start reading it as a process that reduces uncertainty about that answer. The formal objects are unchanged, including the same states, observations, histories, and policy, but the quantity of interest moves from the terminal success predicate to the agent’s evolving belief, and the action space is read by the provenance of the information each action supplies.

Belief and utility.

Let Y denote a task-relevant random variable, such as the answer to a question, the correct next action, or the latent state needed to commit, and let $b_t=p_{\theta }(Y\mid {\tau }_t)$ be the agent’s belief over Y at step t, formed from its parameters $\theta$ and history ${\tau }_t$. Associate with each belief a utility $U\left(b_t\right)$, for example the negative expected loss the user incurs by acting on $b_t$ under a downstream decision. A step is useful to the extent that it raises this utility,

$$\Delta U_t=U\left(b_{t+1}\right)-U\left(b_t\right)>0,$$

(2)

i.e. it resolves uncertainty in a way that improves the downstream decision. Under this reading, “solving a task” is the special case in which $\Delta U$ accumulates until the belief is sharp enough to commit to an action at acceptable risk. Crucially, utility is measured against the ground-truth answer distribution, not against the agent’s own posterior: a step that confidently consolidates probability mass on a wrong answer sharpens the belief but contributes zero or negative $\Delta U$. This is exactly the distinction that task success cannot make, an agent that maintains a meaningful, calibrated belief and one that reports low uncertainty while having learned little are epistemically different even when they emit the same correct output [140].

Agents as epistemic decision-makers.

With belief and utility in place, the agent’s job at each step is no longer simply to act, but to decide whether its current belief is sufficient to commit, or whether the remaining uncertainty should first be reduced and, if so, by what means and at what cost. This framing rests on a robust empirical fact: LLMs carry a non-trivial belief signal even before any external interaction, in the sense that their internal probabilities and hidden representations correlate with answer correctness [61,62]. The agent can therefore, in principle, assess whether further internal computation is worth its cost relative to acting on the world, and the central question of agent design becomes how that assessment is made, not merely what action is taken. This reframing does not discard the POMDP; it relocates the objective. The conventional agent optimizes terminal return; the epistemic agent is analyzed by how intelligently it converts effort into resolved uncertainty?

Reasoning and acting as knowledge acquisition.

The defining structural feature of an LLM agent, under our view, is not the size of its action space but its partition. We split the action set into two semantically distinct subsets that differ in the provenance of the information they supply:

$$\mathcal{A}={\mathcal{A}}_{\mathrm{int}}\cup {\mathcal{A}}_{\mathrm{ext}}.$$

(3)

• ${\mathcal{A}}_{\mathrm{int}}$: internal actions: reasoning steps that update the belief using parametric knowledge and current context alone, such as a chain-of-thought token, a reflection, a self-consistency vote, or an internal simulation. They consult what the model already holds.
• ${\mathcal{A}}_{\mathrm{ext}}$: external actions: tool calls, retrievals, code execution, environment interactions, or communication with other agents. They consult what lies outside the model.

An augmented transition kernel.

The two action types act on the POMDP differently, and writing the transition explicitly makes the distinction precise. An internal action produces an observation generated by the model itself and leaves the environment state untouched; an external action queries or alters the environment:

$$(s_{t+1},o_{t+1})=\left\{\begin{array}{cc}\left(s_t,f_m({\tau }_t,a_t)\right)\hfill & a_t\in {\mathcal{A}}_{\mathrm{int}},\hfill \\ \left(s_{t+1}\sim T(·\mid s_t,a_t),o_{t+1}\sim Z(·\mid s_{t+1},a_t)\right)\hfill & a_t\in {\mathcal{A}}_{\mathrm{ext}}.\hfill \end{array}\right.$$

(4)

Here $f_m$ is the model’s own generative map: an internal action’s "observation" is an output the model computes from its parameters and context ${\tau }_t$, extending the belief without contacting the world. An external action’s observation is drawn from the environment, reflects the true state $s_t$, and may change it. This is a conservative addition to the standard POMDP, serving as an epistemic annotation on the action space rather than a modification of the formalism. This is precisely why prior frameworks such as ReAct [151] are recovered as the special case in which internal and external steps are interleaved without being treated as distinct decisions.

The key claim of this subsection is that reasoning and acting are not different kinds of activity but two implementations of a single primitive: a knowledge operator that maps one belief to the next, $b_t\mapsto b_{t+1}$. Both reduce uncertainty about Y; they differ only in where the reducing information originates. Reasoning consults parametric memory and is bounded by what was internalized at training time; acting consults the environment and is bounded by what is queryable now. They differ in three further respects that the cost analysis of Section 2.4 will formalize: (i) reasoning is approximately free in money but pays in latency and tokens, whereas acting pays in money, time, and irreversibility; (ii) reasoning produces beliefs that are internally consistent but possibly ungrounded, whereas acting produces beliefs grounded in the world but possibly redundant or unsafe; (iii) reasoning is always available, whereas acting is gated by the harness. Recent papers of agentic reasoning and agentic RL already converge on this two-operator decomposition as a useful unit of analysis [1,152]; we adopt it as the foundation of the framework.

2.3. The Knowledge Boundary

The operator partition of Section 2.2 raises the question it cannot itself answer: for a given task, which uncertainties can be resolved by internal operators, and which genuinely require external ones? This is the central ontological object of the framework. We define it following the Theory of Agent [19].

Figure 4. The knowledge boundary of different model or agent.

Figure 4. The knowledge boundary of different model or agent.

Task sets.

Fix an agent model m and an environment $\mathcal{W}$. Let $\mathcal{Q}$ be the space of tasks. We define:

• the internal task set ${\mathcal{Q}}^{\mathrm{int}}(m,\mathcal{W})$: the set of tasks m can complete reliably using internal reasoning alone, without invoking external actions;
• the world task set ${\mathcal{Q}}^{\mathrm{world}}\left(\mathcal{W}\right)$: the set of all tasks in principle solvable in the environment given access to appropriate external interaction.

By construction ${\mathcal{Q}}^{\mathrm{int}}(m,\mathcal{W})\subseteq {\mathcal{Q}}^{\mathrm{world}}\left(\mathcal{W}\right)$. The complement

$${\mathcal{Q}}^{\mathrm{ext}}(m,\mathcal{W})={\mathcal{Q}}^{\mathrm{world}}\left(\mathcal{W}\right)\setminus {\mathcal{Q}}^{\mathrm{int}}(m,\mathcal{W})$$

(5)

is the set of tasks that require external interaction to be completed reliably by this agent. Note that ${\mathcal{Q}}^{\mathrm{ext}}$ being defined by necessity does not forbid using external tools on tasks in ${\mathcal{Q}}^{\mathrm{int}}$; it marks where external action is unavoidable, not where it is merely permitted.

The knowledge boundary.

The knowledge boundary $\mathcal{B}$ of an agent is the separation between its internal task set and the world task set, namely the surface that divides what the agent can resolve by reasoning from what it must resolve by acting. The boundary is the geometric object that the paper’s two core processes act on: internalization pushes $\mathcal{B}$ outward by absorbing competence into $\theta$ and enlarging ${\mathcal{Q}}^{\mathrm{int}}$; externalization leaves $\mathcal{B}$ where it is and offloads the residual onto the harness.

However, the boundary $\mathcal{B}$ is a non-trivial object: 1) It is model-specific: different agents exhibit different internal task sets, because they differ in training data, architecture, memory, and available context. There is therefore no single tool-use policy that is uniformly appropriate across agents: a call that is unnecessary delegation for a strong model may be epistemically necessary for a weak one; 2) ${\mathcal{Q}}^{\mathrm{int}}$ is latent: it cannot be observed directly, so the agent must act on an internal estimate of whether a task lies inside its boundary. Tool use is consequently a belief-based classification of the task relative to $\mathcal{B}$, not a decision against a fixed, known boundary.

Operationalizing the boundary.

Because $\mathcal{B}$ is latent, the agent works with a context-conditioned estimate of internal solvability. For a task q, model m, and history ${\tau }_t$, define

$$p_t^{\mathrm{int}}(q,m;\mathcal{W})\triangleq Pr\left(S=1\mid q,m,{\tau }_t,\mathcal{W},\pi \in {\Pi }_{\mathrm{int}}\right),$$

(6)

the probability that the task is completed successfully ($S=1$) using only internal policies ${\Pi }_{\mathrm{int}}$ given the available context. This is a belief-like surrogate the agent uses to estimate whether the current task lies within its internal task set; it can be approximated in practice by self-consistency, rollout agreement, draft-reasoning confidence, or hidden-state probes [153,154]. A policy-dependent threshold $\alpha \in (0,1)$ then induces the natural decision rule

$$\pi (a_t\mid {\tau }_t)=\left\{\begin{array}{cc}{a}_t\in {\mathcal{A}}_{\mathrm{int}}\hfill & p_t^{\mathrm{int}}(q,m;\mathcal{W})\ge \alpha ,\hfill \\ a_t\in {\mathcal{A}}_{\mathrm{ext}}\hfill & p_t^{\mathrm{int}}(q,m;\mathcal{W})<\alpha .\hfill \end{array}\right.$$

(7)

This rule is not the agent’s true boundary; it is how epistemic belief is translated into action under uncertainty. Different agents may adopt different thresholds, reflecting differences in risk tolerance, efficiency preferences, or deployment constraints, and therefore exhibit different tool-use behavior even when their underlying task sets coincide. The capacity to estimate $p_t^{\mathrm{int}}$ well is exactly boundary self-awareness, which Section 4.1 treats as a calibration target.

The boundary is a moving target.

Although ${\mathcal{Q}}^{\mathrm{int}}$ is latent, it is not static. As the agent reasons or interacts, the context ${\tau }_t$ accumulates intermediate results, clarifications, and retrieved evidence, and for tasks where this context is relevant and non-degrading it can only increase internal solvability:

$${\tau }_t\subseteq {\tau }_{t^{\prime }}(t^{\prime }>t)⟹p_t^{\mathrm{int}}(q,m;\mathcal{W})\le p_{t^{\prime }}^{\mathrm{int}}(q,m;\mathcal{W}),$$

(8)

so that context expansion effectively shifts the boundary outward during execution. This monotonicity is the within-trajectory analogue of the cross-trajectory boundary movement that Section 3 (Learning) and Section 5 (Evolving) study, and it is why a single external probe early in a trajectory can render the remainder internally solvable. The boundary thus appears at three timescales the paper returns to: within a single trajectory (this subsection), across training (Section 3), and over the agent’s deployment lifetime (Section 5).

2.4. Epistemic Effort, Allocation and Intelligence

Task sets and the knowledge boundary define what an agent can resolve internally. They do not yet say how much effort a task demands, when an external action is justified, or how to score the allocation. This subsection supplies those quantities of effort, necessity, and the epistemic intelligence they compose, which together form the normative core that the rest of the paper optimizes. We are careful to frame the target as more than efficiency: an agent is not intelligent merely because it spends few tokens or few calls, but because it allocates effort to the right side of the boundary, abstains when neither side can resolve the task, and improves its own boundary over time. Efficiency is the measurable shadow of that intelligence, not its definition.

Epistemic effort is unavoidable; it can only be reallocated.

Every task imposes an informational burden that must be resolved for successful completion. We call this epistemic effort and decompose it additively along the operator partition,

$$E(q,m)=E_{\mathrm{int}}(q,m)+E_{\mathrm{ext}}(q,m),$$

(9)

where $E_{\mathrm{int}}={\sum }_{t:a_t\in {\mathcal{A}}_{\mathrm{int}}}{c}_t$ aggregates the cost of internal operators along the trajectory and $E_{\mathrm{ext}}={\sum }_{t:a_t\in {\mathcal{A}}_{\mathrm{ext}}}{c}_t$ aggregates the external ones. The central structural fact is that effort is conserved: define the minimal required effort over all successful policies ${\Pi }_{\mathrm{succ}}(q,m,\mathcal{W})$ as

$$E^{★}(q,m)\triangleq \underset{\pi \in {\Pi }_{\mathrm{succ}}(q,m,\mathcal{W})}{inf}\left(E_{\mathrm{int}}^{\pi }(q,m)+E_{\mathrm{ext}}^{\pi }(q,m)\right).$$

(10)

Then for any successful policy, $E_{\mathrm{int}}^{\pi }+E_{\mathrm{ext}}^{\pi }\ge E^{★}(q,m)$: no strategy can eliminate the task’s epistemic difficulty, only redistribute it between reasoning and acting. Stronger agents, namely those with larger internal task sets, satisfy a greater fraction of $E^{★}$ internally; weaker agents lean on external interaction. But the total required effort is a property of the task relative to the agent, invariant to strategy. This invariance is what makes both of the successful and the failure modes diagnosable, e.g., the successful trajectory should care more about ideal allocations between internal thinking and external acting while failure ones should determine which part should be optimized first.

One thing to attention is that internal and external effort are not interchangeable currencies. Writing the deployed cost as

$$E_{\mathrm{total}}^{★}=c_{\mathrm{int}}·E_{\mathrm{int}}+c_{\mathrm{ext}}·E_{\mathrm{ext}},$$

(11)

with $c_{\mathrm{int}}$ capturing per-token cost (reasoning cost, KV-cache growth) and $c_{\mathrm{ext}}$ capturing per-call cost (API latency, monetary fees, irreversibility), exposes that the rational placement of effort depends on the cost ratio $c_{\mathrm{int}}/c_{\mathrm{ext}}$ as well as on the boundary. When internal computation is cheap relative to external calls, rational allocation leans on reasoning even slightly past strict necessity; when it is expensive, it leans on acting. The threshold $\alpha$ of Section 2.3 implicitly encodes this ratio.

Effort allocation and its failure modes.

Putting necessity, boundary, and cost together yields a sequential resource-allocation problem: at each step, with a residual budget and a current belief, commit a fraction to further internal deliberation or to an external probe so as to maximize expected $\Delta U$ subject to cost. The pathologies that motivate the paper are all misallocations: effort spent where it buys no belief change. They trace to a single cause, a miscalibrated boundary estimate $p_t^{\mathrm{int}}$, which can err in two directions, and each direction surfaces as both a reasoning pathology and an acting pathology:

• Boundary over-estimated ($p_t^{\mathrm{int}}$ too high, meaning the agent trusts internal competence it lacks).Overthinking spends $E_{\mathrm{int}}$ on uncertainty that lies in ${\mathcal{Q}}^{\mathrm{ext}}$, reasoning past the edge of its knowledge so that more deliberation only sharpens a fluent but unfounded conclusion (manifesting as hallucination); its acting-side twin, underacting, withholds the external call a task in ${\mathcal{Q}}^{\mathrm{ext}}$ genuinely requires, committing to an internal answer where grounding was needed.
• Boundary under-estimated ($p_t^{\mathrm{int}}$ too low, meaning the agent distrusts competence it has).Overacting spends $E_{\mathrm{ext}}$ on uncertainty already resolvable in ${\mathcal{Q}}^{\mathrm{int}}$, firing retrieval, search, or execution for what its parameters already hold and inflating latency, cost, and irreversible side-effects; its reasoning-side twin, underthinking, abandons internal reasoning prematurely on a task in ${\mathcal{Q}}^{\mathrm{int}}$, thereby delegating or abstaining where a few more steps would have resolved it.

These four are usually catalogued as distinct defects, but they are one defect, namely a miscalibrated boundary, seen from two error directions and two operator types. Reading them together also unifies the two repair literatures: efficient reasoning trims wasted $E_{\mathrm{int}}$ [155] and efficient acting gates or prices unneeded calls [30,73], but pursuing either alone is incomplete, because the same calibration target $p_t^{\mathrm{int}}$ governs all four. The diagnosis is sharpened by a learning consequence developed in Section 3 and Section 5: an agent that over-externalizes or underthinks what it could have internalized receives little learning signal on its own reasoning and stagnates in internal competence even as its task scores hold, so misallocation shapes not only the cost of a single trajectory but the long-run development of the agent’s intelligence.

Figure 5. The epistemic effort allocation between internal reasoning and external acting over the internal task set and external task set. Stronger agents and weaker agents, as well as different members within a team, occupy different ideal points along the same frontier $E_{\mathrm{int}}+E_{\mathrm{ext}}=E^{★}$; over-/under-thinking and over-/under-acting are deviations to either side of an agent’s ideal point.

Figure 5. The epistemic effort allocation between internal reasoning and external acting over the internal task set and external task set. Stronger agents and weaker agents, as well as different members within a team, occupy different ideal points along the same frontier $E_{\mathrm{int}}+E_{\mathrm{ext}}=E^{★}$; over-/under-thinking and over-/under-acting are deviations to either side of an agent’s ideal point.

Epistemic intelligence and its observable signature.

The allocation above is good or bad to the extent that it converts effort into resolved uncertainty, and that conversion has a single scalar signature. Aggregating across a trajectory of length T, with $\Delta U_{\mathrm{useful}}=U\left(b_T\right)-U\left(b_0\right)$ measured against a downstream decision rather than a leaderboard label, we define the epistemic intelligence ratio

$$\eta ={\displaystyle \frac{\Delta U_{\mathrm{useful}}}{E_{\mathrm{int}}+E_{\mathrm{ext}}}}.$$

(12)

All four failure modes above share one signature under $\eta$: the denominator grows while the numerator does not ($\Delta U\approx 0$ paired with rising $E_{\mathrm{int}}$ or $E_{\mathrm{ext}}$). But we stress what this ratio is and is not. It is the observable signature of an epistemically intelligent agent: high $\eta$ describes one that converts its compute and tool-call budget into useful belief change at a favorable rate, but it is not the whole of the property. Epistemic intelligence is the capacity that produces a high $\eta$robustly: knowing where one’s boundary lies, allocating effort to the correct side of it, abstaining when the task lies beyond any feasible operator, and pushing the boundary outward through learning so that future tasks cost less. A policy can momentarily inflate $\eta$ on a fixed task distribution, for instance by refusing to act and getting lucky, without possessing this capacity. We therefore treat $\eta$ as the metric that makes epistemic intelligence measurable, while reserving the term itself for the underlying competence. Because $\Delta U$ is a delta, defined as the reduction from prior to posterior against ground truth, rather than an absolute correctness score, $\eta$ neutralizes a familiar artifact of fixed metrics: an agent that produces a $0.92$-correct answer from a $0.90$ prior has performed less epistemic work than one that reached $0.40$ from a prior of $0.10$, even though a raw accuracy score would rank them in the opposite order.

Connection to the rest of the paper.

The objects defined here recur as the spine of every later chapter. The decomposition $E=E_{\mathrm{int}}+E_{\mathrm{ext}}$ supplies the axis along which Section 3 contrasts internalization (lowering $E_{\mathrm{ext}}$ by absorbing competence into parameters and enlarging ${\mathcal{Q}}^{\mathrm{int}}$) with externalization (lowering $E_{\mathrm{int}}$ by offloading state, skills, and control onto the harness). The knowledge boundary $\mathcal{B}$ and the solvability estimate $p_t^{\mathrm{int}}$ supply the calibration target that Section 4 trains agents to satisfy. The value-of-information criterion supplies the per-step rule whose long-horizon revision Section 5 studies as the agent reshapes its priors, routines, and harness over time. And the efficiency intelligence ratio $\eta$ of Eq. equation 12 is the unifying quantity that Section 6 proposes as a complement to task accuracy in evaluation. With these primitives in place, we turn in Section 3 to how agents acquire each component of this epistemic intelligence in practice.

3. Learning: Internalization and Externalization

LLM-based agent relies on three core cognitive dimensions when interacting with the world: state over time, executing procedures, and environment modeling. Learning in these dimensions collectively determines how an agent’s intelligence is shaped. In human cognition, the capabilities required to accomplish a task can either be internalized through learning as one’s own knowledge and skills, or complemented by external artifacts. For example, translation can be internalized as linguistic competence or externally supported by translation software. Following this analogy, we organize agent learning into two complementary routes. As shown in Figure 6, (1) Internalization realizes learning through epistemic compression: reusable regularities supporting state tracking, planning, tool use, and environment modeling are consolidated into model-side representations and policies, thereby reducing reliance on complex external scaffolding (Section 3.1). (2) Externalization realizes learning through epistemic offloading: task state, reusable skills, and coordination logic are maintained explicitly in memory systems, external operators, and the surrounding agent harness, where they remain inspectable, modular, and updatable without modifying the model itself (Section 3.2). Rather than competing paradigms, internalization and externalization define a design spectrum centered on how capabilities should be allocated across the model-harness boundary. (Section 3.3).

3.1. Internalizing as Epistemic Compression

Internalization is the process of compressing cognitive processes into the agent’s own model state or parameters, thereby turning them into intrinsic capabilities. Following the three cognitive dimensions introduced above, we organize internalization into four forms. ❶ Internalized memory retains information from prior interactions in the model state or parameters, allowing this information to affect future decisions. ❷ Planning and ❸ tool-use internalize the completion procedures of task: planning compresses how a task is decomposed into sub-goals and sequenced into a course of action, while tool-use policy compresses when and how the agent should act on the world through external operators. ❹ World model internalizes environment dynamics, capturing how the world is expected to respond to the agent’s actions so that the agent can simulate possible outcomes before committing to one.

3.1.1. Internalized Memory

Memory internalization addresses the difference between retaining a record of the past and allowing the past to change later computation. External memory can provide continuity by storing conversations, documents, task histories, and reflections, but repeatedly retrieving and interpreting a growing store adds context cost and can surface irrelevant or outdated records. Internalization instead preserves information whose value recurs across later decisions after the original context has been removed. In this subsection, we focus on information retained from prior context and interaction, either in model state or parameters. The procedures and environment dynamics learned from interaction are discussed under planning and tool use (Section 3.1.2 and Section 3.1.3) and the world model (Section 3.1.4), respectively.

Memory in Model State.

One way to retain prior information is to compress it into hidden states, memory tokens, or memory banks that remain available to later inference. The model then uses attention, retrieval, or gating to combine this state with the current input, rather than receiving the original history again. M+ follows this design with layer-wise memory-token pools: new chunks update a short-term pool, displaced tokens enter a larger long-term pool, and a learned retriever returns relevant tokens during generation [156]. G-MemLLM instead attaches a memory bank to a frozen backbone and uses gated updates to preserve, overwrite, or incorporate information as new context arrives [157]. In both cases, the retained information remains distinct from the model parameters and can change as the interaction continues.

The same state need not be updated or used at every step. FlashMem monitors the current reasoning state and generates compact memory embeddings from the model’s existing hidden states and KV cache when additional memory is useful [175]. LatentMem combines external and internal representations: it retrieves raw multi-agent trajectories from an experience bank, then composes role-specific memory tokens for the current agent [176]. These designs place memory closer to the computation that uses it, while retaining different degrees of external support. Updating state as context arrives favors coverage of the interaction history; constructing it only when needed reduces unnecessary memory computation but makes the trigger and the selected source important.

Training therefore has to make a compressed state useful for what follows, not merely similar to what came before. Token prediction can teach a state to support later generation, while task rewards evaluate whether it improves the agent’s decisions. G-MemLLM combines language-model supervision with regularization that discourages redundant use of its memory bank [157]; LatentMem propagates task reward through its memory composer so that the resulting tokens retain information useful to the multi-agent task [176]. JAMEL similarly trains memory together with exploration, using novelty to reward states that help the agent distinguish previously tried behavior from new behavior [177]. Across these objectives, the useful state is not a compressed copy of the whole transcript, but a shorter representation that preserves information needed by later decisions.

Memory in Model Parameters.

Other methods use past experience to update the model or an adapter, allowing it to affect later decisions through the model parameters. The update may modify shared weights, be isolated in a task- or skill-specific module, or last only for the current episode. What these methods share is that experience changes the policy that processes later inputs, rather than only producing a separate state for the policy to read.

(1) Learning from collected experience. Offline consolidation can fine-tune on collected trajectories, but this treats all recorded tokens as useful supervision [159,178]. Much of a trajectory may instead consist of redundant exploration or choices that access to experience would have changed. Experience distillation addresses this mismatch by asking an experience-aware teacher to resample improved actions at decision points in recorded histories. The student learns these revised decisions without receiving the experience at inference time, concentrating supervision on where experience changes a choice without requiring new environment interaction or a simulated continuation [159]. Continual internalization provides a second refinement: principle-level experience transfers more reliably than instance-specific details, and experience selected for the current decision state provides more stable supervision than a fixed set supplied to the whole trajectory [178]. In both cases, the target of learning is the decision improved by experience, rather than the surface form of the original trajectory.

(2) Updating during interaction. Parameters can also change while an agent is carrying out a long task. TMEM extracts supervision from the current session when the working context reaches its capacity and writes it into episode-local LoRA weights; subsequent actions are generated by the updated model [158]. Agentic test-time training likewise updates a local adapter from the agent’s recent output, environment observations, or a summary of its progress [179]. Such updates allow recent experience to influence later steps even as earlier context is removed, but the training stream is partly produced by the policy being updated. Repeated actions and explanations can therefore reinforce a stuck policy. Down-weighting text patterns that have already dominated earlier updates limits this feedback and preserves more influence for new information [179]. The update signal, frequency, and lifetime of the adapter determine whether this mechanism serves as temporary task memory or persists beyond the episode.

Parameter updates need not enter a shared backbone. PEAM stores selected embodied experience in category-specific LoRA adapters, training on both successful trajectories and matched failure–correction pairs so that skills can be updated with less interference across categories [23]. MemGen combines parameter and state memory in a different way: past experience is learned by a memory-weaver adapter, which generates memory tokens from the current reasoning state when a learned trigger requests them [160]. The parameters retain what has been learned across interactions, while the generated tokens express the part relevant to the current computation. These systems show that information can enter parameters without being merged into one shared set of weights, and that parameter updates and model state can serve complementary roles.

What to Internalize.

(1) Selecting experience. Not every record that improves one trajectory should become memory. PEAM evaluates candidates by their expected reuse, stability, redundancy with existing skills, and potential interference before updating an adapter [23]. Continual experience internalization further shows that abstract principles survive repeated updates better than details tied to particular queries or trajectories [178]. Resampling experience-aware actions at recorded decision points provides another form of selection: it retains where experience changes behavior without training on every preceding action [159]. Together, these results favor information that is reusable and decision-relevant over a faithful reproduction of the original record. The same selection principle applies to the training set itself: gradient-aware data selection scores candidate examples by their gradient-level effect and retains those that add new information without overwriting what the model already holds [180].

(2) Controlling memory updates. Selection also concerns when retained information should change computation. For model state, FlashMem invokes memory from signals in the current reasoning process rather than generating it uniformly [175]. For model parameters, PEAM waits until valuable candidates have accumulated, while continual distillation depends on experience that is aligned with the current decision and on coherent teacher trajectories [23,178]. During interaction, repetition-aware test-time training shows why control is necessary: an update can preserve new task information or reinforce behavior that has already failed [179]. Reliable internalization therefore depends on the content and timing of an update as much as on whether memory is retained in state or parameters.

Memory in model state keeps retained information as a distinct and replaceable part of inference; memory in model parameters binds its effect more closely to the policy and can carry it across later inputs. The first is generally easier to update or clear, whereas the second makes the source and revision of a learned behavior harder to trace. Practical agents can combine both with external records, keeping detailed or changing information available for inspection while retaining recurring information in the model. Internalized memory is therefore not the accumulation of all past experience, but the selective retention of experience that continues to matter for future decisions.

3.1.2. Planning Reasoning

Planning decomposes a task into subgoals, orders them, and revises that ordering when execution diverges from expectation. What matters is the distinction between the artifact and the procedure that produces it; internalization concerns the procedure. This is a capability separate from step-wise reasoning, which acts as a greedy policy that takes the locally best next step without weighing its effect on later subgoals; agents that stumble on multi-step tasks more often fail to plan than to execute a correct plan [181,182]. An agent internalizes the procedure when decomposition, weighing alternatives, and replanning on failure become dispositions in its parameters rather than structure supplied by a prompt template, an orchestration loop, or a search controller. The two abilities internalize unevenly. Decomposition transfers readily: a fine-tuned agent breaks a task into sensible subgoals without a template prescribing how. Lookahead, anticipating how a plan unfolds before committing to it, is harder, since the agent must produce within its own forward pass what a search controller once obtained by expanding and scoring candidates. This section reviews how planning is organized at inference time, then asks what training absorbs into the model and where that absorption stays incomplete.

The Organization of Planning.

At inference time, planning is organized in a handful of recurring ways, and a single agent often combines several.

• Workflow / Loop Structures split planning into stages such as perception, reasoning, execution, and verification, arranged either as a plan-then-act pipeline or as a reactive loop that revises decisions from execution feedback [147,151,183,184,185]. In embodied settings, PEAP extends such loops to proactive action-sequence planning that jointly grounds vision and audio perception [186].
• Search / Branching Structures keep several candidate trajectories alive and expand, evaluate, prune, and backtrack over them with a search procedure, whether breadth-first, depth-first, A*, Monte Carlo tree search, or beam search [187,188,189,190,191].
• Process Formalization writes plans in PDDL, programs, or other symbolic forms that a classical planner, verifier, or interpreter can check or run [192,193,194,195,196].
• Decoupling and Decomposition break a task into subproblems, reusable abstractions, or hierarchical goals, keeping plan construction separate from the execution of individual steps [197,198,199,200,201].
• External Assistance brings in retrieved knowledge, structured graphs, world models, or specialized tools where the model on its own cannot reliably supply the required state, constraints, or computation [202,203,204,205,206,207].

Internalizing the Procedure.

The routes to internalizing planning differ in where the training signal comes from: a demonstrated trajectory, the trace of the search that produced it, or the reward from acting. A fourth concerns not how to plan but whether to.

• Imitation from Successful Trajectories. Fine-tuning on successful trajectories lets common patterns of decomposition and action ordering emerge without a detailed prompt each time. FireAct established the route on ReAct-style trajectories [161]; later work focused on the coverage and structure of the data, separating reasoning from formatting [208], scaling to tens of thousands of interactions [209], building verified GUI trajectories [210], and distilling planning traces from larger models [211]. The object imitated is always the same: a trajectory that worked. So is the weakness, since planning, tool use, and formatting stay entangled within it: imitation transfers the surface form of a plan, not the competence that produced it.
• Distillation from Search Procedures. Search lays out candidate generation, evaluation, and backtracking as an explicit trace that can be written out and learned. Searchformer learns the execution dynamics of A* and then shortens them below the planner it trained on [162]; Stream of Search improves past its heuristic teachers, solving problems they could not [163]. That these models do better than the traces they learned from shows that what transfers is the search procedure itself, not a lookup table of its solutions, which resolves the entanglement of imitation but not its coverage: a student still needs outside search where the teacher never went. Symbolic planners resist distillation entirely, since they supply a guarantee that preconditions hold rather than a trace to copy.
• Reinforcement Learning from Consequences. With reward as the signal, no demonstration is needed; the difficulty is assigning credit over a long trajectory. Outcome rewards work at first [212] but weaken over longer horizons, leaving shallow strategies rather than reliable reasoning [213]. Denser signals credit intermediate structure directly, rewarding tool-use completeness [214], subgoal milestones [164], or unresolved subgoals revised after failure [215], part of a broader shift toward process-level reward [31]. Planning quality is a property of intermediate structure, so outcome-only reward reinforces the right answer rather than the right plan.
• Learning When to Plan. Internalization reaches past the procedure to the choice of when to use it. A trained agent may over-plan easy tasks, adding delay without gain [216], so work on fast-and-slow control and explicit planning triggers builds this choice into the policy, saving deliberation for hard cases [165]. Without a learned trigger, internalization merely relocates the cost of planning from context to inference.

Overall, imitation carries over a plan’s form but not the reasoning behind it; distillation carries over a procedure but only where the teacher went; coarse reward finishes tasks without planning reliably. Each route yields plans of the right form while leaving the underlying competence unverified. We take internalized planning to require three things the analysis keeps pointing to: generalization to unseen states, calibrated rather than merely fluent foresight, and early commitments that answer to the global structure of the task. A plan that reads well without these is the appearance of planning, not the capability.

3.1.3. Tool Use Policy

Tools extend an agent’s capabilities, but learning to use them involves more than learning the tools themselves. A calculator, retriever, code interpreter, or API remains external to the model. What the model can internalize is a policy for deciding when to invoke it and how to act on its output. This policy involves four decisions: whether to call a tool, which tool to select, how to invoke it, and how to use the returned result. These decisions receive different forms of supervision. Successful trajectories directly demonstrate tool selection and invocation, making both amenable to imitation. By contrast, they rarely show when a call should be avoided or how an agent should respond when a tool fails. Learning these behaviors therefore requires either deliberately constructed coverage or feedback from interaction.

Whether to call.

The policy must first decide whether external assistance is needed. Existing methods learn this boundary from three sources: the model’s internal state, the observed consequences of calling, and the utility of a call relative to its cost.

• Confidence-based triggering. Active retrieval methods often use uncertainty to decide when internal knowledge is insufficient. FLARE predicts upcoming content and triggers retrieval when that prediction contains low-confidence tokens [217]. MeCo instead extracts metacognitive signals from the model’s hidden representations and uses them to determine whether a tool should be invoked [218]. Both provide inference-time decision rules rather than training the calling policy itself: FLARE relies on an external retrieval controller, while MeCo is explicitly fine-tuning-free. Their contribution is to identify signals from which a calling decision can be made.
• Signals from consequences. Toolformer samples candidate API calls, retains those that reduce language-model loss, and fine-tunes on the resulting data [166]. Search-oriented reinforcement learning replaces token-level improvement with task outcomes. ReSearch and Search-R1 train models to insert retrieval into multi-step reasoning when it contributes to the final answer [219,220], while ZeroSearch uses a retrieval simulator and a curriculum over document quality to avoid repeated calls to a live search engine during training [221]. General tool use also requires explicit negative and ambiguous cases. When2Call distinguishes among generating a tool call, asking for missing information, and admitting that the available tools cannot answer the request; preference optimization is then used to train these choices [167]. SMART-ER alternates parametric reasoning with tool-dependent steps and annotates why a tool is needed, providing supervision for SMARTAgent to balance internal knowledge against external assistance [30]. These methods learn more than a tendency to call tools: they expose the model to alternatives against which a call can be judged.
• Utility-aware calling. Task improvement alone does not determine whether a call is worthwhile. Latency, monetary cost, and the risk of an unhelpful result also affect the decision. Cost-aware objectives incorporate these factors into the calling policy, favoring a tool only when its expected contribution offsets its cost [72,73]. The goal is calibrated use rather than either maximal or minimal invocation.

Which tool to call.

Once external assistance is judged necessary, the policy must match the current need to an available operator. This is not simply classification over tool names: descriptions may overlap, multiple tools may offer similar capabilities, and the best choice may depend on earlier results. Existing approaches address this problem either by structuring the tool space presented to the model or by learning a routing policy from trajectories and execution outcomes.

• Structuring the tool space. Gorilla retrieves relevant API documentation [146]; ToolkenGPT represents tools with learned embeddings [8]; Meta-Tool separates hypothesis, retrieval, and invocation [222]; and ToolGen represents tools as generatable tokens [48]. ToolExpNet further organizes tool similarities, dependencies, and usage experience into a network [223]. These methods make a large catalog easier to navigate rather than forcing direct selection from an unstructured list.
• Learning the routing policy. ToolLLM supplies single- and multi-tool solution paths over 16,464 APIs [46]. Tool-Star combines synthesized trajectories with hierarchical reinforcement learning for coordinated multi-tool use [108], while ToolRL studies how reward granularity affects selection and application [41]. Training can internalize how task needs map to tool capabilities, but the available tools and their descriptions remain external.

How to call it.

Tool selection identifies an operator but not an executable action. The model must still translate its intent into the schema, arguments, and format required by the interface, often for tools not observed during training. Work on this problem follows two complementary routes: conditioning the model on external specifications and training it on demonstrations whose calls have been checked for validity.

• Conditioning on tool specifications.Models can use unseen tools from documentation [224]. EASYTOOL improves this process by standardizing heterogeneous instructions [225], while guided function calling separates intent interpretation, tool grounding, and parameter construction [226]. RoTBench shows that perturbing tool descriptions disrupts both parameter identification and argument filling, and uses noisy environments to improve robustness [227].
• Learning from executable demonstrations.TALM bootstraps tool use through self-play from a small demonstration set [228]. API-Bank covers 2,138 APIs [47], and ToolAlpaca uses simulated interactions to study transfer to unseen tools [229]. APIGen filters generated calls through format checks, execution, and semantic verification [230]; ToolACE combines dialogue generation with rule- and model-based validation [231]. These methods train a specification-to-invocation procedure whose strongest test is valid calling on interfaces absent from training.

How to proceed from the result.

A tool result updates the agent’s state and may prompt it to continue, revise the call, switch tools, or stop.

• Learning under nominal execution.ReAct interleaves reasoning, actions, and observations [151]. WebGPT learns browser interaction from demonstrations and human preferences [232], while WebShop provides trajectories and task rewards for comparing imitation with reinforcement learning [233]. ReTool and ToRL optimize reasoning interleaved with code execution [24,107]; MINT evaluates the use of tool and language feedback across multiple turns [234]. Such interaction teaches how one result should shape the next action.
• Learning from execution failures.ConAgents distributes selection, execution, and calibration across cooperating agents [235], whereas ToolReflection trains a single model on errors and corrections collected from API feedback [236]. CRITICTOOL evaluates failure identification and diagnosis [237]. Tool-MVR constructs error–reflection–correction examples [238]; PALADIN pairs injected failures with recovery demonstrations [239]; and Fission-GRPO converts on-policy errors into recovery rollouts [168]. These signals teach recovery from explicit failures. Plausible but incorrect outputs remain harder because they require independent verification.

3.1.4. World Model

In partially observable environments, an agent rarely knows the full consequence of an action before taking it. Feedback may be delayed, and an incorrect operation may be costly or difficult to reverse. A world model equips the agent to anticipate how an action may change the environment and what possibilities may remain afterward. These predictions can shape how the policy is learned or provide evidence for revising an action before it is executed.

The learned object need not be a complete reconstruction of the environment. What matters is a decision-relevant account of action-conditioned change, whose form depends on the environment. A web world model can describe which page elements are added, deleted, or updated after an action, avoiding reconstruction of an almost unchanged page [240]. A computer-use world model may separate a textual description of an interface change from the visual realization of the resulting screenshot [241]. In code environments, execution traces pair program steps with changes in local state, while multi-turn agent trajectories additionally expose the environment responses that follow tool use [242]. Text-based world models instead track such consequences as changed object states, tool responses, and newly available actions while maintaining consistency across multiple steps [170]. These predictions may remain conditional or uncertain because observations can omit hidden state and some environments respond stochastically.

Learning Action-Conditioned Dynamics.

(1) Grounding and expanding transition data. Transition learning begins with an observable consequence: the agent executes an action, and the environment returns the next observation, tool result, or state description. A collected trajectory provides this supervision only for the actions that its policy happened to take. Coverage can be widened by returning to states visited by an expert and executing alternative actions, so that unrealized branches become grounded, reward-free examples of how the environment responds [40]. On-policy collection follows the distribution in the opposite direction, continually acquiring transitions from states the current agent actually reaches and comparing its predicted next state with the realized one [119]. Self-play similarly lets the policy explore beyond expert demonstrations; replacing its predicted current and next states with environment-returned states before fine-tuning prevents its own beliefs from becoming their sole supervision [169]. Together, these sources trade off reliable but narrow expert coverage against broader, policy-dependent exploration.

Coverage also depends on the diversity and representation of the environments in which transitions are collected. Execution traces expose fine-grained program changes, whereas multi-turn agent trajectories reveal how tools respond within longer tasks [242]. Training across terminal, software-engineering, search, web, and GUI interactions can further expose shared transition regularities rather than binding prediction to one interface [171]. Generated interactions may extend this support, but they inherit the assumptions of their generator. They are therefore most useful for proposing states and actions whose consequences are subsequently anchored by execution, rather than as replacements for environment evidence.

(2) Optimizing transition prediction. The training target determines which regularities are retained. Predicting the full next observation supplies dense supervision, but may devote most of the loss to unchanged page content, formatting, or incidental identifiers. Predicting an action-induced delta instead concentrates supervision on what was added, removed, or modified [240]. The objective also need not require an exact textual reproduction: when several descriptions denote the same state, rewarding semantic agreement between the prediction and the realized observation preserves the transition while avoiding a brittle surface match [119]. Optimization can then proceed at different scales. Token-level supervised learning acquires local transitions; continual pretraining over heterogeneous environment trajectories supplies broader world knowledge; supervised fine-tuning elicits explicit next-state reasoning; and rubric- or rule-based reinforcement learning improves fidelity on consequences that are difficult to specify with one reference string [171]. The resulting objective is not simply to reconstruct what appeared next, but to retain changes that distinguish the choices available afterward.

Integrating Dynamics with Policy Learning.

(1) Sequential integration. Dynamics can first be learned and then transferred to action selection. The most direct form applies next-state training to the same model that will later act. Alternative-action experience can warm up transition prediction before expert imitation and policy optimization [40]; self-play transition fine-tuning can likewise initialize a subsequent policy-RL stage [169]. A semantic transition reward provides another initialization: the model is first optimized to align predicted and realized states, after which task-success RL starts from the resulting checkpoint [119]. Text-environment experiments similarly find benefits from placing world-model supervision before agent fine-tuning and RL [170], while cross-domain next-state warm-up can transfer directly to multi-turn tool use even without an additional agent fine-tuning stage [171]. In these cases, the policy inherits dynamics through its own parameters rather than consulting a separate predictor at deployment.

A second, indirect form trains the policy on experience produced by an already learned simulator. A web policy can interact with a synthetic environment to generate on-policy dreamed rollouts, while real expert trajectories are interleaved to stabilize learning against simulator error [117]. Large-scale world-model trajectories can likewise broaden the interactions available for policy fine-tuning beyond those collected from live websites [43]. Although the simulator remains external, its transition regularities are absorbed into the policy through optimization, allowing the policy to act without querying it at deployment.

(2) Joint integration. Coupled training lets each executed action supervise both what the agent chose and what followed. One objective applies policy-gradient loss to action tokens and next-observation loss to environment tokens from the same on-policy transition. Because observation modeling can overwhelm the task objective, uncertain transitions can be selected more heavily, noisy token losses clipped, already predictable tokens masked, and the auxiliary weight reduced as task performance improves [172]. Terminal interaction offers a concrete instance: commands receive outcome-driven policy updates, while stdout, errors, logs, and other returned tokens receive cross-entropy supervision in the same forward pass, so failed rollouts still teach the policy how its actions affect the environment [243]. Prediction can also enter credit assignment rather than a token loss. When the agent predicts several future states during rollout, their agreement with the states later observed can multiplicatively modulate existing task credit, rewarding reliable foresight without allowing a plausible prediction to create reward for a failed trajectory [244]. Joint learning thus turns a transition into two linked signals—action quality and consequence fidelity—while requiring safeguards against noisy or stochastic observations dominating the policy.

Simulating Action Consequences.

(1) Refining actions through simulated consequences. Dynamics become part of the current decision when the agent represents a consequence before acting and uses it to retain, reject, or revise the proposed action. The minimal form predicts the immediate next state of the action being considered, as in self-play traces that place an environment-grounded prediction before the executable answer [169]. A next-state-trained foundation model can similarly anticipate a tool response inside its reasoning, identify an infeasible approach, and modify the action before execution [171]. Richer forms consider multiple steps or alternatives. Search trees built through real environment interaction can be distilled into simulation-guided reasoning so that the trained agent later performs the refinement without the original search procedure [173]. Action-centered reasoning traces can also retain predicted effects of the final action, while later environment outcomes supervise next-state, state-difference, or simulation-critique targets in the same policy model [42]. These mechanisms need not reproduce a full trajectory: even one predicted consequence is useful when it reveals that an action violates a precondition or removes a future affordance.

(2) Controlling when and how far to simulate. Simulation has value only when its decision benefit exceeds its inference cost and prediction error. A learned configurator can decide whether to skip planning, construct a new simulated plan, or continue an existing one, while the same language model predicts the belief states expected along that plan [174]. Reinforcement learning in this setting increases how far the agent plans much more than how often it invokes planning, suggesting that useful control concerns both invocation and horizon. Separate-verifier experiments provide complementary evidence about where this computation is valuable: checking an irreversible action before execution can prevent unrecoverable mistakes, whereas repeated verification eventually shifts the interaction distribution and ceases to help [170]. This verifier remains external, but the result motivates the same internal allocation principle. Familiar and reversible actions can often proceed directly; uncertain or consequential actions justify deeper simulation, subject to the risk that errors compound over a longer imagined horizon.

Whether dynamics enter through sequential transfer, a joint objective, or decision-time simulation, realized consequences remain the calibration point. Policies trained on imagined experience can inherit simulator bias, coupled objectives can overfit incidental observation tokens, and internal rollouts can drift as their horizon grows. Continued interaction closes the loop: prediction shapes action, execution supplies the next observation, and the discrepancy updates both what the agent expects and how much confidence it places in later simulation.

3.2. Externalization as Epistemic Offloading

Externalization offloads epistemic content from the model’s parameters into a persistent external environment, relieving the model of cognitive burden it would otherwise carry in its weights. We organize this route around three objects. ❶ Externalized memory externalizes the agent’s time-varying state, persisting its histories, user preferences, and prior trajectories in an external store so that continuity carries across a single session and across separate ones. ❷ Skill kit externalizes the agent’s procedures, packaging routines, best practices, and operating guidance into reusable artifacts the agent can retrieve and apply without retraining. ❸ Harness engineering externalizes control, providing the orchestration logic, constraints, observability, and feedback loops governing execution. What the three share is that they remain explicit: inspectable, updatable, and verifiable without retraining the model.

Across these three forms, Table 2 compares representative systems at the mechanism level: the evidence and method used to construct an external resource, the artifact and access interface through which it supports the agent, and the way it is updated and adaptively invoked over its lifecycle.

3.2.1. Externalized Memory

A language model is stateless: each call begins from a fresh context, so any continuity across turns, sessions, or tasks must be reconstructed rather than carried forward. For short interactions the context window hides this limitation, but once work spans sessions or interruptions, holding all relevant history in the prompt becomes impossible, and continuity has to live somewhere else. Memory externalizes that state: it moves the agent’s accumulated history into a persistent store outside the model, written and retrieved on demand, so that the agent recalls what it needs rather than regenerating it from scratch. Papers of agent memory [32] chart a progression from raw conversation buffers toward structured hierarchies with explicit write, read, and forget operations, whose shared role is to turn past experience into queryable evidence answerable by recall rather than re-derivation.

What is externalized.

The state worth externalizing falls into three kinds. (1) Working state is the live intermediate context of the current task, such as open files, partial plans, and execution checkpoints; materialized outside the prompt, it lets an interrupted agent resume from where it left off rather than rebuild its workspace, as in coding agents that treat the file system and shell as the authoritative record of task state [257]. (2) Experiential and semantic state records what happened in prior runs and the abstractions distilled from them, from concrete episodes (decisions, tool calls, failures) to general facts and conventions that hold across episodes; Reflexion [185] stores reflective summaries of failed attempts as reusable episodic experience, Generative Agents [246] maintains a memory stream of observations periodically reflected into higher-level summaries, and retrieval-augmented generation [22] is the common vehicle for the semantic case. (3) Personalized state tracks stable information about a particular user or environment, such as preferences and recurring constraints, kept separate from general task knowledge because it obeys different retention and privacy rules.

From storage to control.

How memory is externalized has progressed from passive storage toward active control over what is kept and surfaced. (1) Monolithic context holds everything in the prompt, transparent but bounded by the window and lost when the session ends. (2) Retrieval stores keep only near-term state in context and fetch the rest from an external index [22], which trades the capacity problem for a retrieval problem: surface the wrong records and the model is distracted, miss the right ones and the memory might as well not exist. Work here improves the representation behind retrieval, as in GraphRAG [245], which indexes a corpus as an entity graph with community summaries to answer global queries that flat similarity search misses. (3) Hierarchical memory imposes explicit lifecycle operations on the store: MemGPT [10] borrows OS-style paging to swap state between an in-context tier and external storage, while Mem0 [247] adds extraction, consolidation, and forgetting so that not every trace is retained or retrieved alike. (4) Adaptive memory learns these policies from experience rather than fixing them by hand: A-MEM [248] organizes memories into an evolving linked structure, stepwise experience recall [249] retrieves fine-grained past trajectories at each step of a tool-using interaction rather than only at task start, and experience synthesis [101] treats the store not as a static log but as a substrate the agent continually summarizes and re-indexes so that recall cost scales sub-linearly with experience. The through-line is a shift from storage to control: the difficulty is no longer fitting history into the window but deciding which slice of the past to make legible to the present decision.

Key Insight.  Externalizing memory keeps time-varying state in a persistent, editable store, replacing repeated reconstruction with controlled writing, retrieval, consolidation, and forgetting of decision-relevant evidence.

3.2.2. Externalizing Skills as Epistemic Operators

skills externalize what it can do. Section 3.1.3 internalized the control over tool use, the decision of whether, which, and how to call. The complementary move is to externalize the operators themselves: the tools, APIs, and packaged procedures the agent draws on, kept outside the model as inspectable, reusable resources rather than regenerated from the weights on every task. The core transformation is from generation to composition: instead of improvising a procedure step by step each time, the agent assembles behavior from pre-validated components it can call by name.

What is externalized.

Skills externalize procedural expertise at three levels of packaging. (1) Atomic operators are individual tools and APIs exposed as callable units; the engineering problem is letting an agent select correctly from a large catalog, addressed by retrieval-grounded interfaces [146], by representing each tool as a learned token so that thousands remain addressable [8], or by an orchestrator that routes a request across many expert models [207]. (2) Induced toolboxes are operators the agent creates and curates for itself rather than receiving fixed: LLMs as Tool Makers [250] and CREATOR [251] have the model write reusable utility functions on demand, and TroVE [252] grows and periodically trims a verifiable library so that recurring sub-procedures become named functions. (3) Packaged skills are higher-level expertise bundled with the guidance needed to apply it, the level this section emphasizes, since it is where externalization does the most work: rather than a bare function, a skill carries the operating procedure, preconditions, and conventions for a task, so that loading it transfers know-how the model would otherwise have to reconstruct.

From generation to composition.

The packaged-skill level has moved from libraries built inside a single run toward portable, shareable expertise. Voyager [253] established the pattern with an ever-growing skill library of executable code: behaviors that succeed once are stored as named, composable programs and retrieved later, so the agent’s competence compounds instead of resetting each episode. SkillWeaver [254] extends this to open-ended web environments, having the agent autonomously discover candidate skills, practice them, and distill the successful trajectories into robust APIs, and shows that skills synthesized by a strong agent transfer to weaker ones as plug-and-play modules. The packaging has since been standardized: an agent skill in the sense of recent practice [255] is a self-contained document bundling instructions, examples, and supporting scripts under an explicit description, loaded on demand and portable across agents and platforms. Across these systems the trajectory is the same as for memory, from passive resource to managed asset: a skill is not merely stored but specified, discovered, versioned, and composed, and the agent’s growth comes from enlarging and refining this external repertoire rather than from changing its weights.

Key Insight.  Externalizing skills packages procedural expertise as inspectable, reusable operators, shifting work from regenerating procedures inside the model to selecting and composing verified components on demand.

3.2.3. Externalizing Orchestration as Epistemic Scaffolding

What is externalized.

The harness externalizes the coordination structure of agency along three axes. (1) Control flow is the agent loop itself, the explicit logic that sequences perception, action, and recovery rather than leaving the trajectory to emerge turn by turn from the prompt; multi-agent frameworks make this structure a first-class artifact, with AutoGen [13] programming interaction as conversations among customizable agents and MetaGPT [256] encoding role-based standard operating procedures so that a team of agents follows a defined workflow. (2) Protocols externalize the interaction contracts between the agent and what it talks to, whether tools, other agents, or the user, replacing bespoke per-tool glue with machine-readable interfaces for discovery and invocation; the Model Context Protocol [96] is a representative standardization, giving an agent a uniform way to find and call external capabilities without custom wiring for each one. (3) Governance externalizes the constraints on execution, the sandboxing, permission boundaries, approval gates, and observability that decide what the agent is allowed to do and make its behavior auditable rather than opaque.

From ad hoc coordination to governed execution.

The harness layer has moved from minimal wrappers toward managed runtimes that integrate the other forms of externalization. Early agent systems were thin loops bolting a task queue and memory onto a model; contemporary harnesses are richer, sequencing skill calls, threading memory reads and writes through each step, enforcing protocol schemas at tool boundaries, and mediating the interactions among these modules, since they compete for the same scarce context budget and their effects feed back on one another, an execution trace becomes a memory entry, a retrieved memory conditions the next skill choice. Coding harnesses are the clearest contemporary example, embedding the model among files, shells, tests, and version control so that long-horizon work proceeds through a governed environment rather than a single prompt [257], and the same pattern appears wherever reliability is sought by changing the environment around the model rather than the model itself [253]. Across these systems the harness is what turns a capable model into a dependable agent: it decides which externalized resources are in play at each step and holds the whole arrangement to inspectable rules.

Key Insight.  Externalizing orchestration places control flow, interaction protocols, and governance in the harness, making coordination across memory, tools, and agents inspectable, recoverable, and enforceable rather than implicit in model generation.

3.3. The Internalization-Externalization Trade-Off

The boundary between internalization and externalization concerns how epistemic work is allocated rather than where a component is permanently placed. Moving a capability inward can reduce repeated retrieval, coordination, and execution overhead, whereas keeping it external preserves access to current state, explicit evidence, and controllable operations. A useful allocation therefore asks first whether an external component supplies something the model cannot reliably produce, and second whether that contribution is stable enough to be absorbed without losing properties required at runtime. The first question concerns epistemic necessity; the second concerns the limits of compression.

External Resources as Epistemic Support.

The central criterion for external support is epistemic gain: whether an external operation changes what the agent can reliably decide. Tool use should therefore be evaluated not only by final task success, but also by whether each call contributes useful information, execution, or verification relative to its cost [73,260]. This gain is relative to both the model and the task. The same operator can be redundant within the model’s reliable range yet valuable when computation scales up, knowledge falls outside the parametric boundary, or the task requires a real state change [261,262,263]. This dependence is not always reflected in the agent’s behavior: a model may represent when a tool is needed yet still fail to calibrate its calls accordingly. Indiscriminate call suppression affects easy and hard tasks alike, but omitted calls impose a larger accuracy cost on the latter [264]. Epistemic gain also depends on the quality and type of support. Irrelevant evidence can degrade an otherwise correct closed-book answer, whereas recent and long-tail facts can make external search essential [265]. The appropriate operation also depends on what is missing: absent facts favor retrieval, while unspecified intent or evaluation criteria favor clarification [266]. Conversely, tools with non-positive aggregate marginal contribution can be removed from the tool suite without reducing accuracy [267]. Externalization is thus a routing problem among reasoning, retrieval, clarification, execution, and verification, rather than a preference for more scaffolding.

Internalization Under Epistemic Limits.

Internalization can replace external support only when the contribution of that support can be reconstructed from information already available in the model’s state or parameters. In such cases, what moves inward is a decision procedure: additional deliberation can refine a judgment based on existing knowledge, and the amount of computation can be adapted to task difficulty [268,269]. This inward movement does not, however, expand the evidence available at runtime. Fresh facts, current environment state, unstated user intent, and the result of an action that has not been executed become available only through interaction [260]. The first epistemic limit is therefore one of grounding: internal computation can transform available information, but it cannot supply a missing observation. A second limit arises when the task requires not merely a plausible answer, but externally established correctness. In structured financial verification, combining adaptive language-model search with executable taxonomy traversal, exact computation, and rule checking is substantially more reliable than internal reasoning alone; removing deterministic checks sharply reduces joint accuracy [270]. Even when a model can predict the result, these checks remain part of the epistemic procedure because they make that result reproducible. Internalization is thus well suited to reusable reasoning and decision procedures, whereas runtime evidence and deterministic verification remain external when the task depends on them.

Capability-Dependent Epistemic Boundaries.

The boundary is consequently a property of the model–harness–workload configuration, not a permanent attribute of a component. Scaffolds for planning or context management can become less load-bearing as a model gains coherence and self-correction, while remaining useful near the edge of its autonomous capability [261]. Two mechanisms explain this movement. First, the value of an external artifact depends on the agent’s ability to use it: weak models may fail to invoke or follow it, intermediate models can benefit substantially, and strong models may approach a ceiling [271]. Second, workload scale changes the burden placed on the same internal capability. A model may navigate one document without prepared support, whereas larger corpora benefit from staged disclosure; excessive hierarchy, however, can again consume context [272]. These effects are visible under shared tasks and budgets, where changing the model–harness pairing alters completion, process quality, efficiency, and failure behavior [273]. At the same time, persistent state, permissions, traces, and objective validation retain functions that model scale does not absorb. External support should therefore be assessed against the capability and workload it serves, rather than assigned a context-independent value.

Conditions for Epistemic Allocation.

A practical allocation can be made in two stages. The first asks whether a capability is a viable candidate for internalization. Stability favors resources that remain valid over the learning horizon; recurrence determines whether repeated use can amortize consolidation; and comparative cost weighs external retrieval, calls, orchestration, and verification against data collection, training, updating, rollback, and lost inspectability. These conditions must be considered jointly: a resource can be stable but too rare to consolidate, recurrent but too volatile to learn, or inexpensive to internalize but costly to make opaque. Passing this stage establishes economic viability, not complete placement. The second stage asks which properties must still remain external. Freshness favors runtime access to current state, verifiability and auditability favor evidence and deterministic traces, and irreversibility and user control favor explicit approval boundaries. Even a stable and recurrent capability may retain an external checker when errors are consequential or attribution matters; marginal utility and agent usability further determine whether that checker adds enough value and can be used reliably. When a capability is worth internalizing but still depends on these runtime properties, a hybrid boundary is appropriate. Repeated procedures can be compiled into validated, versioned tools with a general agent available for failures or drift [274]; learned environment priors can guide action while runtime discovery supplies changing enterprise configurations [275]. The resulting allocation internalizes reusable regularities while retaining current state, verification, and governance at runtime.

Summary.  The internalization–externalization boundary is not a fixed division of components, but a configuration-dependent allocation of epistemic work. Stable, recurrent, and amortizable regularities can move inward; fresh state, evidence, execution, and governance remain external when they provide decision-relevant information or control. Hybrid designs connect the two when internalized capabilities still require runtime grounding, verification, or rollback.

4. Aligning: Training Agents for Epistemic Calibration

An agent can act in accordance with human expectations only when it is sufficiently well aligned. For agents, such alignment can be understood from two complementary perspectives: internal alignment and external alignment.

Agent-internal alignment.

Agent-internal alignment concerns whether an agent can calibrate its decisions with respect to two requirements that arise from the agent itself: its cognitive boundaries and its efficiency constraints. These two requirements can be viewed as preferences over the effort space. Within this effort space, two corresponding dimensions can be identified: the reasoning–acting dimension and the effort-allocation dimension. (1) Along the reasoning–acting dimension, the agent must decide whether a problem should be addressed through internal reasoning or external action. (2) Along the effort-allocation dimension, it must decide how much effort is worth spending on the task.

Agent-external alignment.

Given the agent’s action-selection tendency over the available action space, external alignment further constrains and shapes the final action. First, human preferences determine which objectives and behaviors are acceptable to the user, including both safety preferences and personalization preferences. Second, environment constraints define which actions are valid, executable, and safe under the current interface, tool schema, or protocol. Third, an agent must also align with other agents: communication protocols and coordination mechanisms let agents negotiate commitments, avoid miscoordination, and improve collective performance.

Under this view, agent alignment can be decomposed into five dimensions: alignment with the agent’s own decision boundaries, alignment of efficiency, alignment with human preferences, alignment with environment constraints, and alignment with other agents. We organize this chapter accordingly around five dimensions of calibration: the agent’s self-awareness of its own cognitive boundaries (Section 4.1), its efficiency in managing the costs of thinking and acting (Section 4.2), its alignment with human preferences (Section 4.3), its alignment with environment constraints (Section 4.4), and its alignment with other agents (Section 4.5).

Key Insight.  All agent alignment reduces to calibrating one decision rule, whether to reason, act, delegate, or stop, against five constraints: what the agent knows, what effort costs, what humans want, what the environment allows, and what other agents do.

Figure 7. The Overview of Agent Aligning. Agent aligning is organized along two axes: inside the agent, covering (1) aligning with cognitive boundaries (knowledge and ability) and (2) aligning with efficiency constraints (task budget and action cost); and outside the agent, covering (3) aligning with humans (preference, intention, and action reshaping) and (4) aligning with the environment and multi-agent systems.

Figure 7. The Overview of Agent Aligning. Agent aligning is organized along two axes: inside the agent, covering (1) aligning with cognitive boundaries (knowledge and ability) and (2) aligning with efficiency constraints (task budget and action cost); and outside the agent, covering (3) aligning with humans (preference, intention, and action reshaping) and (4) aligning with the environment and multi-agent systems.

4.1. Self-Awareness for Cognitive Boundaries

An agent should be aware of when its existing knowledge is insufficient to answer a question, and when external tools are needed to extend the boundary of its abilities. This form of self-awareness can be understood as self-awareness of cognitive boundaries, which is a key component for an agent as an epistemic decision maker.

Cognitive boundaries can be further divided into two types:❶the knowledge boundary, which enables the agent to recognize when its existing knowledge is insufficient to answer a question; and ❷the ability boundary, which enables the agent to recognize when external tools are needed to expand what it can do.

This view positions self-awareness of cognitive boundaries as a central requirement for agent alignment. The following discussion first explains why cognitive boundaries matter for agents as epistemic decision makers in Sec Section 4.1.1, then reviews how knowledge boundaries can be detected, estimated, and calibrated in Sec Section 4.1.2, and further discusses ability boundary alignment in Sec Section 4.1.3.

4.1.1. Why Self-Awareness of Cognitive Boundaries Matters

As an epistemic decision maker, an agent must determine when it can rely on internal reasoning and when successful task completion requires external evidence, tools, or action. This view implies that external resources should be invoked only when they are task-necessary: either because the agent lacks the relevant knowledge or because the task exceeds what its native ability can accomplish.

The knowledge boundary.

In knowledge-intensive tasks such as question answering, making this decision requires the model to recognize when its internal knowledge and reasoning are insufficient. The decision boundary between answering internally and seeking external evidence therefore corresponds to the model’s knowledge boundary.

Self-awareness of the knowledge boundary is necessary because both over-reliance and under-reliance on internal knowledge can produce misaligned epistemic behavior. When an agent overestimates the sufficiency of its parametric knowledge, it may generate fluent but unsupported claims, summaries, or reasoning chains, which is reflected in studies on hallucination and factual inconsistency [276,277]. External evidence can mitigate such failures, but only when the agent can determine whether retrieval is actually needed rather than applying it indiscriminately [5,278]. The opposite failure also matters: an agent may become overly conservative, abstaining, refusing, or externalizing the task even when its internal evidence is sufficient [279]. Aligning the knowledge boundary therefore means calibrating the decision of when to answer internally, when to seek external evidence, and when to abstain.

The ability boundary.

In action-intensive tasks, however, successful task completion often depends not only on what the agent knows but also on what it is capable of doing. Making this decision requires the agent to recognize when its intrinsic abilities are insufficient and external tools, APIs, or collaborators are needed to accomplish the task. The decision boundary between relying on native abilities and extending them through external resources therefore corresponds to the agent’s ability boundary.

Miscalibration of this boundary can cause both underuse and overuse. ❶Underuse occurs when the agent handles a task internally although external assistance is needed, causing hallucination, task failure, or unsafe action. ❷Overuse occurs when the agent invokes tools although internal execution would suffice, increasing cost, latency, permission exposure, and vulnerability to unreliable tool outputs.

We refer to this alignment problem as Ability Boundary Alignment: whether the agent can judge when its own ability is sufficient and when task execution should be delegated.

As the primary form of agent acting, tool use has been extensively evaluated, and existing tool-use benchmarks show that current agents often struggle to decide whether a tool is needed and which tool should be used, indicating that tool invocation itself constitutes a boundary-calibration problem [280]. Recent work further identifies both directions of capability-boundary miscalibration: tool underuse, where the agent attempts to solve externally dependent tasks through internal reasoning alone, and tool overuse, where it invokes tools for tasks that it can already solve without external support [30,281]. Beyond the binary decision of whether to use a tool, agents may also misjudge the scope, granularity, or completeness of external actions even when tool use is broadly appropriate [282], which can be regarded as action-boundary failures. Aligning the capability boundary therefore means calibrating when to rely on native capabilities, when to invoke external tools, and how far external action should extend.

Taken together, failures along the knowledge and capability boundaries are two manifestations of a unified cognitive-boundary miscalibration problem. The former concerns whether the agent has sufficient epistemic grounds to answer, while the latter concerns whether the task demands capabilities beyond the agent’s own. Aligning cognitive boundaries therefore requires shaping the decision process through which agents estimate what they know, what they can do, and when external resources are necessary.

4.1.2. Aligning Agents for Self-Awareness of Knowledge Boundaries

Aligning an agent with its knowledge boundary requires converting epistemic insufficiency into an actionable decision: whether to answer directly, seek additional evidence, ask for clarification or help, or abstain. Existing work can be understood as a progression from measuring boundary signals, to localizing where epistemic failure occurs, to controlling agent behavior, and finally to internalizing boundary awareness as part of the agent’s own decision process. Table 3 organizes representative knowledge-boundary alignment methods along five orthogonal dimensions: what signal reveals the boundary, where the failure is localized, how control is exerted, what the method applies to, and what outcome it produces.

Predictive and semantic uncertainty.

The most direct approach treats the knowledge boundary as a form of predictive uncertainty. Early work estimates whether a model is likely to be outside its usable knowledge from token- or sequence-level probability distributions in autoregressive generation and conditional generation [283,284]. These methods are attractive because they provide measurable and often model-agnostic scores, but surface-level likelihood is an imperfect proxy for epistemic sufficiency: different wordings can express the same meaning, and low probability may reflect ambiguity or stylistic variation rather than lack of knowledge. Semantic uncertainty methods address this problem by clustering meaning-equivalent generations and estimating uncertainty over meanings rather than surface forms [64,65]. Subsequent work refines this direction through relevance-weighted token- or sentence-level uncertainty, black-box calibration wrappers, and benchmark toolkits that compare uncertainty estimators across tasks [285,300,301]. A related line decomposes total uncertainty into aleatoric and epistemic components, making explicit that knowledge-boundary alignment should primarily target the latter [286]. However, uncertainty scores alone do not solve boundary awareness: they may align with human disagreement only under specific measures, be adversarially manipulated without changing the top prediction, or collapse under genuine ambiguity [67,302]. For this reason, calibration and agent-level papers increasingly treat uncertainty quantification as a useful control substrate rather than a sufficient diagnostic of self-awareness [66,303].

Representation-level boundary probes.

Because output-level uncertainty can be brittle, a second family asks whether knowledge-boundary signals are encoded inside the model’s representations. Hidden-state probes show that internal activations can contain information about statement truthfulness even when the generated answer appears confident [62]. Related work builds real-time hallucination detectors from contextual embeddings, identifies neurons involved in confidence regulation, and uses activation-based probes to approximate semantic entropy or verbal uncertainty mismatch [287,288,289,304]. The value of this family is that it separates what the model internally represents from what it says on the surface, thereby exposing cases in which fluent generation masks latent uncertainty. Its limitation is equally important: the boundary is still interpreted by an external probe or detector. These methods reveal that the model may contain boundary-relevant signals, but they do not by themselves ensure that the agent will use those signals to regulate answering, retrieval, or abstention.

Granular factuality and abstention evaluation.

The next step is to localize boundary failure at the unit where decisions are made. For knowledge-intensive generation, this means moving beyond a single uncertainty score for the whole response and estimating whether specific tokens, claims, summaries, or answers are unsupported. Reference-free hallucination detection, cross-sample consistency methods, and task-agnostic hallucination benchmarks operationalize this idea by estimating factuality without relying solely on external databases or human inspection [63,290,291,305]. Work on summary inconsistency and token-level fact checking further pushes boundary estimation to the granularity at which downstream revision or abstention can be triggered [306,307]. At the agent level, benchmarks such as ACEBench and AbstentionBench shift the focus from raw answer accuracy to whether the agent’s behavior matches epistemic need: whether it uses external resources when needed and whether it avoids overcommitting on unanswerable questions [68,69]. This family is important because it turns the abstract notion of a knowledge boundary into observable failure cases. Yet it remains primarily diagnostic: it tells us where the boundary was crossed, but not necessarily how the agent should regulate itself before making the error.

Uncertainty-gated control.

Once boundary signals can be estimated, they can be used to control agent behavior. In this family, uncertainty or factuality estimates act as gates that decide whether the agent should answer directly, retrieve evidence, ask for clarification, call a tool, or escalate to a human. KNOW NO, for example, uses conformal prediction to decide when a robot planner should ask for help, converting uncertainty into a help-seeking policy with statistical guarantees under the assumed calibration setting [292]. Similarly, uncertainty-aware language agents route among direct answering, tool activation, and human escalation through an uncertainty module outside the base policy [293]. These methods make boundary estimation actionable, which is a major step beyond post-hoc detection. Their limitation is that the agent’s boundary awareness is still delegated to a separate control mechanism. If the uncertainty module is miscalibrated, distribution-shifted, or insensitive to the task cost of externalization, the resulting agent can still over-answer, over-retrieve, or over-abstain.

Prompted self-reflection and deliberation.

To move from externally imposed control toward agent-level self-awareness, another line of work elicits boundary judgments from the model itself. Foundational studies show that language models can often evaluate their own answers or predict whether they know an answer when prompted appropriately, although this ability is incomplete and unevenly calibrated [61]. Further work finds that models can express uncertainty in grey areas, expose some hallucinated references through indirect consistency checks, and reveal mismatches between verbal confidence and probability-based evidence [308,309,310]. Building on this capacity, prompted and deliberative methods use self-consistency, paraphrase consistency, atypicality detection, explanation cycles, multi-LLM debate, or attention-guided reflection to induce the model to reconsider whether its answer is epistemically supported [153,294,311,312,313]. Compared with uncertainty-gated control, these methods better match the notion of self-awareness because the boundary judgment is expressed through the agent’s own reasoning routine. However, they are often prompt-sensitive, computationally expensive, and vulnerable to rationalization: the model may produce a plausible explanation of uncertainty without reliably changing its decision policy.

Training boundary-aware response policies.

Because prompted self-reflection is unstable, training-based approaches attempt to make knowledge-boundary awareness part of the model’s learned behavior. One line teaches models to abstain or say “I don’t know” when their knowledge is insufficient, making refusal an explicit action rather than an accidental failure mode [295,297,314,315]. A second line trains models to express calibrated confidence, provide self-reflective rationales, or support selective prediction, thereby making uncertainty communication itself an alignment target [154,296,316,317]. A third line incorporates uncertainty directly into alignment or learning objectives, for example by weighting examples according to uncertainty [318]. These methods shift the objective from detecting boundary violations after generation to shaping the model’s policy before generation. Their central risk is miscalibrated conservatism: a model trained to acknowledge uncertainty may over-refuse, while a model trained to be helpful may still overcommit. Theoretical and empirical work on self-evaluation therefore asks when such learned introspective routines can themselves be trusted [319].

Agent-level boundary-aware action selection.

For agents, knowledge-boundary awareness ultimately has to govern action selection, not merely answer scoring. The agent must decide when to answer, when to critique itself, when to ask another model, when to retrieve, when to call a tool for epistemic support, and when to stop. Work on calibrated LLM evaluators and multi-agent deliberation improves the reliability of self-critique by aggregating or calibrating rationales before using them in decision loops [320,321]. Recent agent-level studies show that this calibration problem persists when external actions are available: SMART targets tool overuse by training agents to recognize when external calls are unnecessary, while work on the confidence dichotomy in tool-use agents shows that text-level confidence calibration does not automatically transfer to action-level calibration [30,298]. Numerical predictive distributions further enrich the agent’s action vocabulary by allowing it to communicate graded forecasts rather than binary confidence or refusal [299]. Taken together, these methods suggest that aligning agents with their knowledge boundaries requires more than estimating uncertainty. It requires a calibrated policy over answering, abstaining, evaluating, retrieving, and externalizing, where each action is selected according to epistemic sufficiency and the cost of unnecessary intervention.

Summary.  Knowledge-boundary alignment converts epistemic insufficiency into a calibrated policy over answering, retrieving, clarifying, and abstaining, and uncertainty becomes genuine self-awareness only when it reliably changes action selection rather than remaining a post-hoc score.

4.1.3. Aligning Agents for Self-Awareness of Ability Boundaries

The problem of ability boundary alignment is different from knowledge-boundary calibration: knowledge boundaries ask whether the model knows an answer, whereas ability boundaries ask whether the agent can reliably, safely, and efficiently advance a task through its native capabilities. Existing work can therefore be organized around a progression from tool-use gating, to model-adaptive boundary estimation, to trajectory-level boundary control, and finally to governance of boundary failures.

What is aligned.

The object of alignment is not the tool set itself, but the agent’s routing policy between internal execution and external delegation. Given a task state, an aligned agent should decide whether to answer directly, call a tool, ask for clarification, or refuse execution. Benchmarks such as MetaTool and When2Call make this decision boundary explicit by evaluating whether models can determine not only how to call tools, but also whether tools are needed and which tool is appropriate [167,280]. This shifts tool use from a selection problem to a boundary-placement problem: a correct tool call is not merely syntactically valid, but necessary for advancing the task.

From tool availability to tool necessity.

A tool-augmented agent should not call a tool simply because the tool is available or semantically related to the task. The relevant question is whether using the tool improves expected task performance after accounting for cost, latency, uncertainty, and risk. Work on tool-call optimization therefore frames the call/no-call decision as a utility-sensitive problem, where the agent must compare internal execution against external delegation [260]. This perspective also explains both underuse and overuse as boundary errors: underuse underestimates the benefit of external assistance, while overuse overestimates it.

From static labels to model-adaptive boundaries.

Tool necessity cannot be treated as a fixed property of the task alone. The same task may be solvable internally by a stronger model but require tool use for a weaker one. Ability boundaries are therefore model-relative: they depend on the agent’s actual competence, not merely on task type. Model-Adaptive Tool Necessity captures this idea by defining tool necessity with respect to the model’s own ability and reveals a knowing-doing gap, where a model may recognize that a tool is needed but still fail to call it during execution [322]. This suggests that ability-boundary alignment requires both accurate self-assessment and reliable translation of that assessment into action.

From single decisions to trajectory-level control.

In real agents, ability boundaries are not decided once at the beginning of a task. They must be re-estimated throughout a multi-step trajectory as the agent observes intermediate results, updates the task state, and chooses subsequent actions. ToolSandbox highlights this issue by evaluating tool use in stateful, conversational, and interactive settings rather than isolated single-turn calls [92]. Self-Harness further suggests that an agent’s effective ability boundary is shaped by the surrounding harness, prompts, execution environment, and feedback loop, not only by the model parameters themselves [328]. Thus, ability-boundary alignment should be understood as trajectory control: the agent must repeatedly decide what it can still do internally and what should be externalized at the current step.

What fails when the boundary is misplaced.

Boundary failures appear at several layers. At the decision layer, agents may exhibit intrinsic over-calling bias or prefer external tools even when internal knowledge is sufficient [323,324]. At the trust layer, tool outputs may conflict with the model’s parametric memory, forcing the agent to arbitrate between internal and external evidence [325]. At the selection layer, tool preferences may be unstable under small changes in tool descriptions, showing that natural-language tool specifications can distort the boundary judgment [326]. At the permission layer, agents may select tools with higher privileges than the task requires, turning ability-boundary alignment into a problem of least-privilege action selection [327]. These failures are not isolated phenomena; they are different manifestations of the same underlying issue: the agent delegates too little, too much, or to the wrong external operator.

From self-awareness to governance.

Self-awareness of ability boundaries should therefore be action-centered. The agent must not only know whether it knows an answer, but also whether an external operator would improve execution, and whether the cost and risk of delegation are acceptable. Recent work on agentic self-awareness and tool-overuse mitigation moves in this direction by connecting internal ability estimation with external action selection [27,30,218,281]. A unified view of Ability Boundary Alignment should jointly model ability estimation, tool utility, execution feedback, trust calibration, and permission control. Under this view, the goal is not to maximize tool use or minimize tool use, but to align the agent’s delegation behavior with the true boundary of what it can do safely and reliably on its own.

Summary.  Ability-boundary alignment regulates the routing between native execution and external delegation, treating tool necessity as model- and state-dependent and repeatedly recalibrating it from expected utility, execution feedback, trust, permissions, and risk.

4.2. Aligning with Efficiency Constraints

The preceding subsection asked whether an agent can recognize the boundary of what it knows and what it needs. This subsection asks what the agent should do once that boundary has been recognized under efficiency constraints. Beyond deciding whether to rely on internal reasoning or external action, the agent must determine how much additional effort is warranted, where that effort should be allocated, and when its expected marginal benefit no longer justifies its cost.

In practice, aligning with efficiency constraints means regulating goal-directed epistemic effort in a cost-sensitive way. The central problem is therefore not only whether the agent can obtain more information or improve its answer, but whether doing so is worth the additional computational, temporal, monetary, or interaction cost. Recent work on economic evaluation, efficient agent design, dynamic test-time scaling, and adaptive routing converges on the same point: efficiency-aware alignment requires a calibrated agent to decide whether further thinking, acting, or asking has sufficient marginal epistemic value to justify its cost [71,262,329,330,331].

We organize this subsection around two aspects of how efficiency constraints shape agent behavior. ❶Internal efficiency alignment regulates the cost of the agent’s own reasoning, including token budgets, rollout allocation, stopping decisions, and context maintenance. ❷External efficiency alignment governs the cost of world-facing actions, including tool invocation, model routing, action-space shaping, long-horizon tool trajectories, and multi-agent collaboration.

4.2.1. Internal Efficiency Alignment: Budgets, Search, and Context Maintenance

Internal efficiency alignment concerns how an agent manages the cost of its own reasoning process. The central issue is not whether more thinking can improve performance, but when additional internal computation is worth its cost. Existing work forms a clear progression: first, making reasoning budgets visible and controllable within a single trace; second, allocating compute across multiple reasoning paths or search states; and third, reducing the accumulated cost of long-horizon reasoning by stopping, compressing, or maintaining context more efficiently.

Summary.  Internal efficiency alignment turns computation into budgeted epistemic effort, allocating tokens, rollouts, search depth, and context according to task execution state and stopping when the expected reduction in uncertainty no longer justifies additional cost.

Budget as a first-class signal.

The first line of work makes budget a first-class signal within a single reasoning path. Qwen3 [332] and BudgetThinker [74] introduce budget tokens or special control tokens into the reasoning process, allowing the model to condition its reasoning on a target or remaining budget. SelfBudgeter [268] further connects budget use to task difficulty by training models to estimate and follow appropriate token budgets before reasoning. LightThinker [333] approaches the same problem from the compression side: instead of only limiting how much reasoning is generated, it makes intermediate reasoning more compact. These methods share the same role in the efficiency-alignment pipeline: they transform budget from an external deployment limit into an internal variable that the model can perceive, plan around, and obey.

Budget allocation across multiple paths.

The second line expands the unit of efficiency alignment from a single reasoning trace to a distribution over possible traces. Once the agent can reason under a budget, the next question is how that budget should be allocated across candidate solutions, rollout paths, or search states. Star Elastic [334] enables runtime selection among elastic compute levels, so the same model can operate under different reasoning budgets. DUET [335] optimizes token-budget allocation across prompts and rollouts, while dual-dimensional consistency [336] balances budget and answer quality by adapting inference-time scaling across reasoning paths. Complexity-aware Text-to-SQL [337] allocates exploration budget according to query difficulty, and adaptive branching tree search decides whether to expand wider or refine deeper according to the current search state [338]. Compared with the first line, these methods no longer ask only how long one trace should be; they ask how limited internal compute budget should be distributed across a structured space of alternatives.

Budget conservation in long-horizon agents.

The third line addresses a distinct source of internal inefficiency: in long-horizon agents, cost accumulates over time. Even if each individual reasoning step is cheap, the agent may persist longer than necessary or carry forward an ever-growing context. One way to curb this accumulation is to lighten the input burden before reasoning even begins. UI-to-agent-interface optimization simplifies how environment observations are represented, reducing the initial cognitive load [344]. Once reasoning starts, adaptive stopping methods control its depth: MiCP allocates error budgets across turns and terminates early when further thinking is unlikely to reduce uncertainty, cutting inference cost while preserving coverage guarantees [339]. Beyond stopping, the agent must also manage the width of its context. Lightweight strategies such as simple observation masking can match or outperform LLM summarization at lower cost [341], while AgentDiet explicitly removes useless, redundant, and expired trajectory information [340]. More proactive approaches let the agent itself decide when to consolidate and prune history [343], or optimize compression guidelines and distill them into cheaper models [342]. At the extreme, MEM1 learns a compact memory state that jointly supports reasoning and long-horizon efficiency [11]. Together, these works show that internal efficiency alignment is not limited to token budgeting: it also includes deciding when to stop, what to remember, what to discard, and how to represent the task state compactly.

Taken together, internal efficiency alignment follows a coherent trajectory from budget visibility to budget allocation and finally to long-horizon context economy. Early methods make the cost of reasoning observable to the model; allocation methods decide where limited compute should be spent; and stopping, compression, and memory methods prevent past reasoning from becoming a growing burden on future decisions. In ToA terms, these methods align the agent with efficiency constraints by regulating the scale, distribution, and persistence of $E_{\mathrm{int}}$.

4.2.2. External Efficiency Alignment: Tool Use, Routing, and Trajectory-Level Action Costs

External efficiency alignment governs the cost of actions beyond the agent’s internal reasoning. These actions include retrieving evidence, invoking tools, calling other models, interacting with an environment, and delegating work to other agents. The central issue is not whether external resources improve capability in general, but whether a specific external action is worth its cost in the current state. Existing work develops this idea along a natural progression: first, deciding whether externalization is needed; second, selecting the most cost-effective external resource; third, reducing the exposed action space; and fourth, coordinating external actions over long trajectories or multiple agents.

Summary.  External efficiency alignment treats tools, models, collaborators, and actions as priced sources of evidence or capability, deciding whether, where, and how long to externalize effort so that marginal task value exceeds invocation, coordination, and verification cost.

Deciding whether externalization is needed.

The first line calibrates whether externalization is warranted at all. Efficient tool-calling alignment directly penalizes redundant invocations and trains the model to call external functions only when the expected value exceeds the cost [72]. Acting Less is Reasoning More converges on the same objective, showing that agents improve by learning when not to act, which shifts the goal from maximal externalization to cost-effective decision [73]. These works define the basic external efficiency boundary: before asking which tool to use, the agent must decide whether any external action is justified.

Selecting the most cost-effective external resource.

Once externalization is justified, the next question is which external resource should be used. ToolOrchestra trains an orchestrator to coordinate specialized tools and models under outcome-, efficiency-, and preference-aware rewards [9]. Budget-aware agentic routing treats route selection as a sequential, path-dependent decision problem, choosing among models under a cost-quality frontier and explicit budget constraints [345]. Reinforcement-learning approaches similarly turn external resource selection into a learned policy: ReTool teaches strategic code-interpreter use through outcome feedback [107], and Search-R1 trains search-engine use as part of reasoning rather than as a fixed retrieval wrapper [220]. These works move beyond the binary call-or-not-call decision. They align the agent with efficiency constraints by making external resource selection sensitive to task state, expected utility, and budget.

Reducing the exposed action space.

A complementary line reduces external inefficiency by shaping the action space before the model acts. Rather than relying on the agent to choose from a large and redundant tool set, ToolScope merges overlapping tools, retrieves relevant tool descriptions, and presents a narrowed toolset for each query [346]. Dynamic instruction-tool exposure constructs step-specific system instructions and exposes only the tools relevant to the current state [347]. AutoTool reduces repeated tool-selection cost by exploiting tool-usage inertia from historical trajectories and selecting tools through a structured graph rather than invoking the LLM from scratch at every step [348]. Causal Minimal Tool Filtering exposes only the causally sufficient next-step tool frontier according to precondition–effect contracts, avoiding premature or unnecessary tool calls [349]. These methods share a common principle: external efficiency can be improved not only by making the agent choose better, but also by making the available action space smaller, cleaner, and more causally relevant.

Making externalization decision under full trajectories.

The fourth line extends external efficiency alignment from isolated actions to entire trajectories. In long-horizon tasks, the cost of external action is cumulative: every tool call, observation, and environment interaction expands the trajectory and may increase the cost of later decisions. ReWOO reduces the cost of interleaved thought–action–observation cycles by decoupling reasoning from external observations, avoiding repeated prompt expansion during tool-augmented reasoning [183]. Budget-Constrained Tool Learning with Planning estimates candidate-tool utility from past experience and constructs a budget-feasible tool-use plan through dynamic programming before execution [350]. During execution, budget-aware tool use tracks remaining tool-call resources and adapts planning and verification, deciding whether to deepen a promising lead or pivot to alternative paths [75]. INTENT formalizes budget-constrained tool use as sequential decision making with priced and stochastic tool executions, using an intention-aware hierarchical world model to anticipate future tool usage and risk-calibrated cost [76]. Budget-Aware Value Tree Search treats multi-hop reasoning as a dynamic search tree and uses a budget-conditioned node-selection mechanism to shift from exploration to exploitation as resources are depleted, while pruning redundant or uninformative tool calls [351]. Dynamic turn-budget studies in coding-agent settings further show that limiting the number of action–observation cycles can reduce cost compared with unrestricted iteration [352]. These works reveal that external efficiency alignment is fundamentally trajectory-level: the agent must decide not only whether the next action is useful, but how that action affects the remaining budget and future decision space.

Distributing efficiency constraints across agents.

Finally, external efficiency constraints can also be distributed across agents. In multi-agent systems, inefficiency often arises from duplicated exploration, redundant communication, or poorly coordinated division of labor. Co-Saving shows that resource-aware collaboration can reduce duplication in software-development teams [353]. This extends the same efficiency principle from tools to collaborators: the agent should externalize work only when delegation or coordination reduces the total cost of solving the task.

Overall, external efficiency alignment follows a coherent trajectory from externalization decisions to resource selection, action-space shaping, and trajectory-level coordination. The early question is whether the agent should act externally at all; the next is which external resource is worth using; the structural question is how to expose only useful actions; and the long-horizon question is how to spend external effort across a sequence of decisions. In ToA terms, these methods align the agent with efficiency constraints by regulating the necessity, target, feasibility, and temporal distribution of $E_{\mathrm{ext}}$.

4.3. Aligning with Human

Human alignment focuses on ensuring that agent behavior conforms to external human expectations and values. In practice, agents must not only accomplish tasks correctly, but also respect human preferences and accurately interpret human intentions. Accordingly, this section discusses two complementary aspects of human alignment: ❶preference alignment, which shapes agent behavior through learning from human feedback on helpfulness, safety, and personalization, and ❷intention alignment, which enables agents to infer and clarify the underlying goals behind user instructions.

4.3.1. Aligning for Human Preference

Preference-based alignment aims to align agent behaviour with human expectations and values by learning from human feedback. Existing approaches typically optimize agents using human preference comparisons through methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), encouraging behaviours that are helpful, honest, harmless, and consistent with human expectations [79,354,355]. Recent work further explores fine-grained human feedback to model different aspects of human preferences separately [80].

Among these objectives, safety alignment is a fundamental component of general preference alignment, ensuring that agents avoid harmful behaviours while accomplishing user goals. Representative methods explicitly distinguish utility from safety during optimization. For example, BeaverTails [78] separates helpfulness from harmlessness, while Safe RLHF [81] introduces explicit safety costs to optimize task performance under safety constraints. These approaches improve the reliability of agents by integrating safety requirements into preference optimization.

While general preference alignment captures preferences shared across users, real-world users often exhibit diverse values, habits, and interaction styles. Personalized preference alignment therefore aims to adapt agent behaviour to individual users by modeling user-specific preferences and interaction history. AGENTIF [89] shows that agents may gradually drift away from user constraints during long-horizon tool use, highlighting the need for continual preference tracking. To facilitate research on adaptive personalization, HumanLM [90] simulates users with evolving preferences, while FlippingDialog [356] generates adversarial interactions that expose failures in preference inference and adaptation. On the evaluation side, ToolSpectrum [357] evaluates personalized tool utilization, testing whether agents adapt tool selection and call construction to individual user profiles.

Summary.  Preference alignment shapes agent behaviour by learning human-valued objectives from feedback, extending from general helpfulness and safety optimization to personalized adaptation through user-specific preference modeling, continual interaction feedback, and robust preference inference.

4.3.2. Aligning for Human Intentions

Human intention alignment extends beyond instruction following to establishing a shared understanding between humans and agents. Rather than treating humans as static sources of instructions or feedback, recent work increasingly views them as active collaborators whose intentions, guidance, and feedback should be continuously interpreted throughout task execution [358,359].

Aligning Intent Understanding

A central challenge in human intention alignment is recovering users’ underlying goals from imperfect instructions. The URS Benchmark [87] shows that current agents struggle to satisfy multiple user objectives and implicit constraints simultaneously, highlighting the gap between instruction following and genuine intention understanding. To bridge this gap, Mistral-Interact [360] proactively identifies incomplete instructions, asks clarification questions, and refines implicit intentions into executable goals. Besides improving agents’ understanding of users, recent work also emphasizes helping users understand agents. [361] further advocates that agents should explicitly communicate their goals and plans, making their intentions transparent and establishing shared understanding through bidirectional communication.

Dynamic Intention Tracking and Refinement

During long-horizon interactions, users may revise their objectives, reveal new constraints, or change their preferences, requiring agents to continually update their interpretation of user intent rather than relying on a single instruction [362,363,364,365]. [366] suggest that language models exhibit emergent Theory-of-Mind capabilities, providing a foundation for reasoning about users’ latent beliefs beyond explicit instructions. Nevertheless, InterIntent shows that accurately inferring and maintaining users’ intentions remains a challenging aspect of social intelligence for LLM agents, especially in interactive settings [367,368,369]. To address this challenge, recent works increasingly formulate intention alignment as an interactive and personalized process rather than one-shot instruction following. [360] identifies underspecified requests, elicits missing information through dialogue, and refines inferred intentions before task execution. To ensure that such refinements remain consistent across interactions, PUMA [370] augments agents with personalized user profiles derived from historical behaviors, allowing current requests to be interpreted in the context of long-term user preferences. This persistent user modeling is further supported by memory-based frameworks such as EMG-RAG [371], which continuously update editable user memories to retrieve, revise, and accumulate user-specific knowledge as users’ goals and preferences evolve over time. These approaches enable agents to align with user intent both within and across interactions.

Summary.  Human-intention alignment governs how agents interpret and adapt to human goals, evolving from instruction understanding to interactive intention refinement through clarification, bidirectional communication, and persistent modeling of users’ evolving objectives.

4.4. Aligning with Environment Constraints

Unlike agent learning, which equips agents with the capabilities to interact with external environments, environment alignment focuses on ensuring that these capabilities are exercised within environmental constraints. In practice, agents must not only generate actions that conform to interface specifications and execution requirements, but also satisfy safety and security policies during deployment. Accordingly, this section discusses two complementary aspects of environment alignment: ❶operational constraint alignment, which ensures that actions are valid and executable, and ❷safety constraint alignment, which prevents harmful or unauthorized behaviours when interacting with external environments. We summarize the representative methods discussed in this section in Table 6.

4.4.1. Aligning for Operational Constraints

Environment interfaces, such as APIs, tools, and operating system calls, define a constrained action space that agents must follow to produce valid and executable actions. A common approach is to incorporate these constraints into policy learning. Supervised fine-tuning on environment-specific trajectories enables agents to learn valid API formats, argument structures, and invocation patterns [46,47]. Reinforcement learning further optimizes policy generation using rewards based on execution correctness or schema compliance [41,372]. To improve generalization to previously unseen interfaces, some methods also adopt test-time adaptation, allowing agents to adjust to new schemas through in-context interaction or lightweight updates without full retraining [100]. The same constraint problem becomes physical in laboratory automation, where an autonomous chemical robotic platform must map synthesis goals into valid instrument actions and an invalid action wastes materials rather than merely failing a call [379].

However, learning interface constraints alone cannot prevent execution failures caused by invalid actions at deployment time. As a result, many approaches enforce operational constraints during inference by validating actions before execution. This can be achieved through constrained decoding, which restricts token generation to syntactically valid outputs [94], or runtime verification, where external modules check generated actions against interface specifications before execution. For example, InferAct [374] performs look-ahead validation to identify invalid intermediate actions, while subsequent work further detects and corrects off-task actions during execution [375]. Beyond pre-execution validation, execution feedback can also be used to recover from constraint violations. E2CL [373] treats execution failures as signals for revising subsequent actions, enabling agents to recover from invalid trajectories and maintain compliance with operational constraints. Although these methods improve execution reliability, overly strict constraints may reduce agent flexibility and incur an alignment tax in complex tasks [380].

Summary.  Operational-constraint alignment regulates how agents map decisions into executable actions, evolving from learning interface-specific policies to enforcing runtime validity through verification and feedback-based recovery while balancing execution reliability against flexibility.

4.4.2. Aligning for Safety Constraints

Beyond producing executable actions, agents must also ensure that their actions comply with safety and security requirements during deployment. Unlike conventional LLMs that generate isolated responses, agents interact with external environments through tools and APIs, making unsafe actions more likely to produce lasting consequences. Recent benchmarks, including Agent-SafetyBench [82], AgentHarm [84], SafeToolBench [381], and SafeGenBench [382], show that agents may reject harmful requests in isolation yet still pursue unsafe objectives through multi-step planning and tool use. These findings suggest that safety cannot be evaluated solely at the response level, but should instead consider the entire execution process.

To address these challenges, recent work increasingly complements training-time alignment with deployment-time safeguards. Agentic red-teaming and security studies reveal that prompt injection, malicious tool outputs, and other environmental interactions introduce new attack surfaces for agents [99,383,384]. Accordingly, runtime protection mechanisms, such as programmable privilege control [85], restrict the actions available to agents according to predefined security policies by statically confining the tool and argument space an agent may invoke. ShieldAgent [376] extends this idea from single-step permissions to whole trajectories, compiling policy documents into verifiable rule circuits that formally check an agent’s action sequence against the underlying policy. Rather than fixing a rule set in advance, GuardAgent [377] instead uses a separate LLM-based guard agent that interprets a safety request at inference time and synthesizes guardrail code on the fly, allowing the check itself to adapt to each new request. A complementary line of work instead keeps a human or a separate monitor in the loop at inference time: the control tax framework [378] quantifies the overhead of inserting such oversight checkpoints before high-risk actions are committed, making explicit the trade-off between operational flexibility and safety guarantees. Together, these approaches emphasize that safety constraints should be enforced throughout deployment rather than relying solely on policy alignment during training.

Summary.  Safety alignment governs how agents maintain secure behaviour during execution, extending beyond static policy compliance to trajectory-level risk control through runtime safeguards, privilege management, and adversarial evaluation while balancing security constraints with agent flexibility.

4.5. Aligning with Other Agents

Agents have to align with other autonomous agents whose goals, beliefs, and actions are only partially observable. Effective multi-agent collaboration therefore requires progressively developing three complementary capabilities: ❶information exchange, which establishes shared interaction spaces through standardized protocols for message exchange, capability discovery, and context sharing; ❷state understanding, which enables agents to infer and reason about other agents’ latent mental states, including their beliefs, goals, and intentions; and ❸collaborative coordination, which organizes individual plans and actions into coherent team-level behaviors through role allocation, joint planning, and adaptive orchestration. We summarize the representative methods discussed in this section in Table 7.

4.5.1. Information Exchange

Information exchange concerns how heterogeneous agents establish a shared interaction space through standardized communication protocols. The importance of standardized communication has long been recognized in traditional multi-agent systems through agent communication languages such as KQML [394] and FIPA ACL [395], which define common message semantics for distributed agents. As LLM-based agents are increasingly developed by different models, platforms, and organizations, heterogeneous interfaces and communication formats have become a major obstacle to scalable collaboration. Information exchange therefore focuses on establishing standardized mechanisms for capability discovery, context sharing, task delegation, and message exchange across heterogeneous agent systems.

Recent protocols extend these ideas to support richer interactions involving tools, external resources, and long-horizon collaboration. At the tool level, the Model Context Protocol (MCP) [96] standardizes how tool schemas, resources, and contextual information are exposed to agents, improving interoperability across heterogeneous services. Building on this idea, the Agent-to-Agent (A2A) Protocol [97] further defines structured primitives for inter-agent communication and task delegation under a unified messaging schema. Beyond standardized interfaces, Agora [385] explores decentralized coordination mechanisms that enable agents to negotiate, delegate responsibilities, and cooperate in open multi-agent environments. Together, these efforts advance information exchange from standardized communication interfaces to richer interaction mechanisms that support scalable collaboration across heterogeneous agent ecosystems.

Summary.  Information exchange governs how agents establish shared interaction spaces, extending from standardized communication protocols to interoperable interaction mechanisms through unified interfaces, context sharing, and capability discovery across heterogeneous agent ecosystems.

4.5.2. State Understanding

State understanding concerns how agents infer and reason about the latent mental states of other agents, including their goals, beliefs, intentions, and knowledge, from observed interactions and behaviors. Unlike environmental constraints, these latent states are not directly observable and must be inferred from interactions and behaviors. Effective collaboration therefore depends on accurately understanding other agents while remaining robust to uncertainty and potentially misleading signals.

Early work primarily focuses on state inference, where agents estimate the goals and future actions of teammates or opponents from their observed behaviors. Such reasoning reduces coordination uncertainty and enables agents to anticipate the behaviors of others in both cooperative and competitive settings [396]. Building on this idea, recent LLM-based agents incorporate Theory of Mind to infer other agents’ beliefs and internal states, allowing decisions to account not only for what others are likely to do, but also what they know, believe, or expect [386,397].

More recent work further extends state understanding from estimating individual mental states to strategic reasoning, where agents recursively reason about how other agents perceive the environment and respond to their own actions. Such higher-order reasoning supports negotiation, adaptive cooperation, and competitive decision-making by allowing agents to anticipate reactions and adjust their strategies accordingly [387,388]. Recent approaches further incorporate adaptive Theory-of-Mind reasoning, enabling agents to dynamically adjust the depth of recursive reasoning according to the behaviors of their teammates or opponents [389]. Together, these studies advance state understanding from predicting observable behaviors to reasoning about latent beliefs and strategic interactions.

Summary.  State understanding governs how agents infer and reason about the latent mental states of others, extending from predicting observable behaviors to modeling beliefs, intentions, and recursive strategic reasoning for adaptive interaction in uncertain multi-agent environments.

4.5.3. Collaborative Coordination

Information exchange and state understanding are necessary but not sufficient for effective multi-agent collaboration. A group of agents may exchange well-formed messages and correctly infer one another’s goals, yet still duplicate work, violate task dependencies, or leave important subtasks unassigned. Collaborative coordination concerns how multiple agents organize their local beliefs, plans, and actions into coherent team-level behaviors under a shared objective. Existing work progresses from allocating roles and responsibilities, to synchronizing shared epistemic state, to constructing joint plans, and finally to adapting the coordination structure when initial allocation proves insufficient.

Role and responsibility allocation.

The first coordination problem is determining which agent is responsible for which part of the task. Role-based systems address this through explicit division of labor: CAMEL uses role-playing to stabilize interactions, while AutoGen supports configurable conversation patterns among agents with different functions [13,390]. MetaGPT [12] and ChatDev [391] extend this approach by mapping software-development tasks onto specialized roles and structured workflows, so that requirements, design, and implementation are handled by distinct agents. These structures reduce ambiguity about ownership and make intermediate outputs easier to inspect. However, a role label does not guarantee that the assigned agent is the best available solver. Static role assignments may mismatch model capabilities, create bottlenecks, or cause several agents to solve the same subproblem independently. Effective role alignment therefore requires assigning responsibilities according to current capability, task dependency, and workload, with explicit conditions for completion and handoff.

Shared epistemic state and coordination artifacts.

Once responsibilities are distributed, agents need a sufficiently consistent view of the task state to avoid redundant exploration or conflicting assumptions. Multi-agent systems commonly externalize this state through shared message pools, memories, or repositories of reusable experience and skills [35,123]. Such artifacts allow one agent’s observation or partial solution to become another agent’s evidence without requiring every member to repeat the same exploration. However, unrestricted sharing may propagate stale observations or unsupported claims, requiring selective access policies and provenance tracking to preserve information reliability.

Collaborative planning and synchronized execution.

A shared state becomes useful only when agents can turn it into a joint plan whose components are mutually compatible. Process-oriented systems such as MetaGPT[12] use structured stages and handoffs to decompose global objectives into subtasks and specify how local outputs will be integrated. More interactive settings require temporal coordination: agents must decide when to act in parallel, when to wait for another result, and when a downstream step should be invalidated by an upstream change. Benchmarks such as MultiAgentBench and Collab-Overcooked evaluate collaboration under different communication topologies and resource constraints, revealing that locally reasonable actions can still be globally misaligned when they arrive too late or violate another agent’s preconditions [125,126].

Adaptive orchestration and topology selection.

Because task difficulty and agent competence vary across states, a fixed coordination structure is rarely optimal for an entire trajectory. Adaptive orchestration changes which agents participate, how they are connected, and how much redundancy is introduced according to the current uncertainty and expected value of collaboration. GPTSwarm represents multi-agent systems as optimizable interaction graphs, while automated design methods search over agentic structures rather than assuming a hand-written workflow [392,393]. Resource-aware collaboration further shows that coordination should account for duplicated exploration and communication cost rather than maximizing the number of active agents [353]. From an alignment perspective, the relevant decision is whether an additional delegate or communication edge is justified in the current state. Parallel agents may be valuable when independent evidence is needed, but become wasteful when they merely reproduce the same reasoning. Adaptive coordination therefore treats team topology as a control variable: it should expand when specialization can materially reduce uncertainty, and contract when coordination overhead exceeds the expected contribution of another agent.

Team-level calibration and coordination failures.

The final requirement is to calibrate local decisions against their effect on the team rather than against individual success alone. Current benchmarks reveal a persistent collaboration gap: agents that perform well independently may deteriorate when they must synchronize beliefs, timing, and responsibilities with peers [398]. Miscalibration manifests in two forms. Under-coordination produces omissions, conflicting actions, or repeated work, while over-coordination introduces excessive communication and redundant verification that increases cost without improving outcomes [125,126]. Team-level calibration therefore requires evaluating the marginal contribution of each action to the shared objective while retaining redundancy only where verification value exceeds coordination cost.

Summary.  Collaborative coordination governs how agents transform individual capabilities into coherent team behaviors, extending from responsibility assignment and shared state coordination to joint planning, adaptive orchestration, and cost-aware collaboration.

5. Evolving: Revising Epistemic Supports Over Time

An agent’s capabilities are not fixed once a checkpoint is released. As it executes real-world tasks, it can accumulate experience, discover tools, revise memory, interact with other agents and users, and encounter environments whose tasks and feedback shift over time. If these experiences cannot reshape the agent, it either repeats old mistakes or keeps paying external costs for problems it has already learned to handle. Evolution is thus a boundary problem, not merely a matter of making the agent more capable. Where Section 3 asked where competence should live (inside the model as priors, routines, and control, or outside it in memory, tools, workflows, and environments), this section asks the next question: As the agent continually executes tasks in real-world environments, how should it evolve its internal reasoning and external acting capabilities, and how should the boundary between them shift over time?

Figure 8. Epistemic evolution as a three-level feedback loop. Internal evolution amortizes experience into model-side priors, routines, and control; External evolution revises harness-side memory, tools, workflows, and environments; and Ecological evolution feeds population- and environment-level signals back into subsequent rounds of adaptation.

Figure 8. Epistemic evolution as a three-level feedback loop. Internal evolution amortizes experience into model-side priors, routines, and control; External evolution revises harness-side memory, tools, workflows, and environments; and Ecological evolution feeds population- and environment-level signals back into subsequent rounds of adaptation.

Under the ToA framework, evolution does not simply mean a larger model or a richer toolset. A meaningful update reduces the cost of transitioning from uncertainty to action: merely invoking more tools or interacting with the environment more frequently does not necessarily improve epistemic efficiency, as task pass rates may increase while efficiency remains unchanged. We consider an update to constitute evolutionary progress only when it lowers the long-run cost of useful uncertainty reduction.

❶ Internal evolution revises model-side resources (i.e., priors, routines, control); ❷ External evolution revises the harness (i.e., memory, tools, workflows, environments); and ❸ Ecological evolution moves the unit of adaptation outward, from one agent to populations or agent–environment systems. The levels interact: model updates change what is cheap to reason about, harness updates what is cheap to obtain, and ecological updates how evidence is produced, shared, and selected. One test spans all three: adding components is progress only when it makes useful evidence cheaper, more reliable, or better aligned with the agent’s uncertainty.

5.1. Internal Evolution: From Experience Signals to Model-Side Control

Internal epistemic evolution concerns the parts of an agent that can improve on the model side: what it assumes about tasks and tools, how it carries out recurring reasoning patterns, and how it decides whether to keep reasoning or seek external evidence. The motivation is amortization. If a behavior is useful across many episodes but currently requires long prompts, tool traces, external critiques, or multi-agent scaffolds, then it is a candidate for being moved into the model, provided that the behavior is stable enough to reuse.

We discuss three ways this inward update happens. ❶ Experience-driven data evolution collects failures, successful trajectories, and boundary cases that reveal gaps in the current model. ❷ Parameter-level evolution then uses post-training, distillation, or reinforcement learning to turn selected trajectories into model behavior. ❸ Inference-time strategy adaptation is lighter-weight: it changes how the current model spends test-time computation before new data or gradient updates are available. Thus, while Section 3 studied where competence should be located, this subsection studies how that location is revised over time as the agent gathers trajectories, rewards, and deployment feedback. Table 8 compares representative methods along the experience, feedback, and adaptation dimensions used throughout the subsection.

5.1.1. Experience-Driven Data Evolution: Building the Next Training Signal

Before a model can get better on the inside, it first needs the right training data, and that is usually the real bottleneck. Fixed, hand-written datasets mainly show what correct behavior looks like; they do not show where the model currently fails. Evolving agents instead need signals generated through interaction: tasks that probe the edge of what the agent can do, trajectories that record how it worked through uncertainty, and lessons distilled from past attempts. Recent papers describe this shift from static datasets to closed loops in which agents generate tasks, attempt them, verify the results, and reuse the outcome as new training data [2,31,36]. We group this line of work by where in the pipeline a method intervenes: task-side construction decides what the agent should attempt, trajectory-side realization turns a given task into a trainable trace, and post-trajectory conversion turns an existing trace into reusable training material.

On the task side, these methods build or adjust the set of tasks before the agent starts acting, aiming to create cases near the edge of what the agent can currently do, rather than just adding more of the same easy examples. One approach makes existing tasks harder or more agent-like: TaskCraft turns simple seed problems into harder, multi-tool, checkable tasks, while Benchmark Self-Evolving uses multiple agents to rewrite benchmark problems so they keep targeting the model’s current weak points [102,104]. A second approach pulls new tasks out of a richer environment: SPICE turns a text corpus into a self-play setting where one model writes grounded questions and another tries to answer them, while Agent-World does the same with real tool ecosystems and databases, exploring the environment to create checkable tasks at a chosen difficulty level [105,399]. SAGE makes this loop explicit with four roles, a Challenger, a Planner, a Solver, and a Critic, that separately generate, plan, solve, and filter tasks [106]. The value of task construction depends entirely on whether it targets real weak points: if the generator, solver, and verifier share the same blind spots, the resulting tasks stay easy to produce but stop being useful for finding what the model cannot yet do [36].

On the trajectory side, given a task, these methods generate the actual reasoning or acting trace that the agent can learn from. Early work already used this idea: STaR keeps only the reasoning chains that lead to a correct answer, while FireAct turns diverse agent trajectories into fine-tuning data [161,400]. More recent systems move from plain text reasoning to full interaction: TOUCAN generates large numbers of tool-use trajectories from real tool environments, turning multi-turn tool use into a rich, dense training signal instead of a handful of demonstrations [49]. Some methods lower the cost of collecting these traces without needing full live interaction: Experience Synthesis and simulated-environment methods generate rollouts and feedback from past histories or reasoning models, while implicit-experience methods pull tool-use trajectories out of ordinary text [101,115,409]. Despite their different levels of realism, all of these methods do the same job: turning a task into a trace of how the agent reasoned, acted, observed, and corrected itself.

After the trajectory, these methods start from traces that already exist and turn them into something else that is reusable, instead of just replaying the same episode. Trajectory2Task takes successful tool-call trajectories and turns them back into realistic user tasks, including ones with ambiguous, changing, or unreasonable requests, producing new training and test cases for tool-calling agents [103]. Trace2Skill instead picks out the useful lesson inside a trajectory and turns it into a reusable skill, so the agent can reuse what it learned without keeping the whole trace around [401]. Both point to the same next step, covered in Section 5.1.2: deciding which of these new tasks or skills are worth building into the model itself.

5.1.2. Parameter-Level Evolution: Compiling Epistemic Routines into Weights

Once useful trajectories exist, the next question is which parts should be compiled into the model. Parameter-level evolution is the durable form of internal evolution: it changes priors, routines, or control policies so that behavior that once required prompting, search, tool demonstrations, or multi-agent orchestration becomes available through the model’s own forward pass. The gain is amortization over future episodes. The danger is that the model may absorb the wrong part of the trajectory, overfit to a particular tool regime, or forget older competence while learning the new behavior. We discuss two routes through which this compilation happens: agentic reinforcement learning trains the policy directly on reward signals extracted from mixed reasoning and acting trajectories, while distillation and skill internalization instead takes an external procedure that already works and folds it into the model through supervised or preference-based training.

On the reinforcement-learning route, the main challenge is credit assignment over mixed reasoning and acting trajectories. One line of work, represented by ToRL and related tool-integrated RL methods, makes intermediate tool, code, or retrieval actions trainable rather than treating the trajectory as an opaque path to a final answer [24,41,107,108,220,410]. A final answer can tell us whether a trajectory succeeded, but it does not reveal which decomposition step, API call, code execution, retrieval, or verification step actually helped; step-level rewards tied to executable side effects, passing tests, successful tool calls, or validated intermediate states give the optimizer a more local signal. A second line focuses on the optimization dynamics induced by long multi-turn interaction: ARPO and AEPO use entropy-aware rollout branching or entropy balancing to keep exploration concentrated at uncertain tool-use decisions without collapsing the policy [402,411]. A third raises the unit of optimization above individual calls: StraTA abstracts recurring trajectory fragments into reusable strategies, AGILE treats memory use, tool use, reflection, and expert consultation as policy actions, and bottom-up policy optimization treats latent sub-policies inside the language model as objects that can be optimized [403,412,413,414]. In ToA terms, RL turns a trajectory into a sequence of candidate operators whose costs and benefits can be trained separately.

On the distillation route, the starting point is often an external procedure that already works but is expensive: a planner–critic loop, a tool-use scaffold, a multi-agent debate, or a carefully prompted reasoning routine. Procedure-level distillation methods, including Chain-of-Agents, structured agent distillation, and step-wise on-policy distillation, aim to preserve the control procedure itself inside a smaller or more integrated model rather than imitating only the final answer [109,404,415]. Tool- and skill-internalization methods make the migration more explicit: TInR, SKILL0, SIRI, HeavySkill, and SkillRL study when tool knowledge, in-context skills, deliberate reasoning routines, or external skill banks should be absorbed into the policy rather than kept as permanent scaffolds [25,26,57,58,416]. Mid-training pipelines such as daVinci-Dev move the same idea earlier in the training stack by making software-engineering agent behavior part of the model’s native training distribution [405].

The key question is not whether a behavior can be trained, but whether it should be compiled. A routine is a good candidate for parameters only when it is recurrent, costly to elicit externally, and stable over the model’s update cycle. Parameter-level evolution is therefore selective compilation, not simply more training. It should reduce future $E_{\mathrm{int}}$, avoid unnecessary $E_{\mathrm{ext}}$, or place the internal and external boundary more accurately. If the compiled behavior is unstable, tool-specific, or poorly credited, the update may lower apparent inference cost while making the agent less calibrated.

5.1.3. Inference-Time Strategy Adaptation: Scheduling Internal Effort

Once experience and parameter-level knowledge have been injected, a lighter path remains: before new data or gradient updates arrive, it changes how the current model spends computation at inference time, further eliciting the capability already injected. We discuss two ways this scheduling happens: scheduling how much to think, which decides how many reasoning chains, branches, or rollouts to spend on a single query, and scheduling when to reach outside the model, which decides when reasoning should stop and retrieval, simulation, or action should take over.

On how much to think, self-consistency samples multiple reasoning chains and aggregates the answers, trading more internal compute for lower variance [153]. Search-based methods make this allocation explicit: Tree of Thoughts branches and backtracks over candidate ideas, organizing reasoning as a searchable tree [187]; LATS further places the language model inside a tree search over candidate thoughts and actions [52], while ParallelMuse uses parallel rollouts, path reuse, and compressed aggregation for deep information-seeking tasks [406]. The common point is that additional thinking is treated as a resource to schedule, not as a fixed prefix applied to every query.

On when to reach outside the model, a second group of methods adjusts the boundary between reasoning and evidence acquisition. Agentic RAG and reasoning-RAG systems interleave decomposition, retrieval, evidence evaluation, and answer synthesis, replacing one-shot retrieval with a policy over when and what to search; Self-RAG lets the model learn to trigger retrieval on demand and critique the retrieved results, turning “whether to gather evidence” into a learnable decision [6,37,278]. Fishing for Answers shows that retrieval itself has a scheduling problem: iterative retrieval may help, but it also consumes context and can drift away from the original query [417]. Dyna-Think adds internal world-model simulation before external action, training agents to imagine likely consequences before acting in the environment [42]. Aware First closes the loop from the other direction by estimating whether the model is already within its competence boundary and suppressing unnecessary reasoning when further thought has little marginal value [407]. In the same spirit, EAPO trains the policy to learn when not to act, curbing tool abuse by withholding calls whose expected information gain does not justify their cost [408].

The boundary with external evolution matters. FlowBench and OctoTools expose planners, workflows, tool cards, executors, and verification modules; these can be viewed either as inference-time strategies or as harness-level scaffolds [50,53]. We classify them as internal evolution only when the updated object is the policy that chooses among thinking, retrieval, tool use, and stopping. When the reusable graph, protocol, or tool interface itself is edited, the update belongs to external evolution in Section 5.2. This keeps the analysis from collapsing the whole agent stack into “reasoning strategy.”

Together, experience-driven data evolution, parameter updates, and inference-time adaptation form a loop. Strategies generate trajectories; trajectories become filtered experience; filtered experience updates the model; and stronger models support richer strategies. The goal is to make useful reasoning cheaper over time, not to make the agent think longer: better priors reduce avoidable uncertainty, better routines amortize repeated work, and better control places the think, act, or stop boundary closer to the true value of information. These model-side improvements only matter, however, if the harness can supply reliable evidence and actions. That is the role of external epistemic evolution.

Key Insight.  Internal evolution converts interaction into model-side competence: generated tasks expose capability gaps, post-training compiles verified trajectories into stable routines, and inference-time adaptation allocates reasoning and action for the current task.

5.2. External Evolution: Maintain the Harness as an Evolving Interface

External epistemic evolution updates the harness rather than the model. Memory determines which past evidence remains available; tools determine which external operations can be invoked; workflows determine how these operations are composed; and environments determine the cost and fidelity of new observations. The aim is to improve the interface between the model and the world, not to surround the model with more components so that evidence is cheaper, actions are more reliable, and updates remain auditable. This continues the externalization view in Section 3.2: learning asks which burdens should be moved out of the model, while evolution asks how those external supports should be maintained as the agent gains experience, encounters new tools, and operates in changing settings.

We organize this subsection around four kinds of evolvable objects in the harness, which build on one another: memory determines which evidence can be reused, tools and skills determine which actions can be taken, the workflow that coordinates them determines how actions are organized and checked, and the environment in which they sit determines the distribution of evidence the whole interface faces. The four following subsections are therefore not an independent component checklist but four facets of the same interface. ❶ Memory covers persistent traces, profiles, and histories, and determines which past evidence is written, represented, consolidated, retrieved, and forgotten (Section 5.2.1). ❷ Tools and skills, ranging from APIs to specialist agents, determine which external operations can be invoked and how they are indexed, routed, and composed into higher-level skills (Section 5.2.2). ❸ Workflows and protocols govern how memory and tools are used, specifying the order, permissions, and verification procedures under which they are orchestrated, distinct from the internal strategy adaptation in Section 5.1.3 because it edits the reusable harness rather than the model’s own policy (Section 5.2.3). ❹ Environment determines from which world, and at what cost and fidelity, evidence is obtained; here we focus on environments as design choices made by the developer or training system, while the reciprocal case where the environment adapts to the agent is discussed in Section 5.3.3 (Section 5.2.4).

ToA criterion: an external update counts as progress only when it makes useful evidence cheaper, more reliable, or better aligned with the agent’s uncertainty, not merely when it enlarges the set of available components.

Table 9 compares how representative methods validate and update external artifacts, including their consolidation and pruning controls.

5.2.1. Memory Architecture and Management

Memory evolution asks which observations should remain outside the model while still shaping future decisions. A useful memory module is a policy over evidence, not an ever-growing transcript: what to write, how to represent it, when to consolidate repeated traces, what to retrieve for the current decision, and when to forget stale or harmful information. We unpack this policy view in two steps, first looking at concrete systems and papers that already implement it in practice, and then at the granularity trade-off that these systems must all confront: how coarse or fine-grained a stored memory should be.

MemGPT schedules memory between a limited context and external storage in an operating-system style, MemoryBank and generative agents maintain long-term memory that can be written, consolidated, and retrieved, and MemOS abstracts memory management into an explicit operating-system layer [10,246,418,419]. Recent memory papers can be read as a progression toward this policy view. They organize agent memory around dynamic context construction, retrieval-augmented stores, episodic and semantic memory, reflective consolidation, and learned memory-management policies [32,44]. Hu et al. [45] make the lifecycle explicit by separating memory forms, functions, and dynamics, so that formation, revision, and retrieval are treated as coupled decisions rather than separate modules. A dual-process cognitive memory system pushes this lifecycle view further, separating fast associative recall from slow deliberative consolidation so that a self-evolving agent reorganizes what it stores as experience accumulates [421]. This matters for ToA because past experience has no value by itself. It becomes useful only when the harness can judge whether a stored trace is relevant to the uncertainty the agent faces now.

The central design choice is granularity. Raw trajectories preserve the local conditions under which a prior success or failure occurred, but they consume context and retrieval budget. Compressed semantic memories are cheaper to access, but may remove the details that made an episode diagnostic. User and task memories create another trade-off: they help the agent adapt to recurring preferences and environments, but can preserve stale assumptions when the user, task, or interface changes. Stepwise experience recall for self-guided function calling illustrates the fine-grained version of the problem: successful tool-use trajectories are stored in an experience pool and retrieved at intermediate steps of future function-calling tasks, so each tool-selection or argument-generation decision can reuse prior evidence without a manually curated demonstration [249]. Self-evolving skill memory extends the same move from raw traces to distilled skills, accumulating and revising reusable reasoning skills across episodes so that recurring task structure is recalled as competence rather than re-derived each time [420].

Memory evolution therefore optimizes a local economy inside $E_{\mathrm{ext}}$. Storing everything makes retrieval expensive and noisy; storing too little forces the agent to repeat search, re-derive facts, or rediscover old failure modes. Consolidation lowers access cost but risks compression error. Forgetting improves freshness and privacy but may destroy rare evidence needed for long-tail tasks. A memory update is valuable only when its expected future benefit exceeds storage, retrieval, compression, and interference costs. Memory management is a calibration problem for the harness, not bookkeeping: deciding which past observations are likely to matter for the current belief state.

Memory also sits on the boundary between externalization and internalization. Experience may first enter as an episodic store, later be summarized into semantic memory, and eventually be compiled into adapters, prefixes, or latent memory slots, as discussed in Section 3.1.1. The temporal question is which parts should remain editable and attributable, and which have become stable, recurrent, and costly enough to move inward. Recent and volatile evidence should remain outside. Stable preferences, durable task regularities, and frequently reused summaries are stronger candidates for internalization.

5.2.2. Tool and Skill Library Evolution

Tool evolution changes the external action space. Memory determines what the agent can recall from the past; tools determine what it can obtain or do next. A tool library is a changing interface between the model and the world, not just a list of APIs appended to a prompt, defining which external operations are available, how they are represented, and when they should be invoked. We trace this evolution through three stages: expanding and re-representing the toolset to grow the library and make it searchable, orchestrating a large toolset to address the routing problem that a bigger library creates, and structuring tools into skills to consolidate the repertoire itself rather than only growing or routing it.

Toolformer, Gorilla, and ToolLLM establish tool use as a trainable interface to real APIs rather than a hand-written prompt pattern [46,146,166]. Subsequent work shifts the bottleneck from whether a model can call a tool to how a large and changing library is represented: ToolGen, ToolkenGPT, and Chain-of-Tools make retrieval, embeddings, or semantic descriptions part of the tool-selection process, while ToolScope extends the same pressure to multimodal tool use [8,48,51,110]. Beyond re-representing existing tools, task-driven tool creation lets a self-evolving agent synthesize new tools when the current library cannot cover a task, as benchmarked by Tool-Genesis [423]. The common move is to make the external action space mutable, searchable, and learnable.

Once the library grows, the bottleneck shifts from access to orchestration. A larger toolset reduces $E_{\mathrm{ext}}$ only if the agent can find the right operator without spending more effort than the operator saves. ToolOrchestra trains lightweight orchestrators to coordinate models and tools under outcome, efficiency, and preference-aware rewards [9]. Budget-aware tool-use methods make the cost explicit by treating tool calls as a limited test-time resource [75]. When the toolspace itself becomes very large, reinforcement finetuning can scale tool-selection competence without inflating the context window, training the policy to navigate large toolspaces directly [429], while skill-retrieval benchmarks such as SkillRet isolate the retrieval bottleneck of finding the right skill among thousands of candidates [430]. SkillOrchestra raises the same routing problem to the level of specialist agents, treating them as callable skill providers and learning transferable skill abstractions [111]. AnyTool uses self-reflective, hierarchical agents to retrieve and call among thousands of APIs, illustrating that navigating a vast action space is itself a capability that must be organized [431]. In all three cases, tool choice becomes a control problem: the agent must ask not only whether a tool exists, but whether its expected information gain justifies the cost of selecting, parameterizing, waiting for, and verifying it.

A complementary line makes the repertoire more structured rather than merely larger. Divide-then-aggregate reduces redundant calls by decomposing tool-use tasks, sharing intermediate results, and aggregating outputs efficiently [113]. Agent-as-tool hierarchies expose sub-agents as callable modules, while cooperative tool-learning agents let multiple agents acquire and refine tool competence jointly [112,235]. Large-scale tool-agentic trajectory synthesis supplies the supervision needed to grow and stress-test these libraries across diverse multi-turn settings [49]. This is where tools begin to look like skills: repeated external action sequences are abstracted into higher-level operators, redundant tools are merged or pruned, and routing policies expose only the relevant subset at decision time. Dynamic skill lifecycle management makes this explicit by retaining, retiring, and expanding an active skill set over time [424], while SkillPyramid consolidates primitive skills into a hierarchy so that higher-level operators are composed from verified lower-level ones [425]. Tool evolution is progress only when the reduction in primitive external actions outweighs the added cost of navigating the enlarged action space. When the stable procedure around a tool is more valuable than the fresh output it returns, the procedure becomes a candidate for skill internalization, as discussed in Section 3.1.3.

5.2.3. Workflow and Planning Evolution

Workflow evolution edits the control surface of the harness. Memory decides what evidence persists; tools decide which operators exist; workflows decide how these resources are composed over time. A workflow specifies when to decompose a task, retrieve context, call a tool, branch into alternatives, ask another agent, verify an intermediate result, recover from failure, or stop. This is different from the internal strategy adaptation in Section 5.1.3. The model may learn a better think, act, or stop policy, but the harness may also evolve the reusable graph or protocol that implements such choices outside the model. Workflow evolution plays two roles: governed computation turns free-form interaction into inspectable, auditable procedure, and search and allocation decides where the harness should spend each additional action once such a procedure exists.

Protocols and tool cards specify schemas, permissions, side effects, failure reports, and audit trails. Workflow graphs specify the order and conditions under which these interfaces are used. FlowBench formalizes workflow knowledge formats and evaluates how they guide planning across domains [53]. OctoTools implements the same idea through standardized tool cards combined with planning, execution, and verification modules for complex reasoning tasks [50]. The important shift is that planning knowledge is no longer only a prompt instruction. It becomes a reusable artifact that can be inspected, modified, benchmarked, and transferred.

Once a workflow can branch, retrieve, verify, and delegate, the harness must decide where to spend each additional action. Tree-search and parallel-thinking systems, represented by LATS and ParallelMuse, instantiate this as explicit branching, pruning, path reuse, and aggregation over candidate reasoning or action paths [52,406]. A related line places the allocation decision at specific boundaries: Self-DC decides when external retrieval is unnecessary, while Dyna-Think inserts world-model simulation before external action [42,432]. At the workflow-design level, AFlow, GPTSwarm, ADAS, DSPy, LEMON, and AIP treat the control structure itself as an optimizable or governable artifact, using graph search, meta-agent search, declarative compilation, counterfactual RL, or directed skill graphs to revise the scaffold rather than merely prompt the model harder [4,258,259,392,393,426,427,433]. The broader movement is from hand-crafted prompt recipes toward learned, searched, and modularized control structures.

The benefit of workflow evolution is that control becomes editable and auditable. The cost is that every branch, verifier, role, and protocol step consumes tokens, latency, coordination effort, and sometimes expands the attack surface of the system [99]. The ToA criterion is not whether a workflow is more elaborate, but whether it reduces avoidable uncertainty, prevents expensive mistakes, or improves recovery enough to justify its overhead. A brittle workflow can increase $E_{\mathrm{ext}}$ by forcing unnecessary calls. An over-general workflow can increase $E_{\mathrm{int}}$ by making the model reason over too many possible next steps.

Workflows also create a path back to internal evolution. Multi-agent and multi-module procedures often begin externally because explicit graphs, protocols, and role decompositions are easier to inspect and repair. When the same procedure becomes stable and recurrent, it may be distilled into a smaller stack or even a single model. Chain-of-Agents, AgentArk, and latent-agents post-training all attempt to compress multi-agent or debate-like workflows into more compact agents while preserving some of the original benefits [59,60,404]. Agent-as-tool hierarchies and divide-then-aggregate schedulers apply the same logic to nested tools and sub-agents [112,113]. The boundary condition mirrors memory and tools: keep the workflow external when auditability, updateability, and task-specific variation matter; move it inward only when recurrence and stability justify losing the explicit control surface.

5.2.4. Environment Design and Affordance Scaling

Environment evolution changes the conditions under which evidence is generated. Memory, tools, and workflows operate inside a world; environments define the tasks, rewards, observations, reset conditions, simulators, interaction interfaces, and safety boundaries of that world. The environment is therefore not merely an evaluation backdrop. It determines what the agent can experience, how quickly feedback arrives, how costly exploration is, and how trustworthy each observation is for future action. Environments such as WebArena, OSWorld, SWE-bench, and GAIA delimit the experience available to the agent and its cost precisely by fixing tasks, observation interfaces, and success criteria [148,149,434,435]. We discuss two forms this evolution takes: curriculum and feedback design reshapes an existing environment so that it exposes more informative tasks and denser rewards, and surrogate environment construction replaces costly live interaction with a cheaper learned substrate altogether.

Work on environment scaling argues that diverse and well-instrumented environments are a central source of agent learning, because they determine the experience distribution available to the agent [114]. Environment tuning makes this operational: changing task distributions, progress rewards, observability, and augmentation can sometimes improve downstream behavior more than additional fine-tuning on static trajectories [428]. Simulated reasoning environments provide a controlled version of the same idea, with adjustable difficulty and dense feedback that increase the amount of useful signal per episode [115]. In this sense, the environment decides which uncertainties the agent is forced to confront next.

Instead of paying the full cost of live interaction, the harness can build a cheaper substrate that approximates the relevant dynamics. SWE-World replaces costly software-execution environments with a learned surrogate that predicts intermediate outcomes and final test feedback, allowing software-engineering agents to train without rebuilding full containerized environments for every episode [116]. WebWorld builds a large-scale open-web simulator from web interactions, enabling long-horizon web-agent training with lower latency, fewer rate limits, and reduced safety risk [43]. DynaWeb then performs model-based reinforcement learning inside a learned web simulator before acting on the live web [117]. Multimodal agentic papers show a parallel pressure in vision-language settings, where perception interfaces, grounding constraints, and tool-augmented interaction determine whether text-trained agents can act reliably outside text-only tasks [436].

Environment evolution differs from memory, tools, and workflows because it changes the price and fidelity of evidence itself. Better memory reduces repeated search; better tools compress primitive action sequences; better workflows reduce coordination overhead. Better environments can reduce $E_{\mathrm{ext}}$ across all of them by making observations cheaper, feedback denser, exploration safer, and failures easier to recover from. This is why environment design can appear more powerful than local harness edits: it shifts the distribution of future episodes, rather than only improving a single decision pipeline.

The same leverage creates a fidelity risk. A simulator may be cheap but misleading. A curriculum may overfit to the generator’s artifacts. A shaped reward may teach policies that exploit the training substrate rather than solve the intended task. Environment evolution improves long-run $\eta$ only when the new substrate is both cheaper and faithful enough to deployment. This also places environment evolution on the internalization boundary discussed in Section 3.1.4: stable and recurrent dynamics may be compiled into a model-side world representation, while volatile, safety-critical, or attribution-sensitive dynamics should remain external and inspectable. The reciprocal case, where the environment adapts to the agent rather than serving as a unilateral design lever, is the topic of Section 5.3.3.

Taken together, memory lifecycle management, tool and skill library evolution, workflow evolution, and environment design form the harness-side counterpart to internal evolution. Memory changes what past evidence remains available; tools change which operations can be invoked; workflows change how actions are organized; and environments change the substrate from which evidence is obtained. These mechanisms primarily reshape $E_{\mathrm{ext}}$, but they also feed back into $E_{\mathrm{int}}$: a richer harness gives the model more options to select from and verify. External evolution is the long-run optimization of the interface between model and world, not component accumulation. Each added support should make useful belief updates cheaper or more reliable, not merely expand the set of possible actions. When these improvements accumulate across multiple agents or become coupled with an evolving environment, the unit of analysis expands beyond the individual agent. This is the shift addressed by ecological epistemic evolution.

Key Insight.  External evolution lowers the cost of interacting with the world by consolidating memory, organizing tools into reusable skills, optimizing workflows, and designing environments with cheaper and more reliable feedback.

5.3. Ecological Epistemic Evolution: Co-Adaptation

The previous subsections separated the supports that evolve: model-side priors, routines, and control in Section 5.1, and harness-side memory, tools, workflows, and environments in Section 5.2. Ecological evolution changes the level of analysis. The question is no longer only which component is updated, but which system supplies the evidence, feedback, disagreement, and selection pressure that make updates reliable. The unit of adaptation expands from a component inside one agent, to the agent’s lifetime loop, to a population of interacting agents, and finally to an agent–environment system whose task distribution also changes over time. Under ToA, this matters because the cost of reasoning, acting, verification, memory, simulation, and curriculum generation can be distributed across the whole ecology rather than paid by a single model.

We organize this subsection from the smallest unit of adaptation to the largest. ❶ Individual self-evolution views a single agent across its whole lifetime rather than at a single episode: it acts, observes the result, diagnoses success or failure, updates one or more supports, and returns to the environment with a changed configuration, so that trajectories may become data, failures may become memories, and repeated actions may become skills (Section 5.3.1). ❷ Multi-agent co-evolution rewires roles, communication, and shared artifacts across a population of agents, providing structures a single agent cannot easily create for itself, such as role specialization, division of labor, shared memory, peer critique, and distillation, so that it becomes useful when it supplies independent evidence or disagreement that can stabilize individual self-improvement (Section 5.3.2). ❸ Agent–environment co-evolution closes the outermost loop, in which the environment is not only where actions happen but also an adaptive source of tasks, feedback, observability, and constraints, and the task distribution itself moves as the policy improves (Section 5.3.3). The population supplies an individual with disagreement and evidence it could not produce alone, and the environment sets the price and reliability of evidence for the population. Table 10 compares these loops by their evidence sources, validation, co-adaptation, updated unit, and dominant cost and risk.

5.3.1. Individual Self-Evolution

Single-agent self-evolution is the base ecological loop. The agent acts, receives observations or feedback, diagnoses what happened, updates one or more supports, and acts again. Recent papers describe such loops across model updates, data regeneration, memory growth, skill acquisition, and workflow revision [2,36]. We group existing systems into two families: those that update real supports from real interaction, revising the model, memory, skills, tools, or even the agent’s own code based on what actually happened, and those that update from imagined interaction, adding a second loop in which the agent first learns to simulate consequences and then learns from those simulations.

Existing systems can be grouped by the support they revise. A first group studies data- and curriculum-light self-improvement, where carefully selected demonstrations, early coarse interactions, or self-generated tasks bootstrap agency before dense rewards are available [39,40,437,438]. A second group keeps the model fixed but evolves external supports: Steve-Evolving revises experience anchors and control loops in open-world embodied settings, Voyager accumulates executable skills, MetaForge forges tools on demand, and Socratic-SWE distills reusable software-engineering skills from execution traces [253,422,439,445]. A third group pushes self-evolution into the agent implementation itself, as in the Darwin Gödel Machine and MOSS, where improvement is expressed as source-level rewriting rather than only memory or skill growth [440,441]. The common pattern is that the agent’s own traces become editable support for later episodes, whether the edited object is data, memory, skills, tools, or code.

Instead of learning only from real interaction, the agent learns to simulate consequences and uses those simulations to guide future action. Reinforcement World Model Learning trains action-conditioned world models by aligning simulated next states with realized next states, providing a self-supervised signal about environment dynamics [119]. Self-improving latent world models treat hidden dynamics as a planning substrate that can be refined over time [120]. Self-play approaches that internalize world models from generated interactions show how simulated or corpus-grounded experience can be folded back into the agent’s policy [169]. These methods are ecological because they let the agent buy imagined experience before spending external actions in the real environment.

The key question is whether the loop produces new evidence or merely recycles the agent’s current errors. Benchmarks of self-evolving agents therefore track whether successive iterations improve held-out performance rather than only self-generated objectives [2,104]. Minimal-supervision and early-experience settings further show how fragile the loop can be when rewards or verifiers are weak [39,40]. In ToA terms, self-evolution amortizes $E_{\mathrm{int}}+E_{\mathrm{ext}}$ across the agent’s lifetime only when each iteration supplies information whose value exceeds its noise and bias. Productive self-evolution usually needs at least one stabilizer: an external verifier, a curriculum that controls difficulty, or disagreement from another agent or environment model.

5.3.2. Multi-Agent Co-Evolution

Multi-agent co-evolution can provide the stabilizer that isolated self-evolution often lacks. When agents evolve together, they can generate tasks for one another, critique partial solutions, specialize into complementary roles, share experience, and distill group-level discoveries into individual policies. Cooperative multi-agent frameworks such as AutoGen, MetaGPT, CAMEL, and ChatDev already show how role division and communication can solve tasks that exceed a single policy [12,13,35,123,390,391]. Ecological evolution adds time: the roles, messages, and shared artifacts are themselves updated across rounds. We group this line of work into two families: those that treat the group itself as the unit of improvement, and those that focus on how the population is formed and governed.

One family shares discoveries across branches or organizes Challenger–Planner–Solver–Critic loops for task generation, planning, solving, verification, and filtering [106,124]. Another family studies how populations are formed and governed: Role-Agent bootstraps complementary roles from a single model, decentralized-memory systems use a shared experience store instead of a central controller, and hierarchical systems such as Multi² distribute decisions across layers to curb goal drift [244,442,443]. The common object of evolution is no longer one agent’s state, but the population structure that determines which disagreements and discoveries become reusable evidence.

Benchmarks show why this population-level view is needed. CollabOvercooked evaluates LLM agents in collaborative interactive environments and emphasizes process-level collaboration metrics beyond task completion [125]. MultiAgentBench covers both collaboration and competition, making communication topology, milestones, and coordination quality part of the evaluation [126]. These results make an important point: adding more agents is not automatically helpful. Agents that perform well alone may fail when communication, timing, or role assumptions become part of the task. A population can reduce uncertainty through distributed search, but it can also introduce ambiguity, inconsistent beliefs, and coordination overhead.

The same interaction structure creates risks that do not appear in isolated agents. Colosseum studies collusion in cooperative multi-agent systems, where agents may form coalitions and pursue secondary goals that harm the intended objective [127]. The collaboration gap shows that strong individual performance does not necessarily translate into strong group performance [398]. Red-teaming work shows that communication channels themselves can become attack surfaces, because malicious or compromised peers can inject messages that steer the shared context and distort the joint loop [446]. These failures are ecological: they arise from interaction structure, not from any single model in isolation.

The ToA accounting at the population level is therefore non-trivial. A multi-agent system may raise task success while also increasing total reasoning, communication, and tool-use cost. If the target is latency or spend per query, the better update may be distillation: Chain-of-Agents, AgentArk, and latent-agent post-training aim to compress multi-agent intelligence into a smaller or more integrated agent stack [59,60,404]. If the target is coverage over tasks that require simultaneous specialization, the population may remain the right unit. The central question is whether each additional message, role, or peer trajectory buys enough uncertainty reduction to justify its marginal cost.

5.3.3. Agent-Environment Co-Evolution and World Models

Agent–environment co-evolution is the outermost loop. In Section 5.2, environments were treated as harness-side supports that developers can design: better simulators, richer feedback, cheaper observations, or safer exploration. Here the adaptation is reciprocal. The environment changes in response to the agent’s current competence, and the agent changes in response to the resulting task distribution. We discuss this reciprocal loop in two parts: adaptive environments that change directly in response to the agent’s competence, and world-model co-training, which makes the same reciprocal structure explicit by letting the agent learn and use a model of the environment itself.

Adaptive environments can increase difficulty, expose new affordances, adjust observability, or reshape rewards as the policy improves [114,428]. Simulated reasoning environments provide controllable difficulty and dense feedback for reasoning models [115]. Benchmarks such as GAIA2 make non-stationarity and asynchrony part of the evaluation target rather than nuisance variation [122].

On world-model co-training, one line learns predictive dynamics from environment transitions and uses the improved model for later action selection or planning, as in reinforcement world-model learning and self-improving latent world models [119,120]. A second turns model construction itself into an agentic process: Agent2World generates and validates symbolic world models through tool-augmented multi-agent interaction, while WorldMind aligns agentic world models by synthesizing process experience from prediction errors and goal experience from successful trajectories [121,128]. In these systems, the world model works as a moving interface between the agent’s predictions and the environment’s dynamics, rather than a fixed evaluator or a passive store. A third scales the environment generator itself: software, web, and text-based domains all instantiate the same pattern, where learned surrogates or simulators replace expensive live interaction and the resulting generator becomes part of training rather than a fixed benchmark. SWE-World predicts software-engineering outcomes, WebWorld and DynaWeb support web-agent training and model-based RL inside learned web simulators, bilevel world-model synthesis studies outer loops that generate environments for inner-loop policy training, and textual self-play systems such as SPICE treat a corpus as an adaptive task-generation environment [43,105,116,117,169,444]. These systems replace a fixed task set with a generator whose distribution is itself updated.

Because both sides of the loop move, agent–environment co-evolution is powerful but unstable. Test-time adaptation studies how policies can update online as conditions change [100]. Budget-constrained agentic training emphasizes that adaptation must occur under finite interaction and compute budgets, not under unlimited exploration [76]. The ToA interpretation is that the environment sets the price and reliability of evidence. A good co-evolutionary loop lowers $E_{\mathrm{ext}}$ by making informative experience cheaper, and lowers future $E_{\mathrm{int}}$ by shaping better priors and world models. A bad loop lowers apparent cost only inside a distorted substrate, producing policies that overfit to simulator artifacts, generated curricula, or reward shortcuts.

Ecological evolution therefore expands the accounting unit from the model to the harness, from the harness to the agent, from the agent to the population, and from the population to the environment. Single-agent self-evolution is easiest to control but limited by the agent’s own evidence and verifier quality. Multi-agent co-evolution raises the ceiling through specialization, disagreement, and shared experience, but introduces communication cost, collusion, and coordination failure. Agent–environment co-evolution offers the richest source of long-run improvement, but also the hardest evaluation problem because the task distribution itself changes over time. Across all three cases, the ToA question is whether repeated adaptation lowers $E_{\mathrm{int}}+E_{\mathrm{ext}}$ for the tasks that matter while preserving the calibration, abstention, evidence weighting, and boundary placement discussed in Section 4 [2,31,36]. Evolution that raises raw success while eroding these abilities is not genuine epistemic progress; it is optimization of a narrower metric. The next section turns this criterion into the evaluation problem for agentic systems.

Key Insight.  Ecological evolution closes progressively wider feedback loops: individuals learn from verified experience, populations exchange critiques and reusable skills, and adaptive environments generate tasks and feedback matched to current competence.

6. Evaluation: Measuring What Matters for Agents

6.1. Rethinking Agent Metrics: Beyond Task Success Rate

Task success rate remains the most intuitive and widely adopted baseline for agent evaluation: it directly measures whether an agent accomplished what it was asked to do. Yet as a standalone metric it is fundamentally incomplete. A binary completion signal cannot distinguish an agent that reached a correct answer through careful reasoning from one that stumbled upon the same answer via excessive tool calls and hallucinated intermediate steps. Nor can it reveal whether the process was safe, cost-effective, or robust to distributional shift. For single-turn language models these blind spots are tolerable, because the system produces one response and stops. However, agents act, plan, observe, and adapt over extended trajectories, and it is the quality of those trajectories, not merely their terminal states, that determines real-world utility.

Viewed through the ToA lens, an agent continuously allocates epistemic effort between internal reasoning and external acting. Tools do not eliminate cognitive difficulty; they relocate it from the model’s parameters to the environment. This perspective exposes two failure modes that task success rate cannot detect. Overthinking occurs when the agent reasons past its knowledge boundary, generating confident but unsupported conclusions instead of consulting an external source [67,447]. Overacting occurs when the agent reflexively invokes tools for information it could derive internally, inflating cost and, over time, inhibiting the growth of the model’s own competence [30,73]. Both patterns may coexist with a perfect task success score, which makes the score misleading as a quality indicator. The guiding question of this section is therefore: how should an agent be measured so that its score reflects not merely whether it succeeded, but how intelligently it converted epistemic effort into resolved uncertainty?

Under the ToA framework, a metric is informative only when it can separate genuine uncertainty reduction from effort that changes cost without changing belief. We organize the metrics along two complementary families, and the running theme across both is who does the grading and against what: some metrics presuppose a ground-truth label and a programmatic verifier (the setting that verifiable-reward training, RLVR, depends on), while others, such as open-ended generation, or long-horizon workflows, have no reference and must fall back on model judges or label-free proxies. ❶ Outcome evaluation (Section 6.1.1) measures episode-level aggregate quantities: whether the final answer correctly fused internal reasoning with external evidence, and what the episode cost in total to produce it. ❷ Process evaluation (Section 6.1.2) provides trajectory- and policy-level diagnostics: whether the intermediate states it passed through were earned, whether the side conditions it had to honor held throughout, and whether the policy that produced the trajectory is calibrated, safe, and robust, the last two read across runs and perturbations rather than off a single trajectory. The two families catch different kinds of failure, and Section 6.1.3 makes their connections explicit and discusses what happens when either is read alone.

Figure 9. Rethinking agent metrics beyond task success rate: a two-family taxonomy and their joint reading. ❶ Outcome evaluation grades the endpoint of a trajectory along two inseparable axes: correctness (from binary success through execution-based and graded checks to judge-based verdicts) and cost and efficiency (a separate internal-reasoning and external-acting ledger). ❷ Process evaluation opens the trajectory itself, moving from local progress and constraints, through the agent’s boundary calibration and tool-use decisions, to the safety and robustness of its policy and the consistency of the whole chain. ❸ Integrated evaluation reads the two families together: accuracy trades off against cost, boundary calibration is the common cause behind the process family, and process metrics explain the residual that outcome metrics leave. The two do not reduce to a single score but are unified by the epistemic intelligence ratio (resolved uncertainty over total effort), which alone assigns overthinking and overacting the low score that task success rate hides.

Figure 9. Rethinking agent metrics beyond task success rate: a two-family taxonomy and their joint reading. ❶ Outcome evaluation grades the endpoint of a trajectory along two inseparable axes: correctness (from binary success through execution-based and graded checks to judge-based verdicts) and cost and efficiency (a separate internal-reasoning and external-acting ledger). ❷ Process evaluation opens the trajectory itself, moving from local progress and constraints, through the agent’s boundary calibration and tool-use decisions, to the safety and robustness of its policy and the consistency of the whole chain. ❸ Integrated evaluation reads the two families together: accuracy trades off against cost, boundary calibration is the common cause behind the process family, and process metrics explain the residual that outcome metrics leave. The two do not reduce to a single score but are unified by the epistemic intelligence ratio (resolved uncertainty over total effort), which alone assigns overthinking and overacting the low score that task success rate hides.

6.1.1. Outcome Evaluation

Outcome evaluation sits at the output end of the internal–external axis, but it asks two questions about the episode as a whole rather than about a single instant. Did the agent’s final answer correctly combine its internal reasoning with the external evidence it gathered? And what did producing that answer cost in total, summed over the whole episode? Two output-level failure modes can appear in the answer itself. Over-internalization is overthinking observed at the output, where the agent commits to a claim that the available evidence does not support; under-internalization runs the other way, passing a tool’s output through to the final answer without checking it against what the agent already knows, so an unreliable retrieval flows into the answer unchecked. The metrics below fall into two groups: ❶ correctness metrics score the terminal utility of the answer, and ❷ cost metrics score the aggregate effort, internal reasoning plus external acting, spent to reach it.

Correctness metrics.

Correctness measures differ in how much information they retain about a run. Many knowledge and reasoning benchmarks, including GAIA [435], MMLU-Pro [141], GPQA [142], FrontierMath [144], and the contamination-limited LiveBench [145], ultimately reduce each example to a correct or incorrect outcome and aggregate these decisions into an accuracy or success rate. This is easy to interpret, but it cannot distinguish a careful trajectory from one that arrived at the answer by chance. The problem is more pronounced for agents because repeated runs may diverge even under nominally deterministic settings. Reporting only a single run can therefore give a misleading picture of reliability. Repeated-run statistics provide a partial remedy: pass@k records whether at least one of k attempts succeeds, whereas pass^k asks whether all of them succeed [448]. Tasks that admit meaningful intermediate credit can instead use token-level F1 or semantic similarity, while exact match remains appropriate when only a fully matching reference answer counts. For open-ended outputs, rubrics and requirement checklists make the scoring criteria explicit, as in TravelPlanner [449] and AGENTIF [89]. These choices determine the granularity of the correctness signal; how that signal is assigned is a separate question.

The second axis, orthogonal to granularity, is who assigns the score, and it is dictated by whether a ground-truth label exists. Where a task admits a programmatic verifier (a test suite, an exact answer, a satisfiable constraint set), correctness is cheap, reproducible, and directly reusable as a training signal, which is why verifiable-reward (RLVR) settings lean almost exclusively on it: SWE-bench [131] executes a hidden test suite over a generated patch, and data-analysis suites such as InfiAgent-DABench [132] and IDA-Bench [133] run the produced code on held-out inputs. Such execution-based verification usually returns a binary pass/fail, which is exactly why it belongs on the verifier axis and not on the granularity one. Where no verifier exists, evaluation defers to a model judge: single-model LLM-as-a-Judge is scalable but suffers from score inconsistency, position and verbosity bias, and knowledge-cutoff drift [319,320], and the agent-as-a-judge variant mitigates these by composing specialized sub-agents under a reflective coordinator, exemplified by RATE for non-literal MT evaluation [450] and by general judging frameworks [451,452,453]. Because a strong judge is itself costly, cost-aware judge routing [331] spends judge effort only where the verdict is uncertain, echoing the effort-allocation logic the agent under test must itself follow. When even a judge is unreliable, label-free proxies such as self-consistency and cross-sample agreement estimate correctness from answer stability rather than from a reference. The two axes are orthogonal: the same rubric can be scored by execution, by a reference, or by a judge, and execution can return either a bare bit or a graded value. A benchmark therefore picks a cell, a granularity paired with a verifier, and the right cell is fixed by label availability and by the downstream decision the score feeds, not by any single ordering of discrimination.

Cost and efficiency metrics.

Producing a correct answer is only half of an outcome report; the other half is what producing it cost. Following the effort decomposition of Section 2.4, outcome evaluation splits aggregate cost along the internal–external axis into internal reasoning cost and external acting cost. Internal-cost metrics count reasoning and output tokens, thinking-budget expenditure, and chain-of-thought latency; external-cost metrics count tool and API calls, monetary fees, network wall-clock latency, and the irreversible side effects an action leaves on the environment. These units are not interchangeable: tokens are cheap and reversible, whereas external actions are dear and sometimes cannot be undone, so a faithful report keeps the two ledgers separate rather than collapsing them into a single dollar figure [454].

This decomposition turns the two motivating failure modes into cost signatures one can actually see. Overacting shows up as high external cost (many tool calls, large API spend) without a matching drop in internal cost, the agent pushing out work its parameters could have done; the “tool-use tax” measures this overhead [263]. Overthinking shows up as inflated internal cost (long chains of thought, repeated reasoning loops) without a matching gain in accuracy [28,216,447]. Either can sit behind a high task-success rate, which is why cost has to be reported next to correctness and not after it.

Because correctness and cost are incommensurable, no single number captures their trade-off, and the literature’s response has been to fold the two into a joint object rather than average them. The variants differ in the shape they impose: an economic score that prices a correct answer in dollars so that rankings cannot be inflated by burning compute [71], an explicit accuracy–cost Pareto frontier that refuses to collapse the two axes and instead makes dominated systems visible as such [454], or the epistemic intelligence ratio of Section 2.4, which divides useful belief change by total effort and so penalizes effort that buys no accuracy on either side of the axis. What ties them together is that cost is not a fixed property of a system but a pressure applied at some chosen point in its life cycle, and where that pressure is applied is itself a design decision. The same parsimony can be imposed after the fact by an evaluation-time score, built into the policy by training-time penalties on redundant calls [73,353], or enforced online by controllers that re-route or re-plan as costs change [9,77,329,330,340]. Seeing these as one continuum rather than three separate tricks is what makes a cost figure interpretable: an agent that is cheap at run time may have been made cheap by expensive training, and the ledger has to say which.

Outcome evaluation, however, still looks at the agent only in aggregate. It does not inspect the sequence of think-or-act decisions the agent went through to reach its answer. Process evaluation is what fills that gap.

6.1.2. Process Evaluation

Process evaluation turns from the aggregate outcome to the trajectory and the policy behind it. It asks whether the intermediate states the agent passed through were earned, whether the side conditions it had to honor held all the way through, and whether the decision policy is sound enough to be trusted unsupervised, a judgment that for calibration and robustness is read across runs and perturbations rather than off any single trajectory. Where an outcome score collapses an episode to a scalar, process metrics open the trajectory black box and grade the decisions inside it. We group them into four families, ordered from the most local (individual states and steps) to the most global (the trajectory as a whole): ❶ progress and constraint satisfaction, ❷ boundary calibration and tool-use decisions, ❸ safety and robustness, and ❹ trajectory consistency.

ToA criterion: a process metric earns its place only when it reads something outcome metrics cannot: the agent’s estimate of its own knowledge boundary, the necessity of a step, an irreversible side effect, or a stability gap. Otherwise it is only re-scoring the final answer under a new name.

Progress and constraint satisfaction.

The most local process metrics grade what a trajectory did on the way to its end: whether it made earned progress, and whether it honored the conditions it was bound by. Progress metrics start from a simple observation. A long-horizon task is not one pass/fail trial but a chain of them, so collapsing it to a terminal bit throws away most of the signal and rewards a lucky shortcut over patient work. Milestone and partial-credit scoring put that signal back by crediting completed sub-goals [130,134,135], and taking the same idea down to the individual step exposes something a final score hides: a weak agent’s high ratio of “correct” steps often just means it stopped early, before it had a chance to err [455]. Constraint metrics cover the other axis. Side conditions such as formatting, resource limits, safety invariants, and instruction adherence apply to the whole trajectory rather than to any single step, so one violation anywhere fails them, and benchmarks report a constraint-pass rate separately from success [89,449]. Read together, the two make a point neither makes alone: progress and compliance are both orthogonal to success, so an agent can clear every milestone while breaking a global constraint, or keep every constraint while failing the task. That is why a deployment with hard side conditions cannot be signed off on task success alone. The recurring cost of this family is reference specification. Partial credit and constraint sets both assume someone has written down the legitimate sub-goals and conditions, so on tasks with many valid paths an over-specified reference punishes the creative-but-correct trajectories it should reward.

Boundary calibration and tool-use decisions.

This family is the epistemic core of process evaluation, and it has two layers: whether the agent knows where its own knowledge boundary lies, and whether its behavior is actually driven by that knowledge. The first layer is calibration, the agreement between the confidence an agent reports and the accuracy it achieves. Standard measures such as expected calibration error carry over from probabilistic forecasting, and for agents they specialize into boundary self-awareness: does stated confidence rise on exactly the tasks the agent can in fact solve on its own [66,298,303,315]? A stricter competence is abstention, saying “I don’t know” at the right time, which is scored on its own because calibrated confidence is useless unless the agent can turn low confidence into declining to answer [69,140,279]. The second layer is tool-use decision quality, where the think-versus-act choice becomes measurable: the accuracy of the call-or-not decision, the accuracy of tool and argument choice once a call is warranted, and the separate rates of over- and under-calling that one accuracy number hides [30,167,280]. Measured this way, agents turn out to call far more often than they need to [323,324,326], and the excess is not ignorance. Agents often recognize that a task is solvable on their own and still reach for a tool, a “knowing–doing gap” [281,322]. Probing the hidden state makes this concrete: tool necessity is linearly decodable from a model’s internal representation far more reliably than from its own stated reasoning, so the boundary signal is there but does not control the action [264]. Keeping the two layers together makes the connection plain. Calibration asks whether the agent’s estimate of its boundary is accurate; tool-use quality asks whether it acts on that estimate; and both read the same underlying quantity, the solvability judgment of Section 2.3. A miscalibrated boundary is then the shared cause behind both failure modes: the over-confident agent over-internalizes and overthinks, the under-confident one over-externalizes and overacts.

Safety and robustness.

The earlier families ask whether the agent decides well; this one asks whether the resulting policy can be trusted once deployed, meaning whether its actions are tolerable and whether its behavior holds up under stress. Safety turns on an asymmetry that the internal–external axis makes explicit. Internal missteps are cheap and reversible, but external actions such as a deletion, a transaction, or a sent message may not be, so risk sits on the external side of the axis and grows with tool access rather than with task difficulty. Evaluation therefore stops scoring answers and starts eliciting and counting harmful actions, by surfacing them in instrumented sandboxes [136], measuring harm rates at scale [82,84], or flagging actions that drift off the stated goal [374,375]. Robustness looks past a single action to the policy behind it, and its metric is a difference rather than a level: the gap between clean and perturbed runs under adversarial observations, noisy tool outputs, reordered inputs, or reworded instructions [98,99,122,384,446]. Two things connect the pair. First, both are trade-offs, not boxes to check after the fact. Control Tax measures the accuracy an agent gives up to stay under control [378], and a robustness gap can flip a ranking, since an agent that scores lower but degrades gracefully is often more deployable than one that peaks higher and then collapses under mild shift. Second, this family is where the single-agent picture runs out. In multi-agent and socially situated settings the perturbation is no longer environmental noise but the strategies of other agents the policy cannot inspect, and norms that shift mid-episode [125,126,138,139], so staying reliable there means extending boundary management from the self to other minds, a direction current metrics only begin to probe.

Trajectory consistency.

The last and most global family steps back from any single decision and asks whether the trajectory holds together, that is, whether the actions taken actually follow from the plan the agent stated and the goal it was given. Goal–Plan–Action alignment makes this concrete: it splits a trajectory into declared goals, planned sub-steps, and executed actions and scores the consistency between neighboring layers [456]. That exposes a failure the earlier families miss, where an agent succeeds while quietly rewriting its plan at every step, so its stated reasoning is a story told after the fact rather than the cause of what it did. Even the precondition for these judgments, telling where one action ends and the next begins, is not free and has to be measured on its own [282]. The point is that a trajectory can be right at every local step and still incoherent overall, which is why consistency is a natural place to close: it checks the account an agent gives of its own behavior, the one thing the other families take on trust.

Read together, these families move evaluation from a capability question (can the agent solve the task?) to a deployment-readiness question (can it be trusted to do so unsupervised?). They also nest from local to global: progress and constraints check what the trajectory did step by step, calibration and tool-use check whether its epistemic decisions were sound, safety and robustness check whether the resulting policy is tolerable and stable, and consistency checks whether the whole chain holds together. Process evaluation complements outcome evaluation by making sure a correct, cost-effective answer was reached through a reliable trajectory. The two, however, do not add up to a single deployable score, which sets up the discussion next.

6.1.3. Toward Integrated Evaluation

The two families of metrics are complementary, but they do not add up to a single deployable score, and seeing why exposes the connections among them. Their units do not line up: correctness on a bounded zero-to-one scale, tokens and calls as counts, dollars and latency as rates, irreversibility as a categorical hazard. Several process metrics are also trajectory-dependent, needing step-by-step assessment that current benchmarks rarely provide. An agent can top the outcome metrics while failing the process ones, so any aggregate must be reported alongside its components rather than in place of them [152,454,457]. Emerging trajectory-aware frameworks nonetheless move in this direction: TRACE composes accuracy, process efficiency, and cognitive quality into a single trajectory-utility score for deep-research agents, explicitly to counter the “high-score illusion” that a lone pass@1 creates [458].

The metrics are nonetheless tightly connected, in three ways. First, accuracy and cost trade off: pushing one usually moves the other, so neither is interpretable alone and the honest object of comparison is a Pareto frontier rather than a point (Section 6.1.1). Second, calibration is the common cause behind the process metrics: a miscalibrated estimate of the agent’s knowledge boundary (Section 2.3) at once degrades tool-use decision quality, inflates cost, and, on the external side, creates safety exposure, so the process families are not independent axes but downstream readouts of one underlying quantity. Third, process metrics explain outcome metrics: when two agents post the same success rate at different costs, the gap always shows up in some process metric, whether a redundant-call rate, an overthinking loop, or an uncalibrated confidence, which is why reading the two families together is diagnostic where either alone is not.

This joint reading is exactly what exposes the two failure modes that motivate this chapter. Overthinking and overacting share one signature, visible only when outcome and process are read together: effort grows while resolved uncertainty does not. An agent that spends abundant reasoning tokens to re-derive a fact one API call would have supplied is internally inefficient yet accurate; an agent that issues many redundant queries when one would do is externally inefficient yet accurate. Pass-rate detects neither. The epistemic intelligence ratio of Section 2.4 is the natural unifier, since it places useful belief change over total effort and thus assigns both failures the same low score, but computing it requires the decomposed cost ledger and an estimate of belief change that only joint outcome–process instrumentation provides.

Because a fully unified score remains out of reach, recent work has approached the gap from the agent side rather than the metric side, training agents to allocate effort more carefully instead of defining one number. Thinking-budget controllers such as BudgetThinker [74] truncate chain-of-thought once marginal information gain drops below a threshold; budget-aware tool-use policies [75,350,351] allocate a finite invocation budget by expected information value; and intention-based planners [76] decide what information is needed before deciding how to obtain it. All share the epistemic necessity criterion of Section 2.4 [19]: invoke an external tool only when no internal path can resolve the uncertainty within an acceptable cost. Which metrics to foreground is finally set by the deployment context. Verifiable-reward training leans on functional correctness plus a cost ledger, safety-critical deployment foregrounds irreversibility and robustness, and open-ended assistance relies on judge-based correctness plus calibration. But every context still needs at least one metric from each family to avoid the blind spots a single-family report leaves open.

This points to the empirical question that the rest of this chapter takes up: what parts of the internal-external boundary do current benchmarks make visible, and how can outcome and process evidence be combined to judge whether external support was necessary? Section 6.2 re-reads the benchmark landscape through the ToA lens. It first establishes the internal baseline through internalization benchmarks, then characterizes the environments and support systems captured by externalization benchmarks. It finally turns to trade-off-oriented benchmark design, where controllable information conditions, controlled access settings, full trajectory observation, multi-axis reporting, and longitudinal measurement are combined to evaluate whether the agent used external support at the right time and at a justified cost.

Summary.  Agent evaluation should combine episode-level outcome metrics with trajectory- and policy-level process diagnostics. Outcome evaluation reports correctness together with separate internal- and external-cost ledgers, while process evaluation examines progress and constraints, boundary calibration and tool-use decisions, safety and robustness, and trajectory consistency. Read jointly, the two families expose overthinking and overacting as effort that grows without a corresponding reduction in uncertainty. Their components should remain visible rather than being collapsed into a context-free score; the epistemic intelligence ratio provides a unifying lens only when joint outcome-process instrumentation is available.

6.2. Benchmarks Through the ToA Lens

The previous subsection shows that outcome and process metrics must be read together: a benchmark should reveal both whether an agent succeeds and how its effort was spent. This subsection applies the same view to the benchmark landscape by asking what part of the internal-external boundary each benchmark makes visible. ❶ Internalization benchmarks (Section 6.2.1) estimate the agent’s internal baseline: what can be answered or inferred before outside support is used, and whether the agent can notice its own boundary. ❷ Externalization benchmarks (Section 6.2.2) evaluate the other side of the boundary: how agents use environments and external support systems such as tools, memory, skills, and harnesses. ❸ Trade-off-oriented benchmark design (Section 6.2.3) therefore treats the boundary judgment as the object of benchmark design. It specifies how to control information and access conditions, observe complete trajectories, report success, effort, and boundary quality together, and track how this allocation changes as the agent learns. Existing benchmarks characterize capabilities on the internal and external sides of the boundary; trade-off-oriented design connects them through a protocol that evaluates whether and when the agent’s use of external support is justified, and at what cost. Figure 10 summarizes this organization: internalization benchmarks establish the agent’s internal baseline, externalization benchmarks characterize the environments and support systems through which it acts beyond its parameters, and trade-off-oriented design specifies how benchmarks can evaluate the resulting internal-external boundary decision.

6.2.1. Internalization Benchmarks

Internalization benchmarks ask what an agent can solve before it calls any outside support. For ToA, they provide the internal baseline: we first need to know what the model can handle by itself, then judge whether external action is justified. We group this line of work into four parts. ❶ Knowledge benchmarks test whether the model already has the knowledge needed for hard academic questions. ❷ Reasoning benchmarks test whether the model can use that knowledge through multi-step inference. ❸ Live evaluation keeps this baseline reliable as benchmarks become saturated or leaked. ❹ Epistemic self-monitoring tests whether the model can inspect its own beliefs. This order follows a simple path: what the model knows, what it can derive, whether our measurement is still trustworthy, and whether the model can recognize its own boundary.

(1) Knowledge benchmarks. Knowledge benchmarks ask whether a model can answer difficult questions from its own knowledge. MMLU-Pro [141] starts from the saturation of MMLU and makes the task harder through more reasoning-heavy questions and a larger answer set. GPQA [142] raises the bar with expert-written science questions that are difficult for strong non-experts even with web access. Humanity’s Last Exam [459] extends this idea to a broader expert frontier, with verifiable questions written across many academic areas.

These benchmarks solve a clear problem in earlier evaluation: standard academic tests became too easy to separate strong models. They give ToA a first estimate of the internal knowledge boundary. If a task falls inside this boundary, external lookup may add little value. If it falls outside, answering from memory alone may lead to confident errors. The missing piece is that these benchmarks usually judge the final answer after the task. They do not show whether the agent knew during the task that internal knowledge was enough.

(2) Reasoning benchmarks. Reasoning benchmarks ask whether the model can derive an answer through internal reasoning. BIG-Bench Hard [143] selects difficult BIG-Bench tasks and shows that chain-of-thought prompting can reveal reasoning ability hidden by direct answering. BIG-Bench Extra Hard [460] follows the same path after BBH becomes less discriminative, replacing tasks with harder variants that probe similar skills. FrontierMath [144] moves this pressure to research-level mathematics, where problems are new, difficult, and automatically checked.

This group addresses a different limit of internalization. A model may have relevant knowledge but still fail to combine it across steps. Reasoning benchmarks therefore help locate the boundary of internal computation: how far the agent can go by thinking before outside action is needed. For ToA, the remaining issue is effort control. A correct answer does not tell whether the model reasoned efficiently, stopped too early, or kept reasoning after the useful signal had faded.

(3) Live evaluation. Live evaluation asks whether the internal baseline itself is still reliable. LiveBench [145] responds to test-set leakage and benchmark saturation by using frequently updated questions, objective scoring, and recent sources. Its role is slightly different from knowledge and reasoning benchmarks. It is less about one specific ability and more about keeping internal evaluation meaningful over time.

This matters for ToA because the internal boundary cannot be estimated from a stale benchmark. If the test has leaked or become too easy, the model may look more internally capable than it really is. Live evaluation strengthens the baseline by making the measured ability harder to memorize and easier to verify. Its limitation is that it still measures task success more directly than boundary awareness. It helps us trust the internal score. It does not yet tell us whether the agent knows when to rely on that score.

(4) Epistemic self-monitoring. Epistemic self-monitoring moves from task accuracy to self-knowledge. Reflection-Bench [140] evaluates whether a base model can form beliefs, revise them, reason about counterfactuals, and reflect on its own state. Complementary benchmarks isolate narrower components of this capacity. SelfAware [461] tests whether models recognize questions that are unanswerable or unknowable, while UA-Bench [462] evaluates whether they can attribute uncertainty to deficiencies in the available information or limitations in their own knowledge and reasoning. MIRROR [463] further connects monitoring to control by examining whether models can use self-knowledge to predict their own performance and select appropriate agentic actions. This is important because an agent uses the model as its cognitive core. The model must answer questions, but it must also track what it believes, when those beliefs should change, and when its own uncertainty warrants a different course of action.

This category connects most directly to ToA. The internal-external boundary is a decision problem, so the agent needs some sense of its own uncertainty before it can choose well. Reflection-style benchmarks make this ability visible. Their current scope is still mostly internal: they test belief management in controlled tasks. A full ToA benchmark would take the next step and ask whether this self-monitoring leads to the right behavior, such as continuing to reason, seeking evidence, verifying an answer, or stopping.

Taken together, these benchmarks estimate the internal side of the ToA boundary. Knowledge benchmarks show what the model can answer. Reasoning benchmarks show what it can derive. Live evaluation keeps the estimate current. Epistemic self-monitoring asks whether the model can see its own limits. This makes internalization benchmarks a necessary starting point for externalization benchmarks. Without a clear internal baseline, we cannot tell whether outside action is useful or simply extra effort.

6.2.2. Externalization Benchmarks

Externalization benchmarks evaluate how agents use resources outside their parameters to resolve uncertainty, update state, and act on the world. We organize them along two axes. ❶ Environment-facing externalization asks where the agent acts. This axis covers tools and APIs, computer interaction, domain-specific workflows, general agent tasks, and social or safety-critical interaction. ❷ Support-facing externalization asks what support remains outside the model. This axis covers memory, skills, and harnesses. The first axis describes the setting in which the agent acts. The second describes the support system around the model. This separation prepares the ToA question: after we know that an agent can externalize, we still need to ask whether that externalization was necessary.

Environment-facing externalization.

Environment-facing benchmarks differ in the scope and consequences of external action. They range from structured tool interfaces to domain-specific workflows and, eventually, to general, social, and safety-critical environments. Table 11 compares representative benchmarks in terms of their interfaces and executable environments, task interaction and horizon, and evaluation targets and methods. The discussion below focuses on how each line of work developed and how its primary object of evaluation changed over time.

(1) Tool-use and API environments. Tool-use benchmarks first treated external resources as task-specific aids. The evaluation target was whether access to an external resource improved the final answer. This left tool dependence ambiguous because some tasks remained solvable from parametric knowledge alone [464]. Evaluation then shifted toward selecting tools, constructing arguments, and composing multiple calls. Some benchmarks assessed generated calls or traces against fixed references [46,47,146,465,466,467,502]. Others emphasized reproducible execution rather than a single prescribed trace [91]. More recent settings make tool use a sustained interaction process. They test whether agents can act across multiple turns, incorporate feedback, manage application state, and complete trajectories with several valid paths [68,92,234,468,469].

(2) Web, mobile, and desktop environments. Computer-interaction evaluation began with simplified websites and narrowly defined interface actions [233]. Static demonstrations later introduced realistic pages and diverse human tasks, but they could not test how an agent changed the environment through its own policy [147,470]. Executable web environments made functional completion observable and reproducible [148]. Visual grounding then expanded the evaluation target from element selection to perception-guided action [471]. The same trajectory extended to enterprise web software, mobile devices, and full desktop operating systems [14,149,472,503]. Cross-version, cross-platform, and cross-application settings further turned transferability into an explicit object of evaluation [473].

(3) Software engineering environments. Functional correctness was the initial focus of software evaluation. Models generated self-contained functions or short programs, and test cases determined whether the output worked [474,475,476,477]. Interactive execution later allowed agents to observe program behavior and revise their actions  [478]. Repository-level benchmarks expanded the unit of evaluation from isolated code to issue resolution, codebase navigation, and dependency-aware modification [131,479,504]. End-to-end settings broadened it again. They assess project design, environment setup, multi-file implementation, testing, and requirement satisfaction across a complete development cycle [480,481,482,505].

(4) Data science and machine learning environments. In data science and machine learning, executable-code evaluation provided the initial bridge from textual answers to verifiable outcomes. The first agent benchmarks concentrated on running established tasks or improving a single experimental script  [483,484]. Later work increased the scope of the workflow. Agents must inspect data, select methods, write and execute code, and optimize against an empirical objective [485,486]. The evaluation object therefore moves from code validity to the quality of an experiment produced through iterative interaction with data, libraries, and compute.

(5) Scientific research environments. Scientific research benchmarks evolved from bounded scientific programming tasks to larger discovery workflows [487,488]. Subsequent settings evaluate data analysis and evidence-based discovery, including whether agents can implement methods and produce objectively checkable results [18,489,490]. Repository-based tasks add environment setup, dependency resolution, and experimental execution [491,492]. Research-replication benchmarks extend the horizon further. They ask agents to connect literature understanding, implementation, experimentation, and result analysis over an entire project [150,493]. The evaluation target thus shifts from solving a scientific subproblem to reconstructing and validating a research process.

(6) Workplace and enterprise environments. Workplace evaluation first modeled bounded knowledge-work tasks within a particular application [494]. It then moved toward operational settings with persistent records, role-specific constraints, and workflows spanning several business objects [495]. Broader enterprise environments combine browsers, terminals, code execution, communication, and organizational data. They also introduce collaboration and intermediate checkpoints [130,496]. Evaluation consequently changes from application-specific task completion to coherent performance across heterogeneous systems and professional roles.

(7) General agent environments. The move toward general agent evaluation responded to the narrow scope of domain-specific benchmarks. Multi-environment suites test the same agent across operating systems, databases, games, and web tasks [497]. Open-ended assistant benchmarks reduce dependence on a fixed API set and require agents to combine retrieval, reasoning, coding, and interaction as needed [435]. Constraint-rich planning adds a complementary target: agents must integrate information into feasible plans rather than recover one fixed answer [449,506]. The evaluation object becomes cross-domain task completion under heterogeneous forms of external access.

(8) Social and strategic environments. By treating other agents as part of the environment, social benchmarks make beliefs, incentives, and communication central to evaluation. Hidden-information games test whether an agent can infer roles and coordinate or deceive under uncertainty [498,499]. Strategic environments add resource allocation and competition, so success depends on anticipating how other participants respond [500]. Open-ended social simulations broaden the target from winning a game to maintaining plausible, goal-directed interaction over dialogue [138]. Evaluation therefore moves from individual task success to behavior that is contingent on other agents’ beliefs and actions.

(9) Agent safety and security environments. Safety evaluation for agents emerged because external actions can create consequences that text-only safety tests do not capture. Initial settings study accidental harm when benign requests are executed through imperfect or emulated tools [136]. Security-oriented benchmarks then introduce adversarial content inside tool outputs and test resistance to indirect prompt injection [501]. Misuse benchmarks change the threat model by giving the agent directly harmful goals. They evaluate whether safety persists across multiple turns and executable actions [82,84]. The evaluation object consequently expands from refusing unsafe text to controlling harmful behavior throughout an action trajectory.

Across these categories, environment-facing benchmarks expose increasingly complete forms of externalization. Isolated calls become grounded observations and state-changing actions, while short episodes become long-horizon and consequential trajectories. Under the ToA lens, however, most benchmarks begin after an external interface has already been supplied. They evaluate whether the agent can act through that interface and whether its trajectory reaches an acceptable outcome. They do not systematically determine whether external action was necessary in the first place. Without matched internal-only, external-enabled, and adaptive-access conditions, success cannot reveal whether the agent recognized the boundary of its internal competence. It also cannot show whether an external action reduced enough uncertainty to justify its cost and risk. The shared missing object is therefore the externalization policy itself: when to externalize, which intervention to choose, when to verify, and when to stop.

Support-facing externalization.

(1) Memory benchmarks. Memory benchmarks evaluate externalized state. They ask what information about users, conversations, tasks, and prior experience should be stored outside the model and reused later. LoCoMo [507] and LongMemEval [508] make this state measurable through long-term conversational settings. The agent must recover relevant evidence, handle temporal updates, and synthesize information across sessions. These benchmarks address a limitation of single-turn evaluation: an agent that cannot use history cannot sustain continuity.

Later benchmarks move from recall to memory management. Evo-Memory [509] evaluates test-time memory evolution in streaming tasks. EvoMemBench [510] connects knowledge and execution across both within-episode and cross-episode settings. The focus shifts from whether a stored item can be recalled to whether memory can evolve into reusable experience.

This shift already points to ToA. Memory is useful when it reduces future uncertainty, yet every memory operation has cost through storage burden, retrieval noise, interference, or stale evidence. Current benchmarks show whether agents can use memory. They say less about the memory lifecycle policy, including when experience should be stored, refreshed, discarded, or internalized. A ToA-aligned benchmark should evaluate these operations as allocation decisions and measure when memory access is worth its cost.

(2) Skill benchmarks. Skill benchmarks evaluate externalized procedures. A skill packages reusable know-how into an artifact that can be invoked across tasks. This matters because agents often face recurring workflows. Reconstructing the same procedure from prompt context or model parameters wastes effort, while a well-formed skill can preserve procedural knowledge outside the model.

The first question is whether skills help. SkillsBench [511] compares no-skill, curated-skill, and self-generated-skill settings. This establishes whether a skill artifact adds value beyond the base model and agent scaffold. The next question is retrieval. SkillRet [430] isolates the problem of selecting the right skill from a large library. SkillGenBench [512] then evaluates pipelines that generate reusable skills before downstream execution. The sequence is useful: efficacy, retrieval, and generation become separable evaluation targets.

From the ToA perspective, skills externalize procedural support. The open issue is the policy governing their use. Skills vary in utility, freshness, reliability, and invocation cost. The agent must decide when to solve internally, when to retrieve a skill, when to generate one, and when repeated use should become an internalized routine. A ToA-aligned benchmark should judge skill use by whether it reduces uncertainty or execution cost enough to warrant that external dependency.

(3) Harness benchmarks. Harness benchmarks evaluate externalized control. A harness is the runtime scaffold that turns model outputs into agent behavior. It manages three kinds of execution conditions: interface exposure, stateful context, and governance mechanisms such as tracing or recovery. This layer matters because the same model can behave differently under different execution arrangements.

The need for harness evaluation becomes visible in executable and workflow benchmarks. SWE-Bench [131], WebArena [148], and OSWorld [149] place agents in environments where actions change state. AgentBench [497] and GAIA [435] add long-horizon execution, persistent state, and live workflow pressure. Harness-Bench [273] turns this observation into a diagnostic benchmark by varying harness configurations while controlling task conditions. It measures how scaffold choices affect task success, efficiency, safety, and traceability of failures.

For ToA, harnesses are the control layer that governs other forms of externalization. They schedule external support and decide when validation or recovery begins. Existing harness work shows that scaffolds matter. The next question is whether each scaffolded intervention is epistemically justified. A ToA-aligned benchmark should evaluate whether the harness invokes external support at the right time, with acceptable cost, and with a clear contribution to uncertainty reduction, safety, or traceability.

Externalization benchmarks show that agents can act beyond their parameters by using environments and external support systems. Their main gap is the necessity of that use. Most protocols make external support available and measure whether it improves task success, but they leave open whether the agent needed that support in the first place. Externalization benchmarks therefore provide the environments and support systems on which a ToA-aligned benchmark can build. Their limitation is that they mostly show whether external support can be used, while the necessity of that use remains hidden. This motivates the trade-off-oriented design discussed next, where task boundaries, access conditions, trajectories, and costs are evaluated together.

6.2.3. Toward Trade-Off-Oriented Benchmark Design

Internalization benchmarks tell us what an agent can solve before outside support is used. Externalization benchmarks measure whether the agent can complete tasks with external resources and supports. Reporting the two sides separately still leaves the main ToA question open: did the agent choose internal reasoning or external action at the right time? Trade-off-oriented evaluation therefore treats the boundary judgment itself as the object of evaluation. It asks how much benefit external support brought, how much it cost, and whether the same task could have been handled internally.

The design discussion is organized into five layers. ❶ Design principles state what a ToA benchmark must make observable. ❷ A unified protocol turns these principles into matched tasks, paired access settings, trajectory records, and multi-axis reports. ❸ Retrofitting existing benchmarks explains how the protocol can be added to current benchmark environments. ❹ Designing new benchmarks shows how to build boundary-aware tasks from the start. ❺ Longitudinal extension asks whether the boundary moves as the agent learns.

Design Principles.

A trade-off-oriented benchmark should satisfy five connected requirements. These requirements form a chain: the task must expose the need for external support, the run must estimate the value of that support, the trajectory must show how the agent used it, and the report must keep success and cost visible together.

(1) Controllable information conditions. The benchmark should control where the necessary evidence is located. A task family should include prompt-sufficient cases, external-required cases, and external-trap cases. Prompt-sufficient cases test whether the agent can stay internal when the context is enough. External-required cases test whether the agent can seek evidence when the context is missing a decisive fact or state. External-trap cases test whether the agent can resist an available resource when that resource adds no value. High-risk or dynamic settings need two further cases: tasks that are answerable but still require verification, and tasks where neither internal reasoning nor external support can justify an answer.

(2) Controlled access settings. The value of external resources should be estimated through paired access conditions. A no-support run estimates internal solvability. A full-support run estimates the gain that outside resources can provide. An adaptive-access run tests the agent’s own choice. The boundary is relative to the model and context. A task may be internally solvable for a stronger model, externally dependent for a weaker model, and verification-dependent when the cost of a wrong answer is high.

(3) Full trajectory observation. The evaluator should observe both the path and the answer. The log should record internal reasoning effort, external access, environment feedback, verification, and stopping. Open tasks should allow multiple valid paths. The evaluator can then use executable checks, acceptable evidence sets, and step deletion tests to ask whether a step changed the support for the answer. This is important because a solved task may still contain unnecessary browsing, repeated API calls, or validation that only repeats what was already known.

(4) Multi-axis reporting. The result should report success together with effort and decision quality. A compact metric vector can contain outcome success, resource effort, and boundary quality. Boundary quality should separate underuse, overuse, verification failure, and poor stopping. Costs and risks rarely collapse cleanly into one context-free number. The default output should therefore be a metric vector or a Pareto comparison. A scalar utility is appropriate only when the deployment setting supplies weights.

(5) Longitudinal measurement. Agents that learn or evolve should be evaluated over time. The benchmark should ask whether internal success improves, avoidable external calls decrease, boundary judgment becomes more accurate, and the change survives delayed tests or environment shifts. The desired pattern is better allocation. Stable and repeated knowledge can move inward. Fresh, changing, risky, or audit-sensitive evidence should remain external.

A Unified Evaluation Protocol.

The protocol has four visible parts: matched task families define comparable cases by changing only the location or value of the necessary evidence; counterfactual access compares no-support, full-support, and adaptive-access runs; trajectory instrumentation records the final answer together with the resource path that produced it; and multi-axis reporting reads task success together with cost and boundary quality. The basic family contains internal-sufficient, external-required, and external-trap cases. When risk or refusal matters, it can also contain verification-required and unanswerable cases.

The key is to evaluate support need under a given task, model, and setting. ToolQA [464] is a useful starting point because it was built around questions that require external tools and sources unlikely to be memorized. A ToA protocol would add matched versions: one where the decisive evidence is placed in the prompt, one where it remains behind the tool, and one where the tool returns only irrelevant evidence. The same question can then expose three different failures: answering internally when evidence is missing, calling a tool when the prompt was already enough, or continuing to call tools after the resource has shown that it cannot help.

The same protocol can be stated for executable environments. In WebArena [148], the decisive state may be visible on the current page, require one controlled navigation step, or be absent from the self-hosted site. In SWE-Bench [131], the issue statement may localize the bug, leave localization to repository inspection, or require a test run to separate two plausible fixes. These paired versions keep the original task flavor, while making the marginal value of external action measurable.

Retrofitting Existing Benchmarks.

Retrofitting asks how far the protocol can be added to existing benchmarks while preserving their original evaluation target. This is useful because existing benchmarks already provide realistic task distributions, environment states, and mature verifiers. The price is weaker control: the original task may mix task difficulty, interface complexity, and resource need. For this reason, retrofitting is best treated as a realistic stress test of the protocol.

(1) Tool-use benchmarks. API-Bank [47] is suitable for retrofitting because its original evaluation already separates planning, retrieval, and API calling over executable tools. A ToA version can keep the same API schemas while changing the information condition. In a prompt-sufficient variant, the user request already contains the answer or all parameters needed for a direct response. In an external-required variant, one necessary value must be obtained through an API. In an external-trap variant, a tempting API exists but returns a value that is irrelevant, stale, or redundant. The resulting score can distinguish three behaviors that ordinary call accuracy merges together: correct abstention from API use, necessary API use, and wasteful API use.

ToolSandbox [92] gives an even sharper retrofit path because it already models stateful tool execution, implicit state dependencies, a simulated user, and intermediate milestones. A ToA retrofit can reinterpret milestones as evidence checkpoints and minefields as boundary failures. In an insufficient-information scenario, asking the user can be counted as a justified external action, while calling a state-changing tool before clarification becomes a boundary error. In a state-dependency scenario, a repeated call is useful only if the previous call changed the world state or reduced uncertainty about it. ACEBench [68] can be used in the same spirit: its special cases with ambiguous or incomplete instructions can be split into answerable, clarification-needed, and refusal-needed variants.

(2) Web and operating-system benchmarks. WebArena [148] is a good retrofit target because it is self-hosted and uses programmatic validation. A ToA version can vary where the decisive evidence is placed. If the current page already contains the needed information, extra browsing is overuse. If the evidence is one page away, navigation is justified only until the relevant state is reached. If the evidence is absent, continued browsing should be penalized through stopping quality. The same task can therefore measure success, page-read cost, and whether navigation reduced uncertainty.

OSWorld [149] extends this idea to real computer environments. Its tasks already involve initial state setup and execution-based evaluation scripts, so the retrofit can control whether the decisive state is visible in the current window, hidden in a nearby file, or only knowable after running an application action. The trajectory should record screenshots, clicks, commands, file reads, and app switches. A successful final state would then be read together with the path: a model that reaches the goal after ten blind clicks is different from a model that observes the right state and acts once.

(3) Software and app-world benchmarks. SWE-Bench [131] already grades patches by running tests in a repository. For ToA, the important retrofit is to make file access and test access part of the boundary. Some issues can be rewritten so the report contains a precise localization cue. Others should require reading a small set of files, and harder variants should require running tests to choose between plausible fixes. The score should then report patch success together with file-inspection cost, test cost, and stopping after sufficient evidence. This matters because a correct patch reached after broad repository search and repeated test runs may have lower epistemic efficiency than a patch reached through focused inspection.

AppWorld [469] provides another useful case because its state-based tests check both task completion and unwanted collateral changes. A ToA retrofit can attach costs to API calls, confirmation requests, and state mutations. If the user goal is already determined by the current app state, a confirmation round may be redundant. If the goal is underspecified, acting without confirmation can be unsafe. This lets the benchmark separate correct task completion from a well-calibrated action path.

(4) Memory, skill, and harness benchmarks. LongMemEval [508] is especially compatible with ToA because it marks evidence sessions and provides oracle, short-history, and long-history settings. A retrofit can turn these settings into access conditions. The oracle setting estimates performance when the relevant memory is already isolated. The long-history setting tests retrieval under noise. An unavailable-memory variant can test whether the agent knows to abstain when no stored evidence supports the answer. This directly measures the value of memory access, not just the ability to answer after memory is supplied.

SkillRet [430] can be retrofitted by adding no-skill, correct-skill, and distractor-skill access for the same user request. The original task asks whether the right skill can be retrieved from a large library. The ToA version asks whether retrieval should happen at all, and whether the selected skill adds value over internal execution. Harness-Bench [273] is similar at the control layer. Since it already measures model-harness configuration effects over sandboxed workflows, a ToA retrofit can compare disabled, fixed, and adaptive harness interventions. The question becomes whether validation, recovery, memory, or tool exposure was invoked at the step where it changed the outcome.

(5) Safety-sensitive benchmarks. AgentHarm [84] shows why trade-off evaluation needs more than utility and cost. In harmful agent tasks, external action may increase task completion while decreasing safety. A ToA retrofit can pair harmful tasks with benign capability controls and score refusal, verification, and action suppression as boundary decisions. Here, task completion alone is insufficient. The boundary decision is whether the agent can recognize when external action should be blocked even if it is technically feasible.

Designing New Benchmarks.

New benchmarks can control the boundary from the task-generation stage. The design should begin by deciding what evidence is needed before choosing a task domain. For each task, the builder specifies the decisive evidence, the context given to the agent, the outside resources, the expected check, and the stopping point. The task generator then produces matched versions by moving the same evidence across these locations. This design emphasizes task and outcome validity by making the relevant evidence and evaluation criteria explicit during benchmark construction [513]. This makes the comparison cleaner: the evaluator knows which resource could have helped and which resource was merely available but unnecessary.

(1) Make matched versions before writing prompts. The builder should first define the needed evidence. For a knowledge task, this can be a fact with a source and timestamp. For an action task, it can be a state that must change. For a memory task, it can be a stored preference or event. The prompt is written only after this evidence is fixed. This avoids a common failure in benchmark design: two tasks look like a boundary pair, but they differ in topic, wording, and difficulty. Counterfactual question-answering retains the underlying question while varying the contextual evidence and corresponding answer, whereas situated question-answering varies the temporal or geographical conditions under which the answer is determined [514,515]. A clean pair likewise changes where the decisive evidence is placed while keeping the surface problem similar.

(2) List available supports and their costs. Each task should expose a small set of available supports. The set can include a tool, a memory store, a skill library, or a checker, and each support needs a cost and a known risk. Existing benchmarks have separately operationalized customizable tool costs and dynamic cost changes, as well as high-stakes tool-use risks such as privacy leakage and financial loss [77,136]. The cost can be a call count, waiting time, permission level, or risk tag. The risk records what can go wrong: stale memory, misleading retrieval, invalid tool output, or harmful side effects. This makes the agent choose among supports under explicit cost, so outside help no longer appears free.

(3) Keep a hidden answer key for the evaluator. The evaluator should know which evidence is needed, which outside support is allowed, and when the agent should stop. In a ToolSandbox-like setting, the hidden key can use intermediate milestones and minefields [92]. In a WebArena-like setting, it can use environment states and final functional checks [148]. In a SWE-bench-like setting, it can use fail-to-pass and pass-to-pass tests [131], supplemented by an evaluator-side record of the files containing decisive evidence. The key is hidden from the agent. It lets the evaluator judge whether a trajectory step was necessary.

(4) Check whether each step mattered. For open tasks, a single reference path is too brittle. State-based, process-agnostic evaluation already provides a way to accept multiple valid solution paths [469]. The benchmark should instead test whether a step mattered through counterfactual interventions on the trajectory. Adapting deletion-based comprehensiveness and sufficiency tests from rationale evaluation [516], one can delete a tool call and ask whether the final answer loses support. Similarly, robustness tests using noisy and counterfactual retrieved contexts motivate replacing a retrieved memory with a distractor and observing whether the answer changes [517]. One can also remove a verification step and check whether the remaining evidence still justifies stopping. This keeps the evaluation path-sensitive without forcing every strong agent to imitate one human trace.

(5) Report success and cost side by side. The final benchmark should report paired outcomes, with a single leaderboard number only as an optional view. Multi-metric evaluation exposes trade-offs that accuracy alone obscures, while agent-specific evaluation work has argued explicitly for considering accuracy and cost together [518,519]. A model may score high under full-support and low under adaptive-access, which means it benefits from resources yet fails to decide when to use them. Another model may use fewer resources while missing external-required cases. These are different failures. Multi-turn agent evaluation also shows the diagnostic value of reporting intermediate progress rather than final success alone [135]. The benchmark should therefore release task success, boundary errors, effort, verification quality, and stopping quality as separate fields.

Longitudinal Extension.

The same protocol should be repeated when agents can learn from experience. Initial evaluation estimates the agent’s internal baseline and external dependence; post-update evaluation measures the effect of memory updates, skill acquisition, or harness changes; delayed evaluation checks whether the change persists; and shifted evaluation changes resources or environments to see whether the agent can recalibrate.

LongMemEval [508] offers a useful starting point because its evidence-only oracle setting and longer history settings separate relevant memory from noisy history. A longitudinal ToA extension can repeat related sessions over time and ask whether stable user facts become cheaper to access, either through internalization or through a better memory index, while recent updates still require external retrieval. The useful signal is more precise than a raw drop in memory calls. A drop is desirable only when answer quality and sensitivity to new information are preserved.

SkillRet [430] gives the same test for procedural support. Since it evaluates retrieval from large skill libraries, a longitudinal extension can ask whether repeated related tasks lead to better skill selection and fewer irrelevant retrievals. A strong agent should also detect when the library no longer contains a reliable match. Harness-Bench [273] gives a control-layer version of this question: by comparing model-harness configurations across repeated runs, the evaluator can ask whether added validation, recovery, or tool exposure reduces real failures or only adds routine overhead.

The desired long-term pattern is higher task utility with less unnecessary external effort. Stable and repeated abilities can move inward. Fresh, dynamic, high-risk, or audit-sensitive evidence should remain external. A good benchmark should measure that movement and avoid rewarding a fixed preference for reasoning or acting.

Summary. This subsection reads benchmarks through the internal-external boundary. Internalization benchmarks establish the internal baseline, while externalization benchmarks characterize the environments and support systems available beyond the model. Trade-off-oriented design connects these sides through controlled information and access conditions, full trajectory observation, multi-axis reporting, and longitudinal measurement, making it possible to assess whether the agent used external support when it was justified and avoided it when it was not.

7. Open Problems and Future Directions

The preceding sections have treated LLM-based agents as epistemic decision-makers whose behavior is governed by the placement, calibration, revision, and measurement of the internal–external boundary. This view suggests that future work should not be organized only around stronger models or richer harnesses, but around the mechanisms that determine when an agent should reason internally, act externally, revise its supports, or stop. We discuss four open problems that follow from this perspective: jointly learning reasoning and acting, achieving long-term personalized alignment, assigning credit in multi-agent evolution, and evaluating epistemic intelligence.

7.1. Learning: Jointly Improving Reasoning and Acting

Existing work on agent learning has made substantial progress on both sides of the internal–external boundary. On the internal side, learning methods compress memory, reasoning procedures, tool-use policies, and world-model knowledge into the model, turning recurring epistemic operations into more persistent model-side competence. On the external side, agent harnesses offload state, skills, orchestration, and environment interaction into memory systems, tool libraries, workflow graphs, and simulators. Recent agentic reinforcement learning and tool-integrated training methods further make retrieval, code execution, tool invocation, and search behavior directly trainable, while self-aware and budget-aware methods begin to regulate unnecessary tool calls and excessive reasoning [24,30,41,73,107,108,220,407]. However, much of this literature still improves reasoning and acting as separable objectives: one line teaches models to reason better or longer, while another teaches agents to call tools more accurately, more cheaply, or less often.

The open problem is joint learning of reasoning and acting. A capable agent should not only improve its internal reasoning ability or its external action ability in isolation; it should learn how the two interact under a shared epistemic objective. Future work should therefore train agents on mixed trajectories in which decomposition, internal reasoning, retrieval, memory access, tool calls, environment interaction, verification, delegation, abstention, and stopping are all treated as candidate epistemic operations. The main technical bottleneck is trajectory-level credit assignment: final success does not reveal whether a reasoning step, tool call, verifier, or retrieval actually reduced task-relevant uncertainty. A promising direction is to combine boundary-aware objectives with counterfactual trajectory analysis, estimating the marginal contribution of each operation to $\Delta U_{\mathrm{useful}}$ relative to its $E_{\mathrm{int}}$ or $E_{\mathrm{ext}}$ cost. Such learning signals would support selective internalization: stable, recurrent, and costly external routines can be moved inward, while information that requires freshness, provenance, execution grounding, or safety oversight should remain external [25,26,42,169].

7.2. Aligning: Long-Term Personalized Boundary Calibration

Agent alignment has expanded from response-level preference optimization to trajectory-level calibration. RLHF, DPO, and safety-oriented alignment methods remain central for learning helpful, honest, and harmless behavior from human feedback, while agent-specific alignment further constrains whether, when, and how an agent should reason, retrieve, call tools, execute actions, or abstain [78,79,81,354,355]. In the ToA view, this problem is not only preference satisfaction but epistemic calibration: the agent must align with its knowledge boundary, capability boundary, efficiency constraints, human preferences, environment constraints, and safety constraints. Recent work on personalized alignment, user simulators, user-centric benchmarks, and agent memory further shows that real-world agents must adapt to particular users rather than only to population-level preferences [54,55,56,86,87,90,520]. At the same time, deployment-time safety work shows that agentic actions introduce risks that cannot be handled by response-level alignment alone [82,84,85,99,383].

The open problem is long-term personalized boundary calibration under sparse and changing evidence. Future agents should adapt their boundary decisions to individual users in real time: when to answer directly, when to ask for clarification, when to retrieve user memory, when to invoke tools, when to verify, when to require confirmation, and when to abstain. This should not be reduced to simply fine-tuning the deployed model on a few user examples. The more general problem is to decide which part of the alignment state should be updated: external memory, a user profile, a compressed summary, a personal adapter, a verifier, a runtime policy, or model parameters. Future directions should therefore study few-shot user-conditioned boundary estimation, preference drift detection, and selective personalization under privacy and safety constraints. Stable, recurrent, and low-risk preferences may be internalized or compressed; volatile, sensitive, legally consequential, or revocable information should remain external and auditable. Personalized alignment also needs a hierarchy between user-conditioned utility and non-negotiable safety constraints: a user may prefer faster execution or fewer confirmations, but irreversible or high-risk actions should still require verification, sandboxing, or human approval when the expected harm exceeds the epistemic value of acting.

7.3. Evolving: Multi-Agent Evolution and Credit Assignment

Work on evolving agents has begun to treat agent improvement as a long-horizon process rather than a one-time training event. Internal evolution updates model-side priors, routines, and control policies; external evolution revises memory, tools, workflows, and environments; ecological evolution expands the unit of adaptation to individual agents, agent populations, and agent–environment systems [2,36,40,101]. Multi-agent systems make this evolutionary view more complex. Agent populations can specialize roles, share experience, critique one another, generate tasks, and distill collective discoveries into individual policies [35,106,111,123,124,125,126]. However, collaboration also introduces coordination overhead, misleading communication, collusion, attack surfaces, and a gap between individual capability and team performance [127,380,398,446].

The open problem is fine-grained credit assignment for multi-agent evolution. In a multi-agent system, one agent’s internal competence may function as another agent’s external epistemic resource: a planner externalizes verification to a critic, a reasoner externalizes execution to a tool-operator, and the team externalizes long-term state to a memory agent. Future systems should therefore learn agent-specific boundary profiles rather than relying only on prompt-level roles such as planner, critic, retriever, or executor. Each sub-agent should be characterized by what it can solve internally, what it can obtain externally, how reliable its boundary estimates are, and what cost or risk it introduces. The central future direction is team-level epistemic credit assignment: identifying which agent, role, message, retrieval, tool call, or verification step actually reduced uncertainty. Counterfactual agent ablations, message masking, role swapping, delayed delegation, and tool-call interventions can help determine whether a sub-agent contributed useful evidence, introduced misleading information, or merely increased coordination cost. This would allow differentiated evolution: agents specialized for internal epistemic work can improve decomposition, abstraction, proof search, or self-critique, while agents specialized for external epistemic work can improve retrieval, execution, environment interaction, memory management, or verification. A further question is when a useful multi-agent workflow should remain external for auditability, disagreement, and role separation, and when it should be distilled into a smaller agent or single model [59,60,404].

7.4. Evaluation: Measuring Epistemic Intelligence

Agent evaluation has moved beyond simple pass rates, but remains fragmented across outcome metrics, process metrics, cost-sensitive metrics, and domain-specific benchmarks. Internalization benchmarks estimate what an agent can solve through its own knowledge and reasoning; externalization benchmarks test whether the agent can use tools, memory, environments, or interfaces; trade-off benchmarks expose trajectories in which internal reasoning and external action must be coordinated. Recent work has begun to evaluate epistemic agency, tool-use necessity, multi-turn trajectories, agent-as-judge protocols, and cost-sensitive performance [68,71,77,92,134,135,140,451]. Nevertheless, current benchmarks still tend to emphasize terminal success more than the epistemic quality of the trajectory, and evaluation papers and leaderboard proposals indicate that agent evaluation lacks a unified protocol for comparing correctness, cost, safety, calibration, and process quality [152,454,457].

The open problem is to make epistemic intelligence empirically observable. In ToA, the relevant target is not raw accuracy or minimal cost, but useful uncertainty reduction per unit of internal and external effort. A correct answer may still be epistemically poor if it required redundant retrieval, unnecessary tool calls, unsafe execution, or excessive deliberation; a failed trajectory may still contain useful reasoning, a correct warning, or a justified abstention. Future benchmarks should therefore make the internal–external boundary explicit through boundary-aware task construction, including internally solvable tasks, externally necessary tasks, ambiguous tasks, stale-information tasks, dynamic tasks, and unanswerable tasks. They should also use counterfactual access protocols, evaluating the same task under no-tool, tool-enabled, and adaptive regimes to distinguish overacting, underacting, overthinking, and underthinking. At the trajectory level, benchmarks should annotate whether intermediate operations are useful, necessary, redundant, premature, unsafe, misleading, or epistemically justified. Finally, evaluation should become causal and longitudinal: causal analysis should test how removing, delaying, masking, or replacing a step changes belief state, final utility, safety, and cost, while longitudinal evaluation should test whether an evolving agent becomes better calibrated over time through fewer redundant retrievals, fewer unnecessary tool calls, more appropriate refusals, better stopping decisions, and more selective internalization of stable knowledge.

8. Conclusion

This paper has reviewed roughly six hundred recent studies of LLM-based agents under the Theory of Agent. Our central claim is that the internalization–externalization boundary is a governing problem of agent design. It decides which knowledge, skills, and control sit inside the model, which stay in the surrounding harness, and how much effort goes to internal reasoning versus external acting.

From this perspective, our taxonomy categorizes current methods by how they shift this boundary. Learning distributes capabilities between the model and harness, while evolution updates both over time. Alignment determines when an agent reasons versus acts under cognitive, efficiency, human, environmental, and multi-agent constraints. Evaluation tests whether these decisions yield enough uncertainty reduction to justify their cost, risk, and process quality.

Moving forward, boundary calibration should be a primary objective in agent design. Effectiveness depends not just on model capacity, available tools, or task throughput, but on knowing when an agent should rely on itself, when it should query the world, and how to coordinate internal and external abilities.