Agentic Representation Learning: How Should AI Agents Be Represented and Learned from Black-Box Agents?

1. Introduction

An autonomous agent is more than a passive predictor. Situated in an environment, an agent acts over time toward objectives and often interacts with other agents [1]. Representation learning has repeatedly advanced AI by turning complex objects into reusable computational representations [2], from perceptual representations learned by convolutional neural networks [3] to semantic representations in language models [4], multimodal representations in vision-language models [5], and predictive state representations in world models [6]. Following this trajectory, understanding how to depict an agent becomes a fundamental research question.

Recent benchmarks such as AgentBench [7], MultiAgentBench [8], WebArena [9], AgentBoard [10], OSWorld [11], and $\tau$-bench [12] have substantially advanced the realistic evaluation of agent behavior. These benchmarks primarily characterize agents through their observable task performance, measuring how they accomplish goals across complex environments. Such agents are still primarily described in terms of their implementation details. Building on and extending this perspective, we frame Agentic Representation Learning (ARL) by investigating which aspects of an agent should be captured in its representations (Q1) and how to learn such representations efficiently from black-box interaction (Q2). ARL provides a representation-centric view for scalable multi-agent systems under constrained conditions.

Figure 1. From feature, language, multimodal, and world-model representations to agentic representations. Five columns compare neural networks, language models, vision-language models, world models, and agents.

Figure 1. From feature, language, multimodal, and world-model representations to agentic representations. Five columns compare neural networks, language models, vision-language models, world models, and agents.

2. From Policies & Profiles to Representations

Early policy representation work explicitly casts modeling other agents from interaction data as representation learning. Neural methods learn opponent models and policy embeddings from interactions [13,14], whereas subsequent works use policy representations for value generalization and exploration [15,16]. Agent evaluation and profiling characterize capabilities and predefined tendencies [10,17], and research works show that agent-level regularities are measurable, but identity recognition or a fixed trait score is weaker than a reusable representation that transfers to unseen situations. Black-box agent modeling infers policy behavior and task- or security-specific properties from exposed interactions, with active probing used to acquire informative evidence [18,19,20].

Table 1 summarizes these research lines and typical objectives, while also highlighting their connections to ARL. Unlike prior profiling or identification methods, ARL focuses more on predictive representations that can be learned efficiently and transferred beyond the limited interactions used to infer them.

3. Q1: What Representations Should an Agent Have?

3.1. Structure: How the Agent Is Built

Figure 2 highlights five representative components of an agent A: (1) Model (M): processing observations and generating responses or actions; (2) Planning (P): organizing actions and subgoals toward a task objective; (3) Memory ($Mem$): storing and retrieving information across interactions; (4) Tools (T): providing interfaces to access external resources and execute actions; and (5) Verification (V): checking intermediate results and task outputs. Together with instructions I, these give a description of the agent:

$$A=(I,M,P,Mem,T,V,\dots ).$$

(1)

Instructions I guide how the agent uses these components since component availability alone does not determine behavior.

3.2. Representation: What Should Be Captured

The representation $z_A$ summarizes behavioral tendencies, not necessarily an internal neural representation. Consider two agents facing the same uncertain tool result, where one checks it and the other proceeds. Predictions should account for the task context and the cost of checking, since more checking is not inherently better. Figure 2 illustrates five potentially overlapping dimensions:

• Behavioral Patterns: Regularities in task completion and interaction.
• Strategic Preferences: Choices between exploration and exploitation, and preferences concerning risk.
• Memory Usage: Patterns of retaining and using past information, including memory reads and writes when exposed.
• Tool Use: Tool selection and the frequency of tool calls.
• Recovery Ability: Error handling and self-correction following failures.

These dimensions need not be independent coordinates. They can be jointly represented as vectors, textual profiles, or probabilistic summaries to support useful prediction.

3.3. Agent Type

White-box and black-box refer to different levels of access: $\left(\mathrm{i}\right)$white-box access provides access to internal structure and configuration; $\left(\mathrm{ii}\right)$ black-box access is limited to observable interactions. Internal information may inform white-box representations, but their behavioral predictions still require validation.

3.4. External Signals: What Can Be Observed

Under black-box access, the only evidence is what the interface exposes. Six potentially overlapping signal categories are: (1) Interaction Trajectories ($\tau$): visible inputs, actions, observations, responses, and exposed states; (2) Tool Usage ($\mathcal{T}$): tool selection, call frequency, and invocation patterns; (3) Memory Access ($\mathcal{M}$): exposed memory read/write events and use of retrieved information; (4) Resource Consumption ($\mathcal{R}$): API calls, token usage, elapsed time, and other observable costs; (5) Error-Recovery Behavior ($\mathcal{E}$): retries, backtracking, and self-correction; and (6) Task Outcomes ($\mathcal{O}$): success, partial success, failure, or task-specific results. These signals map to the representation as:

$$z_A=f(\tau ,\mathcal{T},\mathcal{M},\mathcal{R},\mathcal{E},\mathcal{O}).$$

(2)

The mapping f specifies no particular architecture and need not use independent channels. Hidden reasoning and memory operations are not observable by default.

3.5. State: What The Agent Is Currently Carrying

Following the distinction between agent characteristics and mental state in Machine Theory of Mind [23], separate relatively stable agent tendencies from state that changes within an interaction, such as current goals and beliefs. State that is not directly observable must be inferred from behavior. For evaluation, track interaction history and state validity, while accounting for the fact that probing can alter the agent state and preclude clean resets.

3.6. Downstream Tasks

The representations can guide a separate learning or adaptation procedure for a new agent $A^{\prime }$. Task families include (1) Coding: code generation, debugging, and program analysis; (2) Web Interaction: browsing, information retrieval, and data extraction; (3) Planning: task decomposition, plan generation, and resource allocation; (4) Tool Use: external tools and APIs for data processing; and (5) Multi-Step Decision Making: long-horizon reasoning and sequential decisions.

4. Q2: Learning Agent Representations from Limited Black-Box Interaction

4.1. Characterizer Agent

Q1 defines the observable evidence, while Q2 addresses how to learn from limited evidence and determine which additional interactions are worth their cost. A characterizer agent selects a permitted query, observes the resulting trajectory and the exposed signals, and updates the target representation. Let $D_t$ store the ordered query contexts, initialization conditions, trajectories, and associated signals after t rounds. We use $z_{A,t}=f\left(D_t\right)$ to represent the evidence-to-representation mapping in Equation (2), with t indicating updates. Prior work has explored probing other agents [25] mainly in grid worlds, while ARL pursues reusable predictions for composite agents across tasks and time.

Figure 3 illustrates the whole process of active black-box characterization. Continuing the verification example, the loop runs as follows. (1) Probe: present a visible uncertainty cue and an optional check with a stated cost. (2) Observe: record checking, proceeding, or clarification, together with context, cost, and outcome. (3) Update: revise the behavioral estimate and its uncertainty before choosing another query. Adjusting the checking cost while controlling available evidence helps distinguish cost sensitivity from a general tendency to verify.

4.2. Learning from Limited Evidence

A limited number of target interactions does not require training an encoder from scratch. One approach is to learn an encoder $f_{\varphi }$ and predictor $p_{\theta }$ across source agents, following policy-representation and character-modeling methods [14,23]. For a source agent A, let $S_A$ contain characterization evidence and $V_A$ contain separate prediction examples. x denotes a task context with interaction history and y a specified observable behavior. These give the objective function

$$\mathcal{L}(\varphi ,\theta )=\mathbb{E}\left[-log{p}_{\theta }\left(y\mid x,f_{\varphi }\left(S_A\right)\right)\right].$$

(3)

Here, $\varphi$ and $\theta$ are the encoder and predictor parameters, and the expectation samples source agents, their characterization records, and examples $(x,y)$ from $V_A$. Examples for continuing agents must respect temporal order and the histories created by earlier interactions. Supervision is based on observed behavior and does not require ground-truth latent traits. During deployment, shared parameters can remain fixed as target-specific evidence accumulates.

4.3. Information Gained per Unit Cost

Adaptive experimental design provides a natural strategy for selecting informative interactions [26,27]. BED-LLM similarly uses belief updating and information-guided question selection with LLMs [28].

In a fixed representation space, let $Z_A$ denote the unknown target representation, over which the characterizer holds a belief, and $z_{A,t}$ its current point estimate based on the evidence collected so far. Let ${\mathcal{Q}}_t$ denote the feasible query set under the interface and budget constraints, and let ${\tau }_{t+1}$ denote the observable record returned by the next query. Issuing $q_{t+1}$ extends $D_t$ with that query and its record. We select the next query by maximizing information gain per unit cost:

$$q_{t+1}\in arg\underset{q\in {\mathcal{Q}}_t}{max}{\displaystyle \frac{\mathrm{I}(Z_A;{\tau }_{t+1}\mid q,D_t)}{c(q\mid D_t)}},$$

(4)

where $\mathrm{I}$ measures the expected information gained about the target representation $Z_A$ and $c(q\mid D_t)>0$ denotes the expected interaction cost.

This rule provides a way to balance information gain against cost. Not all information equally improves downstream prediction, so probe selection may prioritize expected improvement in predicting target behaviors. When probes may alter agent state, representation updates should account for such changes. Probing stops when the budget is exhausted or the expected benefit becomes too small.

4.4. Behavioral Sufficiency and Evaluation

A learned representation is behaviorally sufficient if it predicts pre-specified agent behaviors on held-out tasks within a declared task family. Accurate agent identification alone does not establish behavioral sufficiency. Evaluation should compare acquisition strategies under matched interaction cost, without using final test outcomes during characterization.

5. Related Work

Agent modeling uses behavioral observations for prediction and interaction [29]. Grover et al. [14] learn policy embeddings from limited interactions for prediction and downstream learning. Policy representations also support value generalization [15] and exploration in policy space [16]. For robotic control, BehaviorVLA [30] uses behavioral prototypes derived from demonstrations and utilizes execution progress to guide actions. Opponent modeling and theory of mind also address context and adaptation. He et al. [13] incorporate opponent representations into value estimation, while ToMnet [23] separates character and mental-state embeddings for behavior prediction. To shape future learning, LOLA [31] differentiates through anticipated opponent updates, whereas Xie et al. [32] learn latent strategy dynamics to steer a partner that is still adapting. Hypothetical Minds [33] uses an LLM to refine natural-language hypotheses about other agents for planning, and Shapira et al. [24] use features from a separate observer LLM to predict, from limited target interactions, the decisions of unfamiliar LLM agents in bargaining and negotiation games.

Other work characterizes or evaluates agents through multiple dimensions. For explicit characterization, agentic profiles [21] distinguish autonomy, efficacy, goal complexity, and generality, while TRAIT [17] measures predefined personality dimensions in LLM responses to questions. Leshin et al. [34] instead learn trait directions to score skill-file edits, which require access beyond black-box interaction. Beyond describing traits, FP-Agent [22] identifies browsing-agent systems through browser and behavioral fingerprints. These profiles and signatures offer candidate representations, complemented by performance evidence from AgentBoard [10], which analyzes intermediate task progress. Broader evaluation challenges include granularity, cost, and robustness [35].

For black-box modeling, knowledge compilation [18] and TripleTree [36] construct interpretable models from recorded behavior. To acquire informative evidence, Bayesian active learning selects observations using model uncertainty [37,38]. Shu et al. [25] learn probes rewarded by changes in behavioral representations used for action prediction, whereas Wang et al. [19] use active interaction in driving simulations to infer preferences within a predefined utility model. When behavior changes, differential assessment uses selective queries to update symbolic models under fully observable, deterministic assumptions [39]. For security, AGENTVIGIL [40] optimizes prompt injections using attack feedback, whereas Know Your Agent [20] builds target profiles through probing to guide subsequent attacks. ARL should not become a new name for trajectory embedding or agent profiling. Nor does ARL require a universal latent space, since different downstream tasks may need different behaviorally sufficient representations.

6. Future Directions

6.1. Controlled Agent Zoos And Identifiability

A factorial agent zoo should vary models, planning, memory, verification, tool policies, and instructions within shared environments. These controlled variations test whether representations predict behavioral changes beyond recovering configuration labels or writing style. Crucially, they test whether verification effects transfer to unseen models and whether combined effects of memory and verification are predictable from their individual effects.

6.2. Active Characterization Benchmarks

The decisive experiment should plot held-out behavioral prediction versus characterization cost. A characterizer receives a budget to probe a hidden target agent, and the learned representations are evaluated on unseen tasks or environments. Stronger tests also withhold both target-agent configurations and environment families. While recent work predicts counterpart decisions from few interactions and demonstrates the feasibility of this regime [24], the open challenge lies in developing adaptive observations and transferable representations beyond a single game family.

6.3. Representation-Based Multi-Agent Systems

Agents could estimate representations of unfamiliar teammates or opponents before high-stakes collaboration [33], route tasks to complementary agents, adapt negotiation policies to predicted counterpart behavior, or detect when a long-running partner has drifted [39].

6.4. Strategic Behavior, Privacy, and Governance

Characterization introduces both strategic and privacy risks. Target agents may behave differently during calibration, probing may reveal proprietary information [20], and learned representations may facilitate tracking [22]. These risks call for safeguards such as consent, limits on probing, privacy preservation, and robustness evaluation under strategic behavior.

7. Conclusions

We propose ARL as a new perspective for representing AI agents as learnable behavioral objects. ARL asks what agent representations should capture and how to learn them efficiently from limited black-box interaction. We advocate predictive representations acquired through active, cost-aware characterization rather than static profiles or exhaustive evaluation. ARL can provide a foundation for scalable multi-agent systems in which agents efficiently understand, compare, and adapt to one another.