Agent Governance for Self-Evolving AI Agents: A Literature Review on Runtime Adaptation, Self-Modification, and Safe Autonomy

1. Introduction

Agentic artificial intelligence extends generative models from response generation to goal-directed interaction. In a typical language-model agent, a foundation model is embedded in a loop that observes a state, reasons about a goal, selects an action, invokes tools, and incorporates feedback. ReAct made the reasoning–action interleaving pattern explicit [1]; Toolformer demonstrated that language models can learn when and how to call external tools [2]; and subsequent systems added long-term memory, reflection, planning, and multi-agent coordination.

The governance problem becomes qualitatively different when an agent can change itself. A static model may still behave unpredictably, but at least its main components remain fixed between evaluation and deployment. A self-evolving agent can rewrite prompts, curate or delete memories, synthesize new tools, alter workflows, update policies, generate code, or learn from interaction. Reflexion, for example, stores verbal feedback to improve later trials [3]; Voyager accumulates an executable skill library in an open-ended environment [4]; and generative-agent architectures use memory streams and reflection to shape later behavior [5]. These mechanisms are useful precisely because they change future behavior. They also create a moving target for assurance.

This review treats agent governance as the set of technical and institutional mechanisms that constrain, observe, authorize, audit, and, when necessary, reverse an autonomous system’s behavior and adaptation. Governance is broader than alignment training and narrower than society-wide AI policy. It includes access control, tool permissions, runtime monitors, behavioral contracts, human approval gates, provenance, logging, rollback, red teaming, and recertification after material changes.

The paper addresses four questions:

RQ1.

What parts of an AI agent can change during deployment, and through which mechanisms?

RQ2.

What risks are introduced when adaptation occurs after initial evaluation?

RQ3.

Which governance and verification mechanisms are technically available?

RQ4.

What research agenda could make self-evolving agents auditable and safely governable?

The main contribution is a lifecycle synthesis. Rather than treating memory, reflection, tool use, safety, and governance as separate topics, we connect them through the sequence propose change–evaluate change–authorize change–deploy change–monitor consequences–rollback or retain. This framing highlights an underdeveloped principle: learning capability and action authority should be governed separately.

Although this review focuses on agentic AI, it is informed by broader traditions in computer vision, medical AI, and machine learning systems engineering. Earlier work on face recognition and biometric verification demonstrated the importance of robust feature representations under changing conditions [6,7,8,9,10,11,12,13,14,15,16]. Subsequent studies on clinical prediction and healthcare AI highlighted the need for reliable, interpretable, and deployable machine learning in high-stakes environments [17,18,19,20,21]. These lessons motivate governance mechanisms that emphasize safety, monitoring, and accountability for adaptive AI agents.

2. Review Methodology

This project uses a structured narrative review design. It is not yet a completed systematic review because the final database export, deduplication record, and reviewer agreement statistics must be generated by the author. The search protocol is designed to be reproducible.

2.1. Search Strategy

Recommended databases are Scopus, Web of Science, IEEE Xplore, ACM Digital Library, arXiv, and Google Scholar for citation chaining. A core Boolean query is:

(agentic AI OR LLM agent* OR autonomous agent*) AND (self-evol* OR self-modif* OR runtime adapt* OR continual learn* OR reflection OR memory) AND (governance OR safety OR verification OR oversight OR monitoring OR alignment)

The recommended publication window is 2022 onward, supplemented with earlier foundational work in runtime verification, adaptive systems, and AI governance.

2.2. Eligibility

Include papers that (i) study agents capable of sequential action; (ii) introduce or analyze adaptation after deployment; or (iii) propose controls relevant to changing agents. Exclude papers limited to single-turn prompting, ordinary fine-tuning without an agent loop, or governance discussions with no identifiable implications for agent design.

2.3. Coding Framework

Each study should be coded along five dimensions:

1.

agent substrate and environment;

2.

object and mechanism of adaptation;

3.

degree of autonomy;

4.

governance control and evidence type;

5.

evaluation horizon, failure model, and reproducibility.

2.4. Threats to Validity

The literature is preprint-heavy and terminology is unstable. “Self-evolving” may refer to memory updates, prompt rewriting, skill acquisition, code synthesis, or parameter learning. Evidence from benchmark tasks may not transfer to persistent deployment. Publication bias favors capability demonstrations over negative results. The final review should therefore distinguish peer-reviewed evidence from preprints, demonstrations, position papers, and conceptual proposals.

3. Foundations: From LLMs to Adaptive Agents

3.1. Agent Loop

A useful abstraction is

$$s_{t+1}=E(s_t,a_t),a_t={\pi }_{{\theta }_t}(o_t,m_t,g,c_t),$$

where $s_t$ is environmental state, $o_t$ an observation, $m_t$ memory, g the goal, $c_t$ the control context, and ${\theta }_t$ the agent configuration. A self-evolving agent permits some component of ${\theta }_t$, $m_t$, or $c_t$ to change over time.

ReAct operationalized a loop in which language-model reasoning traces are interleaved with actions [1]. Toolformer showed how tool use can become part of model behavior through self-supervised learning [2]. AutoGen later provided a framework for conversational multi-agent workflows [22]. These systems establish the architectural substrate on which runtime adaptation is built.

3.2. Memory and Reflection

Memory transforms isolated interactions into persistent behavior. Generative Agents introduced memory streams, retrieval, reflection, and planning for simulated social agents [5]. Reflexion uses linguistic feedback as episodic memory rather than updating model weights [3]. This distinction matters for governance: external memories can be inspected and rolled back more easily than opaque parameter updates, but they also create poisoning, privacy, and provenance risks.

3.3. Skills, Tools, and Code

Voyager demonstrates open-ended skill acquisition by generating, validating, and storing executable programs [4]. Such skill libraries are a form of self-modification at the scaffold level. They may be safer than unconstrained weight updates because each skill can be tested independently, yet composition creates new behavior not observed during single-skill evaluation.

3.4. Multi-Agent Adaptation

In multi-agent systems, adaptation includes changes to role assignment, communication topology, delegation, trust, and shared memory. Governance must therefore consider not only the internal state of one agent but also emergent behavior at the system level. A locally compliant agent may contribute to globally unsafe outcomes through coordination failures, collusion, or cascading misinformation.

Table 1. Objects of self-evolution and associated governance concerns.

Object of change	Typical mechanism	Primary governance concern
Memory	retrieval, summarization, reflection	poisoning, privacy, irreversible forgetting
Prompt or policy	self-critique, prompt search, constitutional revision	objective drift, hidden rule conflict
Tools and skills	API discovery, code generation, skill libraries	privilege escalation, unsafe composition
Workflow	planner revision, role reassignment, graph search	loss of traceability, untested control flow
Model parameters	reinforcement learning, continual fine-tuning	catastrophic drift, difficult rollback
Social organization	delegation, trust updates, coalition formation	collusion, diffuse accountability

Table 1. Objects of self-evolution and associated governance concerns.

Object of change	Typical mechanism	Primary governance concern
Memory	retrieval, summarization, reflection	poisoning, privacy, irreversible forgetting
Prompt or policy	self-critique, prompt search, constitutional revision	objective drift, hidden rule conflict
Tools and skills	API discovery, code generation, skill libraries	privilege escalation, unsafe composition
Workflow	planner revision, role reassignment, graph search	loss of traceability, untested control flow
Model parameters	reinforcement learning, continual fine-tuning	catastrophic drift, difficult rollback
Social organization	delegation, trust updates, coalition formation	collusion, diffuse accountability

Additional representative work supporting this discussion includes [23,24,25,26].

4. Mechanisms of Self-Evolution

4.1. Reflection and Self-Critique

Reflection methods convert outcomes into textual feedback that influences future decisions. Their appeal is operational simplicity: no parameter update is required, and the adaptation artifact can be logged. However, reflection may amplify incorrect self-diagnoses. A governance layer should therefore preserve the evidence that triggered a reflection, record its provenance, and test whether the change improves performance without violating constraints.

4.2. Memory Adaptation

Memory systems may add, merge, summarize, rank, or delete experiences. Governance questions include who may write to memory, whether high-impact memories require validation, how contradictory memories are resolved, and whether users can inspect or erase retained information. Provenance-aware memory should link each item to its source, confidence, timestamp, and downstream uses.

4.3. Tool and Skill Acquisition

Agents may discover APIs, synthesize programs, or retain reusable procedures. Tool acquisition changes the action space and can increase effective authority. A safe design treats new tools as untrusted software: execution occurs in a sandbox, permissions are least-privilege, outputs are validated, and promotion to persistent use requires tests and approval.

4.4. Workflow and Architecture Search

An agent may alter decomposition strategies, add specialized subagents, or rewrite orchestration graphs. Because these changes affect system structure, unit-level testing is insufficient. Governance should include graph-level invariants, resource budgets, termination conditions, and checks for unsafe delegation.

4.5. Parameter-Level Learning

Continual fine-tuning or reinforcement learning changes the policy itself. Such updates can improve adaptation but are the hardest to interpret and reverse. Safe deployment requires versioned checkpoints, held-out safety evaluations, canary release, change-impact analysis, and automatic rollback when monitored behavior crosses predefined thresholds.

4.6. Degrees of Self-Modification

Not all adaptation is equally consequential. We distinguish:

1.

ephemeral adaptation: temporary context or scratchpad changes;

2.

persistent scaffold adaptation: memory, prompts, tools, or workflows;

3.

policy adaptation: updates to decision rules or model parameters;

4.

authority adaptation: changes to permissions, credentials, or reachable systems.

The fourth category should generally be prohibited from self-authorization. An agent may propose a wider permission but should not grant it to itself.

Additional representative work supporting this discussion includes [27,28,29,30].

5. Governance Mechanisms

5.1. Separation of Learning and Authority

The core design principle proposed in this review is that an agent’s ability to learn should not automatically expand its authority. A system may acquire a new skill in a sandbox while remaining unable to use that skill against production systems. Authority should be represented explicitly through capabilities, credentials, budgets, and policy rules controlled outside the agent’s editable state.

5.2. Constitutional and Policy-Based Controls

Constitutional AI demonstrates how explicit principles can structure model critique and revision [31]. For agents, a constitution can guide planning and refusal, but a text-only constitution is not an enforcement mechanism. High-assurance governance requires executable policy checks at the tool boundary and environment boundary.

5.3. Human Oversight

Human-in-the-loop control is useful when consequences are high and action frequency is manageable. Approval gates are less effective when humans face alert fatigue, cannot inspect long reasoning chains, or routinely approve requests. Oversight should be risk-adaptive: low-impact actions may be automated, while irreversible, financial, privacy-sensitive, or safety-critical actions require stronger authorization.

5.4. Capability Security

Least privilege, scoped tokens, network isolation, rate limits, and budget caps reduce the damage caused by erroneous or malicious plans. Tool calls should be mediated by a trusted reference monitor that the agent cannot rewrite. Inputs and outputs should be typed and validated rather than passed as unrestricted natural language.

5.5. Logging, Provenance, and Audit

Every material adaptation should produce an audit record: prior version, proposed change, triggering evidence, evaluator results, authorizer, deployment time, and rollback target. Logs must be tamper-evident and stored outside the agent’s writable environment. Auditability is especially important when multiple agents contribute to a decision.

5.6. Change Management

Self-evolution can be governed like safety-critical software deployment:

1.

generate a candidate change;

2.

test it in isolation and in representative scenarios;

3.

check security and policy invariants;

4.

authorize based on risk tier;

5.

deploy gradually;

6.

monitor consequences;

7.

retain, quarantine, or roll back.

Figure 1. Governed adaptation lifecycle.

Figure 1. Governed adaptation lifecycle.

Additional representative work supporting this discussion includes [32,33,34].

6. Runtime Assurance, Verification, and Evaluation

6.1. Runtime Monitoring

Runtime monitors observe actions, tool calls, state transitions, and resource use. Deterministic checks are preferable for hard constraints such as forbidden domains, transaction limits, data schemas, or credential scope. Model-based monitors may help classify ambiguous risks but introduce correlated failure and should not be the sole enforcement layer.

6.2. Formal and Semi-Formal Methods

Full formal verification of open-ended language-model behavior is generally infeasible, but narrower properties can be checked. Examples include temporal constraints on workflows, information-flow rules, preconditions for tool calls, and invariants over resource budgets. Assurance should therefore be compositional: formally verify trusted boundaries and use empirical testing for open-ended cognition.

6.3. Sandboxing and Staged Deployment

Candidate tools, code, and policies should run first in environments with synthetic data, restricted networks, and disposable credentials. Canary deployment exposes only a limited fraction of tasks to the modified agent. Rollback must restore not only code but also memory and policy state.

6.4. Benchmarking Limitations

AgentBench evaluates agents across multiple interactive environments [35]; WebArena provides realistic web tasks [36]; and SWE-bench evaluates software-engineering problem solving [37]. These benchmarks are valuable, but governance requires additional measures:

• stability across repeated and long-horizon runs;
• safety under distribution shift;
• resistance to memory and tool poisoning;
• audit completeness;
• rollback effectiveness;
• human intervention load;
• behavior after cumulative adaptation.

6.5. Evaluation Matrix

A mature evaluation program should cross four dimensions: capability, safety, governance, and resilience. Capability asks whether the agent completes tasks. Safety asks whether it avoids prohibited outcomes. Governance asks whether actions and changes are authorized and auditable. Resilience asks whether controls continue to work under adversarial pressure and system drift.

Additional representative work supporting this discussion includes [38,39].

7. Comparative Analysis

Table 2. Representative systems and their governance implications.

Work	Adaptation mechanism	Persistent artifact	Governance strength	Main limitation
ReAct [1]	reasoning/action loop	usually none	action trace supports inspection	no persistent-change governance
Reflexion [3]	verbal feedback	episodic reflections	changes are externalized	self-critique may be incorrect
Generative Agents [5]	retrieval and reflection	memory stream	inspectable memory architecture	privacy and provenance risks
Voyager [4]	code generation and validation	skill library	executable skills can be tested	unsafe skill composition
Toolformer [2]	learned tool invocation	model behavior	tool use can be mediated	tool-selection policy is opaque
Constitutional AI [31]	principle-guided critique	trained policy	explicit normative rules	principles alone do not enforce actions
AutoGen [22]	multi-agent orchestration	conversation/workflow	modular roles and messages	no intrinsic safety guarantee
AgentBench [35]	interactive evaluation	benchmark traces	multi-environment testing	limited cumulative adaptation
WebArena [36]	realistic web tasks	benchmark trajectories	grounded action evaluation	not a governance framework
SWE-bench [37]	repository-level tasks	code patches	objective test suites	safety scope is narrow

Table 2. Representative systems and their governance implications.

Work	Adaptation mechanism	Persistent artifact	Governance strength	Main limitation
ReAct [1]	reasoning/action loop	usually none	action trace supports inspection	no persistent-change governance
Reflexion [3]	verbal feedback	episodic reflections	changes are externalized	self-critique may be incorrect
Generative Agents [5]	retrieval and reflection	memory stream	inspectable memory architecture	privacy and provenance risks
Voyager [4]	code generation and validation	skill library	executable skills can be tested	unsafe skill composition
Toolformer [2]	learned tool invocation	model behavior	tool use can be mediated	tool-selection policy is opaque
Constitutional AI [31]	principle-guided critique	trained policy	explicit normative rules	principles alone do not enforce actions
AutoGen [22]	multi-agent orchestration	conversation/workflow	modular roles and messages	no intrinsic safety guarantee
AgentBench [35]	interactive evaluation	benchmark traces	multi-environment testing	limited cumulative adaptation
WebArena [36]	realistic web tasks	benchmark trajectories	grounded action evaluation	not a governance framework
SWE-bench [37]	repository-level tasks	code patches	objective test suites	safety scope is narrow

The comparison shows a recurring asymmetry. Capability papers often provide concrete adaptation mechanisms but limited governance. Safety papers articulate constraints but may not test persistent self-modification. Framework papers support orchestration and logging but do not guarantee policy compliance. The research opportunity lies in integrating these strands into systems where every material change has a machine-readable specification, evidence record, authorization decision, and rollback path.

Three patterns emerge. First, externalized adaptation—memory, prompts, and skill libraries—is easier to inspect than parameter updates, although it remains vulnerable to poisoning and unsafe composition. Second, tool boundaries are natural enforcement points because actions become concrete there. Third, long-horizon governance is not reducible to single-episode benchmark success; it requires observing cumulative change and delayed effects.

8. Open Problems and Research Agenda

8.1. Governance-Aware Agent Architectures

Future agents should represent policies, permissions, and adaptation state as first-class architectural objects. The trusted control plane should remain outside the agent’s self-editable workspace. Research is needed on standard interfaces between an adaptive cognitive layer and a non-adaptive enforcement layer.

8.2. Behavioral Contracts

Agents need machine-readable contracts specifying allowed actions, required approvals, resource bounds, data-handling rules, and postconditions. Contracts should be checked at runtime and versioned alongside the agent. A major challenge is translating high-level goals into constraints that are neither too weak nor so restrictive that they eliminate useful autonomy.

8.3. Provenance-Aware Memory

Memory systems should support source attribution, confidence, expiry, conflict resolution, and right-to-delete semantics. Research should evaluate whether provenance improves resistance to poisoning and whether memory rollback can restore prior behavior.

8.4. Compositional Verification

A safe tool may become unsafe in combination with another tool. Verification must address sequences and compositions, not merely individual calls. Promising approaches include typed action languages, temporal logic over workflows, model checking of bounded plans, and information-flow control.

8.5. Long-Horizon and Cumulative Evaluation

Most benchmarks reset state between episodes. Self-evolving agents require persistent testbeds in which memories, skills, and policies accumulate over weeks or months. Metrics should measure drift, compounding error, delayed harms, intervention frequency, and recovery after rollback.

8.6. Multi-Agent Accountability

When a group produces an action, responsibility is distributed across planners, critics, executors, and shared memories. Needed research includes contribution tracing, secure inter-agent communication, coalition-risk detection, and governance of dynamically created agents.

8.7. Adaptive Oversight

Static approval rules cannot scale across all contexts. Oversight should adapt to estimated risk, uncertainty, novelty, reversibility, and affected stakeholders. However, the mechanism that estimates risk must itself be monitored to prevent strategic underestimation.

8.8. Institutional Integration

Technical controls must connect to organizational roles, incident response, compliance, and legal accountability. A useful deployment standard would specify minimum evidence for approving a self-modifying system, mandatory logs, rollback objectives, and thresholds for human escalation.

9. Conclusions

Self-evolving AI agents challenge a basic assumption of conventional assurance: that the evaluated system is the deployed system. Reflection, memory, tool acquisition, workflow search, and continual learning allow agents to improve after deployment, but they also create behavioral drift and new attack surfaces. The literature suggests that no single technique—alignment training, a written constitution, human approval, sandboxing, or benchmarking—is sufficient. The central synthesis of this review is a governed adaptation lifecycle in which candidate changes are proposed, tested, verified, authorized, deployed gradually, monitored, and either retained or rolled back. This lifecycle depends on separating learning from authority. Agents may generate knowledge, skills, and proposed policies, but permission to affect high-consequence environments should remain under an external control plane with least privilege, tamper-evident logs, and explicit recertification. The field now needs persistent evaluation environments, provenance-aware memory, compositional verification, behavioral contracts, and multi-agent accountability. Progress should be judged not only by whether agents become more capable, but by whether their evolution remains observable, reversible, and institutionally accountable.