Submitted:
15 November 2026
Posted:
17 November 2026
You are already at the latest version
Abstract
Embodied foundation models produce impressive demonstrations, but executing long, complex tasks depends on software outside the model, just as in the ransition from ChatGPT to Codex. We call that software the embodied agent harness and divide it into planning, skill acquisition, memory, monitoring, and self-evolution. The papers come from robotics, machine learning, and embodied-AI venues, 2022--2026. About a fifth of the citations are preprints or non-archival manuscripts, mostly for new directions. Open problems remain in efficiency, robustness, generalizability, model availability, multi-robot orchestration, and evaluation.
Keywords:
embodied agents
; foundation models
; robot software architectures
; skill acquisition
; agent memory
; execution monitoring
; continual learning
MSC: • Computer systems organization → Robotics; • Computing methodologies → Artificial intelligence; Planning and scheduling.
1. Introduction
The use of language models has moved from the model alone to the agent. A chat model such as the original ChatGPT works end to end: it takes a question and returns text, and the user copies the code, runs it, and reports the error back. Coding agents such as Codex keep the same kind of model at the center and add software around it that edits files, runs commands, reads the results, and keeps the working state across many steps. We call this software the harness. The shift matters because the harness affects results independently of the model: with the model held fixed, changing the interface through which an agent edits and searches code changes how many software issues it resolves [1].
Robots built on embodied foundation models are beginning to make the same move. Vision-language-action models such as RT-2 [2], OpenVLA [3], and [4] map camera images and a language instruction directly to robot actions, and they carry out a manipulation task from a single instruction. A long, complex task, such as tidying a kitchen the robot has not seen before, needs more than the next action. The task has to be split into steps, the robot has to remember what it has done and where things are, a failed grasp has to be noticed and retried, and what was learned should still be available next time. In its basic form, a policy that maps recent observations to the next few actions has no explicit plan, only a short history, and no record of past failures to draw on.
The agentic robot is a practical route to such tasks: the model keeps perceiving and acting, and software around it organizes the task. Hi Robot places a vision-language model over to turn an open-ended request into short commands [5]. General-purpose models are now used in the same way. In a recent evaluation, an agent built on GPT-6 Astra controls a robot through a harness that supplies its prompt, observations, and tools; the agent can control the robot directly or call a trained vision-language-action policy, and calling the policy raises overall task success [6]. Anthropic’s Frontier Red Team tested Claude on robot tasks through four control interfaces, from direct control to supervising a pretrained policy, and reports that a model’s score depends as much on the robot body and the control interface as on the model itself [7].
There is not yet an agreed account of what this software must contain. The systems above each build their own, and earlier work supplies individual pieces without describing the whole: SayCan scores how feasible each skill is before choosing one [8], Code as Policies generates programs that call robot APIs [9], ConceptGraphs keeps a graph of the objects the robot has observed [10], DoReMi checks constraints during execution [11], and SOAR improves a policy from data the robot collects on its own [12]. We bring these pieces together as the embodied agent harness and divide it into five responsibilities. Planning decomposes the task and revises the steps. Skill acquisition builds the behaviors a plan can call, by generating programs or training policies. Memory keeps the environment, the task state, and past experience available. Monitoring checks progress and triggers correction. Self-evolution turns accumulated experience into changes that later tasks use.
Existing surveys are organized differently. Surveys of language models, foundation models, and vision-language-action models in robotics classify papers by what the model does within perception, planning, and control [13,14,15], so a learned policy and a scene memory can land in the same section. Two recent surveys study the software around the model, and both define the harness as a runtime. Zeng et al. describe the system that keeps task and execution state for a base model, divided into six responsibilities [16]; Huang et al. describe a control plane that governs how skills are invoked [17]. In both, an action policy enters as a capability to be called, and Zeng et al. leave the training of the base model outside the harness. That boundary suits a coding agent, whose tools, such as a shell and a file editor, already exist. A robot’s skills do not: each must be programmed or learned for a particular body, and systems such as RECAP [18] and HERO [19] improve by retraining the policy on the robot’s own experience. We therefore treat building skills and updating learned parameters as responsibilities of the harness, next to the runtime ones.
The contributions of this survey are as follows.
- A reference architecture of the embodied agent harness with five responsibilities, in which skill acquisition and self-evolution stand alongside planning, memory, and monitoring (§ 2.2).
- A review of the methods for each responsibility, grouped into families with what each family requires and where it fails, and of how the responsibilities depend on one another (§ 3).
- A discussion of open challenges in efficiency, robustness, generalizability, model availability, multi-robot orchestration, and evaluation (§ 4).
2. Scope, Architecture, and Related Surveys
2.1. Literature Selection
Period and venues. We consider technical work released from 2022 through September 30, 2026, together with selected earlier studies that establish foundations for the methods reviewed. Seven preprints released in October 2026 are also included for the most recent evidence on failure recovery, the validation of self-improvement, and direct control by general-purpose models. We prioritize peer-reviewed AI and robotics publications, including RSS, CoRL, ICRA, IROS, RA-L, T-RO, IJRR, NeurIPS, ICML, ICLR, and CVPR. Preprints, non-archival manuscripts, and technical reports that report a clear method and supporting evidence extend the coverage to recent developments; they make up about a fifth of the cited works, mainly in self-evolution and monitoring. Two concurrent surveys of the harness are discussed in §2.3.
Inclusion criteria. A paper is included when it meets three conditions. It concerns an embodied agent and is evaluated on a physical robot or in an interactive simulation. It builds on a foundation model; methods without one, such as symbolic planners, task-specific policies, scene maps, and safety filters, are cited where they perform the same function in the harness. Its contribution lies in how a task is organized, executed, or improved, and not only in the design of a model: a paper that only proposes a model architecture or a training procedure is cited where the model serves as a skill or is updated from experience.
From topics to responsibilities. Candidates were collected under provisional topics. Titles and abstracts were screened with robotics terms and keywords for these topics, and full texts were assessed for the mechanism and its supporting evidence. The topics were revised as the papers were read, and the result is the five responsibilities of § 2.2. The electronic supplement provides screening details, classification and rating criteria, extended comparisons, and the links between reviewed mechanisms and the open challenges (§ S1–S4).
2.2. Reference Architecture
The reviewed systems differ in what they build around the model, but the same functions recur from one system to the next. We grouped these functions into five responsibilities, planning, skill acquisition, memory, monitoring, and self-evolution, and arranged them in the reference architecture of Figure 1.
Consider a robot asked to move an apple from the living room to the refrigerator in the kitchen; later sections return to this example. The online phase is the time during which the robot carries out the task. Planning breaks the task into subgoals such as grasping the apple and opening the door, checks that they are feasible, and revises them when needed. Each subgoal is carried out by a skill, a reusable behavior that the robot executes with feedback. Memory keeps the environment, the task state, and past experience available, such as where the refrigerator is [10,20]. Monitoring checks progress against requirements and constraints and triggers correction, for example after a failed grasp [11]. The offline phase is the time when the robot is not carrying out a task. Skill acquisition builds the skills, usually before the task, by generating a program or training a policy [9,21]. Self-evolution turns recorded experience, the failed grasp included, into knowledge, skills, and strategies for later tasks [12,20,22]. Two things link the phases: memory carries what is recorded online to self-evolution, and the skill library carries the skills built or revised offline back to planning.
The boundary between the phases is not strict. Code as Policies writes its program while the task runs [9], and DROC distills a human correction into reusable knowledge during the task [20]. Nor is any of the five mandatory. A vision-language-action model that carries out a single instruction is one learned skill with nothing around it; Hi Robot adds planning above such a model [5], and SayCan plans over a fixed set of pretrained skills [8].
The architecture thus has three features: (1) it consists of five responsibilities, which together cover how skills are built, how a task is carried out, and how the agent improves afterwards; (2) it works in two phases, online while the robot carries out a task and offline otherwise, with memory and the skill library connecting them; and (3) every responsibility is optional, and a system may implement only some of the five.
2.3. Related Surveys
Model-centered surveys. Most surveys of foundation models in robotics are organized around the model. Kim et al. place large language models in communication, perception, planning, and control [13]. Kawaharazuka et al. place foundation models at stages of the perception–planning–control pipeline and separately review how a robotic foundation model is trained [14]. Ma et al. review the vision-language-action model itself, from its components to the low-level policy and the high-level planner [15]. These surveys describe what the model does. Under their groupings a learned policy and a stored scene graph can fall in the same section, and the software around the model is not a subject of its own.
Harness surveys. Two concurrent surveys take that software as their subject, and both treat it as a runtime. Zeng et al. divide the runtime around a base model into six functional responsibilities and group methods by the runtime step they serve [16]. Huang et al. define a control plane between an agent’s proposals and physical execution and group methods into five mechanism families [17]. This survey differs in three ways. It divides the architecture into an online and an offline phase. Its five responsibilities include skill acquisition and self-evolution, parameter updates among them, whereas in both surveys an action policy is a capability that the runtime calls, and Zeng et al. leave the training of the base model outside the harness. And it groups the methods under each responsibility by technique.
3. The Embodied Agent Harness
3.1. Planning
Planning breaks a task into subgoals or skill calls and decides their order. An end-to-end policy has no explicit plan, which limits it on long tasks in three ways. Its errors accumulate over execution steps [23,24]. It follows short instructions, so an open-ended request has to be split into such instructions first [5]. And the order of its steps is not written down anywhere, so it can be neither checked before execution nor corrected when a step fails. An explicit plan lets a policy execute one step at a time and makes the dependencies among steps available for inspection and correction [25,26,27]. In the refrigerator task of § 2.2, plan generation orders the skills (localize, grasp, open the door, place), feasibility checking tests whether each step is reachable and free of collision, and replanning adjusts the affected steps if the apple moves or the grasp fails.
Organization. We review planning in these three stages: plan generation (§ 3.1.1), feasibility checks (§ 3.1.2), and dynamic replanning (§ 3.1.3). Figure 2 lists the method families of each stage.
3.1.1. Plan Generation
Methods differ in what produces the plan: a prompted language model, a solver that searches a hand-built model, a combination of the two, or a model trained to plan.
Methodology I. LLM task decomposition. A pretrained large language model (LLM) is prompted to write the plan, without training. LLM-Planner [28] retrieves relevant demonstrations and adds environment observations to generate subgoals; Zero-Shot Planner [29] maps each generated step to the closest admissible action. ProgPrompt [26] writes the plan as a program: the prompt declares the available actions and objects as code, and the model completes a function for the new task. General-purpose multimodal models are now used in the same way. An agent built on GPT-6 Astra is given a trained vision-language-action policy as a tool and decides when to invoke it [6]; the study traces part of the remaining failures to these decisions, since the agent invokes the policy even on tasks where the policy alone rarely succeeds, and the policy receives the fixed task instruction instead of the agent’s subgoal. Prompted plans draw on broad pretrained knowledge, but they can contain steps the robot cannot execute [29], and the models’ spatial understanding is weak [59].
Methodology II. Symbolic planning. Task and motion planning (TAMP) predates foundation models and is the basis of the hybrid methods below. PDDL-based representations [60] specify the preconditions and effects of each action, and PDDLStream [30] and COAST [31] combine task-level search with geometric sampling and motion checks. A plan found this way is consistent with the model, but the domain, the constraints, and the robot interfaces are written by hand, which limits use in unfamiliar environments [55].
Methodology III. LLM–symbolic hybrid planning. Here the language model translates the instruction and a structured algorithm constructs the plan. AutoTAMP [32] translates spatial and temporal requirements into Signal Temporal Logic for trajectory solving. OLP [33] represents object-state changes in a Functional Object-Oriented Network and converts them into PDDL subgoals. Behavior-tree approaches make the execution structure explicit: LLM-as-BT-Planner [34] combines prompting with supervised fine-tuning, and BTGenBot [35] fine-tunes a language model to generate trees. The solver answers the specification it is given, so a mistranslated instruction yields a plan for the wrong task.
6 Methodology IV. Learning-based planning. The planner is trained, and the methods differ in how much of the plan’s structure is given in advance. FILM [36] fills human-defined task templates with predicted arguments. RoboBrain [38] and RoboGPT-R1 [37] train a vision-language model to output subtasks, the first with multi-stage supervised training and the second with added reinforcement fine-tuning. Hi Robot [5] trains a vision-language model to issue short commands to a vision-language-action policy, and [39] puts both levels in one model, which predicts a subtask before it predicts actions. Hierarchical task learning [40] gives the least in advance: it infers skill boundaries, constraints, and a task graph jointly from a few demonstrations by inverse reinforcement learning.
3.1.2. Feasibility Checks
A plausible plan must also be executable. The checks below are ordered by the strength of the answer they give: an estimate from learned scores, a solution that satisfies modeled physical constraints, or a proof against a formal specification.
Methodology I. Grounded scoring. A learned model estimates whether a step will succeed in the current state. SayCan [8] multiplies an LLM’s estimate of a skill’s usefulness by a learned value function for that skill, and selects from a predefined skill library. Grounded Decoding [41] applies grounding at token generation, so that affordance and preference models shape the sequence as it is produced. Text2Motion [25] scores a whole skill sequence instead of a single step, using the skills’ value functions to test whether the sequence is geometrically feasible. Spatial methods score positions instead of skills: VoxPoser [42] composes 3D value maps for trajectory optimization, and LLM-GROP [43] samples geometric arrangements and assesses navigation and manipulation with learned feasibility maps. These scores guide selection or optimization; they estimate executability and do not prove it.
Methodology II. Constraint-based checks. Feasibility is enforced while the candidate is constructed, by solving or searching under physical constraints. PRoC3S [44] fills the open parameters of an LLM-generated plan template by continuous constraint solving that checks kinematics, collisions, and stability, and a failed check prompts regeneration. LGMCTS [45] searches over object poses with geometric patterns and motion checks, and ReKep [46] optimizes poses and trajectories under relational keypoint constraints. ASAP [47] checks candidate sequences in physics simulation and guides the search with a graph neural network. A plan that passes satisfies the constraints that were modeled, and nothing more.
Methodology III. Formal verification. The plan is checked against a temporal-logic specification. AutoTAMP [32] expresses spatial and temporal requirements as Signal Temporal Logic constraints, and Lang2LTL-2 [48] supports checking the logical consistency of a translated task specification. The result holds for the specification: it depends on a correct translation, and logical consistency alone does not establish physical feasibility.
3.1.3. Dynamic Replanning
Execution can invalidate a plan that was feasible when it was made. Replanning methods differ in what triggers the revision, a detected failure or a new observation, and in what is revised: a single step, the remaining plan, or the knowledge from which the plan was generated.
Methodology I. Reactive replanning. The plan is revised after a failure has been detected. CAPE [51] inserts corrective actions to resolve an unmet precondition, and BETR-XP-LLM [54] expands a behavior tree with the missing preconditions or parameters. Inner Monologue [27] uses detector and human feedback to guide further planning, ReplanVLM [49] uses visual feedback to revise the plan or the task code, and RoboRepair [53] generates a recovery program from the execution trace without repeating completed work. CoPAL [50] routes physical, logical, and semantic errors to the matching level of its planning hierarchy. LASP [55] revises the knowledge instead of the plan: it diagnoses the error and expands the preconditions, objects, and properties of the symbolic domain before planning again. How targeted a repair can be depends on the evidence about the failure: REFLECT [52] summarizes the execution history and explains the failure, and DoReMi [11] ties a detected constraint violation to interruption and replanning. These methods act only on failures that monitoring has detected, and the new plan has to start from the state the failed attempt left behind. Switching skills and correcting motion within a skill are treated under monitoring (§3.4.3).
Methodology II. Proactive replanning. The plan is revised when a new observation shows that it no longer fits, before a step fails. FLARE [56] revises subgoals against the current visual observation, and SayNav [57] incrementally updates a scene graph and refines a short-horizon navigation plan as new areas are observed. HUME [58] revises knowledge as well: it treats unknown object properties and action effects as hypotheses, tests them during execution, and updates its abstract world model before replanning when one fails.
8 Discussion. The three stages classify mechanisms, so a system can appear under more than one: AutoTAMP both generates and verifies a plan. Nor are all three required; explicit feasibility checking is optional. The methods’ strengths differ: AutoTAMP offers formal-model assurance, RoboBrain open-vocabulary generation, and CAPE targeted action repair (Table Section S3).
3.2. Skill Acquisition
In the refrigerator task of § 2.2, each step of the plan, such as “grasp the apple” or “open the door,” has to be turned into motor commands for a particular robot. A skill does this: it is the smallest unit that a planner can invoke, and it runs in closed loop on the robot. A skill may be a generated control program, a learned visuomotor policy, or a vision-language-action (VLA) model [2,9]. This section asks where the behavior inside a skill comes from. Recent surveys cover foundation models for robot learning and VLA architectures [15,61]; we review only how a skill is acquired.
Organization. We group the methods into three paradigms by the source of the knowledge that becomes the skill (Figure 3). Program synthesis (§ 3.2.1) writes the skill directly from the knowledge of a foundation model, without training. Learning from demonstrations (§ 3.2.2) supervises a policy with recorded robot actions. Learning from reward feedback (§ 3.2.3) obtains actions by optimizing a reward or tracking objective, either online or on logged data. The last two differ in whether the data contain the robot’s actions: a humanoid controller trained by reinforcement learning (RL) to track human motion capture belongs to the third, because the human motion defines its objective and supplies no robot actions. Improving a skill after deployment is covered under self-evolution (§ 3.5).
3.2.1. Program Synthesis
Synthesis acquires a skill without training a model. A language or vision-language model writes a program or an objective, and existing perception, planning, and control modules execute it, so a new behavior costs a prompt instead of a dataset. A generated program counts as a skill here when it reads perception and issues motion commands in closed loop; a program that only orders existing skills is a plan (§ 3.1.1). The two families differ in what the model generates: a program that runs directly, or an objective that a solver turns into motion.
Methodology I. Generating programs. The model generates code that runs on a fixed robot API. Code as Policies [9] writes Python whose branches and loops sequence perception queries and motion primitives, and VLMaps [62] applies the same idea to navigation, generating calls that resolve spatial goals for an existing navigation stack. The API sets how far the generated logic can reach: GenCHiP [63] exposes impedance parameters and force-triggered termination conditions, which brings contact-rich insertion within reach. First drafts often fail, so ASPIRE [64] revises the code from execution errors and visual outcomes over several rounds, and LRLL [65] abstracts recurring structure in successful programs into parameterized functions that later programs can call.
Methodology II. Generating objectives for a solver. The model states what the motion should achieve and leaves how to an optimizer. VoxPoser [42] writes code that builds 3D value maps of targets and obstacles, from which a motion planner synthesizes end-effector trajectories, and ReKep [46] expresses a task as relational keypoint constraints solved by hierarchical optimization; planning uses both as feasibility checks (§ 3.1.2). The objective can also target a controller: Language to Rewards [66] turns instructions into reward parameters for a model-predictive controller, on a quadruped as well as a manipulator. The solver enforces the kinematic and dynamic constraints it models, so the motion satisfies them even when the model’s numerical reasoning is weak.
Verifiability. The two families trade expressiveness against reliability. A program can express arbitrary logic, but it is limited by the primitives it can call and by the accuracy of the perception it relies on, and many of its errors appear only when the code runs on the robot. An objective delegates feasibility to the solver, but it can express only what the solver’s cost terms represent. Generated programs are used mostly in manipulation and navigation and have little role in locomotion, where joint-level balance control is not a sequence of API calls; there the model generates objectives instead.
3.2.2. Learning from Demonstrations
Imitation learning trains a policy from demonstrations. It needs no reward, and it can teach precise behaviors that are hard to specify in code. The three families differ in what stands behind the policy: demonstrations of one task, robot data from many tasks and robots, or a vision-language model (VLM) pretrained on web data.
Methodology I. Task-specific imitation. One policy is trained per task from demonstrations on the target robot. ACT [21] predicts chunks of future actions with a conditional variational autoencoder, and Diffusion Policy [67] generates action sequences by iterative denoising, which captures several valid strategies without averaging them. DP3 [68] conditions the same process on compact point-cloud features, and NoMaD [69] carries it to navigation, using goal masking to unify goal reaching and exploration in one diffusion policy. Two variations lower the cost of supervision or reuse it: NavDP [70] replaces human operators with privileged experts in simulation, and LOTUS [71] segments long demonstrations into sub-skills that can be recombined. Such a policy is limited to what its demonstrations cover.
Methodology II. Pretrained robot policies. A single policy is trained on robot data from many tasks and robots, without a VLM backbone, and then adapted to new settings. ViNT [72] trains a navigation foundation model on data from multiple robots through GNM’s normalized action space [96] and adapts it to new tasks by prompt tuning. In manipulation, Open X-Embodiment [97] pools demonstrations from many robots, Octo [73] pretrains a transformer policy on this data that can be fine-tuned to new sensors and action spaces, and RDT-1B [74] maps different robots’ actions into a unified physical action space before fine-tuning on bimanual demonstrations.
Methodology III. VLM-based policies. A pretrained VLM serves as the policy backbone, so the skill inherits semantic knowledge from web-scale data in addition to robot demonstrations. RT-2 [2] represents actions as tokens and fine-tunes on web and robot data, and OpenVLA [3] provides an open model trained on multi-robot demonstrations and adapted by low-rank fine-tuning. Later work revisits how actions are generated: [4] adds a flow-matching action expert that outputs continuous actions, FAST [75] compresses action chunks into fewer tokens, and [39] predicts a semantic subtask before it decodes actions, which places a planning step inside the model (§ 3.1.1). In navigation, NaVid [76] and NavFoM [77] map video history and instructions to actions, the latter across quadrupeds, drones, and vehicles. For legged robots, NaVILA [78] lets the VLA output mid-level commands that a separate low-level controller executes.
Limits of imitation. Moving from task-specific policies to VLM-based policies trades precision for breadth. A larger prior improves generalization to new objects and instructions, but it leaves open how actions should be represented in a language-model backbone, how long visual histories should be handled, and whether inference can meet control rates (§ 4.1). All three families share one limit: a policy trained only on demonstrations rarely exceeds its demonstrator, and it has no signal for states the demonstrations do not cover.
3.2.3. Learning from Reward Feedback
Here the robot’s actions are obtained by optimizing a reward or a tracking objective, through interaction or on logged data. This reaches behaviors that are difficult to demonstrate, such as agile locomotion, and performance beyond that of the demonstrator. A skill can be learned from scratch, or an existing pretrained policy can be fine-tuned.
Methodology I. Learning skills from scratch. Learning from scratch has to settle what to optimize and where the experience comes from. Designing rewards by hand is slow and error-prone, and foundation models can now supply them. Eureka [79] and Text2Reward [80] generate reward code from a task description and revise it using training statistics, and vision-language reward models such as RoboReward [81] score task progress directly from video. Rewards can also be learned from human preferences [82]. In each case, the model’s knowledge informs the objective, while the skill itself is still learned through RL.
Experience usually comes from simulation, where it is cheap, and the skill is then transferred to the real robot. A common recipe trains a teacher with privileged information, such as exact terrain geometry, and distills it into a student that uses only onboard sensing, as in humanoid parkour [83]. Scale also helps: PoliFormer [84] trains a transformer navigation policy with large-scale on-policy RL, and LocoFormer [85] trains across procedurally varied robots so that a single policy adapts using its interaction history during deployment. For humanoids, motion data is a common source of objectives: BeyondMimic [86] learns to track large motion-capture libraries, and BFM-Zero [87] learns a behavior space, regularized by motion data, that supports prompt-based control.
When simulation is inaccurate, as with contact-rich manipulation, learning moves to the real robot. SERL [88] combines sample-efficient off-policy RL with learned success classifiers and automatic resets, and HIL-SERL [89] adds human interventions that both correct the robot and supply training data, reaching precise and dexterous skills within hours. Offline RL needs no new interaction: ReViND [90] reuses existing navigation logs and obtains different behaviors from the same data by changing the reward.
Methodology II. Fine-tuning pretrained policies. RL also serves as a second training stage on top of the pretrained policies of § 3.2.2. PIRLNav [91] fine-tunes an imitation-pretrained navigation policy with task reward, and DPPO [92] makes diffusion policies trainable by policy gradients. The same idea reaches large models: SimpleVLA-RL [93] trains VLAs from task-success rewards, and ConRFT [94] keeps an imitation constraint while learning from real-world rewards and interventions. The prior can also be an engineered skill whose parameters or residual corrections are learned from reward [95]. Fine-tuning counts as acquisition here when it is a training stage that produces the skill; updating a deployed policy from the experience of its own tasks is self-evolution (§ 3.5.2).
8 Discussion. Synthesis suits tasks expressible through existing APIs or solver objectives, demonstrations suit behaviors easier to show than to specify, and reward feedback suits behaviors that are hard to demonstrate or must exceed the examples. The paradigms also combine: demonstrations initialize a policy and rewards refine it.
3.3. Memory
Memory stores what the robot has observed and done, and returns what is relevant to the current decision. In the refrigerator task of § 2.2, the robot has to know where the refrigerator is and where it last saw the apple.
Organization. We review memory organization, which structures what is stored (§ 3.3.1); memory maintenance, which records new observations and revises or prunes old ones (§ 3.3.2); and memory retrieval, which selects what the current decision needs (§ 3.3.3). Figure 4 lists the method families of each.
3.3.1. Memory Organization
Methods differ in the structure that organizes what is stored: spatial representations, entity graphs, or experience histories.
Methodology I. Spatial representations. VLMaps [62] associates visual-language features with mapped locations, whereas CLIP-Fields [100] learns a continuous field of semantic features. Open-Fusion [98] combines open-vocabulary semantics with 3D reconstruction, and ConvBKI [99] represents uncertainty when fusing semantic evidence. Such representations locate a previously observed object by position, but deciding whether it has moved requires linking observations from different times.
Methodology II. Entity graphs. ConceptGraphs [10] associates multiview observations with object nodes and relations; Hydra [101] and HOV-SG [102] organize places, rooms, and objects at different levels. Event-Grounding Graph [103] links observed events to objects and places, and LT-Mem [104] tracks object identity across sessions.
Methodology III. Experience histories. ReMEmbR [105] stores descriptions with time and location for later event queries, while 3D-Mem [106] retains visual snapshots of explored space. HELPER [107] and ICAL [108] preserve reusable task experience as language–program pairs or abstracted multimodal examples. Other systems encode history in learned features: 3DLLM-Mem [109] combines temporal and spatial features, HAMLET [110] aggregates moment tokens, and [111] encodes visual and proprioceptive history for action prediction. Explicit records let a system revisit a particular observation; once history is compressed into features, later decisions can use only what the encoder retained.
3.3.2. Memory Maintenance
As observations accumulate, the agent has to decide what to record, how to revise an existing estimate, and what to discard.
Methodology I. Memory recording. ConceptGraphs [10] links a new view to an existing object or creates a node, and HELPER [107] adds new examples during deployment. For feature histories, HAMLET [110] creates moment representations, and [111] samples a finite history window before encoding it; the sampling rate and the window then limit which short-lived events remain available.
Methodology II. State revision. When a new observation conflicts with a stored estimate, the system must assess whether the earlier estimate was wrong or the world has changed. FM-Fusion [112] refines object instances through probabilistic label fusion, and ConvBKI [99] updates its semantic estimates as new evidence arrives. Khronos [113] models scene changes over time, Embodied VideoAgent [118] revises object memory using egocentric video and sensor inputs, and LT-Mem [104] updates object state across sessions. If the apple moves from the living-room table to the kitchen counter, its current location should be updated without erasing the earlier observation, which still answers where it was before.
Methodology III. Memory pruning. Storage and retrieval costs grow as records accumulate. Forgetting in Robotic Episodic Long-Term Memory [114] examines both filtering at insertion and deleting stored episodes using similarity, age, or access frequency; Learning to Forget [115] adapts retention through relevance estimates and user feedback. 3D-Mem [106] filters at insertion, keeping only informative snapshots when it builds its scene memory.
3.3.3. Memory Retrieval
Retrieval selects the stored information relevant to the current decision, either by matching the content of a query or by applying constraints on attributes and relations.
Methodology I. Content matching. HELPER [107] retrieves related examples, and DROC [20] matches past corrections using textual and visual similarity. 3DLLM-Mem [109] and MemoryVLA [116] instead use learned features to read from historical representations. A similar past episode can suggest how to act, but it cannot establish where an object is now.
Methodology II. Constrained queries. Scene graphs support selection by object identity, room, or relation, as in HOV-SG [102]; SayPlan [117] searches a hierarchy for a task-relevant subgraph, and EmbodiedRAG [119] retrieves task-relevant graph context. ReMEmbR [105] uses an LLM-guided query loop when additional evidence is needed.
Discussion. A robot’s memory differs from the text memory of language-model agents [120,121] in four ways. Its records are multimodal: maps, graphs, images, and learned features are stored alongside text. They are inferred from perception, so the same object seen in two views has to be recognized as one [10] and semantic labels are fused under uncertainty [99,112]. Sensors add to it continuously, so systems select snapshots [106], bound the history window [111], or forget stored episodes [114]. And each record holds for a place and a time, so it has to be revised when the scene changes [113] and can be retrieved by where and when it was made [105].
3.4. Monitoring
Monitoring watches execution and decides whether to continue, intervene, or recover. Even a feasible plan can fail when a grasp slips [11], a policy repeats ineffective actions [122], or an otherwise valid behavior becomes unsafe [123]. In the refrigerator task of § 2.2, stopping a motion toward a nearby person is avoidance, noticing a dropped apple is detection, and calling a regrasping skill is recovery.
Organization. We review fault avoidance by where safety constraints are enforced (§ 3.4.1), fault detection by how failure is judged (§ 3.4.2), and fault recovery by what is changed to restore execution (§ 3.4.3). Figure 5 lists the method families of each.
3.4.1. Fault Avoidance
Avoidance enforces safety constraints by rejecting, modifying, or replacing what is about to be executed. It works at two levels: the behavior level decides whether an instruction, plan step, or skill may run, and the motion level controls how a permitted behavior moves the robot. Refusing a request to put a metal bowl in the microwave is behavior-level; slowing the arm when a person steps into its path is motion-level.
Methodology I. Behavior-level safety enforcement. The constraints here are rules about what the robot must not do, stated in language or in logic. ThinkSafe [123] screens proposed steps, SafePlan [124] checks requests and plan conditions, and AutoRT [150] filters generated tasks against a robot constitution. Safety Chip [125] instead translates constraints into linear temporal logic and prunes violating actions through an automaton; RoboGuard [126] grounds predefined rules and resolves conflicts between them and the proposed behavior through control synthesis. In the first group a model judges whether a behavior is safe; in the second an executable monitor enforces a stated rule, which still depends on correct rule translation and grounding. ASIMOV [151] supplies resources for assessing safety, while KnowNo [152] asks for clarification when the action is ambiguous.
Methodology II. Motion-level safety enforcement. The constraints here are physical, such as collision and contact energy, and are enforced by filtering trajectories or control commands. Gatekeeper [127] keeps a committed safe trajectory when a candidate cannot be verified; Shield-MPPI [128] combines predictive control with barrier-function shielding. UNISafe [129] uses latent-state uncertainty and reachability, so it can intervene when a predicted future is unreliable as well as when it approaches a known failure. SHIELD [131] modifies a humanoid locomotion controller’s reference commands using learned stochastic dynamics. For contact-rich manipulation, PaCo-VLA [130] constrains energy exchange through a passivity shield. Collision filters differ in the geometry they protect: Multi-Link filtering [132] covers several arm links, and VLPSA [133] additionally protects carried objects. Each guarantee holds only under the modeled constraints, dynamics, and perception, so a safe end-effector motion does not by itself make the whole body safe.
3.4.2. Fault Detection
Detection identifies failures or warning signs. Task requirement checks judge whether execution satisfies the task; statistical detectors look for anomalies or learned signs of failure.
Methodology I. Task requirement checks. Explicit checks compare observations with required conditions, such as the apple staying in the gripper while it is carried. DoReMi [11] generates constraints alongside a plan and checks them with a VLM; Code-as-Monitor [134] compiles spatiotemporal constraints into executable checks over tracked geometry. ConditionNET [135] learns preconditions and effects from demonstrations, but suspends checking during ambiguous core action phases. Other evaluators judge whether execution matches the task without listing individual conditions: AHA [136] learns success judgments and explanations from generated failures, and the VLM in Sentinel [122] checks task progress.
Methodology II. Statistical failure detection. Without failure examples, FAIL-Detect [137] compares the policy’s signals with those from successful executions, and Sentinel [122] tests consistency between overlapping action predictions. With outcome labels, SAFE [138] learns failure scores from VLA features, while Hide-and-Seek [139] localizes failure evidence from trajectory-level supervision. AutoIntervene [153] compares execution with retrieved successful examples to decide when a human takes over and when control is handed back. Such detectors err in both directions: familiar-looking behavior can still fail, and unfamiliar but successful behavior raises a false alarm, so they need calibration on data from deployment.
3.4.3. Fault Recovery
Recovery restores task progress by changing which skill runs or the actions within a skill. Changing subgoals or their dependencies is dynamic replanning (§ 3.1.3).
Methodology I. Skill-level recovery. A different skill takes control after the failure: a separate controller or a different mode of the task policy. RecoveryChaining [140] learns local recovery policies together with the handover to nominal controllers; Sequential Dexterity [142] uses transition-feasibility estimates to switch among chained policies. Adaptive behavior trees [141] state explicitly when a strategy ends and which one is tried next. Methods also differ in when the task resumes: SPR [143] switches to a learned rewind instruction for a fixed duration, whereas Recova [144] returns control once the scene is in an arrangement the task can continue from, which need not be the original one, and asks for human help if recovery fails.
Methodology II. Motion-level recovery. These methods correct grasp poses, contact directions, or trajectory segments within a skill. RACER [145] and Phoenix [146] give language or motion feedback to actors trained to follow it, which limits their use with a frozen policy. AIC MLLM [148] revises contact poses using visual and textual cues, and FailSafe [147] trains on executable corrective actions. ProbeAct [149] steers a frozen VLA’s actions away from locations of repeated failure, using a trained localization probe and a barrier-function filter.
Discussion. Avoidance prevents a fault, detection reports one, and recovery repairs it. How much of each is needed depends on the deployment. An autonomous robot needs a recovery it can execute itself. Under supervision a human can do the repair: model-based runtime monitoring [154] requests attention before a problematic execution, and confidence-aware modular recovery [155] weighs assistance against its cost. In both cases an alarm helps only if it comes while there is still time to intervene.
3.5. Self-Evolution
Self-evolution turns execution experience into improvements that last and can be reused in later tasks. Planning and monitoring act on the current task; self-evolution changes the system for later ones, so that it succeeds more often, works faster, or gains new skills. In the refrigerator task of § 2.2, failed grasps can lead to a revised procedure, a corrected explanation of the failure, or a new low-level policy, each of which can later help with other fruit and containers.
Organization. We review experience acquisition from interaction, feedback, and simulation (§ 3.5.1); capability improvement through external updates, parameter training, and selection and retention (§ 3.5.2); and recursive self-improvement (RSI), in which these steps repeat over rounds (§ 3.5.3). Figure 6 lists the method families of each.
3.5.1. Experience Acquisition
Acquisition records observations, actions, outcomes, and corrections for later reuse or training; methods differ in where the experience comes from.
Methodology I. Real-world interaction. The robot practices on its own, so fewer trials and labels have to be specified by hand. SOAR [12] uses foundation models to propose tasks and assess outcomes, then trains a goal-conditioned policy on successful, hindsight-relabeled trajectories. FIND [156] picks, from a predefined library, feasible tasks with low recent success, uses the resulting scene for further practice, and learns residual corrections around a frozen VLA; people are still needed for occasional physical recovery. RAX [157] instead keeps what it finds in online exploration as executable programs, with the coding model frozen.
Methodology II. External demonstrations and feedback. People can also supply demonstrations, action interventions, or language corrections during execution. TRANSIC [158] learns residual corrections from online human interventions to correct for differences between simulation and reality. DROC [20] accepts language corrections about task order or skill parameters, distills them into knowledge, and retrieves it for new tasks or object instances.
Methodology III. Simulation and generative experience. Simulation makes resets easy, so failures can be practiced repeatedly and data collected where it is needed. LRLL [65] abstracts self-guided simulated exploration into composable skills; EmbodiRSI [160] builds task-specific simulations to generate corrective trajectories and to collect demonstrations aimed at observed failures. RISE [159] improves a policy from imagined rollouts of a pretrained, frozen world model. The value of such experience depends on how closely scenes, contact dynamics, and predicted states match real execution.
3.5.2. Capability Improvement
To have a lasting effect, experience has to change something that is reused: external knowledge or code, learned parameters, or the set of capabilities that is kept.
Methodology I. Model-external update. These updates change knowledge or code outside the foundation model, which is not retrained. For planning, ICAL [108] refines imperfect demonstrations into generalized programs with causal explanations and subgoals, which are retrieved later. For skills, ASPIRE [64] keeps validated repair patterns as contextual guidance, which reduces time and token use over successive real-robot episodes. For memory, RoboHarness [161] compares successful and failed histories to revise its failure attributions and intervention plans, so that earlier explanations can be corrected. For monitoring, Human-Feedback Shield Synthesis [162] adjusts how conservative a shield is from human judgments, while an inner loop trains a constrained RL policy.
Methodology II. Model-internal parameter training. These updates train model weights. For planning and skill selection, LOTUS [71] trains both a meta-controller that selects skills and subgoals and the skill policies beneath it. For action execution, InSight [163] collects successful attempts at missing primitives and applies low-rank adaptation (LoRA) to the language model and the action expert, with other weights frozen. HERO [19] moves from heuristics and exemplar reuse to a learned executor by fine-tuning a -based policy on accumulated experience. In [111], coaching episodes adapt the high-level language policy that instructs the action model, while the action model is trained separately, as in [164]. For auxiliary models, RECAP [18] trains a value function whose advantage estimates condition policy training, so that the resulting policy, , learns behaviors that improve on its current performance.
Methodology III. Capability selection and retention.Validation tests whether a candidate change should be adopted. RPG [165] evaluates shared revisions across practice tasks, requires higher average success with bounded regression on each task, and retests merged changes. RoboFoundry [166] promotes a task-level repair to a general rule only after checking it against the recorded repair traces of other tasks. Versioned retention keeps a configuration that can be restored: RegenHarness [167] specifies regression checks, release at mission boundaries, and rollback versions.
Modular retention keeps earlier modules intact. TAIL [168] uses task-specific adapters; CLARE [169] adds adapters and routes among them by feature similarity, without task labels at test time; LiMoDE [170] combines frozen prior experts with trainable lifelong experts. Shared-model consolidation instead keeps all capabilities in one deployed policy: RETAIN [171] interpolates pretrained and fine-tuned weights, and When Robots Sleep [22] defines consolidation objectives on stored states that preserve earlier skills, using frozen evaluators for RL and earlier policy copies for imitation learning. These approaches pay for less forgetting with extra modules, stored experience, or consolidation effort.
3.5.3. Recursive Self-Improvement
RSI repeats these steps over rounds: what is kept from one round shapes execution, acquisition, and updating in the next. EmbodiRSI [160] feeds the policy’s failures back into collection before the next policy update, and RAX [157] accumulates its programs across exploration sessions. ENPIRE [172] extends revision to training infrastructure and algorithm code. The learning procedure can change as well: Eureka [79] revises reward code using RL feedback, and HARBOR [173] distills reward terms and failure modes from each round’s logs to revise later training.
The evidence for sustained improvement is limited. EmbodiRSI reports two updates with matched-trajectory comparisons, which support feedback-directed adaptation but not sustained improvement. SEAR [174] tests retained experience against budget-matched restarts and no-library controls, then evaluates cross-task, cross-simulator, and physical transfer. Who Improves the Robot? [175] distinguishes capability-level from meta-level updates and concludes that its strongest category, independently verified recursive gain, has not been established. What Stops Recursive Self-Improvement in Robotics? [176] reports 123 rounds without completing its refrigerator task and traces this to weak relational perception, concentration on early skills, and misleading evaluation and memory. More rounds of revision therefore do not by themselves give greater end-to-end capability.
Discussion. Self-evolution is limited by validation more than by acquisition: practice and simulation supply experience, but it is harder to show that a retained change helps later tasks without harming earlier ones. Twenty of the thirty-two works reviewed here are preprints, so their reported gains have not yet been peer reviewed.
3.6. Dependencies Among Responsibilities
The five responsibilities run together and meet at a few interfaces. Table 1 lists what each one reads and what it writes.
Three dependencies recur. (1) Planning can use only what skills expose: SayCan [8] scores only skills that carry a value estimate, and PRoC3S [44] checks only preconditions that can be expressed as its constraints. (2) Monitoring depends on the conditions that planning produces: DoReMi [11] generates constraints alongside the plan, and Code-as-Monitor [134] compiles the same kind of constraint into an executable check. (3) Self-evolution depends on both planning and monitoring. To keep a repair, it has to know that a step failed, which monitoring reports, and what the step was meant to do, which the plan records. DROC [20] learns from corrections given when a step goes wrong, and InSight [163] learns the primitives that a task needs but the robot lacks.
4. Open Challenges
The methods reviewed in § 3 were each demonstrated on their own tasks and robots. Deployment asks for more: the harness has to act in time, withstand failures and attacks, move to a different body, keep working when its model is out of reach, coordinate several robots, and be measured in a way that explains its results. These are the six challenges of this section: (a) efficiency (§ 4.1), (b) robustness (§ 4.2), (c) generalizability (§ 4.3), (d) model availability (§ 4.4), (e) multi-robot orchestration (§ 4.5), and (f) evaluation (§ 4.6). A useful test for each is to ask whether a stronger model would make it go away. For several of them the answer is no, and for two, efficiency and model availability, a larger model makes the problem harder.
4.1. Efficiency
A robot that waits for its model before every motion is slow. A robot that responds late once, at a critical moment, has failed. The first is a matter of average speed and the second of worst-case response, and the two call for different remedies.
4.1.1. Inference Latency and Execution Speed
In a synchronous inference–execution loop, the robot finishes an action segment and then waits for the next inference result, so model latency slows the task directly [3,177]. Predicting several actions at once shortens the wait, but a longer segment leaves fewer chances for a new observation to correct it [21,67]. Raising the update rate moves the cost back to inference [178,179], which may then re-process a scene that has barely changed. Fast–slow policy pairs [180] and asynchronous execution [177] absorb part of the delay. Invoking the model only when it is needed and separating high-level reasoning from fast control [181] are directions for further work. How should a harness decide that the scene has changed enough to be worth another inference?
4.1.2. Temporal Guarantees and Real-Time Scheduling
Real-time operation asks whether the worst-case response meets a deadline: a critical process that overruns even occasionally can make the robot miss the moment to avoid an obstacle [182]. Perception, control, and inference share processors and memory, so at peak load the worst case can exceed average latency by a wide margin [182,183]. Deadlines therefore call for response-time analysis, but existing analyses rest on models of task structure, execution times, and scheduling policy, whereas arrivals, execution times, and contention are stochastic in deployment [184,185]. When inference completes, the new action also waits for the current segment to finish [177]. The open problem is to schedule the whole path from perception to action so that it meets its deadlines within a stated range of hardware and load.
4.2. Robustness
A coding agent that makes a mistake can revert the file. A robot cannot undo a collision, a fall, or a misgrasp. A harness therefore has to hold up against failures that arise on their own (safety) and against inputs crafted to cause them (security).
4.2.1. Safety
Monitoring (§ 3.4) reviews how failures are avoided, detected, and recovered from; the difficulty is that detection and recovery constrain each other. An unusual execution is not always a failure, and a failed one can look normal [122,137]. Combining task checks [11,134] with statistical detectors [138] catches more failures but raises the false alarms that interrupt successful executions [122], while waiting for more evidence can cost the moment at which correction is still possible. Recovery in turn covers only the failures it was built for: recovery policies [140], skill switching [142], and feedback-driven correction [145,146] depend on dedicated recovery skills or correction training. How many interrupted successes is one caught failure worth?
4.2.2. Security
Embodied systems can be attacked through several of their inputs [186]: instructions, camera images, and training data. Jailbreak prompts make LLM-controlled robots carry out harmful physical actions [187], including a deployed commercial robot [188]. A small patch placed in the camera’s view can make a VLA policy fail [189], and a backdoor planted during training stays hidden until its trigger appears [190]. Guardrails outside the model check a proposed plan against safety rules [126]. What protects the other channels: what the camera sees, what memory stores, and what self-evolution keeps?
4.3. Generalizability
A coding agent reaches different machines through a shared operating system. Robots have no such common layer: sensing, locomotion, and control interfaces differ from body to body, and copying a program or a set of weights does not remove the differences. For a skill installed on robot A and on robot B to accomplish the same task, two things have to move: the skill, and the harness that plans, remembers, and monitors around it.
4.3.1. Transferring Skills Across Embodiments
Robot datasets differ in viewpoint, action dimensionality, and control frequency, and the same action value can produce different physical effects on different bodies [74,97]. Shared data and pretraining give transfer a starting point, but a new sensor configuration and action space usually require changes to model inputs and outputs, together with adaptation on data from the target robot [3,73]. Skills extracted from video carry task knowledge that transfers, yet executing them still depends on action policies learned on the robot itself [191]. Transfer therefore requires separating what a skill accomplishes from how a body performs it; the latter has to satisfy the kinematics and dynamics of the new body. Whether that separation belongs inside the skill or in the harness that invokes it remains unsettled.
4.3.2. Adapting the Harness to a New Robot
A transferred skill may still be called by a harness that assumes the old robot. Planning depends on which skills are available and feasible [8], program generation on the sensing and control interfaces [9], and constraint solving on a matching kinematic model [32]. A new body changes reach, grasping, and observation conditions, so goals, manipulation constraints, and completion criteria need rechecking [42,134]. Two interfaces with the same name need not have the same capability behind them. A skill that declares its inputs, preconditions, and expected effects gives the harness what it needs to recheck feasibility on the new body.
4.4. Model Availability
What can a robot still do when its model is out of reach? Many harnesses consult a foundation model while a task runs, for example to score skills at each step [8] or to check constraints [11]. The largest models do not fit on the robot and are served remotely [192,193], over wireless links that may be slow, intermittent, or absent. Efficiency (§ 4.1) concerns how quickly a reachable model responds; this challenge concerns a model that cannot be reached.
4.4.1. Dividing the Harness between Robot and Remote Model
Cloud robotics platforms move computation to remote servers [194], secure the link [195], and replicate services to mask faults [196], but replication assumes stateless services, and a harness carries plans and progress across model calls. Harness functions differ in how often they need the model: a generated program [9] or a compiled constraint automaton [125] needs it once, per-step skill scoring [8] throughout the task. Hierarchical designs that separate high-level reasoning from low-level control [5] suggest where to split. Safety enforcement has to stay onboard: a fallback to a verified backup trajectory [127] protects the robot during an outage only if it never waits on a remote call. Existing systems decide placement case by case, for example by signal strength and resources [192]; a rule that derives it from how long each function can run without the model is missing.
4.4.2. Continuing and Resuming Tasks across Connection Loss
Short delays can be absorbed by continuing the current execution (§ 4.1), but a prolonged outage can invalidate the task-level context the cloud supplies, so a local fallback policy is required [193]. An onboard model has kept a navigation mission going after the cloud connection was lost [192], but it is less capable, so the harness has to decide which steps to perform, which to defer, and when to hold the robot in a safe state. Asking a person for help [152] may fail too, since a remote operator often depends on the same network. After reconnection, the remote plan and memory may no longer match what the robot did alone. Multi-robot mapping merges maps built out of contact [197]; what would do the same for task progress and plan revisions?
4.5. Multi-Robot Orchestration
A team can stall even when every robot in it can execute its own skills, because each member has to choose actions that fit the others using only local observations. The research reviewed in § 3 concerns single robots; this challenge extends its responsibilities to teams.
Each robot observes from its own viewpoint, so members hold different views of the scene, and exchange is limited by communication bandwidth and interaction cost [197,198]. Current methods compress maps, select whom to address, or exchange only intended actions, but sending less can drop a change that a teammate’s decision depends on [198,199], and a message that arrives intact still misleads if the scene description it carries is wrong [200]. Dividing the work is constrained at once by what each body can do, by the order in which steps must occur, and by the shared workspace, so actions that are each feasible can conflict in combination [200,201,202]. Some methods schedule tasks and motion together; others start a later action asynchronously once its predecessors finish [203]. Delay and changes in capability during execution invalidate the original division, and one stalled member can block every member that depends on it [204]. The open problem is to keep a team’s plan valid as information and capabilities change: deciding which changes are worth sending and to whom, and revising one robot’s assignment without creating a new conflict for its teammates.
4.6. Evaluation
Suppose one agent completes more tasks on a benchmark than another. The result does not say whether the difference comes from the model, the harness, the sensors, or the computation budget, and it does not say where the weaker system failed.
Base models, observation conditions, and resource budgets differ across agents, so results cannot be compared directly [205,206,207]. Some studies compare component combinations through a common interface [208], but planning, memory, skills, and execution depend on one another, and the contribution of a single component remains hard to isolate. The success rate also hides the execution: a local error propagates and accumulates, so the final result alone rarely locates its source [206] and says little about recovery or resource cost [209]. Recent work scores partial completion [210] or evaluates fault diagnosis, recovery, and the cost of intervention [209,211], but the execution process and resource cost are rarely evaluated together.
An evaluation that explains its results would hold the base model, the observation conditions, and the execution interface fixed, match resource budgets [207], and substitute components to locate the source of a difference. It would also report subtask and recovery success, use execution traces to identify where errors arise and how the system recovers from them [206,209], and report inference latency, inference frequency, and computation cost [212].
5. Conclusion
Long-horizon robot tasks depend on the software around the model as well as on the model. This survey summarizes that software as the embodied agent harness, a reference architecture with five responsibilities: planning, skill acquisition, memory, monitoring, and self-evolution. The responsibilities work in an online and an offline phase, and a system may implement only some of them. For each responsibility, we review the main method families, what they require, and where they fail. The open challenges are efficiency, robustness, generalizability, model availability, multi-robot orchestration, and evaluation.
References
- Yang, J.; Jimenez, C.E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; Press, O. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Proceedings of the Advances in Neural Information Processing Systems. Curran Associates, Inc., 2024, Vol. 37, pp. 50528–50652. [CrossRef]
- Zitkovich, B.; Yu, T.; Xu, S.; Xu, P.; Xiao, T.; Xia, F.; Wu, J.; Wohlhart, P.; Welker, S.; Wahid, A.; et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. In Proceedings of the Proceedings of The 7th Conference on Robot Learning. PMLR, 2023, Vol. 229, Proceedings of Machine Learning Research, pp. 2165–2183.
- Kim, M.J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al. OpenVLA: An Open-Source Vision-Language-Action Model. In Proceedings of the Proceedings of The 8th Conference on Robot Learning, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 2679–2713.
- Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.R.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; et al. π0: A Vision-Language-Action Flow Model for General Robot Control. In Proceedings of the Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025. [CrossRef]
- Shi, L.X.; Ichter, B.; Equi, M.R.; Ke, L.; Pertsch, K.; Vuong, Q.; Tanner, J.; Walling, A.; Wang, H.; Fusai, N.; et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. In Proceedings of the Proceedings of the 42nd International Conference on Machine Learning, 2025, Vol. 267, Proceedings of Machine Learning Research, pp. 54919–54933.
- Yao, Y.; Zou, Y.; Jiao, G.; Gao, X.; Gao, N.; Wang, Y.; Chen, J.; Zhu, Z.; Li, X.; Yan, N.; et al. Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy, 2026, [2610.35832].
- Berman, S.; Ilie, M.; Deng, J.; Freeman, D. Claude Plays Robotics, 2026. Anthropic Frontier Red Team report, July 9, 2026.
- Ichter, B.; Brohan, A.; Chebotar, Y.; Finn, C.; Hausman, K.; Herzog, A.; Ho, D.; Ibarz, J.; Irpan, A.; Jang, E.; et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. In Proceedings of the Proceedings of the 6th Conference on Robot Learning, Auckland, New Zealand, 2023; Vol. 205, Proceedings of Machine Learning Research, pp. 287–318.
- Liang, J.; Huang, W.; Xia, F.; Xu, P.; Hausman, K.; Ichter, B.; Florence, P.; Zeng, A. Code as Policies: Language Model Programs for Embodied Control. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 9493–9500. [CrossRef]
- Gu, Q.; Kuwajerwala, A.; Morin, S.; Jatavallabhula, K.M.; Sen, B.; Agarwal, A.; Rivera, C.; Paul, W.; Ellis, K.; Chellappa, R.; et al. ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation, 2024, pp. 5021–5028. [CrossRef]
- Guo, Y.; Wang, Y.J.; Zha, L.; Chen, J. DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024.
- Zhou, Z.; Atreya, P.; Lee, A.; Walke, H.R.; Mees, O.; Levine, S. Autonomous Improvement of Instruction Following Skills via Foundation Models. In Proceedings of the Proceedings of The 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 4805–4825.
- Kim, Y.; Kim, D.; Choi, J.; Park, J.; Oh, N.; Park, D. A Survey on Integration of Large Language Models with Intelligent Robots. Intelligent Service Robotics 2024, 17, 1091–1107. [CrossRef]
- Kawaharazuka, K.; Matsushima, T.; Gambardella, A.; Guo, J.; Paxton, C.; Zeng, A. Real-World Robot Applications of Foundation Models: A Review. Advanced Robotics 2024, 38, 1232–1254. [CrossRef]
- Ma, Y.; Song, Z.; Zhuang, Y.; Hao, J.; King, I. A Survey on Vision-Language-Action Models for Embodied AI. IEEE Transactions on Neural Networks and Learning Systems 2026, 37, 3031–3051. [CrossRef]
- Zeng, C.; Guo, Z.; Cui, Z.; Tang, Y.; Xin, Y.; Han, Z.; Yuan, Y.; Yang, L.; Zhang, S.; Zhang, C.; et al. Where the Harness Meets the Physical World: A Survey of the Physical Agentic Harness, 2026. Preprint, version 1, October 15, 2026.
- Huang, Y.; Chen, Y.; Yang, Z.; Ding, G. Embodied Agentic Harnesses for Physical Intelligence: A Survey, 2026. engrXiv preprint, September 24, 2026, . [CrossRef]
- Physical Intelligence. π0.6*: a VLA That Learns From Experience, 2025, [2511.14759].
- Li, J.; Wang, Y.; Li, H.; Zhang, G.; Ye, Y.; Liu, Q.; Liang, H.; Lian, W. Practice Makes Policies: Bootstrapping and Consolidating Robotic Capabilities from Zero Human Demonstrations, 2026, [2607.26809].
- Zha, L.; Cui, Y.; Lin, L.H.; Kwon, M.; Arenas, M.G.; Zeng, A.; Xia, F.; Sadigh, D. Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 15172–15179. [CrossRef]
- Zhao, T.Z.; Kumar, V.; Levine, S.; Finn, C. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Proceedings of the Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. [CrossRef]
- Jayasinghe, N.; Gontero, D.; Trivedi, A.R. When Robots Sleep: Offline Skill Consolidation for Shared-Policy Robot Learning. In Proceedings of the 10th Annual Conference on Robot Learning, 2026.
- Mees, O.; Hermann, L.; Rosete-Beas, E.; Burgard, W. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. IEEE Robotics and Automation Letters 2022, [2112.03227].
- Ross, S.; Gordon, G.; Bagnell, D. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Proceedings of the Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 2011; Vol. 15, Proceedings of Machine Learning Research, pp. 627–635.
- Lin, K.; Agia, C.; Migimatsu, T.; Pavone, M.; Bohg, J. Text2Motion: From Natural Language Instructions to Feasible Plans. Autonomous Robots 2023, 47, 1345–1365. [CrossRef]
- Singh, I.; Blukis, V.; Mousavian, A.; Goyal, A.; Xu, D.; Tremblay, J.; Fox, D.; Thomason, J.; Garg, A. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11523–11530. [CrossRef]
- Huang, W.; Xia, F.; Xiao, T.; Chan, H.; Liang, J.; Florence, P.; Zeng, A.; Tompson, J.; Mordatch, I.; Chebotar, Y.; et al. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Proceedings of the Proceedings of the 6th Conference on Robot Learning, Auckland, New Zealand, 2023; Vol. 205, Proceedings of Machine Learning Research, pp. 1769–1782.
- Song, C.H.; Wu, J.; Washington, C.; Sadler, B.M.; Chao, W.L.; Su, Y. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023, pp. 2998–3009.
- Huang, W.; Abbeel, P.; Pathak, D.; Mordatch, I. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In Proceedings of the Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022, Vol. 162, Proceedings of Machine Learning Research, pp. 9118–9147.
- Garrett, C.R.; Lozano-Pérez, T.; Kaelbling, L.P. PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning. In Proceedings of the Proceedings of the International Conference on Automated Planning and Scheduling. AAAI Press, 2020, Vol. 30, pp. 440–448. [CrossRef]
- Vu, B.; Migimatsu, T.; Bohg, J. COAST: Constraints and Streams for Task and Motion Planning. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 14875–14881. [CrossRef]
- Chen, Y.; Arkin, J.; Dawson, C.; Zhang, Y.; Roy, N.; Fan, C. AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 6695–6702.
- Paulius, D.; Agostini, A.; Quartey, B.; Konidaris, G. Bootstrapping Object-Level Planning with Large Language Models. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 16233–16239. [CrossRef]
- Ao, J.; Wu, F.; Wu, Y.; Swikir, A.; Haddadin, S. LLM-as-BT-Planner: Leveraging LLMs for Behavior Tree Generation in Robot Task Planning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Atlanta, GA, USA, 2025; pp. 1233–1239. [CrossRef]
- Izzo, R.A.; Bardaro, G.; Matteucci, M. BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 9684–9690. [CrossRef]
- Min, S.Y.; Chaplot, D.S.; Ravikumar, P.; Bisk, Y.; Salakhutdinov, R. FILM: Following Instructions in Language with Modular Methods. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
- Liu, J.; Nie, B.; Li, B.; Chen, Y.; Wang, Y.; He, S.; Li, H. RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning. In Proceedings of the Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 2026, pp. 2827–2837. [CrossRef]
- Ji, Y.; Tan, H.; Shi, J.; Hao, X.; Zhang, Y.; Zhang, H.; Wang, P.; Zhao, M.; Mu, Y.; An, P.; et al. RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025, pp. 1724–1734.
- Black, K.; Brown, N.; Darpinian, J.; Dhabalia, K.; Driess, D.; Esmail, A.; Equi, M.R.; Finn, C.; Fusai, N.; Galliker, M.Y.; et al. π0.5: A Vision-Language-Action Model with Open-World Generalization. In Proceedings of the Proceedings of The 9th Conference on Robot Learning. PMLR, September 2025, Vol. 305, Proceedings of Machine Learning Research, pp. 17–40.
- Willibald, C.; Lee, D. Hierarchical Task Decomposition for Execution Monitoring and Error Recovery: Understanding the Rationale Behind Task Demonstrations. The International Journal of Robotics Research 2026, 45, 369–396. [CrossRef]
- Huang, W.; Xia, F.; Shah, D.; Driess, D.; Zeng, A.; Lu, Y.; Florence, P.; Mordatch, I.; Levine, S.; Hausman, K.; et al. Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents. In Proceedings of the Advances in Neural Information Processing Systems, 2023, Vol. 36, pp. 59636–59661. [CrossRef]
- Huang, W.; Wang, C.; Zhang, R.; Li, Y.; Wu, J.; Fei-Fei, L. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. In Proceedings of the Proceedings of The 7th Conference on Robot Learning. PMLR, 2023, Vol. 229, Proceedings of Machine Learning Research, pp. 540–562.
- Zhang, X.; Ding, Y.; Hayamizu, Y.; Altaweel, Z.; Zhu, Y.; Zhu, Y.; Stone, P.; Paxton, C.; Zhang, S. LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models. The International Journal of Robotics Research 2025. [CrossRef]
- Curtis, A.; Kumar, N.; Cao, J.; Lozano-Pérez, T.; Kaelbling, L.P. Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction. In Proceedings of the Proceedings of the 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 1362–1383.
- Chang, H.; Gao, K.; Boyalakuntla, K.; Lee, A.; Huang, B.; Yu, J.; Boularias, A. LGMCTS: Language-Guided Monte-Carlo Tree Search for Executable Semantic Object Rearrangement. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 13607–13612. [CrossRef]
- Huang, W.; Wang, C.; Li, Y.; Zhang, R.; Fei-Fei, L. ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. In Proceedings of the Proceedings of the 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 4573–4602. Conference held in 2024; proceedings published in 2025.
- Tian, Y.; Willis, K.D.D.; Al Omari, B.; Luo, J.; Ma, P.; Li, Y.; Javid, F.; Gu, E.; Jacob, J.; Sueda, S.; et al. ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical Feasibility. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 4380–4386. [CrossRef]
- Liu, J.X.; Shah, A.; Konidaris, G.; Tellex, S.; Paulius, D. Lang2LTL-2: Grounding Spatiotemporal Navigation Commands Using Large Language and Vision-Language Models. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 2325–2332. [CrossRef]
- Mei, A.; Zhu, G.N.; Zhang, H.; Gan, Z. ReplanVLM: Replanning Robotic Tasks With Visual Language Models. IEEE Robotics and Automation Letters 2024, 9, 10201–10208. [CrossRef]
- Joublin, F.; Ceravola, A.; Smirnov, P.; Ocker, F.; Deigmoeller, J.; Belardinelli, A.; Wang, C.; Hasler, S.; Tanneberg, D.; Gienger, M. CoPAL: Corrective Planning of Robot Actions with Large Language Models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 2024; pp. 8664–8670. [CrossRef]
- Raman, S.S.; Cohen, V.; Idrees, I.; Rosen, E.; Mooney, R.; Tellex, S.; Paulius, D. CAPE: Corrective Actions from Precondition Errors using Large Language Models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 2024; pp. 14070–14077. [CrossRef]
- Liu, Z.; Bahety, A.; Song, S. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. In Proceedings of the Proceedings of the 7th Conference on Robot Learning, Atlanta, GA, USA, 2023; Vol. 229, Proceedings of Machine Learning Research, pp. 3468–3484.
- Schlesinger, C.; Guha, A.; Biswas, J. Creating and Repairing Robot Programs in Open-World Domains, 2024, [2410.18893].
- Styrud, J.; Iovino, M.; Norrlöf, M.; Björkman, M.; Smith, C. Automatic Behavior Tree Expansion with LLMs for Robotic Manipulation. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 1225–1232. [CrossRef]
- Chen, G.; Yang, L.; Jia, R.; Hu, Z.; Chen, Y.; Zhang, W.; Wang, W.; Pan, J. Language-Augmented Symbolic Planner for Open-World Task Planning. In Proceedings of the Robotics: Science and Systems XX, 2024, Vol. 20. [CrossRef]
- Kim, T.; Kim, B.; Choi, J. Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples. In Proceedings of the Proceedings of the AAAI Conference on Artificial Intelligence, 2025, Vol. 39, pp. 4329–4337. [CrossRef]
- Rajvanshi, A.; Sikka, K.; Lin, X.; Lee, B.; Chiu, H.P.; Velasquez, A. SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments. In Proceedings of the Proceedings of the International Conference on Automated Planning and Scheduling, 2024, Vol. 34, pp. 464–474. [CrossRef]
- Xiao, A.; Zhang, H.; Hu, T.; Hsu, D. Hypothesis-driven Model Expansion under Uncertainty for Open-World Robot Planning. In Proceedings of the Robotics: Science and Systems XXII, 2026, Vol. 22. [CrossRef]
- Yang, J.; Yang, S.; Gupta, A.W.; Han, R.; Fei-Fei, L.; Xie, S. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025, pp. 10632–10643.
- McDermott, D.; Ghallab, M.; Howe, A.; Knoblock, C.; Ram, A.; Veloso, M.; Weld, D.; Wilkins, D. PDDL—The Planning Domain Definition Language. Technical Report CVC TR-98-003/DCS TR-1165, Yale Center for Computational Vision and Control, 1998.
- Firoozi, R.; Tucker, J.; Tian, S.; Majumdar, A.; Sun, J.; Liu, W.; Zhu, Y.; Song, S.; Kapoor, A.; Hausman, K.; et al. Foundation Models in Robotics: Applications, Challenges, and the Future. The International Journal of Robotics Research 2025, 44, 701–739. [CrossRef]
- Huang, C.; Mees, O.; Zeng, A.; Burgard, W. Visual Language Maps for Robot Navigation. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation, 2023, pp. 10608–10615. [CrossRef]
- Burns, K.; Jain, A.; Go, K.; Xia, F.; Stark, M.; Schaal, S.; Hausman, K. Genchip: Generating robot policy code for high-precision and contact-rich manipulation tasks. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 9596–9603.
- Lu, R.; Wu, Y.; Kou, E.; Lin, H.; Fu, L.; Xiao, W.; Mandlekar, A.; Xu, Y.; Shi, G.; Goldberg, K.; et al. ASPIRE: Agentic /Skills Discovery for Robotics, 2026, [2607.00272]. CoRL 2026 Agentic Robotics Workshop accepted paper; author manuscript, version 2.
- Tziafas, G.; Kasaei, H. Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 515–522. [CrossRef]
- Yu, W.; Gileadi, N.; Fu, C.; Kirmani, S.; Lee, K.H.; Arenas, M.G.; Chiang, H.T.L.; Erez, T.; Hasenclever, L.; Humplik, J.; et al. Language to Rewards for Robotic Skill Synthesis. In Proceedings of the Proceedings of the 7th Conference on Robot Learning. PMLR, 2023, Vol. 229, Proceedings of Machine Learning Research, pp. 374–404.
- Chi, C.; Feng, S.; Du, Y.; Xu, Z.; Cousineau, E.; Burchfiel, B.C.M.; Song, S. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of the Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. [CrossRef]
- Ze, Y.; Zhang, G.; Zhang, K.; Hu, C.; Wang, M.; Xu, H. 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. In Proceedings of the Proceedings of Robotics: Science and Systems, 2024. [CrossRef]
- Sridhar, A.; Shah, D.; Glossop, C.; Levine, S. NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 63–70. [CrossRef]
- Cai, W.; Peng, J.; Yang, Y.; Zhang, Y.; Wei, M.; Wang, H.; Chen, Y.; Wang, T.; Pang, J. NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance. In Proceedings of the 2026 IEEE International Conference on Robotics and Automation (ICRA), 2026, pp. 5233–5240. [CrossRef]
- Wan, W.; Zhu, Y.; Shah, R.; Zhu, Y. LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024. [CrossRef]
- Shah, D.; Sridhar, A.; Dashora, N.; Stachowicz, K.; Black, K.; Hirose, N.; Levine, S. ViNT: A Foundation Model for Visual Navigation. In Proceedings of the Proceedings of The 7th Conference on Robot Learning. PMLR, 2023, Vol. 229, Proceedings of Machine Learning Research, pp. 711–733.
- Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; Luo, J.; et al. Octo: An Open-Source Generalist Robot Policy. In Proceedings of the Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. [CrossRef]
- Liu, S.; Wu, L.; Li, B.; Tan, H.; Chen, H.; Wang, Z.; Xu, K.; Su, H.; Zhu, J. RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation. In Proceedings of the International Conference on Learning Representations, 2025, Vol. 2025, pp. 29982–30009.
- Pertsch, K.; Stachowicz, K.; Ichter, B.; Driess, D.; Nair, S.; Vuong, Q.; Mees, O.; Finn, C.; Levine, S. FAST: Efficient Action Tokenization for Vision-Language-Action Models. In Proceedings of the Proceedings of Robotics: Science and Systems, 2025. [CrossRef]
- Zhang, J.; Wang, K.; Xu, R.; Zhou, G.; Hong, Y.; Fang, X.; Wu, Q.; Zhang, Z.; Wang, H. NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation. In Proceedings of the Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024; Robotics: Science and Systems. [CrossRef]
- Zhang, J.; Li, A.; Qi, Y.; Li, M.; Liu, J.; Wang, S.; Liu, H.; Zhou, G.; Wu, Y.; Li, X.; et al. Embodied Navigation Foundation Model. In Proceedings of the Proceedings of the 14th International Conference on Learning Representations (ICLR), 2026.
- Cheng, A.C.; Ji, Y.; Yang, Z.; Gongye, Z.; Zou, X.; Kautz, J.; Biyik, E.; Yin, H.; Liu, S.; Wang, X. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. In Proceedings of the Proceedings of Robotics: Science and Systems, 2025. [CrossRef]
- Ma, Y.J.; Liang, W.; Wang, G.; Huang, D.A.; Bastani, O.; Jayaraman, D.; Zhu, Y.; Fan, L.; Anandkumar, A. Eureka: Human-Level Reward Design via Coding Large Language Models. In Proceedings of the The Twelfth International Conference on Learning Representations, Vienna, Austria, 2024.
- Xie, T.; Zhao, S.; Wu, C.H.; Liu, Y.; Luo, Q.; Zhong, V.; Yang, Y.; Yu, T. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. In Proceedings of the International Conference on Learning Representations, 2024, Vol. 2024, pp. 35663–35699.
- Lee, T.; Wagenmaker, A.; Pertsch, K.; Liang, P.; Levine, S.; Finn, C. Roboreward: General-purpose vision-language reward models for robotics. arXiv preprint arXiv:2601.00675 2026.
- Seneviratne, G.; An, J.; Ellahy, S.; Weerakoon, K.; Elnoor, M.B.; Kannan, J.D.; Sunil, A.T.; Manocha, D. HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation. In Proceedings of the Proceedings of The 9th Conference on Robot Learning. PMLR, 2025, Vol. 305, Proceedings of Machine Learning Research, pp. 3267–3284.
- Zhuang, Z.; Yao, S.; Zhao, H. Humanoid Parkour Learning. In Proceedings of the Proceedings of the 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 1975–1991.
- Zeng, K.H.; Zhang, Z.; Ehsani, K.; Hendrix, R.; Salvador, J.; Herrasti, A.; Girshick, R.; Kembhavi, A.; Weihs, L. PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators. In Proceedings of the Proceedings of The 8th Conference on Robot Learning. PMLR, 2024, Vol. 270, Proceedings of Machine Learning Research, pp. 408–432.
- Liu, M.; Pathak, D.; Agarwal, A. LocoFormer: Generalist Locomotion via Long-context Adaptation. In Proceedings of the Proceedings of the 9th Conference on Robot Learning. PMLR, 2025, Vol. 305, Proceedings of Machine Learning Research, pp. 532–546.
- Liao, Q.; Truong, T.E.; Huang, X.; Gao, Y.; Tevet, G.; Sreenath, K.; Liu, C.K. BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion. Science Robotics 2026, 11. [CrossRef]
- Li, Y.; Luo, Z.; Zhang, T.; Dai, C.; Kanervisto, A.; Tirinzoni, A.; Weng, H.; Kitani, K.; Guzek, M.; Touati, A.; et al. BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning. In Proceedings of the International Conference on Learning Representations, 2026.
- Luo, J.; Hu, Z.; Xu, C.; Tan, Y.L.; Berg, J.; Sharma, A.; Schaal, S.; Finn, C.; Gupta, A.; Levine, S. Serl: A software suite for sample-efficient robotic reinforcement learning. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16961–16969.
- Luo, J.; Xu, C.; Wu, J.; Levine, S. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics 2025, 10, eads5033.
- Shah, D.; Bhorkar, A.; Leen, H.; Kostrikov, I.; Rhinehart, N.; Levine, S. Offline Reinforcement Learning for Visual Navigation. In Proceedings of the Proceedings of The 6th Conference on Robot Learning. PMLR, 2022, Vol. 205, Proceedings of Machine Learning Research, pp. 44–54.
- Ramrakhya, R.; Batra, D.; Wijmans, E.; Das, A. PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 17896–17906. [CrossRef]
- Ren, A.Z.; Lidard, J.; Ankile, L.L.; Simeonov, A.; Agrawal, P.; Majumdar, A.; Burchfiel, B.; Dai, H.; Simchowitz, M. Diffusion Policy Policy Optimization. In Proceedings of the International Conference on Learning Representations, 2025, Vol. 2025, pp. 77288–77329.
- Li, H.; Zuo, Y.; Yu, J.; Zhang, Y.; Yang, Z.; Zhang, K.; Zhu, X.; Zhang, Y.; Chen, T.; Cui, G.; et al. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. In Proceedings of the International Conference on Learning Representations, 2026, Vol. 2026, pp. 125846–125869.
- Chen, Y.; Tian, S.; Liu, S.; Zhou, Y.; Li, H.; Zhao, D. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. In Proceedings of the Proceedings of Robotics: Science and Systems, 2025. [CrossRef]
- Wu, Z.; Lian, W.; Wang, C.; Li, M.; Schaal, S.; Tomizuka, M. Prim-lafd: A framework to learn and adapt primitive-based skills from demonstrations for insertion tasks. IFAC-PapersOnLine 2023, 56, 4120–4125.
- Shah, D.; Sridhar, A.; Bhorkar, A.; Hirose, N.; Levine, S. GNM: A General Navigation Model to Drive Any Robot. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 7226–7233. [CrossRef]
- O’Neill, A.; Rehman, A.; Maddukuri, A.; Gupta, A.; Padalkar, A.; Lee, A.; et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6892–6903. [CrossRef]
- Yamazaki, K.; Hanyu, T.; Vo, K.; Pham, T.; Tran, M.; Doretto, G.; Nguyen, A.; Le, N. Open-Fusion: Real-Time Open-Vocabulary 3D Mapping and Queryable Scene Representation. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 9411–9417. [CrossRef]
- Wilson, J.; Fu, Y.; Friesen, J.; Ewen, P.; Capodieci, A.; Jayakumar, P.; Barton, K.; Ghaffari, M. ConvBKI: Real-Time Probabilistic Semantic Mapping Network With Quantifiable Uncertainty. IEEE Transactions on Robotics 2024, 40, 4648–4667. [CrossRef]
- Shafiullah, N.M.M.; Paxton, C.; Pinto, L.; Chintala, S.; Szlam, A. CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory. In Proceedings of the Proceedings of Robotics: Science and Systems, 2023. [CrossRef]
- Hughes, N.; Chang, Y.; Carlone, L. Hydra: A Real-Time Spatial Perception System for 3D Scene Graph Construction and Optimization. In Proceedings of the Proceedings of Robotics: Science and Systems, 2022. [CrossRef]
- Werby, A.; Huang, C.; Büchner, M.; Valada, A.; Burgard, W. Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation. In Proceedings of the Proceedings of Robotics: Science and Systems, 2024. [CrossRef]
- Nguyen, P.; Verdoja, F.; Kyrki, V. Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations. IEEE Robotics and Automation Letters 2026, 11, 5286–5293. [CrossRef]
- Lee, Y.; Ju, H.; Kim, G. LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understanding. In Proceedings of the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026. Accepted at IROS 2026, as reported by the authors; author manuscript.
- Anwar, A.; Welsh, J.; Biswas, J.; Pouya, S.; Chang, Y. ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation, 2025, pp. 2838–2845.
- Yang, Y.; Yang, H.; Zhou, J.; Chen, P.; Zhang, H.; Du, Y.; Gan, C. 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 17294–17303.
- Sarch, G.; Wu, Y.; Tarr, M.; Fragkiadaki, K. Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 2023; pp. 3468–3500. [CrossRef]
- Sarch, G.; Jang, L.; Tarr, M.J.; Cohen, W.W.; Marino, K.; Fragkiadaki, K. VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought. In Proceedings of the Advances in Neural Information Processing Systems, 2024, Vol. 37, pp. 75942–75985.
- Hu, W.; Hong, Y.; Wang, Y.; Gao, L.; Wei, Z.; Yao, X.; Peng, N.; Bitton, Y.; Szpektor, I.; Chang, K.W. 3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model. In Proceedings of the Advances in Neural Information Processing Systems, 2025.
- Koo, M.; Choi, D.; Kim, T.; Lee, K.; Kim, C.; Seo, Y.; Shin, J. HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy. In Proceedings of the The Fourteenth International Conference on Learning Representations, 2026.
- Physical Intelligence. π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities. arXiv preprint arXiv:2604.15483 2026. Technical report: https://www.pi.website/download/pi07.pdf.
- Liu, C.; Wang, K.; Shi, J.; Qiao, Z.; Shen, S. FM-Fusion: Instance-Aware Semantic Mapping Boosted by Vision-Language Foundation Models. IEEE Robotics and Automation Letters 2024, 9, 2232–2239. [CrossRef]
- Schmid, L.; Abate, M.; Chang, Y.; Carlone, L. Khronos: A Unified Approach for Spatio-Temporal Metric-Semantic SLAM in Dynamic Environments. In Proceedings of the Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. [CrossRef]
- Plewnia, J.; Peller-Konrad, F.; Asfour, T. Forgetting in Robotic Episodic Long-Term Memory. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation, 2024, pp. 6711–6717. [CrossRef]
- Bärmann, L.; Plewnia, J.; Waibel, A.; Asfour, T. Learning to Forget – Hierarchical Episodic Memory for Lifelong Robot Deployment. arXiv:2604.11306, 2026.
- Shi, H.; Xie, B.; Liu, Y.; Sun, L.; Liu, F.; Wang, T.; Zhou, E.; Fan, H.; Zhang, X.; Huang, G. MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation. In Proceedings of the The Fourteenth International Conference on Learning Representations, 2026.
- Rana, K.; Haviland, J.; Garg, S.; Abou-Chakra, J.; Reid, I.; Sünderhauf, N. SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. In Proceedings of the Proceedings of the 7th Conference on Robot Learning, 2023, Vol. 229, Proceedings of Machine Learning Research, pp. 23–72.
- Fan, Y.; Ma, X.; Su, R.; Guo, J.; Wu, R.; Chen, X.; Li, Q. Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Honolulu, HI, USA, 2025; pp. 6342–6352.
- Booker, M.; Byrd, G.; Kemp, B.; Schmidt, A.; Rivera, C. EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planning. arXiv:2410.23968, 2024.
- Park, J.S.; O’Brien, J.C.; Cai, C.J.; Morris, M.R.; Liang, P.; Bernstein, M.S. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, San Francisco, CA, USA, 2023; pp. 1–22. [CrossRef]
- Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; Yao, S. Reflexion: Language Agents with Verbal Reinforcement Learning. In Proceedings of the Advances in Neural Information Processing Systems, Red Hook, NY, USA, 2023; Vol. 36, pp. 8634–8652.
- Agia, C.; Sinha, R.; Yang, J.; Cao, Z.; Antonova, R.; Pavone, M.; Bohg, J. Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress. In Proceedings of the Proceedings of the 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 689–723.
- Yin, S.; Pang, X.; Ding, Y.; Chen, M.; Bi, Y.; Xiong, Y.; Huang, W.; Xiang, Z.; Shao, J.; Chen, S. SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents, 2024, [2412.13178]. Version 5, revised October 2025.
- Obi, I.; Venkatesh, V.L.N.; Wang, W.; Wang, R.; Suh, D.; Amosa, T.I.; Jo, W.; Min, B.C. SafePlan: Leveraging Formal Logic and Chain-of-Thought Reasoning for Enhanced Safety in LLM-based Robotic Task Planning, 2025, [2503.06892].
- Yang, Z.; Raman, S.S.; Shah, A.; Tellex, S. Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024.
- Ravichandran, Z.; Robey, A.; Kumar, V.; Pappas, G.J.; Hassani, H. Safety Guardrails for LLM-Enabled Robots. IEEE Robotics and Automation Letters 2026.
- Agrawal, D.R.; Chen, R.; Panagou, D. gatekeeper: Online Safety Verification and Control for Nonlinear Systems in Dynamic Environments. IEEE Transactions on Robotics 2024, 40, 4358–4375. [CrossRef]
- Yin, J.; Dawson, C.; Fan, C.; Tsiotras, P. Shield Model Predictive Path Integral: A Computationally Efficient Robust MPC Method Using Control Barrier Functions. IEEE Robotics and Automation Letters 2023, 8, 7106–7113. [CrossRef]
- Seo, J.; Nakamura, K.; Bajcsy, A. Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures. In Proceedings of the Proceedings of the 9th Conference on Robot Learning. PMLR, 2025, Vol. 305, Proceedings of Machine Learning Research, pp. 4442–4472.
- Cao, H.; Li, Z.; You, Z. PaCo-VLA: Passivity-Shielded Compliance Prior for Contact-Rich Vision-Language-Action Manipulation, 2026, [2606.00515]. Version 2, September 2026.
- Yang, L.; Werner, B.; Cosner, R.K.; Fridovich-Keil, D.; Culbertson, P.; Ames, A.D. SHIELD: Safety on Humanoids via CBFs In Expectation on Learned Dynamics. In Proceedings of the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. [CrossRef]
- Agarwal, Y.; Raghunathan, V. Multi-Link Safety Filtering for VLA Policies Around Moving Hazards, 2026, [2609.40007].
- Wilkinson, M.; Fourney, E.; Burdick, J.W.; Ames, A.D. VLPSA: Vision-Language-Poisson-Safe Actions for Full-Body Safety of Learned Policies, 2026, [2609.22462].
- Zhou, E.; Su, Q.; Chi, C.; Zhang, Z.; Wang, Z.; Huang, T.; Sheng, L.; Wang, H. Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
- Sliwowski, D.; Lee, D. ConditionNET: Learning Preconditions and Effects for Execution Monitoring. IEEE Robotics and Automation Letters 2025, 10, 1337–1344. [CrossRef]
- Duan, J.; Pumacay, W.; Kumar, N.; Wang, Y.R.; Tian, S.; Yuan, W.; Krishna, R.; Fox, D.; Mandlekar, A.; Guo, Y. AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation. In Proceedings of the The Thirteenth International Conference on Learning Representations, Singapore, 2025.
- Xu, C.; Nguyen, T.K.; Dixon, E.; Rodriguez, C.; Miller, P.; Lee, R.; Shah, P.; Ambrus, R.A.; Nishimura, H.; Itkina, M. Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies. In Proceedings of the Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025. [CrossRef]
- Gu, Q.; Ju, Y.; Sun, S.; Gilitschenski, I.; Nishimura, H.; Itkina, M.; Shkurti, F. SAFE: Multitask Failure Detection for Vision-Language-Action Models. In Proceedings of the Advances in Neural Information Processing Systems, 2025, Vol. 38. [CrossRef]
- Park, S.; Li, W.; Oh, C.; Yeh, S.; Kira, Z.; Hagenow, M.; Li, S. Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring. In Proceedings of the Advances in Neural Information Processing Systems, 2026. Accepted at NeurIPS 2026; author manuscript, arXiv:2605.30834v2.
- Vats, S.; Jha, D.K.; Likhachev, M.; Kroemer, O.; Romeres, D. RecoveryChaining: Learning Local Recovery Policies for Robust Manipulation. In Proceedings of the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025. [CrossRef]
- Cloete, J.; Merkt, W.; Havoutis, I. Adaptive Manipulation using Behavior Trees. In Proceedings of the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025, pp. 19031–19038. [CrossRef]
- Chen, Y.; Wang, C.; Fei-Fei, L.; Liu, K. Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation. In Proceedings of the Proceedings of The 7th Conference on Robot Learning. PMLR, 2023, Vol. 229, Proceedings of Machine Learning Research, pp. 3809–3829.
- Dai, T.; Han, M.; Du, T.; Liu, Z.; Zhang, Z.; Li, Z.; Khan, S.; Yu, J.; Chang, X. See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation, 2026, [2603.09292]. Version 2.
- Liu, I.; Cheng, A.C.; Bjorck, J.; Yu, Z.; Yin, H.; Kautz, J.; Fan, L.; Zhu, Y.; Liu, S. Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation, 2026, [2610.01178].
- Dai, Y.; Lee, J.; Fazeli, N.; Chai, J.Y. RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025.
- Xia, W.; Feng, R.; Wang, D.; Hu, D. Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
- Lin, Z.; Duan, J.; Fang, H.; Fox, D.; Krishna, R.; Tan, C.; Wen, B. FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026. Accepted paper; author manuscript, arXiv:2510.01642v4.
- Xiong, C.; Shen, C.; Li, X.; Zhou, K.; Liu, J.; Wang, R.; Dong, H. Autonomous Interactive Correction MLLM for Robust Robotic Manipulation. In Proceedings of the Proceedings of the 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 3139–3156.
- Zhang, F.; Park, S.; Mirzasoleiman, B.; Talebi, S.; Sehatbakhsh, N. ProbeAct: Probe-Guided Training-Free Failure Recovery in Vision-Language-Action Models, 2026, [2606.09740v1].
- Ahn, M.; Dwibedi, D.; Finn, C.; Arenas, M.G.; Gopalakrishnan, K.; Hausman, K.; Ichter, B.; Irpan, A.; Joshi, N.; Julian, R.; et al. AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents, 2024, [2401.12963]. Presented at the ICRA 2024 VLMNM Workshop; non-archival manuscript.
- Sermanet, P.; Majumdar, A.; Irpan, A.; Kalashnikov, D.; Sindhwani, V. Generating Robot Constitutions & Benchmarks for Semantic Safety. In Proceedings of the Proceedings of the 9th Conference on Robot Learning, 2025, [2503.08663].
- Ren, A.Z.; Dixit, A.; Bodrova, A.; Singh, S.; Tu, S.; Brown, N.; Xu, P.; Takayama, L.; Xia, F.; Varley, J.; et al. Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners. In Proceedings of the Proceedings of the 7th Conference on Robot Learning, Atlanta, GA, USA, 2023; Vol. 229, Proceedings of Machine Learning Research, pp. 661–682.
- Tang, J.; Zhi, W. AutoIntervene: Calibrated Intervention for Action-Chunking Imitation Learning Policies, 2026, [2608.07065v1].
- Liu, H.; Dass, S.; Martín-Martín, R.; Zhu, Y. Model-Based Runtime Monitoring with Interactive Imitation Learning. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.
- Banerjee, R.; Palempalli, K.; Yang, B.; Fang, J.; Abdullah, A.; Silver, T.; Dean, S.; Bhattacharjee, T. A Human-in-the-Loop Confidence-Aware Failure Recovery Framework for Modular Robot Policies. In Proceedings of the Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2026.
- Fang, Y.; Li, Z.; Tong, H.; Liu, P.; Chalvatzaki, G. Find Something You Can’t Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models, 2026, [2609.32069].
- Huang, R.; Zhang, X.; Chen, H.; Li, Y.; Wang, Y.J.; Lyu, B.; Ye, H.; Hong, C.; Wang, Y.; Su, C.; et al. Scaling Online Exploration for Robot Agents, 2026, [2610.22496]. CoRL 2026 Late-Breaking Results; non-archival manuscript.
- Jiang, Y.; Wang, C.; Zhang, R.; Wu, J.; Fei-Fei, L. TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction. In Proceedings of the Proceedings of The 8th Conference on Robot Learning. PMLR, 2025, Vol. 270, Proceedings of Machine Learning Research, pp. 1691–1729.
- Yang, J.; Lin, K.; Zhang, W.; Li, J.; Lin, T.; Wu, L.; Zhang, Y.Q.; Zhao, H.; Luo, P.; Su, Z.; et al. Self-Improving Robot Policy with Compositional World Model. In Proceedings of the Proceedings of Robotics: Science and Systems, 2026. [CrossRef]
- Lang, H.; Lu, H.; Sang, S.; Luo, H.; Chen, G.; Li, Q.; Yu, J.; Shi, Y.; Wang, J. EmbodiRSI: Recursive Self-Improvement for Data-Efficient Robot Adaptation, 2026, [2609.38905].
- Li, Z.; Li, Z.; Zhou, K.; Gu, J. RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation. In Proceedings of the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026. Accepted at IROS 2026, as reported by the authors; author manuscript.
- Marta, D.; Pek, C.; Melsion, G.I.; Tumova, J.; Leite, I. Human-Feedback Shield Synthesis for Perceived Safety in Deep Reinforcement Learning. IEEE Robotics and Automation Letters 2022, 7, 406–413. [CrossRef]
- Wang, M.; Osterberg, L.; Tian, S.; Shorinwa, O.; Bowkett, J.; Pastor, D.; Wu, J.; Schwager, M. InSight: Self-Guided Skill Acquisition via Steerable VLAs, 2026, [2606.24884].
- Physical Intelligence. π0.6 Model Card. Technical report, Physical Intelligence, 2025.
- Wang, Y.J.; Jiang, H.; Deng, S.; Xue, H.; Ye, W.; Duan, R.; Haghtalab, N.; Sastry, S.S.; Abbeel, P.; Qi, H. Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents, 2026, [2610.02204].
- Liang, J.; Liao, S.; Zhang, S.; Hou, D.; Cai, Y.; Deng, X.; He, C.; Huang, W.; Tan, R.; Wang, Z.; et al. RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents, 2026, [2609.32862].
- Wang, K.; Jie, H.; Yan, Y.; Heng, Z.; Li, Z. RegenHarness: A Robot Agent Harness with Evidence-Gated Recursive Self-Improvement, 2026, [2609.27612].
- Liu, Z.; Zhang, J.; Asadi, K.; Liu, Y.; Zhao, D.; Sabach, S.; Fakoor, R. TAIL: Task-Specific Adapters for Imitation Learning with Large Pretrained Models. In Proceedings of the International Conference on Learning Representations, 2024.
- Römer, R.; Zhang, Y.; Li, Y.; Schoellig, A.P. CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion. IEEE Robotics and Automation Letters 2026, [2601.09512]. [CrossRef]
- Gu, Z.; Wang, L. LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspective, 2026, [2606.26183].
- Yadav, Y.; Zhou, Z.; Wagenmaker, A.; Pertsch, K.; Levine, S. Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging. In Proceedings of the International Conference on Learning Representations, 2026.
- Xiao, W.; Xie, J.; Zhang, T.; Lin, H.; Fu, L.; Xue, H.; Lu, J.; Yang, Y.; Dai, C.; Wang, Z.; et al. ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. In Proceedings of the 10th Annual Conference on Robot Learning, 2026.
- Li, Z.; Jin, Y.; Liu, X.; Liu, P.; Prasad, V.; D’Eramo, C.; Chalvatzaki, G. HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning. In Proceedings of the Proceedings of the Conference on Robot Learning, 2026, [2606.08610].
- Wang, X.; Li, B.; Wang, H.; Zhang, L.; Cui, A.Y.; Tang, Z.; Buch, S.; Xu, D. SEAR: Self-Evolving Agentic Robotics, 2026, [2610.18792].
- Zhou, H.; Yang, F.; Chen, Z.; Zhang, L.; Fu, C.; Li, J.; Zhu, J.; Liu, X.; Yang, Y.; Meng, L.; et al. Who Improves the Robot? Toward Physical Recursive Self-Improvement, 2026, [2610.18842].
- Wang, J. What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery, 2026, [2609.31760].
- Black, K.; Galliker, M.; Levine, S. Real-Time Execution of Action Chunking Flow Policies. In Proceedings of the Advances in Neural Information Processing Systems, 2025, Vol. 38. [CrossRef]
- Zhou, K.; Yan, X.; Wei, X.; Li, X.; Gu, J. ASDSV: Multimodal Generation Made Efficient with Approximate Speculative Diffusion and Speculative Verification. In Proceedings of the Advances in Neural Information Processing Systems, 2025, Vol. 38. [CrossRef]
- Zhou, K.; Chen, Q.; Peng, D.; Li, Z.; Li, X.; Gu, J. Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment. In Proceedings of the Proceedings of the 43rd International Conference on Machine Learning, 2026.
- Xue, H.; Ren, J.; Chen, W.; Zhang, G.; Fang, Y.; Gu, G.; Xu, H.; Lu, C. Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation. In Proceedings of the Proceedings of Robotics: Science and Systems. Robotics: Science and Systems Foundation, 2025, [2503.02881].
- Chen, G.; Schott, A.; Zhong, L. TypeGo: An OS Runtime for Embodied Agents, 2026, [arXiv:cs.SE/2607.05482].
- Ali, S.W.; Angelopoulos, A.; Massey, D.; Haddix, S.; Georgiev, A.; Goh, J.; Wagle, R.; Sarathy, P.; Anderson, J.H.; Alterovitz, R. On the Necessity of Real-Time Principles in GPU-Driven Autonomous Robots. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025. [CrossRef]
- Mei, T.; Pei, C.; Canini, M.; Xiao, Y. MCSched: Memory-Controller-Aware Scheduling for Embodied LLM Workloads on NVIDIA Jetson. In Proceedings of the Proceedings of the 17th ACM SIGOPS Asia-Pacific Workshop on Systems, New York, NY, USA, 2026; APSys ’26, p. 46–52. [CrossRef]
- Arafat, A.A.; Vaidhun, S.; Wilson, K.M.; Sun, J.; Guo, Z. Response time analysis for dynamic priority scheduling in ROS 2. In Proceedings of the Proceedings of the 59th ACM/IEEE Design Automation Conference. Association for Computing Machinery, 2022, pp. 301–306. [CrossRef]
- Teper, H.; Günzel, M.; Ueter, N.; von der Brüggen, G.; Chen, J.J. End-To-End Timing Analysis in ROS2. In Proceedings of the 2022 IEEE Real-Time Systems Symposium (RTSS). IEEE, 2022, pp. 53–65. [CrossRef]
- Xing, W.; Li, M.; Li, M.; Han, M. Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks. ACM Computing Surveys 2026, 58, 1–36. [CrossRef]
- Zhang, H.; Zhu, C.; Wang, X.; Zhou, Z.; Yin, C.; Li, M.; Xue, L.; Wang, Y.; Hu, S.; Liu, A.; et al. BadRobot: Jailbreaking Embodied LLMs in the Physical World. In Proceedings of the International Conference on Learning Representations, 2025.
- Robey, A.; Ravichandran, Z.; Kumar, V.; Hassani, H.; Pappas, G.J. Jailbreaking LLM-Controlled Robots. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 11948–11956. [CrossRef]
- Wang, T.; Han, C.; Liang, J.; Yang, W.; Liu, D.; Zhang, L.X.; Wang, Q.; Luo, J.; Tang, R. Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 6948–6958.
- Zhou, X.; Tie, G.; Zhang, G.; Wang, H.; Zhou, P.; Sun, L. BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization. In Proceedings of the Advances in Neural Information Processing Systems, 2025, Vol. 38. [CrossRef]
- Kim, H.; Kang, J.; Kang, H.; Cho, M.; Kim, S.J.; Lee, Y. UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations. In Proceedings of the Proceedings of the 9th Conference on Robot Learning, 2025, Vol. 305, Proceedings of Machine Learning Research, pp. 4269–4294.
- Navardi, M.; Gao, S.; Walczak, M.; Camacho, F.; Mohsenin, T. Metareasoning for Edge-Cloud Collaborative LLM Planning for Efficient Autonomous Navigation. ACM Transactions on Embedded Computing Systems 2026, 25, 1–27. [CrossRef]
- Peng, D.; Ma, F.; Wang, B.; Wang, S.; Ma, J. Latency-Tolerant Cloud-Edge Collaborative Vision-Language-Action Models via Emergent Representational Specialization, 2026, [2608.00569].
- Ichnowski, J.; Chen, K.; Dharmarajan, K.; Adebola, S.; Danielczuk, M.; Mayoral-Vilches, V.; Jha, N.; Zhan, H.; LLontop, E.; Xu, D.; et al. FogROS2: An Adaptive Platform for Cloud and Fog Robotics Using ROS 2. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, [2205.09778].
- Chen, K.; Hoque, R.; Dharmarajan, K.; LLontop, E.; Adebola, S.; Ichnowski, J.; Kubiatowicz, J.; Goldberg, K. FogROS2-SGC: A ROS2 Cloud Robotics Platform for Secure Global Connectivity. In Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023, [2306.17157].
- Chen, K.; Hari, K.; Chung, T.; Wang, M.; Tian, N.; Juette, C.; Ichnowski, J.; Ren, L.; Kubiatowicz, J.; Stoica, I.; et al. FogROS2-FT: Fault Tolerant Cloud Robotics. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, [2412.05408]. [CrossRef]
- Tian, Y.; Chang, Y.; Arias, F.H.; Nieto-Granda, C.; How, J.P.; Carlone, L. Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems. IEEE Transactions on Robotics 2022, 38, 2022–2038. [CrossRef]
- Gu, Q.; Ye, Z.; Yu, J.; Tang, J.; Yi, T.; Dong, Y.; Wang, J.; Cui, J.; Chen, X.; Wang, Y. MR-COGraphs: Communication-Efficient Multi-Robot Open-Vocabulary Mapping System via 3D Scene Graphs. IEEE Robotics and Automation Letters 2025, 10, 5713–5720. [CrossRef]
- Zhao, Z.; Guo, H.; Chen, S.; Xu, K.; Jiang, B.; Zhu, Y.; Zhao, D. Empowering Multi-Robot Cooperation via Sequential World Models. In Proceedings of the International Conference on Learning Representations, 2026, pp. 13262–13280.
- Mandi, Z.; Jain, S.; Song, S. RoCo: Dialectic Multi-Robot Collaboration with Large Language Models. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 286–299. [CrossRef]
- Luo, X.; Zavlanos, M.M. Temporal Logic Task Allocation in Heterogeneous Multirobot Systems. IEEE Transactions on Robotics 2022, 38, 3602–3621. [CrossRef]
- Ren, Z.; Rathinam, S.; Choset, H. CBSS: A New Approach for Multiagent Combinatorial Path Finding. IEEE Transactions on Robotics 2023, 39, 2669–2683. [CrossRef]
- Huang, P.; Liu, R.; Aggarwal, S.; Liu, C.; Li, J. APEX-MR: Multi-Robot Asynchronous Planning and Execution for Cooperative Assembly. In Proceedings of the Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025. [CrossRef]
- Neville, G.; Chernova, S.; Ravichandar, H. D-ITAGS: A Dynamic Interleaved Approach to Resilient Task Allocation, Scheduling, and Motion Planning. IEEE Robotics and Automation Letters 2023, 8, 1037–1044. [CrossRef]
- Li, X.; Li, P.; Qian, L.; Liu, M.; Wang, D.; Liu, J.; Kang, B.; Ma, X.; Wang, X.; Guo, D.; et al. What Matters in Building Vision–Language–Action Models for Generalist Robots. Nature Machine Intelligence 2026, 8, 158–172. [CrossRef]
- Li, M.; Zhao, S.; Wang, Q.; Wang, K.; Zhou, Y.; Srivastava, S.; Gokmen, C.; Lee, T.; Li, L.E.; Zhang, R.; et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. In Proceedings of the Advances in Neural Information Processing Systems. Curran Associates, Inc., 2024, Vol. 37. [CrossRef]
- Kapoor, S.; Stroebl, B.; Siegel, Z.S.; Nadgir, N.; Narayanan, A. AI Agents That Matter. Transactions on Machine Learning Research 2025.
- Chen, J.; Shen, J.; Kang, H.; Hong, Z.; Jiang, Q.; Bose, S.; Zhang, Y.; Leng, L.; Vyas, A.; Mao, L.; et al. AgentSpec: Understanding Embodied Agent Scaffolds Through Controlled Composition. arXiv preprint arXiv:2606.14674 2026.
- Ahmedt-Aristizabal, D.; Khamis, A.; Armin, M.A.; Petersson, L. RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors, 2026, [2610.13516].
- Chen, T.; Chen, Y.; Li, Z.; Tang, J.; Su, K.; Lu, H.; Wan, W.; Chen, B.; Liu, S.; Yan, H.; et al. RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. arXiv preprint arXiv:2607.04434 2026.
- Yan, B.; Xiao, J.; Liu, K.; Zhang, J.; Zhang, Z.; Jia, Q.; Jia, Z.; Song, H.; Li, C.; Zhao, B.; et al. Can VLMs Diagnose and Recover from VLA Manipulation Faults? In Proceedings of the Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, Vol. 306, Proceedings of Machine Learning Research, pp. 143967–143983.
- Yang, Y.; Wang, Y.; Wen, Z.; Zhongwei, L.; Zou, C.; Zhang, Z.; Wen, C.; Zhang, L. EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models. In Proceedings of the Advances in Neural Information Processing Systems, 2025, Vol. 38. [CrossRef]
Figure 1.
Reference architecture of the embodied agent harness, with its five responsibilities in an online and an offline phase.
Figure 1.
Reference architecture of the embodied agent harness, with its five responsibilities in an online and an offline phase.

Figure 2.
Planning: method families and representative works for plan generation, feasibility checks, and dynamic replanning.
Figure 2.
Planning: method families and representative works for plan generation, feasibility checks, and dynamic replanning.

Figure 3.
Skill acquisition: method families and representative works for program synthesis, learning from demonstrations, and learning from reward feedback.
Figure 3.
Skill acquisition: method families and representative works for program synthesis, learning from demonstrations, and learning from reward feedback.

Figure 4.
Memory: method families and representative works for organization, maintenance, and retrieval.
Figure 4.
Memory: method families and representative works for organization, maintenance, and retrieval.

Figure 5.
Monitoring: method families and representative works for fault avoidance, detection, and recovery.
Figure 5.
Monitoring: method families and representative works for fault avoidance, detection, and recovery.

Figure 6.
Self-evolution: method families and representative works for experience acquisition, capability improvement, and recursive self-improvement.
Figure 6.
Self-evolution: method families and representative works for experience acquisition, capability improvement, and recursive self-improvement.

Table 1.
Interfaces between harness responsibilities: what each one reads and what it writes.
| Responsibility | Reads | Writes |
|---|---|---|
| Planning | task, scene memory, available skills | subgoals, feasibility conditions |
| Skill acquisition | demonstrations, rewards, model knowledge, errors | skills and their interfaces |
| Memory | observations, task state, corrections | time-stamped scene and experience records |
| Monitoring | observations, plan conditions, constraints | progress and failure evidence, interrupts |
| Self-evolution | traces, failures, retained results | revised knowledge, skills, parameters |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.