When LLMs Take Action: A Survey on the Safety and Alignment of Language-Conditioned Embodied Agents

1. Introduction

The integration of Large Language Models (LLMs) and Vision-Language Models (VLMs) into embodied agents marks a profound paradigm shift: language is no longer physically inert Ahn et al. (2022). By translating open-vocabulary semantics directly into kinetic actuation, foundation models expose a critical epistemic gap between fluent text generation and rigid physical constraints Soh and Lim (2026). This transition fractures the core assumption of traditional NLP safety. While conversational jailbreaks or hallucinations merely cause digital confusion, embodied vulnerabilities undergo immediate semantic-to-physical amplification, precipitating irreversible real-world hazards Li et al. (2026a). Furthermore, current alignment paradigms (e.g., RLHF Ouyang et al. (2022), DPO Rafailov et al. (2023)) prioritize toxic text suppression, often defaulting to over-refusal. In dynamic physical settings, such text-centric "harmlessness" introduces the fatal risk of inaction—an agent politely refusing an instruction while freezing on a busy highway remains a catastrophic threat. Consequently, semantic safety must evolve from lexical filtering to rigorous kinetic alignment.

This survey introduces a core thesis to bridge this gap: the physical consequences of language-level attacks shatter the assumption of physical compliance, as semantic hallucinations translate instantly into kinetic hazards. A singular adversarial prompt—such as coercing a planner to bypass spatial constraints—undergoes physical amplification, yielding divergent kinetic outcomes. It may cause a manageable path deviation in a wheeled robot, trigger severe fall fragmentation in a bipedal humanoid, or result in high-energy collisions for a fixed-base manipulator. Securing embodied agents therefore demands a shift from text-centric filtering to a physically-grounded safety paradigm, one that accounts for the specific kinetic failure modes of each embodiment.

We present a comprehensive, NLP-centric examination of safety and alignment for language-driven embodied agents, structured around a unified taxonomy (Figure 2). By synthesizing adversarial red-teaming, representation engineering, and 3D physical grounding, our contributions are threefold:

1.

A Semantic-to-Physical Threat Surface: We map how cognitive layer exploits bypass digital boundaries to trigger embodiment-specific physical vulnerabilities, demonstrating that the same linguistic attack yields different catastrophic outcomes depending on the robot’s form-factor.

2.

An Integrated Defense-in-Depth Architecture: As illustrated in Figure 4, we construct a layered defense that synergizes intrinsic white-box interventions with dual semantic-physical grounding to robustly filter semantic hallucinations.

3.

A Cognitive Error Evaluation Framework: Moving beyond mechanical routing metrics, we propose a unified error taxonomy and collate NLP-centric benchmarks (e.g., FoMER Dissanayake et al. (2025), SAFEL Son et al. (2025)) to rigorously audit logic consistency and abstention recall.

As previewed in Figure 1 and detailed in our taxonomy (Figure 2), the remainder of this paper is structured as follows: §2 deconstructs language grounding vulnerabilities. §3 analyzes morphology-amplified attacks. §4 surveys defense architectures. §5 establishes our unified error taxonomy and collates diagnostic benchmarks. §6 outlines future trajectories for physical-consequence-aware alignment.

Figure 1. Teaser: Safety and Alignment of Language-Conditioned Embodied Agents. Panel ① maps cognitive-layer exploits to task paradigms and attacks (§2, §3); Panel ② illustrates semantic-to-physical amplification and the defense-in-depth architecture (§4); Panel ③ shows how identical language attacks produce divergent physical outcomes across robot embodiments (§3.2). The bottom bar previews the fault taxonomy and diagnostic benchmarks (§5.1–§5.2).

Figure 1. Teaser: Safety and Alignment of Language-Conditioned Embodied Agents. Panel ① maps cognitive-layer exploits to task paradigms and attacks (§2, §3); Panel ② illustrates semantic-to-physical amplification and the defense-in-depth architecture (§4); Panel ③ shows how identical language attacks produce divergent physical outcomes across robot embodiments (§3.2). The bottom bar previews the fault taxonomy and diagnostic benchmarks (§5.1–§5.2).

Figure 2. Comprehensive Taxonomy of Vulnerabilities, Attacks, and Defenses in Language-Conditioned Embodied AI.

Figure 2. Comprehensive Taxonomy of Vulnerabilities, Attacks, and Defenses in Language-Conditioned Embodied AI.

Figure 3. Adversarial physical outcomes in language-conditioned embodied agents. Logical progression organized by grid columns: Academic Blue titles and borders for Sec. 2–2.2, and Academic Red for physical consequences in Sec. 3.2.

Figure 3. Adversarial physical outcomes in language-conditioned embodied agents. Logical progression organized by grid columns: Academic Blue titles and borders for Sec. 2–2.2, and Academic Red for physical consequences in Sec. 3.2.

2. Language Task Paradigms and Grounding Challenges

As language grounding remains a primary bottleneck in embodied AI Lisondra et al. (2026), we categorize task paradigms by interactivity to analyze how linguistic ambiguity manifests as specific physical failure modes.

2.1. Language-Driven Task Paradigms

Rather than treating language grounding merely as a functional bottleneck Lisondra et al. (2026), we categorize task paradigms by their interactivity to expose a structural misalignment: how discrete linguistic ambiguities deterministically collapse into continuous physical failures.

• Single-Turn Tasks. A solitary command forces autonomy without corrective feedback. Goal-oriented tasks (ALFRED Shridhar et al. (2020), REVERIE Qi et al. (2020)) demand planning under uncertainty, compelling agents to resolve latent planning burdens by hallucinating missing sub-steps. Yet, this grounding difficulty is non-monotonic: Huang et al. (2026) reveals a “Complexity Paradox” where coarser instructions paradoxically improve success rates by triggering shallow, vision-dominant policies. This exposes a critical vulnerability—language is often not causally grounding behavior, but acting as a superficial prior that the agent easily bypasses. Route-oriented tasks (Room-to-Room Chang et al. (2017)) enforce strict sequential dependencies, where a single misgrounded landmark shatters the execution chain.
• Multi-Turn Tasks. Progressive instruction delivery introduces an interaction axis. Interactive tasks (JustAsk Chi et al. (2020), RobotSlang Banerjee et al. (2021)) mitigate ambiguity by permitting clarification, shifting the cognitive burden to the ask-or-act dilemma: querying incurs overhead, but acting on uncertainty risks catastrophic grounding errors. However, overall task success often masks brittle interaction logic; thus, Zorzi et al. (2026) forcefully advocates for decoupling interactive reasoning from navigation metrics, arguing that true collaborative intelligence requires calibrated, uncertainty-driven communication rather than brute-force trial and error. Conversely, Unidirectional execution tasks (CEREALBAR Suhr et al. (2019)) strictly forbid queries, precipitating cumulative misalignment.

2.2. Vulnerabilities in Language Grounding

In Vision-Language-Action (VLA) systems, NLP errors transcend text to become kinetic hazards Li et al. (2026a). This semantic-to-physical transmission exposes a fatal tension between low-dimensional linguistic abstraction and rigid physical constraints, driven by three structural vulnerabilities.

• Referential Ambiguity and Hallucination. Beyond mere scene-task inconsistencies—which inflate object hallucinations by 40× Chakraborty et al. (2025)—the deeper crisis lies in action hallucinations Soh and Lim (2026). Rather than mere category confusion, this exposes a fundamental topological barrier: mapping continuous latent priors to disconnected physical free spaces unavoidably creates “seams” of invalid actions (e.g., grasping through solids). Consequently, generative VLAs inherently synthesize affordance violations that bypass digital safeguards.
• Spatial Relation Deficits. Language fundamentally bottlenecks 6-DoF geometric reality. State-of-the-art models systematically fail on metric-semantic queries Padhan et al. (2026), producing plans that are semantically fluent yet geometrically disastrous due to unresolved frame-of-reference ambiguities. Alarmingly, static benchmarks mask this spatial blindness: VLAs exhibit semantic feature collapse Xu et al. (2026), exploiting lexical-kinematic shortcuts that instantly shatter under causal layout shifts, exposing apparent spatial comprehension as a fragile statistical illusion.
• Cumulative Instruction Deviation. In long-horizon tasks, early semantic misalignments do not merely persist; they metastasize. Formalized as cascading failures Zeng et al. (2026) and identified as the definitive VLA bottleneck Zhang et al. (2026b), these compounding errors expose severe flaws in autoregressive physical control. Confronted with physical dissonance, models regress to behavioral inertia Xu et al. (2026) and “over-reasoning” Lim et al. (2026)—consuming tokens to rationalize failed states rather than adapting. This epistemic drift traps the agent in a self-justifying narrative, where fluent text generation masquerades as physical planning, irreversibly decoupled from kinetic reality.

3. Adversarial Language Attacks with Morphology-Aware Consequences

In standard NLP, adversarial attacks terminate at toxic text generation. In embodied AI, semantic jailbreaks bypass digital boundaries to become irreversible kinetic hazards—a paradigm-shifting threat in Vision-Language-Action (VLA) deployment Li et al. (2026a,b). This semantic-to-physical transmission exploits a profound output-action mismatch Zhang et al. (2025b): models may maintain linguistic safety while simultaneously leaking hazardous physical commands. We deconstruct these attack vectors, tracing how each subverts the VLA’s cognitive alignment to dictate asymmetric physical consequences.

3.1. The Cognitive Layer as an Attack Entry Point

Adversaries target embodied planners along a spectrum of access and modality, each exploiting a distinct cognitive vulnerability.

• White-Box Reasoning Layer Attacks. When adversaries breach the inference pipeline, they bypass semantic reasoning to surgically manipulate the planner’s mathematical substrate. Logits-based hijacking (e.g., VulMine Li et al. (2025)) reveals that safety alignments merely suppress, rather than erase, harmful priors, making them easily amplified. Furthermore, this cognitive subversion transcends language: FreezeVLA Wang et al. (2025b) demonstrates that adversarial visual noise can deterministically trap agents in an “action-freezing” state. This severs the digital mind from physical execution, weaponizing persistent inaction to bypass standard safety monitors entirely.
• Black-Box Adversarial Prompts. Without weight access, attackers weaponize the VLA’s foundational mandate for compliance. Static obfuscations like nested narratives (DeepInception Li et al. (2023a)) exploit the model’s “self-losing under authority” to bypass filters. Meanwhile, automated red-teaming (RoboPAIR Robey et al. (2025)) and voice-based exploits (BadRobot Zhang et al. (2025b)) expose a fatal conceptual deception: because VLAs rely on statistical token matching rather than true ethical reasoning, they seamlessly rationalize consequentially identical but lexically distinct malicious acts. The very autoregressive fluency that drives brilliant physical planning thus enables systematic kinetic sabotage.
• Cross-Modality Semantic Injection. Multimodal architectures enable adversaries to deploy perceptual Trojan horses, embedding malicious payloads within visual Gong et al. (2025); Ma et al. (2024) or acoustic Shen et al. (2024) streams. These perturbations trigger severe cross-modal representation drift, bypassing linguistic safeguards to directly corrupt physical affordances. As revealed by the AgentSafe benchmark Ying et al. (2025b), agents often successfully perceive environmental hazards visually but critically fail to translate this awareness into safe planning. This systemic perception-execution disconnect highlights the profound inadequacy of text-centric defensive paradigms for robust embodied control.

Figure 4. The Defense-in-Depth Architecture for Language-Conditioned Embodied AI. Adversarial inputs are systematically intercepted across four modalities. Pre-processing and post-guardrails (§4.3) provide peripheral shielding, while intrinsic latent space alignment (§4.1) and robust dual semantic-physical grounding (§4.2) form the core cognitive defenses. Errors bypassing early layers are increasingly filtered by physical and kinematic realities.

Figure 4. The Defense-in-Depth Architecture for Language-Conditioned Embodied AI. Adversarial inputs are systematically intercepted across four modalities. Pre-processing and post-guardrails (§4.3) provide peripheral shielding, while intrinsic latent space alignment (§4.1) and robust dual semantic-physical grounding (§4.2) form the core cognitive defenses. Errors bypassing early layers are increasingly filtered by physical and kinematic realities.

3.2. Physical Amplification of Embodied Attacks

Classical NLP alignment falsely assumes embodiment-agnostic consequences. In Embodied AI, a singular adversarial payload undergoes physical amplification: its kinetic manifestation is strictly governed by the agent’s morphology, converting identical semantic errors into asymmetric, irreversible hazards:

• Autonomous Mobility Hazards: For mobile platforms (e.g., autonomous vehicles), semantic hallucinations forcefully override spatial constraints. Rather than mere digital path deviations, this cognitive failure physically translates into fatal pedestrian collisions Li et al. (2026a,b).
• Industrial Kinematic Collisions: In rigid manipulators, semantic jailbreaks bypass high-level filters to disregard designated safety zones Li et al. (2026a). This actively weaponizes the robotic embodiment, translating malicious language tokens into aggressive kinetic strikes against humans Zhang et al. (2025b).
• Irreversible Precision Failures: In fine-grained control morphologies, such as surgical robotics, the abstraction gap is lethal. A single misresolved instructional token physically manifests as a misapplied surgical tool, converting a semantic gap into immediate trauma Li et al. (2026a).

4. Defense: Alignment, Grounding, and Purification

To prevent semantic manipulation from manifesting as irreversible physical harm, embodied agents demand multi-layered safeguards resolving the inherent safety-latency paradox. As depicted in Figure 4, we formalize a decoupled defense-in-depth architecture Li et al. (2026a,b) intercepting adversarial inputs across modalities. This unified pipeline comprises intrinsic latent alignment (§4.1), dual semantic-physical grounding (§4.2), and peripheral purification encompassing input sanitization and execution guardrails (§4.3).

4.1. Intrinsic Alignment and Inference-Time Guardrails

Traditional alignment (RLHF Ouyang et al. (2022), DPO Rafailov et al. (2023)) exposes a fatal cross-domain safety misalignment in robotics: penalizing toxic text fundamentally ignores kinetic consequences. Mere textual refusal is dangerously insufficient; cognitive freezing mid-task transforms into physical peril. True embodied alignment must map unsafe directives to safe fallback physical actions. The SAFEL benchmark Son et al. (2025) exposes this crisis, proving text-aligned LLMs completely fail at transition modeling, routinely hallucinating fatal execution paths for contextually subtle, non-toxic hazards.

Bypassing the catastrophic forgetting of full fine-tuning, the frontier shifts from behavioral pruning to inference-time representation intervention. While RAIN Li et al. (2023c) achieves heuristic self-correction without weight updates, CEE Yang et al. (2025a) and LatentGuard Shu et al. (2025) prove that steering activations via SLERP-based subspace rotation and disentangled latent supervision selectively enforces safety boundaries. This reveals a profound insight: embodied safety is fundamentally a representation geometry problem. Planners already encapsulate safety priors; robust defense merely requires geometric extraction at inference, proving true alignment is a structural property of the latent space, rather than an aftermarket behavioral patch.

4.2. Dual Semantic-Physical Grounding

To prevent physical hallucinations, linguistic commands demand dual grounding—a necessity formally verified by SENTINEL’s Zhan et al. (2025) multi-level temporal logic (LTL/CTL) evaluations.

• 3D Scene Representations as Spatial Anchors. Raw 2D perception is epistemically fragile. Enforcing geometric consensus via 3D Gaussian Splatting Matsuki et al. (2024); Qin et al. (2024) mitigates semantic point ambiguity and inherently filters view-dependent visual trojans. Yet, grounding is not an absolute panacea: StealthAttack Ke et al. (2025) reveals that density-guided point injection easily poisons these explicit representations. This exposes a recursive epistemic vulnerability: while we rely on 3D semantics to anchor physical reality, the spatial anchor itself can be seamlessly and adversarially rewritten.
• Affordance Reasoning as a Kinematic Filter. Perfect geometry does not guarantee kinetic feasibility. Affordance reasoning bridges this gap by mapping abstract semantics into structured geometric and positional flows Su et al. (2025). Crucially, the ADAPT benchmark Chen et al. (2026a) exposes execution vulnerabilities through resource-level affordance failures. By testing agents against absent physical prerequisites (e.g., unavailable ovens), it proves that true physical alignment demands proactive environmental verification rather than relying on ungrounded semantic priors. By proactively filtering kinematically invalid commands Ju et al. (2025); Qian et al. (2024), affordance serves as a strict mechanical safeguard, recalibrating the LLM’s illusion of control against genuine physical boundaries.

4.3. Prompt and Input Purification

When latent alignment fails, sanitization must intercept adversarial payloads before kinetic execution.

4.3.1. Preprocessing and Neutralization

Because autoregressive fluency lets models rationalize obfuscated sabotage Yuan et al. (2024b), defenses must decouple intent from syntax. Rather than naive filtering, ASF Khachaturov and Mullins (2025) excises out-of-distribution suffixes, while DR-Smoothing Lin et al. (2026) pairs local disruption with global rectification to restore inputs to safe in-distribution manifolds. Structurally, unimodal sanitization ignores embodied threats. Since visual noise induces cross-modal representation drift Yan et al. (2025), ECSO Gou et al. (2025) bypasses visual Trojans via image-to-text transformation to reactivate linguistic safety priors, while AdaShield Wang et al. (2024) prepends adaptive shield prompts to enforce holistic safety verification across both modalities.

4.3.2. Post-Processing Execution Guardrails

As a final barrier, execution guardrails intercept latent hazards before kinetic actuation. Instead of heuristic filtering, SmoothLLM Robey et al. (2023) applies randomized input smoothing, while LSD Phute et al. (2023) leverages zero-shot self-examination for harm detection. In embodied contexts, RoboGuard Ravichandran et al. (2026) formally synthesizes open-vocabulary rules into Signal Temporal Logic (STL), and RoboSafe Wang et al. (2025a) generates executable safety predicates via bidirectional memory reasoning. This exposes a necessary architectural paradox: the cognitive planner is inherently treated as an untrusted adversary, enforcing a strict epistemic decoupling where high-level semantic fluency is subordinated to formal downstream containment.

4.4. Cross-Modal Consistency Verification

Beyond semantic constraints, kinetic safety demands strict consensus across independent sensor streams. While adversarial payloads can spoof visual perception via optimized visual perturbations Wang et al. (2025b), they mathematically struggle to simultaneously corrupt orthogonal metric modalities (e.g., LiDAR or proprioception). To operationalize this asymmetry, embodied architectures deploy decoupled dual-loop mechanisms Li et al. (2026a). When the slow, high-level VLA planner hallucinates a semantically safe but physically disastrous trajectory, the high-frequency “fast reflex” loop utilizes Control Barrier Functions (CBFs) to trigger immediate kinematic overrides. This strict sensorimotor decoupling ensures language-driven statistical illusions never override rigid geometric realities.

5. Evaluation: Unified Taxonomy and Diagnostic Benchmarks

5.1. A Unified Error Taxonomy for Embodied Language Agents

Transcending reductive binary success metrics that mask compensatory over-reasoning Lim et al. (2026); Xu et al. (2026) and evade text-only safety protocols Ying et al. (2025b), we decompose embodied vulnerabilities into three epistemic domains, tracing the structural degradation from latent linguistic intent to physical execution.

• Cognitive and Semantic Faults (“Brain” Failures). Originating pre-execution, these errors reflect severe implicit intent misresolution Lim et al. (2026). RoboInspector Ying et al. (2025a) formalizes this structural collapse, proving that instruction granularity deficits and cognitive limits drive models to generate Disorder and Nonsense plans. This exposes a profound epistemic flaw: LLMs merely approximate physical syntax through compensatory over-reasoning, blindly generating ungrounded scripts that expose their semantic comprehension as a statistical illusion.
• Kinematic and Execution Faults (“Body” Failures). Logically fluent plans predictably shatter against physical friction. Formalized within a goal-conditioned POMDP framework Garrabé et al. (2025), Badpose and Infeasible Ying et al. (2025a) execution errors expose disembodied planners ignoring continuous kinematics. These breakdowns reveal a dangerous paradigm: the cognitive layer assumes absolute physical compliance, fatally treating the robotic embodiment as a deterministic text-renderer.
• Contextual and Integration Faults (“Nervous System” Failures). These sensorimotor disconnects between latent plans and dynamic environments trap agents in cascading failures. Moving beyond static simulators, AgentSafe Ying et al. (2025b) and HazardArena Chen et al. (2026b) expose a profound alignment crisis: agents successfully perceiving hazards still execute fatal actions under semantic shifts. This cross-modal decoupling reveals that while models statistically master what to do, they remain dangerously oblivious to when they must abstain.

5.2. Diagnostic Dimensions and Systemic Benchmark Gaps

Table 1 maps benchmarks across seven diagnostic dimensions within our System 2-Immune-Somatic framework (metric details in Appendices A1 and A2). Yet, a topological analysis reveals a profound systemic flaw: an overwhelming epistemic bias toward static semantic reasoning at the expense of dynamic kinematic integration, masking severe vulnerabilities in continuous physical control.

• The Bias Toward Static “Brain” Evaluation. While foundational functions are densely covered, they operate in a vacuum. REI-Bench Jiang et al. (2025) and ECBench Dang et al. (2025) effectively expose coreferential vagueness and robot-centric spatial blindness. Concurrently, hallucination diagnostics (MIRAGE Zhang et al. (2025c), AbstainEQA Wu et al. (2025)) map metacognitive boundaries. However, this static paradigm treats agents as passive, disembodied conversationalists. By severing semantic reasoning from active execution, these metrics fail to capture how linguistically “safe” plans predictably collapse into physical violations under mechanical friction.
• The Sparsity of Cognitive Consistency. Furthermore, evaluations for cognitive consistency remain alarmingly sparse. Over extended horizons, autoregressive planners undergo severe epistemic drift, fatally treating rigid physical states as mutable text. Although FoMER Dissanayake et al. (2025) scrutinizes reasoning trail correctness, PRISM Lim et al. (2026) explicitly isolates memory retention failures, and RoboCerebra Han et al. (2026) evaluates System-2 performance in dynamically evolving environments, dynamic red-teaming is virtually absent. Consequently, we cannot audit how early semantic misalignments metastasize into cascading kinetic failures, masking the fundamental inability to maintain a causal world model.
• The Nascent Frontier: End-to-End Physical Consequences. The most glaring revelation is the severe underdevelopment of diagnostics for Semantic-Physical Sync and adversarial Physical Feasibility, exposing the historical neglect of the “Somatic System”. While pioneering frameworks like RoboBench Luo et al. (2025) have finally begun to measure embodied feasibility beyond symbolic matching, and HazardArena Chen et al. (2026b) introduces controlled semantic risk scenarios, they represent exceptions rather than the rule. The vast majority of existing metrics still validate the linguistic plausibility of generated plans but ignore true physical execution limits. By failing to broadly test whether language-driven trajectories trigger cross-modal conflicts or synchronize with high-frequency proprioceptive feedback, the field dangerously equates statistical text generation with genuine sensorimotor intelligence.

6. Conclusion

Integrating foundation models into robotic systems highlights a critical limitation in current AI alignment: linguistic harmlessness does not guarantee physical safety. Standard textual safeguards fail to protect against semantic ambiguities or spatial misunderstandings that can result in real-world kinetic hazards. By reframing embodied alignment to focus on physical consequences, this survey demonstrates that robust defense requires tightly coupling the model’s cognitive constraints with mechanical realities. This is achieved by combining targeted internal representation engineering with dynamic semantic-physical grounding, and by evaluating system reliability through structured cognitive fault taxonomies rather than relying solely on binary task success rates.