Audio Embodied Intelligence: Auditory Perception, Reasoning, and Interaction for Multimodal Agents

1. Introduction

Embodied intelligence refers to an agent’s capacity to perceive, reason, and act through continuous interaction with the physical and social world. Its defining characteristic is the coupling between perception and action: sensory observations inform the agent’s internal state and behavior, while the consequences of action generate new observations that support subsequent reasoning and adaptation. Embodiment therefore involves more than applying perceptual models to robot data; intelligence must be situated in an agent, grounded in an environment, and expressed through action or interaction.

Research in embodied AI has progressed from task-specific navigation and manipulation toward multimodal agents that connect perception, language, planning, and control. Interactive simulation and robot learning have enabled the systematic study of situated behavior, while foundation models and vision–language–action models have expanded the range of instructions, environments, and skills that a single system can address [1,2,3,4,5]. Despite this progress, most embodied systems remain centered on vision, language, and action, leaving other sources of sensorimotor evidence comparatively underexplored.

Audition provides a complementary channel that is temporally fine-grained, non-contact, and available when relevant objects or events are occluded or outside the camera’s field of view. Impact and contact sounds reveal manipulation dynamics, spatial audio provides cues about source locations and acoustic scene structure, speech conveys task intent, and prosodic or other vocal cues provide evidence about user state. These properties make sound a natural modality for connecting perception, reasoning, action, and interaction within an embodied loop.

Figure 1 places this work in the context of recent publication growth. The OpenAlex counts show a marked increase in work indexed under embodied AI and embodied intelligence after 2020, together with growth in the corresponding audio-related subsets. Rather than establishing the maturity of individual tasks, the trend indicates that a previously dispersed body of auditory embodied research has become large enough to warrant a focused synthesis.

The central difficulty is that auditory learning for embodied intelligence does not correspond to a single task, but spans a family of problems with different inputs, inferred states, and behavioral consequences. A microphone array for sound source localization, a binaural navigation policy, a spoken-instruction robot, a contact-audio manipulation model, and an empathy-aware social robot all use sound, but they contribute to different stages and roles within situated perception–action or perception–interaction processes. Some works provide an auditory capability used by an embodied system, whereas others explicitly use auditory feedback to adapt subsequent action or interaction and thereby implement a closed loop. If these works are grouped only by modality, task name, or neural architecture, the logic of embodiment is obscured. We therefore organize the field by asking three questions. First, what does the agent listen for? Second, what state, meaning, or user condition does it infer? Third, how does this inference inform physical action, task execution, communicative interaction, or their evaluation?

In this work, we use embodied audio to denote auditory input or output that is explicitly connected to a situated agent, environment, task, or action-relevant evaluation, rather than sound that is merely recognized or generated as an isolated offline category. Under this view, audio plays three complementary roles. First, it serves as physical evidence: impact sounds, echoes, motion noise, and contact acoustics reveal hidden events, material properties, spatial layout, and action consequences. Second, it serves as semantic and paralinguistic evidence: speech conveys task intent and constraints, while prosody, rhythm, pauses, hesitation, laughter, sighs, intensity, and tone provide cues about how an instruction is expressed and whether the user may be uncertain, urgent, engaged, or in need of repair. Third, audio can be an interaction signal: spoken responses, nonverbal sounds, sonification, and spatialized feedback can shape action timing, turn-taking, safety warnings, clarification, trust calibration, and social adaptation. The key distinction is therefore not whether a system contains an audio encoder or completes an entire closed loop, but whether auditory information makes an explicit, situated contribution to embodied perception, reasoning, action, interaction, or evaluation.

Figure 2. Definition of embodied audio and its roles in an embodied loop. Audio can serve as physical evidence, semantic and paralinguistic evidence, and an interaction signal that changes belief, policy, feedback, action, and social response.

Figure 2. Definition of embodied audio and its roles in an embodied loop. Audio can serve as physical evidence, semantic and paralinguistic evidence, and an interaction signal that changes belief, policy, feedback, action, and social response.

Existing surveys provide substantial coverage of several related research traditions, including robot audition and acoustic localization, audio-visual perception and navigation, spoken human–robot interaction, affective and social robotics, and embodied AI. However, these traditions are generally reviewed within their respective task or disciplinary boundaries. What remains less developed is a shared framework for explaining how physical, semantic, and socio-affective forms of auditory information contribute to situated perception, reasoning, action, and social interaction.

This perspective motivates a two-axis loop–stage taxonomy. The loop axis distinguishes three coupled forms of auditory embodiment: the Physical Interaction Loop, in which sound provides evidence about the environment and the agent’s actions; the Semantic Grounding-and-Action Loop, in which speech, sound, and language convey task-relevant meaning; and the Socio-Affective Interaction Loop, in which paralinguistic and expressive cues inform human-centered adaptation. The stage axis distinguishes Percept, Reason, and Interact, corresponding respectively to the extraction of situated cues, the construction of task-relevant state or meaning, and the use of that state to guide action or communication. Crossing the two axes yields nine loop–stage categories, summarized in Table 1 and formally defined in Section 2.

Figure 3. The proposed unified framework for auditory-centered embodied intelligence. Three coupled loops, the Physical Interaction Loop, Semantic Grounding-and-Action Loop, and Socio-Affective Interaction Loop, span the stages of Percept, Reason, and Interact, with physical, semantic, and social information mutually informing one another.

Figure 3. The proposed unified framework for auditory-centered embodied intelligence. Three coupled loops, the Physical Interaction Loop, Semantic Grounding-and-Action Loop, and Socio-Affective Interaction Loop, span the stages of Percept, Reason, and Interact, with physical, semantic, and social information mutually informing one another.

The taxonomy characterizes the embodied role of auditory information and the stage to which a work primarily contributes. It does not require every included system to implement a complete closed loop, nor does it assume that a paper belongs exclusively to one category.

This work makes the following four contributions:

• We introduce a two-axis loop–stage taxonomy for auditory-centered multimodal learning in embodied intelligence, comprising three embodied loops and three functional stages.
• We provide a structured review of representative methods and systems across the nine loop–stage categories, showing how auditory information supports embodied perception, grounding, action, and interaction.
• We summarize task families, benchmarks, and evaluation metrics for assessing whether audio improves embodied behavior, rather than only improving standalone perception accuracy.
• We discuss key challenges and future directions for developing robust, real-time, and socially aligned audio-aware embodied agents.

The remainder of the work is organized as follows: Section 2 formalizes the taxonomy and the embodied auditory learning problem; Section 3, Section 4, and Section 5 review the Physical Interaction Loop, the Semantic Grounding-and-Action Loop, and the Socio-Affective Interaction Loop, respectively; Section 6 summarizes datasets, benchmarks, and evaluation metrics; Section 7 discusses representative application settings; and Section 8 synthesizes open challenges and future directions.

Table 1 summarizes the resulting loop–stage categories, representative research directions, and relative maturity.

1.1. Scope and Related Surveys

Table 2 positions this work relative to five families of related reviews: robot audition and acoustic perception, embodied AI and robotics, foundation models for intelligent robots, speech and multimodal human–robot interaction, and social and affective robotics. The comparison considers whether each family explicitly addresses physical, semantic, and social roles of audio, evaluates its contribution to an embodied loop, covers foundation-model or VLA approaches, and provides an organizing taxonomy.

The comparison reveals complementary but fragmented coverage. Robot-audition surveys provide the strongest treatment of acoustic sensing, yet typically stop before grounded reasoning, action, or social interaction. General embodied-AI and foundation-model surveys cover planning, control, and language-conditioned behavior, but usually treat audition as peripheral input or preprocessing. Speech and multimodal-HRI surveys examine dialogue and interaction, while social-robotics reviews address affect, empathy, and ethics; neither tradition generally connects these concerns to physical acoustic sensing and action. Taken together, the literature lacks a common framework that follows auditory information across physical, semantic, and socio-affective roles and from perception through reasoning to interaction. This is the gap addressed by the two-axis loop–stage taxonomy proposed in this survey.

We focus primarily on work published since 2020, while including earlier studies that established foundational tasks, methods, datasets, and evaluation protocols. This emphasis reflects the recent convergence of embodied acoustic simulation, large-scale audio-visual learning, foundation-model-based reasoning, and affect-aware human–robot interaction. The manuscript represents a June 2026 snapshot; works listed for 2026 may therefore be online preprints or early-access publications whose metadata can change before archival publication.

Our corpus deliberately excludes generic audio-language foundation models when they are not embodied, even if they are useful as upstream encoders. A general audio-text model is included only when a paper uses it in navigation, manipulation, human-robot interaction, or another robot/agent loop. This decision keeps the work focused on embodied intelligence rather than general-purpose audio representation learning.

Finally, the work is not intended to replace specialized reviews on robot audition, speech processing, audio-visual learning, or social robotics. Rather, it provides a complementary view that asks how auditory signals enter embodied loops: what is sensed, what is inferred, and how the inference changes action or interaction. Across the three loops, we compare how different technical traditions address problems within the same loop-stage taxonomy.

2. Taxonomy and Scope

2.1. Scope and Unit of Analysis

We use an auditory embodied loop as the unit of analysis. A work is included when sound is not merely a signal to be recognized, but part of a situated loop that connects auditory input or output to embodied perception, grounded reasoning, control, interaction, or evaluation. Under this criterion, robot audition, audio-visual navigation, contact-audio manipulation, spoken-instruction grounding, and affect-aware human-robot interaction belong to the scope because audio changes what the agent estimates, decides, says, or does.

This scope also defines the boundary of the review. Generic audio classification, speech synthesis, audio captioning, or audio-language pretraining is treated as background unless it is connected to an embodied agent, a physical environment, a robot platform, a situated task, or an action-relevant feedback process. The taxonomy therefore organizes papers by the role that audition plays in an embodied loop, rather than by audio modality, network architecture, or benchmark family alone.

Formally, this loop can be written as a three-stage mapping from situated observations to an internal state and then to action or expression:

$$\begin{array}{cc}\hfill z_t& =f_{\mathrm{percept}}(a_{\le t},v_{\le t},l_{\le t},h_{\le t}),\hfill \\ \hfill s_t& =f_{\mathrm{reason}}(z_{\le t},x_{\le t}),\hfill \\ \hfill (u_t,y_t)& =f_{\mathrm{interact}}(s_t,x_t),\hfill \end{array}$$

(1)

where $z_t$ denotes auditory or multimodal cues extracted from situated observations, $s_t$ denotes task-relevant physical, semantic, or socio-affective state, $u_t$ denotes physical action, and $y_t$ denotes communicative or expressive output. The three functions correspond directly to the stage axis of the taxonomy: Percept extracts $z_t$, Reason constructs $s_t$, and Interact uses $s_t$ to produce $u_t$ or $y_t$. The loop axis specifies what $s_t$ represents: physical state in the Physical Interaction Loop, grounded task meaning in the Semantic Grounding-and-Action Loop, and user or social state in the Socio-Affective Interaction Loop.

2.2. Stage Axis: Percept, Reason, and Interact

The proposed taxonomy is defined by two axes. The first axis specifies the stage of the embodied loop: Percept, Reason, and Interact. Percept denotes the extraction of reliable auditory or multimodal cues from situated observations, such as source direction, event boundaries, speech content, or affective cues. Reason denotes the transformation of these cues into task-relevant state, grounded meaning, plans, user models, or social interpretations. Interact denotes the use of inferred state or meaning to produce physical action, spoken response, expressive behavior, feedback, or recovery. This distinction separates methods that share similar audio encoders but differ in the loop they close: localization as an observation, localization as evidence for navigation, and localization as a cue for recovery or user coordination are different embodied roles.

2.3. Loop Axis: Physical, Semantic, and Socio-Affective Roles

The second axis specifies the type of embodied loop in which audition is used. The Physical Interaction Loop addresses the role of sound as physical evidence. In this loop, auditory observations are used to infer where events occur, how objects and environments behave, and whether an ongoing action is proceeding as expected. Its progression is therefore from acoustic sensing, to audio-physical state estimation, to feedback that can alter navigation, manipulation, recovery, or human-facing sonic interfaces. The detailed task families associated with this loop are summarized in Table 1.

The Semantic Grounding-and-Action Loop addresses the role of sound, speech, and language as carriers of task meaning. Here, the agent must recover what is being requested or described, bind that meaning to objects, places, actions, and constraints, and then produce an appropriate action or verbal response. This loop is not limited to speech recognition: it includes the broader problem of grounding auditory and linguistic evidence into situated plans, long-term context, multimodal reasoning, clarification, and action-aware language generation.

The Socio-Affective Interaction Loop addresses the role of auditory and expressive cues in human-centered interaction. In this loop, speech acoustics and multimodal behavior are interpreted not only as task information, but also as evidence about user state, identity, engagement, relationship, social norms, and affective context. The resulting interaction problem is to adapt the robot’s speech, gesture, facial expression, or social policy in a way that is expressive, appropriate, privacy-aware, and aligned with the ongoing relationship. Table 1 provides the corresponding task-level decomposition.

Crossing the three embodied loops with the three functional stages yields nine loop–stage categories:

• Physical Interaction Loop maps to P1 Physical Acoustic Sensing (Percept), P2 Audio-Physical Understanding (Reason), and P3 Audio-Physical Interaction (Interact).
• Semantic Grounding-and-Action Loop maps to Se1 Speech and Audio Understanding (Percept), Se2 Grounded Multimodal Reasoning (Reason), and Se3 Spoken Interaction (Interact).
• Socio-Affective Interaction Loop maps to So1 Affective Perception (Percept), So2 User and Social State (Reason), and So3 Expressive Output (Interact).

2.4. Implications for Evaluation

These nine categories also provide a basis for evaluation. The relevant performance question is not limited to recognition accuracy; it is whether auditory information improves closed-loop behavior, including navigation success, manipulation reliability, dialogue efficiency, safety, social appropriateness, or user experience. Evaluation should therefore include matched ablations whenever possible. For example, an audio-aware navigation policy should be compared against audio-off, vision-off, corrupted-audio, delayed-audio, or oracle-audio variants; an audio-aware manipulation policy should measure whether contact sound reduces failure latency or improves recovery; and a social robot should test whether vocal affect improves response selection without reducing user comfort or privacy.

3. Physical Interaction Loop: Sound as Physical Evidence for Action

Section 2 defines an auditory embodied loop as a progression from perceptual cues $z_t$, to task-relevant state $s_t$, and then to action or output $(u_t,y_t)$. The Physical Interaction Loop instantiates this formulation with sound as physical evidence. Here, $z_t$ denotes acoustic observations such as source directions, event boundaries, separated signals, contact transients, microphone-array features, and uncertainty estimates. The inferred state $s_t$ denotes physical beliefs such as source memory, room geometry, acoustic maps, object material, container state, contact state, or manipulation progress. The resulting action or output $(u_t,y_t)$ may be a navigation motion, manipulation correction, failure-recovery decision, warning, spatial audio cue, or sonic feedback signal.

This loop is therefore not a collection of audio perception tasks. It describes how sound enters physical agency: first as an observation that must be stabilized under motion, ego-noise, reverberation, and overlapping sources; then as evidence for latent physical state; and finally as a signal that changes behavior or feedback. Figure 4 should be read in this sense: it shows a progression of embodied roles rather than a fixed processing pipeline.

3.1. Physical Acoustic Sensing

Physical acoustic sensing corresponds to the Percept mapping $z_t=f_{\mathrm{percept}}(·)$ in Section 2. Its goal is to convert situated sound into actionable acoustic cues, rather than to solve recognition in isolation. In the Physical Interaction Loop, these cues include source direction, event timing, separated or enhanced signals, microphone-array spatial features, contact transients, and uncertainty estimates that can support later physical reasoning or action.

Methodologically, this stage contains three recurring paradigms. Geometry-based robot audition uses microphone placement, array processing, localization, tracking, separation, and beamforming to stabilize acoustic observations. Learning-based perception estimates event boundaries, sound classes, enhancement targets, or separable sources from noisy observations. Active audition treats the robot’s motion and sensing geometry as part of the perceptual process. Representative works provide two kinds of support: LOCATA offers a benchmark for localization and tracking under dynamic source-sensor geometries, while ODAS shows how localization, tracking, separation, and post-filtering can be integrated into an embedded audition stack [6,7].

The methodological shift is from passive acoustic estimation to active and situated sensing. Recent systems place microphones on mobile robots, UAVs, robot arms, or robot swarms, so the listening geometry itself becomes part of the problem. Outdoor HRI sound localization, humanoid array optimization, UAV sound detection, adaptive beamforming with moving objects, swarm active audition, and robotic speech enhancement with a movable microphone array all reflect this trend [47,48,49,50,51,52]. Active audio-visual separation further shows that an embodied agent can move to improve the separability of a target sound, rather than treating separation as a fixed-sensor preprocessing step [8,9]. CAVER further connects sensing with exploration by letting a robot interact with objects to build audio-visual representations under uncertainty [53].

The value of physical acoustic sensing depends on whether the sensed cues can support later behavior. A source direction can guide search, a separated signal can improve speech or event recognition, an event boundary can trigger safety monitoring, and an uncertainty estimate can help the agent decide where to attend next. A recurring limitation is that many systems report localization, separation, or enhancement accuracy, but do not always measure how much these estimates improve navigation, manipulation recovery, or human-robot interaction.

3.2. Audio-Physical Understanding

Audio-physical understanding corresponds to the Reason mapping $s_t=f_{\mathrm{reason}}(z_{\le t},x_{\le t})$. The central question is how acoustic cues become beliefs about physical state. In this loop, $s_t$ may represent a navigation belief, source memory, acoustic map, room geometry, object material, container state, contact mode, or failure hypothesis. This stage is therefore defined by interpretation: the agent must explain what physical situation produced the observed sound and how that belief should constrain future action.

The most mature line is audio-visual navigation, where an agent uses spatial audio and egocentric vision to infer room geometry, source location, and a path toward an emitting target. SoundSpaces established audio-visual navigation in acoustically simulated 3D environments, and waypoint-based navigation showed that an acoustic memory can support structured planning instead of short-horizon reactive motion [10,12]. SoundSpaces 2.0 broadened this line by enabling continuous spatial sampling, configurable microphones and materials, and downstream tasks such as navigation, mapping, localization, separation, acoustic matching, and far-field ASR [11].

Subsequent work expands the reasoning problem from “follow the sound” to “understand the physical situation behind the sound.” Semantic audio-visual navigation, audio-visual-language navigation, semantic-prior-based navigation, generalizable audio-visual policies, reliability-aware fusion, residual cross-modal fusion, multi-agent collaboration, and adversarial audio-visual navigation all study how audio should be fused, trusted, or ignored under occlusion, distractors, moving sources, and noisy acoustic fields [23,54,55,56,57,58]. Sonicverse and sim-to-real acoustic field prediction emphasize a complementary question: how to build simulators and acoustic models whose reverberation, material response, and frequency-dependent fields are realistic enough for embodied learning [14,59].

A second branch of P2 concerns physical-property and scene understanding. Here, the agent reasons about what an object or environment is like from the sound produced by interaction: material, fill state, hollowness, contact mode, room layout, or acoustically grounded scene state. Audio-visual floorplan reconstruction shows that sound can reveal geometry and room semantics beyond the camera field of view, making it a natural example of acoustic mapping and scene understanding [13]. CAVER learns audio-visual object representations through exploratory interaction, while SonicSense shows that in-hand acoustic vibration can support container-state differentiation, material prediction, shape reconstruction, and object re-identification [53,60]. NaVLA-style multimodal instruction navigation further indicates that sound can serve as a disambiguating cue for physically grounded navigation goals [24]. The key challenge is causal ambiguity: the same sound may arise from different objects, materials, distances, or actions, so robust systems need uncertainty-aware fusion and ablations that test whether audio changes the inferred state, not only the recognition score.

3.3. Audio-Physical Interaction

Audio-physical interaction corresponds to the Interact mapping $(u_t,y_t)=f_{\mathrm{interact}}(s_t,x_t)$. The defining criterion is that auditory evidence changes behavior or feedback. In the Physical Interaction Loop, the inferred state $s_t$ should alter navigation, manipulation, failure recovery, spatial audio generation, or sonic feedback, rather than only improve an offline classifier. This stage is especially important for contact-rich settings because audio can reveal high-frequency events such as impact, scraping, slipping, pouring, and tool chatter before they are visually obvious.

Methodologically, P3 contains three families. Contact-audio policy learning uses sound as part of the state stream for manipulation. Audio-driven failure detection uses acoustic transients or vibration to identify slip, collision, empty pouring, or other process failures. Sonic feedback and spatial audio generation use sound as an output channel that makes robot state, event location, or task progress legible to humans or to the robot itself.

The strongest recent evidence comes from audio-guided manipulation and audio-driven failure detection. Sound-triggered mobile manipulation uses auditory events to initiate and guide task execution [61]. Play it by Ear studies audio-visual imitation learning for manipulation under occlusion, while See, Hear, and Feel shows that visual, auditory, and tactile sensing jointly improve dense packing and pouring [15,16]. Audio-VLA injects contact audio into VLA-style robotic manipulation to perceive contact events and dynamic process feedback [17]. ManiWAV uses an ear-in-hand device to collect in-the-wild demonstrations with synchronized audio and vision, showing that contact sound helps with events, contact modes, surface materials, and object states [62]. Hearing Touch treats contact microphones as an audio-based proxy for tactile information and transfers audio-visual pretraining to contact-rich manipulation [63]. That Sounds Right uses auditory self-supervision for dynamic manipulation, and SonicSense demonstrates that in-hand acoustic vibrations can support object perception during active exploration [60,64]. Together, these works suggest that audio is not only an auxiliary channel; it can capture high-frequency interaction dynamics that are difficult for vision and often expensive for tactile sensing.

A complementary P3 direction studies sonic feedback and spatial audio generation for interaction. Consequential robot sounds, functional robot sounds, and explicit sonification can make contact, motion, or internal state legible to nearby users, while spatialized audio can expose where an event occurred or where attention should move [18,36]. These interfaces are physical because they close the loop between robot action, acoustic consequence, and human or robot adjustment. Future benchmarks should therefore report action-level metrics: task success under audio removal, time-to-detect slip or collision, recovery rate after failure, unsafe-contact rate, and human understanding of robot state. A strong P3 result should show that sound changes behavior, not just that an audio encoder improves an offline classification head.

4. Semantic Grounding-and-Action Loop: Speech, Sound, and Language as Task-Bearing Evidence for Grounded Action

Section 2 defines an auditory embodied loop as a progression from perceptual cues $z_t$, to task-relevant state $s_t$, and then to physical or communicative output $(u_t,y_t)$. The Semantic Grounding-and-Action Loop instantiates this formulation with speech, task-relevant sound, and language as evidence for task meaning. Here, $z_t$ may include lexical content, candidate intents and slots, dialogue acts, prosody, stress, pauses, deictic expressions, detected sound events, temporal structure, and recognition uncertainty. The inferred state $s_t$ may comprise grounded intent, grounded referents, spatial and temporal constraints, dialogue commitments, affordances, candidate skills, task memory, plans, and confidence or risk estimates. The output space $(u_t,y_t)$ includes physical and communicative alternatives such as execution, clarification, confirmation, waiting, refusal, explanation, correction, and repair.

This loop is therefore not a fixed ASR–LLM–TTS pipeline. Rather, it progresses from task-bearing auditory cues through the construction of a grounded task state to the selection of physical or communicative actions. Compared with the Physical Interaction Loop, the emphasis is not only on where a sound originated or what caused it, but on what it means for the current task. Prosody is included here when it changes task interpretation or execution; affective user-state inference is treated in Section 5. Figure 5 should be read as a progression of embodied semantic roles rather than a fixed speech-recognition–planning–response pipeline.

4.1. Speech and Audio Understanding

Speech and audio understanding corresponds to the Percept mapping $z_t=f_{\mathrm{percept}}(·)$ in Section 2. Its goal is to produce a task-bearing auditory representation rather than a transcript alone. This representation should preserve lexical, acoustic, temporal, contextual, and uncertainty information that may alter later grounding or action, while leaving the construction of a complete task state to Se2.

Methodologically, Se1 contains three recurring families. First, ASR–SLU pipelines transcribe or encode speech before estimating candidate intents, slots, commands, or semantic frames. Whisper-style ASR, SLURP, and Fluent Speech Commands provide representative foundations for robust transcription and spoken language understanding [65,66,67]. For embodied use, however, the front end must preserve action-critical entities, numbers, negation, spatial relations, temporal order, and confidence, because small errors in these elements can change a plan even when overall word error rate is low.

Second, end-to-end speech-conditioned methods retain speech representations long enough to preserve information needed by downstream task interpretation or control. VLAS integrates spoken instructions into manipulation, while speech-instructed navigation shows why spoken modifiers and preferences must be retained when they affect terrain constraints and path selection [19,21]. Although these systems span multiple stages, their relevance to Se1 lies in preserving action-relevant speech information before grounding and control.

Third, prosody- and context-aware methods use stress, intonation, pauses, hesitation, deictic expressions, and dialogue history to preserve disambiguating evidence or produce alternative semantic hypotheses. Prosody-aware instruction understanding shows that vocal emphasis can help determine which object, relation, or action a user intends [20].

Task-relevant nonspeech events also belong to Se1 when they are detected and represented with event type, timing, and uncertainty. A doorbell, microwave beep, alarm, or ringing phone may be recognized at this stage; whether that event constitutes a navigation goal, task transition, safety constraint, or required response is determined in Se2. Estimating the physical source or cause of such a sound belongs to the Physical Interaction Loop, whereas representing it as a candidate task cue belongs here.

Se1 should be evaluated not only by transcription accuracy, but by whether action-relevant entities, constraints, prosody, event cues, and uncertainty are preserved under realistic acoustic conditions. Detailed perceptual metrics and stress tests are summarized in Section 6.

4.2. Grounded Multimodal Reasoning

Grounded multimodal reasoning corresponds to the Reason mapping

$$s_t=f_{\mathrm{reason}}(z_{\le t},x_{\le t}).$$

It maps the auditory and linguistic cues produced by Se1, together with visual observations, environment state, dialogue history, and prior actions, into a grounded task state. In this loop, $s_t$ may contain grounded intent, grounded referents, spatial and temporal constraints, dialogue commitments, affordances, source or event memories, candidate skills, executable plans, and confidence or risk estimates.

Methodologically, Se2 contains three families. The first is structured and probabilistic grounding. Classical robotic language grounding links linguistic constituents to objects, places, paths, events, and actions under uncertainty [41,68,69]. R2R and ALFRED later operationalized grounded instruction following through partial observability and sequential action [70,71]. Although these benchmarks are not primarily auditory, they define embodied grounding settings into which speech and task-relevant sound must be integrated.

The second family is spatial, event-memory, and assistance-aware grounding. This family is especially important for auditory embodied intelligence because sounds are often intermittent, outside the field of view, or separated in time from the action they should influence. AVLMaps binds audio, vision, and language in a spatial representation so that sound snippets and language queries can refer to sounding objects, events, and locations [22]. AVLEN combines audio-visual navigation with language assistance and query decisions, making auditory evidence relevant both to spatial belief and help seeking [23]. CAVEN extends conversational navigation to noisy environments in which an agent must decide how to approach an audio goal and when dialogue is needed [26]. NaVLA${}^2$ moves toward a unified vision-language-audio-action formulation for multimodal instruction navigation [24]. Together, these works motivate grounded states that retain time-stamped event memory, object–place bindings, source uncertainty, task phase, dialogue history, and the provenance of auditory evidence.

The boundary between the Physical and Semantic Interaction Loops remains important. A physical module may infer that a click indicates successful insertion, that an alarm originates in another room, or that an impact was caused by a dropped object. Semantic reasoning then determines whether the insertion subtask is complete, the alarm introduces a safety constraint, or the dropped object should become the next recovery goal. Sound becomes semantic evidence when a physically interpreted event revises task state, memory, plan, or action priority.

The third family is foundation-model and VLA reasoning. SayCan constrains language-model skill proposals with robot affordances, while Inner Monologue uses environmental and human feedback for iterative replanning [72,73]. Representative multimodal and VLA systems such as RT-2, PaLM-E, and OpenVLA further shift the field toward generalist skill selection, embodied representations, and language-conditioned policy learning [3,4,5]. These systems are not primarily auditory, but they provide architectures in which speech and sound can enter as state evidence. Their relevance to this survey lies in whether auditory information becomes a first-class part of policy-level reasoning rather than remaining ASR preprocessing.

The central Se2 criterion is causal contribution: removing, corrupting, or delaying a task-relevant auditory cue should measurably alter grounding, confidence, memory, or planning, whereas irrelevant audio should not dominate the decision. A strong system should also revise an incorrect auditory hypothesis when later observations or feedback contradict it.

4.3. Spoken Interaction

Spoken interaction corresponds to the Interact mapping

$$(u_t,y_t)=f_{\mathrm{interact}}(s_t,x_t).$$

It closes the semantic loop by selecting among physical and communicative actions. Given a grounded task state, an agent may execute a plan, wait, ask for clarification, confirm a referent, refuse an unsafe or impossible request, explain uncertainty, accept a correction, repair a failed action, or request human assistance. These are embodied policy choices under uncertainty and risk, not peripheral language-generation functions.

Methodologically, Se3 contains three families. The first is streaming and turn-taking coordination. Endpointing, turn-taking, overlap handling, interruption detection, barge-in recovery, and response latency determine whether a robot can coordinate speech with ongoing action. HRI turn-taking work shows how multimodal conversational cues can support decisions about when to take, hold, or yield the floor [25]. Full-duplex interaction further requires the agent to coordinate listening, speaking, interruption handling, and physical action in real time.

The second family is clarification, correction, and help seeking. TEACh connects dialogue, visual state, and household task execution; DialFRED makes clarification part of embodied instruction following; and TEACh-DA links dialogue acts such as request, confirm, acknowledge, and correct to task progress [74,75,76]. These benchmarks are not primarily auditory, but they define embodied interaction settings into which spoken language must be inserted. AVLEN and CAVEN provide more explicitly auditory examples in which an agent balances navigation progress against the cost of asking for help [23,26]. The central act-versus-ask decision is whether clarification is expected to prevent a costly error without introducing unnecessary queries when the grounded state is already reliable.

The third family is refusal, explanation, and action repair. Unsafe, infeasible, contradictory, or weakly grounded requests may require refusal or escalation rather than execution. After failure, the agent should report what happened, expose relevant uncertainty, update common ground, and replan. Dialogue-based safety explanations and evidence-grounded HRI with persistent memory illustrate how explanations can be tied to observations and action history rather than generated as detached text [27,77].

Se3 should be judged by whether interaction prevents unsafe or incorrect actions, enables timely clarification and repair, and improves task progress with acceptable latency and user burden. Detailed interaction metrics are summarized in Section 6.

Overall, the Semantic Grounding-and-Action Loop converts task-bearing speech, sound, and language into grounded state and then into a selection over physical and communicative actions. Its main open gap is that auditory evidence still too rarely changes the agent’s belief, policy, and recovery mechanism in a causal and transparent way.

5. Socio-Affective Interaction Loop: Affective and Social Cues as Evidence for Adaptive Interaction

Section 2 defines an auditory embodied loop as a progression from perceptual cues $z_t$, to task-relevant state $s_t$, and then to physical or communicative output $(u_t,y_t)$. The Socio-Affective Interaction Loop instantiates this formulation with affective, paralinguistic, and interactional cues as evidence for human-centered adaptation. Here, $z_t$ may include prosody, hesitation, laughter, silence, overlap, turn timing, and multimodal synchrony. The inferred state $s_t$ is a bounded user and social belief state that may include engagement, confusion, trust, user preference, relationship history, applicable norms, and uncertainty. The output $(u_t,y_t)$ may include physical choices such as pausing, assisting, yielding, or waiting, together with expressive choices such as speech style, gaze, gesture, facial expression, nonverbal sound, or sonified feedback.

This loop is not a fixed emotion-recognition–user-state-classification– emotion-generation pipeline. It progresses from uncertain affective evidence, through a temporally grounded and norm-constrained social state, to adaptive behavior. Section 4 treats speech and sound primarily as task-bearing evidence; this section focuses on how the user participates in interaction and how the robot should adapt its timing, style, channel, and intensity. Prosody belongs to Section 4 when it changes task meaning or grounding, and to this loop when it indicates engagement, urgency, confidence, or comfort. Figure 6 should be read as a progression of socio-affective embodied roles rather than a fixed processing pipeline.

5.1. Affective Perception

Affective perception corresponds to the Percept mapping $z_t=f_{\mathrm{percept}}(·)$ in Section 2. Its goal is to extract reliable affective, paralinguistic, and interactional cues from situated multimodal observations. The output of So1 should not be a definitive assertion that a user is happy, angry, anxious, or disengaged. Instead, it should preserve uncertain evidence for competing interpretations in So2, because a pause, raised voice, laugh, or interruption depends on the speaker, task phase, acoustic environment, and recent interaction.

Methodologically, So1 contains three recurring families. The first is acoustic and paralinguistic affect sensing. These methods estimate discrete affect categories, continuous dimensions such as arousal and valence, or observable interactional cues from prosody, voice quality, speaking rate, hesitation, laughter, silence, overlap, and turn timing. Speaker attribution is relevant when it supports speaker-conditioned calibration, helping distinguish personal vocal baselines from interaction-related changes rather than enabling unrestricted identity inference.

The second family is multimodal affect and engagement perception. Auditory evidence is fused with facial expression, gaze, gesture, posture, touch, dialogue context, and temporal synchrony because individual cues are often ambiguous. Speech-emotion-motion and emotion-aware gesture systems illustrate how vocal, visual, and motion representations can be connected in embodied platforms [29,30]. Although these systems span perception and expression, their relevance to So1 lies in extracting affective evidence before a response is selected.

The third family is personalized and uncertainty-aware perception. Vocal expression varies across speakers, languages, accents, cultures, devices, and environments, so a universal acoustic-to-social mapping is rarely reliable. Personalized speech emotion recognition in HRI demonstrates the value of adapting to individual vocal characteristics [28]. A practical system should expose calibrated confidence, represent disagreement across modalities, and abstain when evidence is weak or contradictory.

The embodied setting adds far-field microphones, reverberation, ego-noise, overlapping speakers, moving users, and the robot’s own sounds. A strong So1 system should preserve calibrated evidence under these shifts and show that the cues improve later timing, comfort, clarification, or social appropriateness. Evaluation should therefore include speaker and domain shift, overlap, confidence calibration, privacy-preserving processing, and downstream interaction effects; detailed perceptual metrics remain in Section 6.

5.2. User and Social State

User and social state corresponds to the Reason mapping $s_t=f_{\mathrm{reason}}(z_{\le t},x_{\le t})$ in Section 2. It transforms affective cues, dialogue history, task progress, environmental context, prior interaction, and user preferences into an action-relevant social belief state. This state should distinguish transient conditions, such as attention, confusion, stress, or willingness to continue, from stable user preferences and relationship-level variables such as trust, rapport, social role, and interaction history.

Methodologically, So2 contains three recurring families. The first is temporal user-state estimation. A pause may indicate reflection, uncertainty, politeness, distraction, or sensing failure, while a raised voice may reflect urgency, excitement, frustration, or noise. Social reasoning must therefore integrate evidence across turns, represent alternative hypotheses, and revise them when later observations conflict with an earlier interpretation. The goal is not a fixed emotion label, but a temporally consistent belief about interaction conditions relevant to action.

The second family is personalized and relational modeling. A user model may represent preferred explanation detail, interaction pace, accessibility needs, or tolerance for proactive assistance. A relationship model may retain prior corrections, failures, successful adaptations, and changes in trust or rapport across sessions. Work on enculturated companion design emphasizes that personalization and perceived care must be situated within the user’s social and cultural context [32]. Longitudinal studies also show that immediate engagement may not transfer across settings or persist after the robot is removed, making long-term evaluation necessary [34].

The third family is norm-, culture-, privacy-, and safety-constrained reasoning. A robot should not adapt merely because an affect classifier produces a confident score. It must consider whether the inference is relevant, whether the context is sensitive, whether the user has consented to personalization, and whether asking directly would be safer. Social roles and cultural conventions affect silence, gaze, interruption, emotional intensity, and empathy. Work on when and how robots should express empathy shows that empathy is a timing and policy problem: clarification or a neutral response may better preserve user agency [31].

A useful So2 representation is therefore a bounded belief state over interaction hypotheses, including short-term state, longer-term preferences, relationship history, applicable norms, confidence, and evidence provenance. It should also support non-inference: when evidence is weak, culturally ambiguous, privacy-sensitive, or irrelevant, the agent may maintain its current policy, ask directly, or avoid personalization.

A strong So2 system should maintain temporally consistent and calibrated beliefs, improve adaptation relative to history-free or non-personalized baselines, and recover after an incorrect hypothesis. Evaluation should include personalization gain, memory consistency, uncertainty calibration, cultural and language variation, consent and privacy cost, and longitudinal effects on trust, comfort, engagement, and task progress. It should also test whether later evidence corrects an initially wrong social hypothesis. Adaptation-off and memory-off ablations are important for establishing causal contribution.

5.3. Expressive Output

Expressive output corresponds to the Interact mapping $(u_t,y_t)=f_{\mathrm{interact}}(s_t,x_t)$ in Section 2. It closes the socio-affective loop by using the inferred state to modulate physical and communicative behavior. So3 concerns response timing, channel, style, and intensity. The central question is not whether output appears emotional in isolation, but whether it is warranted, understandable, appropriately timed, safe, and aligned with the robot’s capabilities.

Methodologically, So3 contains three recurring families. The first is expressive speech and interaction timing. A robot can adapt speaking rate, intensity, utterance length, pauses, backchannels, endpointing, and latency. It may slow down when the user appears confused or issue a concise warning in a safety-critical situation. When the state estimate is uncertain, expressive speech should invite correction rather than assert an inferred emotion as fact.

The second family is coordinated multimodal expression. Speech style can be synchronized with facial expression, gaze, head movement, gesture, posture, and, where appropriate and consented to, affective touch. Co-speech gesture generation connects semantic and affective information with humanoid motion [35]. Work on touch, robot-led emotion regulation, and adaptive facial expression further illustrates the shift from static displays toward interaction-dependent behavior [78,79,80]. The channels should jointly communicate attention, uncertainty, encouragement, warning, or turn ownership without exaggerating the robot’s internal state.

The third family is nonverbal sound and sonified feedback. Designed tones, earcons, spatialized sound, movement sound, and sonification can communicate waiting, uncertainty, completion, warning, or navigation intent when visual attention is divided. Work on explicit sonification, consequential and functional robot sounds, and nonverbal sound in HRI shows that auditory output can improve legibility and coordination, but can also create annoyance, startle, or misleading interpretations [18,36,37].

The boundary with the other loops is important. Se3 determines the task-level communicative act—whether to execute, ask, confirm, refuse, explain, or repair—whereas So3 determines how that act is timed and expressed. Likewise, sound that communicates contact failure or physical state belongs primarily to P3, while sound regulating attention, turn-taking, comfort, trust, or social coordination belongs to So3.

Expressive output is therefore a bounded social control problem, not merely emotional imitation. A robot should not maximize apparent empathy or intensity; a neutral or less expressive response may be more appropriate. Strong So3 systems should be compared with style-neutral, expression-off, and sonification-off baselines and evaluated through timing, coordination, cross-modal consistency, comfort, annoyance, appropriateness, trust calibration, uncertainty communication, and perceived safety.

Overall, the Socio-Affective Interaction Loop reframes social audio from isolated emotion recognition to uncertainty-aware interaction regulation. The agent extracts affective evidence, constructs a temporally grounded and norm-constrained social belief state, and uses that state to modulate physical and expressive behavior. The desired outcome is calibrated, transparent, privacy-aware, and socially bounded interaction.

6. Datasets, Benchmarks, and Metrics

This section summarizes the datasets, benchmarks, and metrics for auditory embodied intelligence. These three elements play different roles: datasets provide sensory observations and annotations, benchmarks define tasks and protocols, and metrics determine what counts as progress. Due to the scarcity of datasets covering complete closed-loops, we organize the discussion around the three stages. At the Percept stage, evaluation asks whether an agent can extract reliable auditory or multimodal cues. At the Reason stage, it asks whether these cues improve state estimation, grounding, planning, or user-state inference. At the Interact stage, it asks whether auditory information meaningfully changes action, dialogue, feedback, safety, or social behavior. Table 3 summarizes representative datasets and benchmarks, and Table 4 summarizes metrics already reported in the literature as well as those still needed for closed-loop auditory embodied agents.

Figure 7. Embodied audio dataset timeline. Colors indicate the primary stage. Circle size encodes the closest reported scale proxy: hours/clips for web or egocentric data, room impulse response density (RIR) density for acoustics, number of recordings, objects, demonstrations, or episodes for embodied resources. Vertical bands indicate data context, and dashed borders mark platforms, devices, or system-style resources.

Figure 7. Embodied audio dataset timeline. Colors indicate the primary stage. Circle size encodes the closest reported scale proxy: hours/clips for web or egocentric data, room impulse response density (RIR) density for acoustics, number of recordings, objects, demonstrations, or episodes for embodied resources. Vertical bands indicate data context, and dashed borders mark platforms, devices, or system-style resources.

6.1. Datasets and Benchmarks

Datasets provide the sensory observations and annotations from which auditory embodied agents learn, while benchmarks turn those resources into comparable tasks, splits, protocols, and success conditions. We discuss them together because many resources serve both roles: they define what is recorded and also how progress is measured. A useful resource should record not only what the agent hears, but also the situation in which the sound occurs, including microphone geometry, robot or human actions, visual context, task or dialogue state, and the interaction partner. Across the three stages, the central difference is whether the resource labels perceived signals, inferred states, or behavioral consequences.

Percept-stage resources. Percept-stage datasets and benchmarks evaluate the listening front end before the agent reasons or acts. They ask whether auditory or multimodal cues can be extracted reliably under noise, reverberation, device variation, motion, and domain shift. Acoustic Perception resources focus on event classes, acoustic scenes, source locations, and temporal activity. AudioSet [81] represents large-scale sound-event supervision, while VGGSound [87] connects visible objects and actions with sound. Spatial and embodied listening resources such as LOCATA [6], STARSS23 [90], and Real Acoustic Fields [111] stress localization, room transfer, distributed sensing, and real acoustic fields; BatVision [116] further illustrates active echo-based egocentric perception. Table 3 gives the fuller list of event, scene, room-acoustic, robot-listening, and transfer benchmarks. Semantic Perception resources connect audio to symbolic content. Speech Commands [118] and SLURP [66] cover command and intent understanding, LibriSpeech [119] anchors ASR-style supervision, and AudioCaps [126] and Clotho [127] map environmental audio to captions. Transfer benchmarks such as SUPERB [131], AudioBench [132], and AIR-Bench [133] test whether such representations generalize across speech, paralinguistic, environmental, and music tasks. Physical Perception resources capture directly observable physical cues in sound. The Greatest Hits [134] studies sounds produced by striking and scraping objects in video, making it useful for learning contact and material cues before moving to state reasoning or control. Social Perception resources represent how people sound and behave in affective displays, speaker recordings, or dyadic interaction. IEMOCAP [135] and RAVDESS [136] are typical emotional-speech resources, while VoxCeleb1/2 [139,140] support speaker recognition. AVEC 2019 [144] illustrates how such front-end cues can be benchmarked for state-of-mind recognition, depression assessment, and cross-cultural affect sensing.

Reason-stage resources. Reason-stage resources go beyond recognizing what is heard. They ask whether audio changes an inferred state, answer, plan, or physical explanation under ambiguity. Acoustic Reasoning resources are led by audio-visual navigation and spatial inference. SoundSpaces [10], SoundSpaces 2.0 [11], Sonicverse [14], and BeDAViN [145] place agents in simulated or benchmarked 3D environments where audio carries information about source location, room geometry, reverberation, distractors, and navigation goals. Hear You Are QA [146] turns simulated visual-acoustic scenes into spatial question answering, while AVLEN [23] adds language feedback and query decisions. The fuller set in Table 3 shows how this line expands from source localization to acoustic memory, map-based grounding, and help-seeking behavior. Semantic Reasoning resources use audio as evidence for questions, dialogue, and multi-step inference. MUSIC-AVQA [150], Clotho-AQA [151], and AVSD [152] cover early audio-visual QA and dialogue settings, while newer benchmarks such as MMAU [153], CompA [156], and MMAR [158] make compositional, open-ended, and rationale-based reasoning more explicit. These resources shift evaluation from recognizing acoustic content to justifying answers over temporal, semantic, and multi-source evidence. Physical Reasoning resources connect sound to latent object properties and cross-modal object state. ObjectFolder [163], ObjectFolder 2.0 [164], and ObjectFolder Real [165] provide multisensory object representations, real object impacts, and tactile-video observations, while RealImpact [166] records impact sound fields with physical and visual metadata. RSAudio [167] adds synthetic audio-visual impact examples for shape and material inference. Together these resources ask whether sound can reveal material, geometry, contact, or out-of-view physical state rather than only event identity. Social Reasoning resources use speech audio together with language, face, body, or context to infer human state. CMU-MOSI [170], CMU-MOSEI [171], and MELD [172] illustrate the move from clip-level affect toward contextual sentiment, emotion, and dialogue understanding. These resources show that social sound is most useful when interpreted with context and interaction history.

Interact-stage resources. Interact-stage resources evaluate behavior after listening and reasoning. They should include audio together with actions, feedback, recovery, dialogue turns, or user responses; because public closed-loop auditory benchmarks are still rare, current evidence often comes from task protocols in robot manipulation, conversational navigation, active listening, and HRI feedback studies. Acoustic Interaction resources test whether movement improves what the agent can hear. Move2Hear [8] and Active Audio-Visual Separation [9] evaluate active listening under changing mixtures and moving sources. Semantic Interaction resources connect audio with task dialogue and decisions about when to act or ask. AVN-Instruct/CAVEN [26] evaluates navigation toward audio goals with help-seeking decisions, and the HRI Turn-Taking Corpus [25] provides spoken human-robot dialogue cues for deciding when to take, hold, or yield the floor. Command corpora remain useful as weakly interactive grounding resources; representative examples are listed in Table 3. Physical Interaction resources test whether contact sound improves action. Play it by Ear [15] studies audio-visual imitation under occlusion, while That Sounds Right [64], ManiWAV [62], SonicSense [60], and Audio-VLA [17] evaluate whether contact audio or vibration improves object-state inference, process monitoring, and manipulation success. AudioPouring-style resources [178] show the same idea in liquid-level and flow estimation. The important distinction from physical reasoning is causal: audio should change the action trajectory, not only improve an offline classifier. Social and Safety Interaction resources capture how people respond to one another and how agents communicate uncertainty, safety, or intent. SEWA DB [182] and K-EmoCon [183] provide multimodal affect and engagement signals, while Xperience-10M [186] represents a broader egocentric interaction stream with synchronized video, depth, pose, motion, IMU, audio, and language annotations. HRI and VLA-adjacent benchmarks such as SafeVLA-Bench [189] and Beyond Task Success [190] identify diagnostics for unsafe actions, confidence, latency, and behavioral failure signatures. A mature auditory Interact-stage benchmark should evaluate not only final success, but also timing, recovery, uncertainty communication, safety, and the effect of removing or corrupting audio.

6.2. Metrics

Metrics define what counts as progress. For auditory embodied intelligence, they should answer three progressively stronger questions. First, at the Percept stage, did the agent extract the right auditory observation under realistic acoustics? Second, at the Reason stage, did this observation change the inferred state, answer, plan, or explanation in a justified way? Third, at the Interact stage, did the use of sound improve action, dialogue, recovery, safety, or human experience? Table 4 therefore groups metrics by stage and by the role of audio. The table also separates core task metrics from embodied diagnostics, because high offline accuracy can hide failures caused by ego-noise, latency, user interruption, domain shift, or unsafe action.

Percept-stage metrics. Percept-stage metrics evaluate the listening front end before its output is used for planning or interaction. Acoustic Perception should combine spatial, event, separation, enhancement, and runtime measures. Direction-of-arrival and localization error, tracking continuity, localization recall, and SELD metrics evaluate where and when sounds occur in protocols such as LOCATA and DCASE/STARSS23 [6,90]. Sound-event and scene analysis should report accuracy, macro/micro F1, mean average precision, event- or segment-based error rate, and threshold-robust scores such as PSDS [191]. Separation and enhancement should not rely on a single signal score: SDR/SIR/SAR from BSS Eval [192], SI-SDR [193], intelligibility metrics such as STOI [194], and perceptual quality metrics such as PESQ, DNSMOS, or ViSQOL [195,196,197] capture complementary failure modes. For embodied systems, these metrics should be sliced by microphone geometry, robot motion, ego-noise, reverberation, source overlap, device shift, compute, and real-time factor. Semantic Perception metrics test whether audio has been converted into action-relevant symbols. ASR should report WER or CER, but also entity WER, semantic preservation, and downstream intent or slot metrics when commands, names, numbers, or spatial relations matter [198,199,200]. Spoken-command and SLU settings use command accuracy, intent accuracy, and slot F1; captioning and audio-language grounding use BLEU, METEOR, CIDEr, SPICE, SPIDEr, and audio-aware caption metrics such as MACE [201]. SUPERB, HEAR, AudioBench, and AIR-Bench further motivate reporting task-family scores separately across speech, paralinguistic, environmental, and music understanding [117,131,132,133]. Physical and Social Perception metrics ask whether observable contact, material, speaker, and affective cues are reliable. Physical perception uses contact/event classification, temporal alignment, material or operation recognition, and generalization across surfaces, tools, and unseen objects. Social perception uses accuracy, unweighted average recall, macro-F1, arousal–valence MAE/RMSE, correlation, and concordance correlation coefficient, as in affective benchmarks such as AVEC 2019 [144]. Speaker-sensitive systems should additionally report verification or diarization errors, overlap-sensitive performance, and cross-speaker, cross-accent, cross-culture, and confidence-calibration slices.

Reason-stage metrics. Reason-stage metrics evaluate whether audio changes a belief, answer, plan, or explanation rather than merely producing a better perceptual label. Acoustic Reasoning is most developed in audio-visual navigation. It uses success rate, SPL, path length, distance to goal, collision rate, and generalization to unseen scenes or unheard sounds [202]; AVLEN adds diagnostics such as success weighted by number of actions, success when silent, distance to goal, and query behavior [23]. These metrics should always be paired with audio-off, corrupted-audio, delayed-audio, distractor-source, and oracle-audio variants so that improvement can be attributed to acoustic evidence rather than dataset priors. Semantic Reasoning metrics include exact match, answer accuracy, multiple-choice accuracy, token F1, compositional subcategory scores, temporal/counting accuracy, abstention accuracy for unanswerable questions, and robustness to paraphrases or distractors. Open-ended AudioQA and AudioLLM evaluation should add evidence attribution, hallucination rate, confidence calibration, human preference, and calibrated LLM-as-judge or rubric scores. Benchmarks such as MMAU, MMAR, and the Audio Reasoning Challenge are useful because they separate final-answer correctness from multi-step reasoning and rationale quality [153,158,162]. Physical Reasoning metrics ask whether latent state estimates are correct: object or material classification, contact-state error, geometry or source-state error, count/temporal error for impacts, and physics-sensitive generation scores. PhyAVBench makes this idea explicit through audio-physics sensitivity tests and contrastive physical-response evaluation [169]. Social Reasoning metrics include sentiment or intent F1, user-state accuracy, arousal–valence error, turn-level consistency, sarcasm or intent scores, personalization benefit, fairness gaps, and uncertainty calibration. For embodied use, these metrics should be checked over interaction histories because a socially adaptive agent can be accurate on isolated clips but unstable, overconfident, or inappropriate across a session.

Interact-stage metrics. Interact-stage metrics measure the behavioral consequence of listening and reasoning. Acoustic Interaction asks whether movement improves perception, using separation gain, localization gain, target-to-interferer improvement, time-to-acquire, movement cost, collision or safety cost during active listening, and performance under moving sources, as in Move2Hear and active audio-visual separation [8,9]. Physical Interaction metrics include task success, task completion rate, contact-state accuracy, failure-detection time, recovery rate, unsafe-contact rate, force or pose error, action latency, and policy change under audio removal or delay [17]. These metrics are strongest when they show that contact audio changes the action trajectory, not only a classifier output. Semantic Interaction metrics include clarification success, correction success, act/ask/refuse accuracy, turn-taking F1, endpointing latency, first-response latency, barge-in detection and recovery, repair turns, dialogue coherence, and explanation faithfulness. Dialogue evaluation work, including PARADISE-style combinations of task success, dialogue cost, and user satisfaction, shows why spoken interaction should not be judged by surface text overlap alone [203,204]. Social and Safety Interaction metrics include trust, rapport, comfort, engagement, perceived intelligence, perceived safety, and satisfaction, with instruments such as the Godspeed questionnaire [205]. Safety diagnostics such as Succ-But-Unsafe and Violation Severity Index in SafeVLA-Bench [189], together with rollout failure signatures and runtime-cost diagnostics from Beyond Task Success [190], are useful templates for auditory agents. The central Interact-stage test is causal: removing, shuffling, corrupting, or delaying audio should reveal whether sound improves timing, recovery, safety, task completion, and human experience.

7. Applications and Systems

Auditory embodied intelligence is most useful when sound changes how an agent behaves in a concrete setting. This section therefore organizes applications by deployment scenario rather than by the three technical loops used in the preceding sections. We group current systems into industrial, educational, and service applications. These categories are not mutually exclusive: a factory assistant may need spoken dialogue, a classroom robot may need affective listening, and a home-care robot may need physical event detection. The point of this organization is to make the application demand explicit: what must be heard, what action should change, and what evidence would show that audio is causally useful. Figure 8 summarizes the resulting application landscape.

7.1. Industrial Applications

Factory-floor perception and anomaly response. Industrial environments give audition a direct operational role because many safety and maintenance events are acoustic before they are visible. Mobile robots or fixed robot workcells can listen for alarms, abnormal machine vibration, air leaks, tool chatter, falling objects, collisions, and human calls for help. Embedded audition stacks such as ODAS [7] illustrate the low-level requirements: real-time localization, tracking, separation, and post-filtering under overlapping sources and reverberation. In smart manufacturing, these front ends must be integrated with visual inspection, proprioception, digital twins, and human-robot collaboration workflows [207,208]. The application metric is not only event accuracy. A useful factory system should reduce time to detection, localize the source well enough for response, lower false alarms per shift, and trigger a safe robot or human action under high background noise.

Warehousing, packaging, and contact-rich manipulation. Industrial manipulation is a second major scenario because the robot’s own actions create informative sound. Packaging, sorting, insertion, wiping, pouring, fastening, and tool-use tasks produce contact signals that reveal slip, scraping, impact, empty containers, blocked motion, or failed engagement earlier than vision alone. Recent contact-audio systems show how interaction sounds can supervise dynamic manipulation, support audio-visual policy learning, and provide object-state evidence [15,16,60,62,63,64]. Audio-VLA further shows how contact audio can be inserted into a vision-language-action policy rather than treated as a separate classifier [17]. For factory deployment, this line connects naturally to packaging and assembly pipelines, where recent VLA case studies report workflow failures and deployment constraints [222]. Strong evidence in this scenario requires audio-off, delayed-audio, and distractor-audio ablations, together with task success, recovery rate, unsafe-contact rate, and audio-to-action latency.

Inspection, maintenance, and collaborative safety. Inspection robots, warehouse robots, and collaborative manipulators also need auditory awareness of people and moving equipment. Speech and paralinguistic cues can signal task requests, urgency, warnings, and misunderstanding; nonverbal sound can communicate robot state, motion intent, or completion to nearby workers. Work on proactive collaboration and human-centric manufacturing emphasizes that the robot should be predictable and responsive, not merely accurate as a perception module [207,209]. Designed robot sounds and sonification can make attention, waiting, uncertainty, or internal state more legible [18,36]. Evaluation therefore includes near-miss reduction, interruption cost, worker response time, perceived safety, and trust calibration, in addition to standard speech or sound-event metrics.

7.2. Educational Applications

Classroom teaching assistants. In schools and universities, auditory embodied systems mainly support interaction rather than physical production. A classroom robot or teaching assistant must hear questions, answers, interruptions, laughter, hesitation, group participation, and signs of confusion while respecting student privacy. Robot-supported collaborative learning uses social robots as facilitators for small-group learning, where turn-taking, prompt timing, and balanced participation are central outcomes [210]. Pepper-like platforms and HRI turn-taking corpora illustrate the platform and interaction infrastructure for this setting [25,211]. Audio is valuable when it helps the robot decide whether to invite a quieter student, ask a clarifying question, slow the pace, or hand control back to a teacher. Learning gain, participation balance, teacher workload, student comfort, and privacy protection are more meaningful than isolated recognition accuracy.

Language learning and conversational practice. Language education is a natural auditory application because the target skill itself is spoken interaction. A robot or embodied tutor can provide pronunciation feedback, conversational practice, listening exercises, prosody-sensitive correction, and adaptive dialogue. Studies of robot interaction styles for second-language conversation and AI-supported L2 education show the importance of engagement, feedback style, and behavioral participation [212,213]. Speech-prosody grounding is also relevant because emphasis, hesitation, and adverbs can change how an instruction should be interpreted or practiced [20,21]. In this scenario, audio should improve practice quality: more turns taken, better pronunciation or fluency over time, appropriate repair after ASR errors, and reduced learner anxiety. The system should avoid overfitting to accent, age, or language background and should expose uncertainty instead of correcting with unwarranted confidence.

Embodied STEM, physical education, and skill tutoring. Educational robots can also teach embodied skills: laboratory procedures, mathematics with manipulatives, music or rhythm activities, sports practice, and physical education. Here audition connects the learner’s action to feedback. Tapping, timing, tool contact, spoken self-explanation, and task sounds can reveal whether a learner is following a procedure or needs help. Work on embodied design for mathematics and voice-interactive educational robots points to this broader role of physical and vocal interaction in learning [214,215]. Sonified feedback can also make invisible process states or robot uncertainty easier to perceive [18]. The key evaluation question is whether the auditory channel improves instruction timing, skill acquisition, retention, or learner agency, not whether the robot can label a sound clip in isolation.

7.3. Service Applications

Home assistance and domestic safety. Domestic service robots use audition for events that happen outside the camera view or before a user explicitly asks for help. Household systems may listen for smoke alarms, carbon-monoxide alarms, glass breaking, knocking, appliance events, running water, dropped objects, or spoken requests. Amazon Astro illustrates the deployed product form of sound-triggered domestic monitoring, where a mobile platform can investigate or notify a user after detecting safety-relevant sounds [216]. Audio-visual navigation and audio-language mapping show how a robot can move toward sounding targets, use sound as a spatial landmark, or answer navigation queries grounded in audio, vision, and language [10,11,22,23]. The main deployment risks are privacy, false alarms, household reverberation, overlapping speakers, and user control over when the robot listens. Metrics should include response latency, false alarms per day, localization quality, task completion, and user override behavior.

Health care, eldercare, and assistive companionship. Health and eldercare services combine safety monitoring, spoken dialogue, reminders, social support, and escalation to caregivers. ElliQ represents a deployed older-adult companion built around proactive conversation, reminders, and engagement [217]; ENRICHME studies perception and interaction for assistive robots in elderly homes [218]. Health-care-oriented HRI also includes robot-patient dialogue and older adult perceptions of robots as communication partners [219,223]. In these applications, audio may provide check-ins, medication reminders, distress cues, conversation history, or evidence that a user did not respond. Because the stakes are high, audio inference should be conservative: the robot should distinguish social engagement from clinical judgment, maintain consent and data minimization, and escalate uncertain cases to humans rather than acting on a fragile affect or health estimate. Evaluation should include longitudinal engagement, comfort, privacy, escalation accuracy, missed-event rate, and caregiver workload.

Public-facing service and social companionship. Service robots in museums, hotels, restaurants, malls, airports, and public institutions depend on robust speech interaction, turn-taking, repair, and socially legible feedback. Pepper, Jibo, and Aibo show different platform forms for voice-centered social interaction and companionship [211,220,221]. Service encounters also expose social risks: users may feel embarrassed, ignored, interrupted, or over-monitored when a robot mishears them or responds with the wrong social tone [224,225]. Vocal affect, hesitation, silence, and overlap can help a robot choose whether to repeat, clarify, apologize, simplify instructions, or yield to a human staff member [43]. Designed sounds can further communicate availability, waiting, completion, or warning, but they must be evaluated for annoyance, accessibility, and cultural interpretability [18,36]. Across public service and companionship, the strongest auditory systems will be those that improve task completion and user comfort while making their listening behavior transparent and controllable.

Across these three application domains, the common lesson is that audio should be evaluated through its behavioral consequence. Industrial robots should detect and recover from physical events earlier; educational robots should improve participation and learning; service robots should provide safer, more natural, and more transparent assistance. Recognition scores remain useful diagnostics, but application evidence requires matched ablations, latency measurements, privacy analysis, and user- or task-level outcomes.

8. Challenges and Future Directions

The preceding sections show that auditory embodied intelligence is no longer a single perception problem. Sound must be extracted from real acoustic fields, grounded into physical and semantic state, and used to change robot action or human interaction. The open problems therefore appear at three coupled levels: the dataset level, where current resources rarely record complete closed loops; the stage level, where perception, reasoning, and interaction each expose different failure modes; and the system level, where audio must be integrated with vision, language, proprioception, safety, and long-term user experience. This section synthesizes these challenges using the same Physical Interaction Loop, Semantic Grounding-and-Action Loop, and Socio-Affective Interaction Loop used throughout the survey.

Figure 9. Summary of open challenges and future directions in auditory-centered embodied intelligence. The central ring follows the structure of this section, while the surrounding panels highlight representative bottlenecks and future directions across datasets, perception, reasoning, interaction, and deployment.

Figure 9. Summary of open challenges and future directions in auditory-centered embodied intelligence. The central ring follows the structure of this section, while the surrounding panels highlight representative bottlenecks and future directions across datasets, perception, reasoning, interaction, and deployment.

8.1. Dataset and Benchmark Bottlenecks

From audio datasets to embodied episodes. Current audio resources are strong for perceptual supervision but weak for embodied causality. Large-scale sound-event and audio-visual datasets such as AudioSet [81], VGGSound [87], and FSD50K [82] cover broad acoustic categories, and captioning or Audio Question Answering (AQA) resources such as AudioCaps [126], Clotho [127], WavCaps [128], Clotho-AQA [151], MMAU [153], PAQA [154], and DCASE 2025 AQA [155] support audio-language understanding. However, most of these datasets are clip-level, observer-centric, and offline rather than action-conditioned resources like embodied navigation, manipulation, or HRI protocols. They rarely include robot pose, action history, actuator state, microphone placement on a moving body, proprioception, recovery behavior, or human feedback. As a result, they can train a model to recognize what a sound is, but not whether that sound should make an embodied agent move, stop, ask, or refuse.

Missing embodied acoustics. Spatial and real-room datasets such as LOCATA [6], STARSS23 [90], CHiME-6 [96], DiPCo [97], DIRHA [98], ACE [107], OpenAIR [103], and MeshRIR [105] are essential for localization, tracking, reverberation, and far-field speech. Yet many resources still underrepresent robot-specific conditions: ego-noise from motors and wheels, changing microphone geometry, self-occlusion, wind and floor noise, asynchronous sensors, moving arrays, and non-stationary sources. Contact and physical-audio datasets are even narrower because existing resources focus on impacts, object identity, or material cues rather than the full closed-loop contact process. Greatest Hits [134], ObjectFolder [163], ObjectFolder 2.0 [164], and RealImpact [166] connect sound to materials, impacts, and multisensory objects, but they cover only a fraction of the contact modes needed for manipulation, such as sliding, pouring, scraping, insertion, tool use, breakage, and incipient failure.

Social data without social deployment. Socio-affective audio also suffers from a gap between labels and deployment. IEMOCAP [135] and RAVDESS [136] provide useful emotional speech supervision, while SEWA DB [182] and NoXi+J [184] add more naturalistic audio-visual interaction and cultural variation. Still, social-robot use requires longitudinal consent, privacy, personalization, relationship dynamics, and culturally situated norms. Short clips or acted labels cannot determine whether an adaptive robot response is appropriate, whether affect inference improves trust, or whether continuous listening creates discomfort. Future social datasets should therefore record not only audio and labels, but also interaction policies, user consent, uncertainty, adaptation decisions, and post-interaction outcomes.

Benchmarking the causal value of sound. The most important benchmark defect is the lack of matched closed-loop ablations that isolate whether sound causally changes behavior. A mature auditory embodied benchmark should ask what changes when audio is removed, delayed, corrupted, shuffled across episodes, replaced by oracle signals, or attacked by distractor sources. Navigation benchmarks such as SoundSpaces [10], SoundSpaces 2.0 [11], AVLEN [23], and Sonicverse [14] show how acoustic simulation can define embodied tasks, while manipulation systems such as That Sounds Right [64], ManiWAV [62], Hearing Touch [63], SonicSense [60], and Audio-VLA [17] show that contact audio can affect action. The next benchmark layer should make the embodied episode, rather than the isolated clip, the basic unit: synchronized audio, vision, proprioception, robot actions, dialogue turns, task state, human feedback, safety events, latency, recovery, and public protocols for audio ablation should be reported together.

8.2. Percept Stage

Physical Interaction Loop: robust robot audition under motion. The physical perceptual challenge is to listen while the robot is acting. Classical robot audition and systems such as ODAS [7] demonstrate real-time localization, tracking, separation, and post-filtering on embedded platforms, but embodied deployment adds stronger disturbances than static benchmarks: robot ego-noise, moving microphones, changing head or arm poses, occluded sources, reflections from nearby objects, and simultaneous speech or impact events. Future perceptual front ends should jointly estimate external sources and self-generated sound, expose calibrated uncertainty, and report performance by motion state, array geometry, acoustic condition, and real-time factor. Without such diagnostics, a high localization or separation score may not transfer to navigation, manipulation, or HRI.

Semantic Grounding-and-Action Loop: reverberant, multi-speaker, action-relevant speech. Semantic listening is not equivalent to generic ASR because robot language understanding must be grounded in action and environment state. A robot must preserve names, objects, spatial relations, negation, urgency, deictic expressions, and prosody under far-field, multi-speaker, and reverberant conditions. Spoken-command and SLU datasets such as Fluent Speech Commands [176] and Snips [177] are useful starting points, while Robots That Use Language [41] emphasizes that robot language understanding must be tied to action and environment state. The challenge is to evaluate semantic preservation rather than transcript quality alone: a word error is harmless if the plan is unchanged, but a small error in an object name, location, or safety constraint can be catastrophic. Future benchmarks should therefore score ASR, intent, slot, grounding, and downstream action jointly.

Socio-Affective Interaction Loop: privacy-preserving vocal state sensing. Social perception from voice is inherently uncertain and privacy-sensitive in embodied HRI. Affect and engagement resources such as IEMOCAP [135], CMU-MOSEI [171], MELD [172], and SEWA DB [182] support recognition of emotion, sentiment, or conversational state, but embodied agents must avoid overclaiming hidden mental states from noisy acoustic cues. The open problem is not only higher affect accuracy; it is calibrated, consent-aware, culturally robust inference that can decide when not to infer or when to ask. Promising directions include on-device feature extraction, privacy-preserving representations, confidence-aware adaptation, and evaluation protocols that measure comfort, trust, fairness gaps, and user control over audio data.

8.3. Reason Stage

Physical Interaction Loop: causal acoustic world models. Physical reasoning requires models that predict how actions, geometry, materials, contact, and robot motion change sound over time. SoundSpaces [10], SoundSpaces 2.0 [11], and Sonicverse [14] provide controlled spatialized environments, while physical-audio resources such as RealImpact [166] and ObjectFolder 2.0 [164] connect sound to objects, impacts, and materials. The reasoning challenge is causal rather than merely transferable: the agent must distinguish external events from self-generated sound, infer which hidden physical state produced an acoustic change, and predict how candidate actions will alter future sight-and-sound observations. Audio-Visual World Models [226] are a promising direction because they can represent these action-conditioned dynamics, but they should be evaluated with interventions that test physical causality rather than video-audio correlation alone. More general audio-aware foundation and VLA models should likewise treat audio as a first-class state channel, as illustrated by Audio-VLA [17] for contact-audio manipulation and NaVLA [24] for audio-conditioned navigation, while supporting streaming audio tokens, event memory, uncertainty estimates, and intervention tests that verify whether acoustic evidence actually changes action.

Semantic Grounding-and-Action Loop: persistent audio-language spatial memory. Semantic reasoning requires memory that binds sounds, words, objects, and places over time. AVLMaps [22] stores audio, vision, and language in spatial maps for robot navigation, AVLEN [23] and CAVEN [26] add query behavior and conversational help in noisy audio-visual navigation, and NaVLA [24] suggests a path toward unified vision-language-audio-action navigation. The challenge is persistence across delays, movements, and intermittent observations. A robot may hear a microwave beep, a person call from another room, or a dropped object outside the camera view, and then need to remember that event after several turns and movements. Future systems should maintain time-stamped audio-language memories with uncertainty, decay, source identity, and provenance, and should be evaluated on delayed grounding, contradiction resolution, and recovery after wrong acoustic hypotheses.

Socio-Affective Interaction Loop: long-term user-state and norm reasoning. Socio-affective reasoning must connect vocal cues to interaction policy across sessions. A Survey on Dialogue Management in Human-Robot Interaction [42] and Recent Advancements in Multimodal Human-Robot Interaction [43] show that social interaction depends on turn structure, task state, multimodal context, and norms rather than isolated utterances. The open challenge is to model user state without turning uncertain affect estimates into rigid labels. Long-term robots should distinguish transient acoustic states from stable preferences, represent uncertainty explicitly, and adapt only when the benefit justifies the privacy and social risk. Evaluation should include longitudinal user studies, opt-in personalization, cultural and language variation, and failure cases such as misread distress, inappropriate empathy, or excessive questioning.

8.4. Interact Stage

Physical Interaction Loop: low-latency contact-audio control. Interaction-stage physical audio must support control at the timescale of contact. Systems such as That Sounds Right [64], Hearing Touch [63], ManiWAV [62], SonicSense [60], and Audio-VLA [17] show that interaction sounds and in-hand vibration can improve manipulation or object-state inference. The next challenge is to close the loop fast enough for recovery: detecting slip, collision, empty pouring, scraping, tool chatter, or unsafe impact before visual evidence is available. Future policies should fuse contact audio with force, touch, proprioception, and vision; report audio-to-action latency; and evaluate recovery rate, unsafe-contact rate, and task success under audio-off, delayed-audio, and distractor-audio conditions.

Semantic Grounding-and-Action Loop: deciding when to ask, act, or refuse. Spoken interaction becomes embodied when the robot must choose between acting, asking for clarification, refusing unsafe instructions, or explaining uncertainty. AVLEN [23] and CAVEN [26] make help-seeking part of navigation, while the HRI Turn-Taking Corpus [25] and dialogue-management work [42] show that timing and floor control affect whether spoken interaction succeeds. Future semantic-interaction systems should optimize act/ask/refuse decisions under acoustic uncertainty and task risk. Metrics should include clarification success, correction success, endpointing latency, repair cost, unsafe action prevention, and user burden, rather than only final task success or response fluency.

Socio-Affective Interaction Loop: streaming turn-taking and sonified feedback. For social robots, audio is both input and output. The robot must listen to overlapping speech, backchannels, silence, and prosody while also producing speech, nonverbal sounds, or sonified feedback that makes its internal state legible. Hearing the Robot’s Mind [18] and The Role of Consequential and Functional Sound in Human-Robot Interaction [36] show that designed sound can make robot state and intent more transparent, while Nonverbal Sound in Human-Robot Interaction [37] highlights the importance of comfort, interpretation, and social context. The challenge is to make feedback useful without becoming intrusive. Future work should evaluate startle, annoyance, perceived safety, timing, interruption cost, and trust, and should test whether sound output improves coordination during real tasks rather than only subjective preference in isolated demos.

8.5. Deployment

Calibration and acoustic sim-to-real protocols. The acoustic sim-to-real gap is primarily a deployment challenge because models that work in controlled simulators can fail under new rooms, microphones, robot bodies, materials, motions, and distractor sources. Future deployment protocols should pair physically grounded simulators with real-world calibration sets, similar in spirit to room-acoustic and impact-field resources. A policy trained in simulation should be tested under shifted acoustics, changed microphone placement, new objects, new robot motions, and adversarial or distracting sounds. Progress should be reported not only by success rate but also by how performance degrades as the acoustic gap increases.

Safety, privacy, and threat models. Auditory systems are vulnerable to false alarms, spoofing, speech injection, deepfake voices, privacy leakage, and unsafe action triggered by misleading sounds. Diagnostic benchmarks such as SafeVLA-Bench [189] and Beyond Task Success [190] provide useful templates for separating task success from safety, latency, and failure signatures, but auditory agents require additional threat models: hidden speakers, replayed commands, inaudible or ultrasonic interference, private conversation leakage, and adversarial environmental sounds. Future evaluation should report refusal precision and recall, attack success rate, privacy leakage, calibration under uncertainty, and safe recovery after wrong acoustic beliefs.

Long-term human-centered deployment. Finally, embodied audio must be studied over time because household monitoring, eldercare, collaborative work, and companion robots involve repeated interaction, changing users, changing rooms, and evolving privacy expectations. Social platforms and studies such as Pepper [211], Jibo [220], Aibo [221], ElliQ [217], and sonified HRI [18] illustrate the demand for auditory interaction, but future work should move from short demonstrations to longitudinal protocols. The core research question is whether audio-aware adaptation improves task performance, trust, independence, and comfort without creating surveillance pressure, dependency, or culturally inappropriate behavior.

9. Conclusions

This work provides a structured review of multimodal auditory learning for embodied intelligence. We introduce a two-axis loop–stage taxonomy whose loop axis comprises the Physical Interaction Loop, the Semantic Grounding-and-Action Loop, and the Socio-Affective Interaction Loop, while its stage axis comprises Percept, Reason, and Interact. Using this taxonomy, we organize and compare how existing approaches acquire and represent auditory information, integrate it with other modalities, and use it to support embodied behavior. We further summarize representative methods, datasets, benchmarks, and evaluation metrics, and discuss current applications, challenges, and future directions.

Collectively, the reviewed literature indicates an ongoing but incomplete transition from treating audition as an isolated perceptual capability to incorporating auditory information as situated evidence for embodied reasoning and behavior. Although numerous studies report improvements in recognition, localization, and multimodal representation, comparatively few establish that these perceptual advances translate into reliable changes in an agent’s internal state estimation, decision-making, action selection, or interaction strategy. Assessing this relationship is further complicated by heterogeneous evaluation protocols, which frequently emphasize task-specific performance without isolating the contribution of auditory information to downstream behavioral outcomes. Advancing the field will therefore require more systematic integration of auditory perception with reasoning and control, evaluation under realistic acoustic and embodied conditions, and matched ablation studies that characterize when, how, and to what extent audition improves agent behavior.