From Terrain to Space: A Survey on Multi-Domain Data Lifecycle for Urban Embodied Agents

1. Introduction

Modern cities are complex systems, dynamically interwoven with physical elements (such as buildings and roads), social structures, economic activities, and environmental factors [1,2]. They are not only the cornerstones of modern civilization but are also continuously evolving, driven by cultural, economic, and technological advancements [3]. Contemporary cities have evolved beyond being collections of physical spaces to become hubs of diverse information sources, composed of Internet of Things (IoT) devices, geospatial data, and sensor data [4,5].

Against this backdrop, the concept of the “Urban Embodied Agent" (UrbanEA) [6] has emerged. It is an intelligent system (such as an autonomous vehicle, robot, or virtual avatar) that possesses a physical or virtual body to directly perceive, reason about, and execute actions within complex urban environments. By leveraging these core capabilities, such agents are envisioned to help address critical urban problems, including delivery and transportation. Compared to conventional urban agents, the core distinction lies in this embodiment: conventional agents may only analyze and predict within digital space (e.g., traffic flow models), whereas an UrbanEA uses its physical form to perceive and execute actions, interacting with the world directly [7].

To fulfill this vision, the UrbanEA’s core capability lies in its data lifecycle, which defines the entire pipeline from data perception to embodied application. As shown in Figure 1, this pipeline contains the following stage:

• Data Perception. The agent perceives multimodal data from multiple domains (handheld, vehicle, drone, satellite) to comprehensively perceive the physical world, marking the starting point of the data lifecycle.
• Data Management. Task-driven storage architectures (such as graph, vector, and spatio-temporal databases) organize and query massive, heterogeneous urban perception data, laying a solid foundation for subsequent fusion and interaction.
• Data Fusion. In this stage, the fusion strategies address the data gaps and construct a unified urban cognition.
• Task Application. The structured data, after being processed through fusion, supports advanced UrbanEA tasks such as Urban Scene Question-Answering (SQA), Vision-Language Navigation (VLN), and Human-Agent Collaboration (HAC), and generates a positive social impact.

Despite its importance, successfully implementing this end-to-end data lifecycle in urban settings remains insufficient. Existing embodied agent research focuses on indoor environments [8], which are relatively controlled, limited in scale, and structurally stable, while urban settings are different. Therefore, UrbanEA presents unique data-centric challenges that impact every stage of this lifecycle:

• Environmental Variability. Urban environments are inherently dynamic and uncertain, unlike controlled indoor settings. Data Perception in outdoor scenes must handle dramatic variations in illumination (diurnal cycles, sunlight-shadow contrasts) and challenging weather conditions (rain, snow, fog), all of which degrade perception performance.
• Limited Observability. Urban environments are vast, but individual sensors have limited coverage. Any single sensor (e.g., a vehicle-mounted camera or LiDAR) suffers from blind spots and occlusions due to its limited Field of View (FoV) and detection range. This results in spatially incomplete perceptual information, making it impossible to achieve a globally consistent scene understanding during Data Fusion.
• Interaction Complexity. An urban environment is beyond a collection of physical spaces but a complex social environment composed of numerous intelligent agents. The behavior of these agents is not simple physical motion but is driven by a multi-layered set of rules. Their behavior follows both explicit rules (e.g., traffic laws) and implicit social norms (e.g., driving habits, pedestrian etiquette, intentions signaled through body language). Understanding these interactions requires interpreting subtle cues like posture, gaze, and intent, which are far harder to capture and model during the Data Fusion and Task Application stages than simple physical motion.

These challenges highlight the complexity of the data lifecycle in urban environments, making it difficult for single-domain data to comprehensively capture the scene. To address these limitations, utilizing multi-domain data for UrbanEA has emerged as a promising strategy [13,15]. By integrating perspectives from diverse domains—such as a vehicle’s terrain perception, a drone’s aerial view, and a satellite’s global map—the system can mitigate the limitations of individual sensors and reduce blind spots to construct a more spatially complete scene understanding.

However, as shown in Table 1, existing reviews tend to adopt a task-centric or model-centric perspective, such as foundation models [10,12] or the graph neural network [9], which may not fully align with the requirements of this multi-domain approach.

Specifically, urban computing surveys primarily focus on digital agents designed for passive analytical tasks, such as traffic and weather prediction. In these frameworks, data is often treated as static inputs for offline reasoning, rather than the interactive streams required by physical agents to actively perceive and intervene in the world. Meanwhile, although existing embodied agent surveys discuss physical interaction, they generally focus on controlled indoor environments, leaving the spatio-temporal gaps inherent in complex, city-scale environments largely unaddressed. Therefore, a systematic review covering the end-to-end lifecycle, from data perception to final application, for UrbanEA is still needed. To fill this gap, we present the first comprehensive survey on the data lifecycle for UrbanEA. Our research focuses on the entire pipeline, investigating how to efficiently store, query, and fuse this complex data to support downstream embodied agent applications. Our contributions are summarized as follows:

1) Unified Data Lifecycle Framework. We propose a systematic framework that organizes the existing UrbanEA research into an end-to-end pipeline. Unlike existing surveys that focus on isolated tasks or specific domain data, our framework integrates Data Perception, Management, Fusion, and Application, providing a holistic view of how urban data flows from raw sensors to an embodied agent.

2) Fine-grained Multi-stage Taxonomy. We establish a taxonomy for each stage of the lifecycle to clarify technical boundaries. Specifically, we categorize perception by modalities and domains, classify management by storage structure and capability, and taxonomy fusion strategies based on the specific Domain Gaps (Representation, Quality, Spatio-temporal, and Semantic) in the real world.

3) Forward-looking Research Roadmap. We identify cross-stage challenges and synthesize them into a strategic roadmap spanning Method-level, System-level, and Societal-level dimensions. By extending the discussion beyond technical metrics to broader social impacts, we aim to provide insights that may inspire future research in the UrbanEA community.

The rest of the survey is organized as follows: Sec. Section 2 reviews the urban sensing and simulation in data perception. Sec. Section 3 discusses the pipeline for data management. Sec. Section 4 surveys data fusion techniques to bridge key multi-domain data gaps. Sec. Section 5 presents downstream task applications, and Sec. Section 6 explores the broader social impacts. Sec. Section 7 discusses future outlook and open challenges. Sec. Section 8 finally concludes the paper.

2. Data Perception: Sensing and Simulation

2.1. How to Perceive the City?

2.1.1. Vision Perception

For the visual sense ability, we divide it into the following main categories. The multimodal data captures distinct aspects of the environment, often complementing each other. We visualize the vision perception in the terrain domain as the example, as shown in Figure 2.

•RGB Images: These are standard color images, akin to what a human eye or a typical camera perceives. They are rich in texture, color, and semantic information, making them invaluable for tasks like object recognition, classification, and scene understanding (e.g., identifying road signs in autonomous driving, describing visual elements in spatial description). An RGB image can be represented as a three-dimensional tensor, denoted as $I_{RGB}\in {\mathbb{Z}}_{256}^{H\times W\times 3}$. Its dimensions are the image height H, width W, and 3 color channels (Red, Green, Blue). However, they are 2D projections and are sensitive to lighting conditions, lacking direct information about the 3D structure or distance to objects.

•Depth Images: Unlike RGB images that capture color, depth images encode distance information. Each pixel value typically represents the distance from the sensor to the corresponding point in the scene. A depth image can be represented as a 2D matrix, denoted as $I_{Depth}\in {\mathbb{R}}_{+}^{H\times W}$. Its dimensions are the image height H and width W. The value at each pixel $(u,v)$ is a scalar representing the distance from the sensor to that point in the scene. This provides explicit geometric cues crucial for obstacle avoidance, navigation, and 3D reconstruction, often used in drone operation and robot tasks.

•Lidar Point Clouds: Light Detection and Ranging (Lidar) sensors actively emit laser beams and measure the reflected light to create a sparse but accurate 3D map of the surroundings. A Lidar point cloud is an unordered set of points, denoted as ${\mathcal{P}}_{LiDAR}=\{p_1,p_2,...,p_N\},p_i=(x_i,y_i,z_i,i_i)\in {\mathbb{R}}^4$. It consists of N points, where N is variable. Each point $p_i$ contains at least its coordinates $(x,y,z)$ in 3D space. It often includes reflection intensity, i, as well. These point clouds provide precise geometric structure and distance measurements over considerable ranges, largely independent of ambient light. While excellent for geometry, they typically lack the rich color and texture information found in RGB images.

•Radar Point Clouds: Radar sensors use radio waves instead of light. Radar data is also a set of points, denoted as $P_{Radar}=\{d_1,d_2,...,d_M\},d_j=(x_j,y_j,z_j,v_{x_j},v_{y_j},v_{z_j})\in {\mathbb{R}}^6$. It consists of M detections, where M is typically much smaller than the number of Lidar points, N. $(v_x,v_y,v_z)$ is the velocity vector of the detection. Similar to Lidar, they can generate point clouds representing detected objects. Radar’s key advantage is its robustness in adverse weather conditions (rain, fog, snow), where Lidar and cameras might struggle. However, Radar typically provides lower resolution and less detailed shape information compared to Lidar or cameras.

2.1.2. Multi-sensory Perception

In urban settings, multi-sensory technology, which integrates auditory (e.g., traffic sounds, water features), tactile (e.g., pavement textures, wind), even olfactory (e.g., floral scents, garbage odor), and thermal (e.g., temperature), is being applied to smart city fields like environmental monitoring [17] and public security [18,19]. Sensing within environments is multi-sensory, extending well beyond the visual sensing [20]. To this end, efforts focused on combining audio and visual information, with various works aiming to train agents that can both see and hear by using integrated audio-visual simulations [21,22,23,24,25]. The domain of visual-tactile learning focuses on building realistic tactile simulation systems to allow agents to understand the world through physical interaction [26,27,28,29,30]. Multiply [31] proposes a multi-sensory sensing simulator. This platform incorporates a wide array of interactive data—including visual, audio, tactile, and thermal information—directly into large language models, thereby establishing a direct and powerful correlation among words, actions, and percepts.

2.2. Where to Perceive the City?

The urban perception requires a system capable of synergistically processing information from different observational dimensions [13,32,33]. We divide the urban perception based on the data domain, including handheld, vehicle, drone, plane, and satellite as shown in Figure 3. These platforms demonstrate spatio-temporal heterogeneity, resulting in a spatio-temporal gap as discussed in Section 4.1.

•Handheld. The data from handheld devices is designed for close-quarters mapping and typically has a shorter range. For example, some professional handheld scanners have a flexible scanning range from $0.4$ to 10 meters, making them ideal for detailed exterior facade work [34]. The terrain handheld scanners can achieve accuracies around 5-10 mm. The use of handheld devices for data acquisition suffers from low efficiency, limited coverage, and data inconsistencies caused by manual handling, as the Quality Gap discussed in Sec Section 4.1.

•Vehicle. These systems are designed for efficient corridor mapping and possess a range optimized for capturing roadside features from a moving vehicle [35,36]. The vehicle systems experience a slight reduction in accuracy compared to handheld static scanners, but they still deliver exceptional results suitable for most urban mapping tasks, capturing high-density data within 30 meters to 100 meters of the vehicle’s path. In urban environments, tall buildings lead to GPS signal drift, while pedestrians and other vehicles create dynamic obstructions. These deficiencies may introduce the Spatio-Temporal Gap as discussed in Sec Section 4.1.

•Drone. Drones operate in a unique low-altitude domain, which allows them to achieve exceptionally high spatial resolutions with both photogrammetric and LiDAR sensors. For most professional urban mapping applications, drones can easily achieve a Ground Sample Distance (GSD) between 1 cm and 5 cm per pixel [37]. While its high flexibility is an advantage, it also causes variations in scale and perspective, posing a challenge for precise camera pose estimation.

•Plane. For urban mapping projects, the aerial plane typically delivers a GSD in the range of 5 cm to 30 cm. A GSD of 5-15 cm is sufficient for creating highly detailed and geometrically accurate city-wide 3D models at LOD2 (differentiated roof structures) and LOD3 (architectural models with major facade elements). At this resolution, it is possible to clearly identify individual buildings, roads, vegetation, and major infrastructure elements [38]. However, it is difficult to capture fine terrain-level details.

•Satellite. Satellites operate from low Earth orbit at altitudes that dwarf aerial platforms, yet technological advancements have enabled them to achieve remarkable spatial resolutions [39,40]. The commercial constellations offer panchromatic imagery with a native spatial resolution of approximately 30 cm. Although it provides wide coverage, it suffers from the spatio-temporal resolution with cloud-based occlusion, which may introduce the Density gap.

2.3. City Scene Simulators

The development of robust and reliable perception for outdoor environments relies on the simulation environments when the Internet agent is towards an embodied agent [41]. Existing indoor simulators [42,43,44,45] collect data from the handheld camera or scan sensors. Compared to indoor settings where controlled experiments can be collected from handheld cameras or scanner sensors, outdoor environments present challenges for real-world experimentation due to their complexity, dynamic nature, and safety concerns. Therefore, as shown in Figure 4, we classify the city simulators based on the perceptual capabilities an agent requires to move from observation to action:

Table 2. Comparison with existing Urban Embodied Agent simulators.

Environment	Year	Kinematics	Platform	Category	Modality				Data Source	Engine
					RGB	Depth	Radar	Lidar
Cityscapes [46]	2016	✗	Terrain	Open-Loop	✓	✗	✗	✗	Street View	-
CARLA [16]	2017	✓	Terrain	Closed-Loop	✓	✓	✓	✓	Vehicle	UE 4
xView [47]	2018	✗	Aviation	Open-Loop	✓	✗	✗	✗	Satellite	-
TouchDown [48]	2019	✗	Terrain	Open-Loop	✓	✗	✗	✗	Street View	-
Nuscenes [49]	2020	✗	Terrain	Open-Loop	✓	✗	✓	✓	Vehicle	Nuscenes-Kit
Waymo [50]	2020	✗	Terrain	Open-Loop	✓	✗	✓	✓	Vehicle	Waymax
KITTI-360 [51]	2022	✗	Terrain	Open-Loop	✓	✗	✗	✓	Vehicle	-
STPLS3D [52]	2022	✗	Aviation	Open-Loop	✗	✗	✗	✓	Drone	-
SensatUrban [53]	2022	✗	Aviation	Open-Loop	✗	✗	✗	✓	Drone	-
UrbanBIS [54]	2023	✗	Aviation	Open-Loop	✓	✗	✗	✓	Drone	-
AerialVLN [55]	2023	✓	Aviation	Open-Loop	✓	✓	✗	✗	Drone	UE 4
GRUTopia [56]	2024	✓	Terrain	Closed-Loop	✓	✗	✗	✗	Virtuality	Isaac Sim
OpenUAV [57]	2024	✓	Aviation	Open-Loop	✓	✓	✗	✗	Drone	UE 4
UnrealZoo [58]	2024	✓	Terrain	Closed-Loop	✓	✗	✗	✗	Virtuality	UE 4/5
MetaUrban [59]	2025	✓	Terrain	Closed-Loop	✓	✓	✗	✓	Virtuality	Gym
OpenFly [60]	2025	✓	Aviation	Closed-Loop	✓	✓	✗	✓	Drone	UE 4, Google Earth, GTA V

Table 2. Comparison with existing Urban Embodied Agent simulators.

Environment	Year	Kinematics	Platform	Category	Modality				Data Source	Engine
					RGB	Depth	Radar	Lidar
Cityscapes [46]	2016	✗	Terrain	Open-Loop	✓	✗	✗	✗	Street View	-
CARLA [16]	2017	✓	Terrain	Closed-Loop	✓	✓	✓	✓	Vehicle	UE 4
xView [47]	2018	✗	Aviation	Open-Loop	✓	✗	✗	✗	Satellite	-
TouchDown [48]	2019	✗	Terrain	Open-Loop	✓	✗	✗	✗	Street View	-
Nuscenes [49]	2020	✗	Terrain	Open-Loop	✓	✗	✓	✓	Vehicle	Nuscenes-Kit
Waymo [50]	2020	✗	Terrain	Open-Loop	✓	✗	✓	✓	Vehicle	Waymax
KITTI-360 [51]	2022	✗	Terrain	Open-Loop	✓	✗	✗	✓	Vehicle	-
STPLS3D [52]	2022	✗	Aviation	Open-Loop	✗	✗	✗	✓	Drone	-
SensatUrban [53]	2022	✗	Aviation	Open-Loop	✗	✗	✗	✓	Drone	-
UrbanBIS [54]	2023	✗	Aviation	Open-Loop	✓	✗	✗	✓	Drone	-
AerialVLN [55]	2023	✓	Aviation	Open-Loop	✓	✓	✗	✗	Drone	UE 4
GRUTopia [56]	2024	✓	Terrain	Closed-Loop	✓	✗	✗	✗	Virtuality	Isaac Sim
OpenUAV [57]	2024	✓	Aviation	Open-Loop	✓	✓	✗	✗	Drone	UE 4
UnrealZoo [58]	2024	✓	Terrain	Closed-Loop	✓	✗	✗	✗	Virtuality	UE 4/5
MetaUrban [59]	2025	✓	Terrain	Closed-Loop	✓	✓	✗	✓	Virtuality	Gym
OpenFly [60]	2025	✓	Aviation	Closed-Loop	✓	✓	✗	✓	Drone	UE 4, Google Earth, GTA V

2.3.1. Open-Loop Simulator

In this category, simulators function as replay platforms for real-world data logs. Their primary role is to evaluate the sensing system. These evaluations range in complexity, from semantic understanding that answers the question, “What is it?" Open-Loop Simulator can be divided into unimodal simulators and multimodal simulators. Unimodal simulators represent the foundational layer of virtual environment design, focusing on generating data for a single sensor modality. The goal is to train and validate specific sensing algorithms in a controlled manner for terrain and aerial simulation domains, such as StreetLearn [61], Cityscapes [46], xView [47], SensatUrban [53], UrbanBIS [54], and STPLS3D [52]. The multimodal simulator involves the integration of multiple, synchronized sensor streams. This is designed to replicate the comprehensive sensor suites of outdoor vision sensing, allowing for the development and testing of algorithms that create a more robust and reliable world model by combining the strengths of different modalities, such as Nuscenes [49], Waymo [50], KITTI-360 [51], AerialVLN [55].

2.3.2. Closed-Loop Simulator

This paradigm completes the sensing-action cycle. By enabling an agent’s behaviors to interact with the simulation world, these simulators are equipped to evaluate the agent’s capability: guiding action based on sensing. They move beyond passive observation to address the interaction problem for an autonomous agent: “How should I react to it?" This simulator focuses on the active, full-stack validation, testing the entire “sensing-to-action" loop and allowing for the evaluation of complex behaviors [62]. For terrain-based systems, including GRUTopia [56], UnrealZoo [58], and MetaUrban [59], leverage powerful physics and rendering engines like Isaac Sim and UE4/5, moving beyond sensing tasks. These environments simulate complex and dynamic weather phenomena (rain, fog, snow), realistic diurnal cycles with changing illumination, and intricate multi-agent interactions. In aviation-based simulation, like OpenUAV [57], and OpenFly [60] facilitate intricate interactions with the environment. A key evolution lies in the fidelity of control. This allows for the simulation of smooth, physically plausible flight dynamics, enabling agents to perform complex maneuvers and precise navigation that mimic real-world dexterity.

2.4. Discussion

Simulators are essential tools for showing the urban perception environment and developing UrbanEAs. However, existing simulators suffer from the “Sim-to-Real" gap. This gap manifests in two key areas: physical realism, such as accurately modeling sensor noise or complex weather and lighting effects, and behavioral realism, which involves simulating unpredictable human behaviors like varied driving habits or pedestrian movements. The core challenge is how to effectively quantify and reduce this gap. One of the future directions is to establish a “Real-to-Sim-to-Real" reinforcement loop. This means using high-fidelity data from the real world to build and continuously refine the next generation of simulators, which in turn can train more capable agents.

3. Data Management: Storage and Querying

UrbanEAs operate in a data-intensive environment as detailed in Section 2. However, this raw data deluge is difficult to use directly for the downstream embodied agent task. For instance, data fusion can fail with misaligned timestamps, semantic relationships between objects remain implicit, and safety-critical tasks may lack the necessary data freshness and consistency. The fundamental challenge is thus to transform this unstructured data torrent into a well-organized and queryable knowledge base that supports both real-time perception and long-term learning [63,64].

Effective data management provides the structured backbone required for robust fusion (Section 4), real-time task execution (Section 5), and auditable systems that respect social considerations (Section 6). Navigating the inherent trade-offs between latency, scale, and query complexity under the data lifecycle challenges (Section 1) requires specialized solutions. We therefore examine six complementary storage architectures that form the basis of modern urban data systems (summarized in Table 3): Data Lakes, Multi-model and Graph Databases, Vector Databases, Time-Series Databases, and Spatio-temporal Databases.

We illustrate their respective roles using a running example in this section, as shown in Figure 5: an autonomous vehicle (ego-agent) approaching a busy, rainy intersection. Its view is partially occluded by a large truck, while a roadside unit (RSU) detects an ambulance coming from the blind spot. This scenario, involving sensor noise (rain), occlusions (truck), and multi-source asynchronous data (vehicle sensors vs. RSU), requires a sophisticated data management strategy to ensure safety. The following subsections will examine how various database paradigms address these specific issues.

3.1. General & Unified Architectures

•Data Lakes. The evolution of UrbanEA data management began with large-scale autonomous driving datasets such as NuScenes [49] and Waymo Open Dataset [50]. These datasets established a baseline practice: organizing multimodal streams (cameras, LiDAR, radar) via relational metadata while storing high-volume unstructured data (images, point clouds) as separate files linked by unique identifiers. NuScenes exemplifies this paradigm by providing temporal indexing for synchronized sensor data, global identifiers enabling cross-modal annotation (“annotate 3D once, project to 2D everywhere"), and standardized coordinate transformation APIs for multimodal fusion. For example, the complete transformation chain from LiDAR to camera frame is formalized as:

$$\begin{array}{c}\hfill T_{\mathrm{cam}\leftarrow \mathrm{lidar}}={\left(T_{\mathrm{ego}\leftarrow \mathrm{cam}}\right)}^{-1}·T_{\mathrm{ego}\leftarrow \mathrm{lidar}}\end{array}$$

(1)

Then, the projection of a LiDAR point $P_{\mathrm{lidar}}$ onto the image plane can be expressed as:

$$\begin{array}{c}\hfill p_{\mathrm{homo}}=K·\mathrm{proj}(T_{\mathrm{cam}\leftarrow \mathrm{lidar}}·P_{\mathrm{lidar}})\end{array}$$

(2)

where K is the camera intrinsic matrix, T represents the rigid body transformation matrices, and $\mathrm{proj}(·)$ denotes taking the first three dimensions of the homogeneous coordinate. Every matrix in these equations is directly supplied by the dataset management system [49].

However, while these datasets excel at providing synchronized data for offline training, they are fundamentally limited to static, post-collection dissemination and cannot support the real-time, streaming requirements of production UrbanEAs systems. This limitation motivated the development of Data Lake architectures [86,87], which provide a principled framework for managing continuous, heterogeneous data streams from city-scale platforms. Unlike the static datasets above, Data Lakes adopt a “schema-on-read" philosophy: raw data is ingested in its native format without predefined schemas, with structure imposed only at query time [67,86]. This design fundamentally shifts the paradigm from “publish once" to “continuously sensing and govern."

The architectural evolution of Data Lakes reflects urban escalating demands for real-time capabilities. The Lambda Architecture [65,88] was designed with parallel batch and speed layers to concurrently support both historical analytics and real-time updates, which in turn allows agents to utilize archival data for training while also accessing live streams for immediate decision-making. Kappa Architecture [66] simplified this by focusing solely on stream processing, optimizing for the low-latency responsiveness critical to reactive UrbanEA. More advanced Zone-Based Governance models, including Data-Pond [89] and Functional Data Lake [90,91], organize data into logical zones (Raw → Curated → Analytics) with progressive quality control, ensuring that agents access appropriately validated data for different tasks.

The Data Lakehouse [67] represents the latest development, integrating the flexibility of data lakes with the robust data management and transactional guarantees of data warehouses, thus providing a promising solution for UrbanEA systems that require both exploratory analytics and reliable operational queries. For city setting, Data Lakes offer three strategic advantages: (1) maximized raw data retention with full lineage tracking, enabling reprocessing as models evolve; (2) real-time streaming ingestion via architectures like Kappa, supporting continuous UrbanEAs; (3) multi-zone governance that enforces data quality standards across the pipeline from raw sensors to agent decision-making. In a rainy urban intersection, multimodal streams typically include multi-camera images, LiDAR point clouds, ego-state signals, and roadside trajectories. A data lake retains these raw assets, while an accompanying catalog captures calibration, timestamps, and lineage. This schema-on-read design preserves weather artefacts and occlusion evidence for later fusion and auditing without imposing premature structure.

•Multi-model Databases. UrbanEAs generate data with fundamentally different structures: structured metadata (relational), spatial relationships (graph), and sensor streams (time-series). Multi-model databases [92,93] offer a unified approach to the heterogeneity problem by natively supporting diverse data models within one system, a capability that data lakes and other single-model architectures only partially provide.

Based on their architectural design, multi-model systems are generally classified into two main categories [94,95]. The first, Polyglot Persistence, involves dedicating separate, specialized database systems to each data type (such as PostgreSQL for metadata, Neo4j for scene graphs, and InfluxDB for sensor streams). While this approach optimizes performance for individual modalities, it introduces significant complexity for holistic tasks that depend on cross-database queries and data integration [96]. Conversely, the second category, Unified Multi-Model Databases [97], integrates various data models within a single instance. This design enables seamless cross-model queries to be executed within a unified transactional context. Such a paradigm is highly suitable for the consolidated data management needs of urban applications; a query like “retrieve all vehicles (relational) that were spatially near (graph) the incident location during the last 5 minutes (time-series)" can be processed as a single atomic operation, eliminating the need for cross-system coordination. In the rainy intersection scenario, such unified queries can simultaneously express occlusion and priority constraints while returning the associated sensor slices for the relevant time window.

For UrbanEAs, the most significant advantage of multi-model databases is their capacity to unify heterogeneous data management. This unification, in turn, simplifies data pipelines and reduces the overhead associated with schema conversion. These systems can also strike a balance between low-latency responses for real-time sensing and high-throughput processing for offline analysis by employing diverse indexing strategies tailored to each specific data model [98]. Notable examples include Sinew [68], which merges document and graph models to achieve flexible schema evolution; NoAM [69], which is tailored for aggregation-oriented NoSQL workloads; and UniBench [70], a comprehensive benchmark used to evaluate the performance of multi-model databases across varied query patterns.

3.2. Semantic & Relational Architectures

•Graph Databases. Beyond data lakes and multi-model paradigms, traditional data models struggle to natively express the complex relationships that govern urban systems [99,100,101]. This limitation manifests as the Semantic Gap, where raw data lacks explicit relational context for high-level reasoning [102]. To bridge this gap, graph-based architectures leverage graph databases [103,104,105,106] and graph neural networks to model entity relationships as graphs $G=(V,E)$, where nodes represent entities and edges encode their relationships [107], such as modeling the `occludes` relationship between the vehicle and the truck. This provides the explicit relational context needed for downstream reasoning tasks like scene question answering. In the intersection scenario, perception outputs are materialized into a temporally indexed scene graph where nodes represent the ego vehicle, the truck, the ambulance, lanes and signals, and edges encode relations such as occludes, located_on_lane, has_priority, and approaching_from_blind_spot. Each relation carries the originating timestamp and pointers to the source frames in the lake, grounding semantic reasoning back to raw data without ambiguity. In UrbanEA, two distinct yet complementary graph representation paradigms have emerged, each serving different scales and purposes. The first paradigm, scene graphs, follows a bottom-up, task-specific approach. Scene graphs are typically generated in an end-to-end manner directly from sensor inputs (e.g., multi-view images or LiDAR scans) to capture the immediate perceptual context of a scene [74,108]. For example, Sg-CityU [74] constructs urban scene graphs that decompose complex 3D environments into structured object-centric representations with spatial relationships, simplifying downstream tasks like spatial reasoning and navigation. T2SG (Traffic Topology Scene Graph) [108] specializes in modeling road-level topology for autonomous driving: it represents lanes as nodes and their connectivity (e.g., predecessor, successor) and control relationships (e.g., governed by a specific traffic light) as edges. The second paradigm, knowledge graphs (KGs), adopts a top-down, ontology-driven approach aimed at building comprehensive, reusable knowledge bases for entire urban domains [107,109,110]. A landmark example is nuScenes Knowledge Graph (nSKG) [73], which transforms the nuScenes autonomous driving dataset into a structured KG comprising approximately 43 million RDF triples. nSKG is constructed by first defining formal ontologies, and then systematically extracting and mapping instance data from the dataset’s annotations into this ontological framework [73]. This dichotomy reflects a fundamental trade-off: scene graphs prioritize speed, task-relevance, and tight integration with perception models, making them suitable for real-time, tactical decision-making; KGs prioritize completeness, semantic richness, and long-term knowledge persistence, making them ideal for strategic reasoning, offline simulation, and cross-domain knowledge integration [111].

Beyond static storage, graph-based architectures enable advanced reasoning through graph neural networks (GNNs) [112]. GNNs operate by iteratively propagating and aggregating information across graph edges, where each node updates its representation by combining information from its neighbors through learnable transformations, enabling the model to reason about both local interactions and global context [9]. In urban applications, (spatio-temporal graph neural networks) STGNNs extend this by jointly modeling spatial topology (e.g., road network connectivity) and temporal dynamics (e.g., traffic flow evolution), making them indispensable for predictive tasks such as trajectory forecasting and traffic prediction [9]. A representative application is SemanticFormer [113], which leverages the rich semantic context provided by nSKG to perform multi-modal trajectory prediction. SemanticFormer employs a hierarchical heterogeneous graph encoder that captures interactions between agents, road elements, and traffic rules by applying attention mechanisms over meta-paths—semantically meaningful sequences of relations (e.g., vehicle →on_lane→ lane →governed_by→ traffic_light).

From a systems perspective, graph databases such as Neo4j [71] and JanusGraph [72] provide the infrastructure for storing and querying large-scale urban KGs using declarative languages like SPARQL and Cypher. These systems support multi-hop traversal queries essential for tasks like multi-step path planning (e.g., “Find all lanes connected to the current lane that are not controlled by a red light") [105]. In practice, hybrid architectures are increasingly common: structured metadata and time-series data are stored in relational or time-series databases (as discussed previously), while complex relational and semantic information is offloaded to graph databases, with both systems queried in a federated manner [107]. Real-world applications demonstrate the power of this paradigm: Urban Region Graph [99] uses graph-based spatial representations for urban village detection; knowledge-driven site selection systems like KnowSite [109] leverage urban KGs to recommend optimal commercial locations by reasoning over multi-hop relationships between venues, demographics, and transportation networks, outperforming purely data-driven approaches [107].

•Vector Databases. Another limitation of relational models is their reliance on exact, token-based queries, making them ill-suited for vague, natural language queries [85,114]. Vector databases address this by encoding multimodal data into high-dimensional vectors and leveraging Approximate Nearest Neighbor (ANN) search algorithms [75,77,78,115] for rapid, content-based semantic retrieval. The core operation is to measure the semantic similarity between a query vector Q and data vectors $D_i$ in the database, often using cosine similarity [116]:

$$\begin{array}{c}\hfill \mathrm{sim}(Q,D_i)={\displaystyle \frac{Q·D_i}{\parallel Q\parallel \parallel D_i\parallel }}\end{array}$$

(3)

The system returns items with the highest similarity scores [117]. In the intersection, retrieving similar rainy intersection segments helps interpret degraded visuals when occlusion and glare lower confidence. This capability is crucial for advanced human-agent interaction, particularly for supporting the Scene Question Answering (SQA) tasks in section V. The demand for such advanced interactive tasks, exemplified by work like NuScenes-QA [118], has spurred the need for vector databases, making it possible to query vast visual scenes using natural language [74,118,119,120].

3.3. Spatio-temporal & Dynamic Architectures

•Time-Series Databases. In data of UrbanEAs, time-series databases serve two primary roles [121]. First, they manage high-frequency sensor data streams from embodied agents. For instance, in autonomous driving research, vehicles generate continuous streams of IMU data (at 100+ Hz), GPS trajectories (at 10 Hz), LiDAR scans (at 10-20 Hz), and vehicle dynamics (velocity, acceleration, steering angle at 50+ Hz). These streams are ingested into time-series databases for real-time monitoring, offline analysis, and model training [49,50]. Systems like Gorilla [79], originally developed by Facebook for operational monitoring, demonstrate the capability to handle billions of data points per day with high write throughput and efficient compression, making them suitable for city-scale deployments. Similarly, Apache IoTDB [80] provides a unified time-series database specifically designed for IoT applications, with native support for irregular sampling intervals and multi-tenant isolation—features particularly valuable for managing heterogeneous sensor deployments across different urban districts or infrastructure operators. CrocodileDB [122] introduces resource-efficient query execution by exploiting temporal slackness in query deadlines, enabling graceful performance degradation under resource constraints—a critical capability for edge deployment scenarios in UrbanEA systems where computational resources may be limited. Second, in the context of interactive and reinforcement learning environments, time-series databases store the complete decision-making history of agents. For example, in EmbodiedCity [123], an agent’s trajectory, observations, actions, and rewards over an entire episode are logged as time-series data, enabling replay, debugging, and offline reinforcement learning. Similarly, in multi-agent driving scenarios studied in DriveLM [120], the synchronized time-series logs of all agents’ states and decisions are crucial for understanding emergent behaviors and training coordination strategies.

Beyond basic storage and retrieval, time-series databases increasingly support advanced analytics directly within the database engine. For instance, TimescaleDB [81] provides built-in functions for continuous aggregation (e.g., computing moving averages in real-time as data arrives), time-bucketing (grouping data into regular intervals for analysis), and hyperfunctions (domain-specific aggregations like percentile estimation over time). Around urgent braking events at the intersection, event-centric windows organize ego-state streams for fusion and audit.

•Spatio-Temporal Databases. In data of UrbanEAs, spatio-temporal databases are critical for the simultaneous management of both spatial and temporal dimensions, a requirement that pure time-series databases do not fully address [124,125,126,127]. A primary role is enabling hybrid systems. Recent empirical studies validate this approach by integrating PostgreSQL with extensions like PostGIS (for spatial capabilities) and Timescale (for temporal optimization) [128,129]. This combination demonstrates substantial query time reductions for both stationary (e.g., parking monitoring) and non-stationary (e.g., railway track monitoring) sensor data. Crucially, these studies reveal that for trajectory data, lightweight BRIN (Block Range Index) indexes can outperform traditional R-trees in both query performance and storage overhead, challenging conventional wisdom in spatial indexing. A second role is trajectory and mobility data management. Urban trajectories (vehicles, drones, pedestrians) are spatio-temporal objects. Specialized systems like MobilityDB [82] extend databases with native support for moving objects, enabling queries such as “find all vehicles that passed through region R between time $t_1$ and $t_2$" or “compute the speed profile of agent A over the last hour," which is foundational for traffic analysis and behavior mining [130]. More recently, the integration with machine learning frameworks has enabled in-database prediction [12,131,132,133]. The emergence of foundation models for Spatio-Temporal Data represents a paradigm shift, allowing pre-trained models to be deployed directly on spatio-temporal data streams for real-time forecasting (e.g., traffic prediction, demand estimation) without moving data to external systems. In the intersection scenario, alignment operators synchronize the RSU ambulance trajectory with the ego timeline and unify coordinate frames, producing per-timestamp bundles ready for fusion.

3.4. Discussion

Effective data management in urban embodied systems transforms raw, heterogeneous sensor streams into actionable knowledge through layered architectures. It begins with Data Lakes that retain raw assets and lineage, continues with Graph Databases that render critical relations explicit and queryable, and relies on Spatio-temporal Databases to synchronize heterogeneous streams in time and space while unifying coordinate frames. Together, these components produce coherent, analysis-ready data for fusion and downstream tasks.

We observe two trends. The first is an evolution from static datasets to dynamic data platforms. The future paradigm of data management will shift from publishing one-off, static datasets to building dynamic, open data ecosystems where the community can continuously upload, annotate, query, and simulate, thereby accelerating iteration across the entire field [42]. The second is the concept of data as a queryable world model. When data management and querying technologies reach their zenith, the managed, structured data asset itself constitutes an implicit, inferable world model, blurring the lines between raw data, information, and knowledge. This offers a novel and exciting avenue for building more general and powerful UrbanEA [32,134].

4. Data Fusion: Bridging Domain Gaps

Data in UrbanEA is beyond the passive feed from a single domain, but an active perception combining multi-domain information. This combination is essential, acting like the assembly of “puzzle pieces" which resolves inconsistencies between sources and fills the blind spots of individual sensors. Only through this process can a scattered collection of data become a coherent and complete urban “tapestry". Without effective fusion, the UrbanEA is left with a collection of scattered, independent, and potentially contradictory perceptual puzzle pieces.

However, this fusion process presents challenges, stemming from the heterogeneity of multi-domain data [13,14]. While the previous section detailed the preparation of each “puzzle piece," including the preprocessing and management of individual data streams, this section explores how to stitch them together into a unified understanding [135]. We will begin by analyzing the four core “gaps" that impede data fusion: at the data level, the Representation Gap and the Quality & Density Gap; at the spatio-temporal dimension, the Spatio-Temporal Gap; and at the cognitive level, the Semantic Gap. After defining these challenges, we will introduce the primary fusion strategies developed to bridge them. These include Early Fusion, Feature-Level Fusion, Late Fusion, and Prior-Based Fusion, which will be illustrated with state-of-the-art research examples [136].

4.1. The Domain Gap between Multi-Domain Data

The challenges in fusing multi-domain data can be analyzed from four perspectives: data structure, data quality, spatio-temporal basis, and information level. These perspectives reveal four obstacles, or “gaps," that must be addressed: the Representation Gap, the Quality & Density Gap, the Spatio-temporal Gap, and the Semantic Gap [137], as shown in Figure 6.

•Representation Gap. The representation gap refers to the fundamental differences in the most basic data structures, mathematical representations, and organizational methods across various data domains, as shown in Figure 6 (a). For example, satellite images are regular 2D grids composed of pixels, while point clouds are sparse, unordered sets of 3D points in a Cartesian coordinate system [138]. Meanwhile, 3D meshes are frequent in vehicles. They consist of vertices, edges, and faces, and implicit representations are continuous field functions [139]. These data structures are not compatible, requiring algorithms to incorporate specialized modules to process and convert these heterogeneous representations for effective fusion [140].

•Quality & Density Gap. The quality & density gap, as shown in Figure 6 (b), describes the inconsistencies across different data domains in terms of their precision, completeness, noise levels, and data point density, as well as the disparity between real-world collected data and idealized (e.g., synthetic) data [141]. This gap is manifested in several ways. For instance, point clouds from low-altitude domain offer high geometric precision, and the reconstruction in the ideal world needs complete and clean data. However, in the real-world data collection, they face challenges such as noise, holes, and uneven density among domains [142,143]. While image data from vehicle is dense, its quality degrades under complex illumination conditions (such as overexposure, shadows, and reflections), and it cannot directly provide geometric information [144,145,146]. Furthermore, the artifacts and imperfections common in real-world data create a “sim-to-real" gap when compared to clean, idealized synthetic training data [147,148,149].

•Spatio-Temporal Gap. The spatio-temporal gap refers to the difficulty of aligning data from different sensors and different points in time within a unified spatial coordinate system and temporal sequence, as shown in Figure 6 (c). This gap has two aspects. Spatially, the coordinate systems of different domains are difficult to unify perfectly. The perception algorithms are sensitive to accurate sensor parameters, and even minor pose errors can severely impact quality [150,151]. Temporally, because urban scenes contain moving objects (like vehicles), the content of data collected at different moments will be inconsistent, leading to deformative and elongated artifacts in the final projection [152].

•Semantic Gap. The semantic gap refers to the cognitive disparity between low-level, raw sensory data (such as pixels and points) and high-level, human-understandable scene semantics (such as object categories, functional attributes, and spatial relationships), as shown in Figure 6 (d). [153]. The algorithms can directly process pixel colors and 3D coordinates $(x,y,z,r,g,b)$, but they cannot understand concepts like “the window", “the door", “the stair". To bridge this gap, researchers introduce techniques like semantic segmentation and language models as semantic priors [154,155]. This injects high-level knowledge into the sensing pipeline to meet the demands of advanced tasks, such as scene understanding, which is a key requirement from industrial partners [156].

Figure 7. The different fusion strategy in the multi-domain data for Urban Embodied Agents.

Figure 7. The different fusion strategy in the multi-domain data for Urban Embodied Agents.

4.2. Fusion Strategies for Multi-Domain Data

The fusion strategies designed to bridge domain gaps can be primarily classified into the following categories according to the stage at which they occur in the model fusion pipeline, including Raw-data-level Fusion, Hierarchical Feature Fusion, and Decision-level Fusion [157,158].

•Raw-data-level Fusion. In Raw-data-level Fusion, raw or independently preprocessed data from different modalities are merged, such as combining RGB images from satellites and depth maps from drones into a unified RGB-D data stream. This unified representation is then fed into a single encoder to extract high-level features. For instance, Muturi et al. [159] handle heterogeneous representations from the outset to bridge the Representation Gap. Additionally, the depth information can compensate for the lack of texture in images, improving the overall quality and robustness of the data to bridge the Quality & Density Gap. Shang et al. [160] integrate RGB images and depth information (from pre-trained depth models or consumer-level depth sensors) to enhance the performance in few-shot novel view synthesis.

•Hierarchical Feature Fusion. Hierarchical Feature Fusion combines features from different domains at various levels of abstraction [115], such as enriching a geometric model with high-level context from a pre-trained semantic segmentation backbone (such as CNN [161,162], ViT [163], PointNet [164]). For instance, the SUDS model [165] utilizes a joint optimization framework through a composite loss function to fuse multiple data domains, enabling the understanding of large-scale dynamic urban scenes and overcoming the limited observability to achieve a city-scale dynamic scene. GAANet [166] unifies cross-domain features in graph space via global graph interaction and local attention alignment, eliminating geometric and semantic gaps for robust fusion. To handle large-scale scenes, VastGaussian [167] and CityGaussian [168] also adopt a “divide-and-conquer" strategy, partitioning the scene into multiple cells for parallel optimization before merging. To address common Quality & Density Gap, such as illumination variation, they introduce a method that uses a CNN during optimization to separate the scene’s stable geometric colors from transient lighting effects.

•Decision-level Fusion. Decision-level Fusion, also known as Late Fusion, is a straightforward strategy where each data domain is processed independently through separate network streams. These individual networks extract high-level features or form initial predictions, and the fusion only occurs at the very end of the decoder or classifier of individual sub-networks. For instance, a model might separately process features corresponding to different levels of detail. Horizon-GS [169,170] uses a “coarse-to-fine" training strategy to establish a geometric model before addressing the gap between the aerial data (sparse 3D data) and terrain data (real street-view images). The City3D [171] method uses a height map from terrain to guide the reconstruction of aerial LiDAR point clouds. This not only ensures the semantic correctness of the result (addressing the Semantic Gap) but also helps handle issues like missing walls (addressing the Quality & Density Gap). CrossView-GS [172] uses a multi-branch architecture, training models on different view sets independently to serve as priors. It tries to capture Internet video to bridge the Quality & Density Gap when using cross-view data with large disparities.

5. Task Application: From Perception to Social Interaction

Having established the challenges of multi-domain data management in Section III and the fusion strategies in Section IV, this section introduces how to apply this processed data to UrbanEA tasks. Traditional urban tasks, such as Traffic Flow Prediction and Point-of-Interest (POI) Recommendation, primarily operate as “Digital Agents" [173,174]. These systems typically process aggregated data from a global perspective to mine patterns or forecast trends within a digital space. Their output is generally limited to information or suggestions, without direct physical intervention. In contrast, UrbanEA are defined by their physical or virtual body and their ability to interact with the environment. They perceive the environment, reason about physical constraints, and execute actions that actively change their state or the environment.

We introduce UrbanEA tasks, including Urban Scene QA (SQA), Vision-Language Navigation (VLN), and Human-Agent Collaboration (HAC). These tasks need to bridge the four Gaps. For instance, an agent cannot answer a complex question (SQA) without first bridging the Semantic Gap between raw sensor data and human-level concepts. Similarly, a navigation agent (VLN) inherently fails if it cannot resolve the Spatio-Temporal Gap to form a coherent world model. This section will explore how each of these applications leverages structured and fused data to achieve sophisticated cognitive capabilities in complex urban environments.

5.1. Urban SQA

Figure 8. Definition of Urban Scene QA (SQA) for Urban Embodied Agents.

Figure 8. Definition of Urban Scene QA (SQA) for Urban Embodied Agents.

1) Definition. Urban SQA enables intelligent systems to answer queries about their spatial context, making it a key task for environmental interpretation [175]. The goal is to develop a model ($\mathcal{F}$) that takes a scene representation (S) and a query (Q) as input to produce a textual answer (T). Optionally, the model can also output spatial grounding (B) via bounding boxes to localize entities. The scene representation S includes a point cloud ($S^{\left(p\right)}$) or multi-view images ($S^{\left(m\right)}$), while a query Q can be text ($Q^{\left(t\right)}$) or an egocentric image ($Q^{\left(e\right)}$). The task can thus be expressed as:

$$\begin{array}{c}\hfill (T,B)=\mathcal{F}(S,Q).\end{array}$$

(4)

Effectively performing SQA requires the model to fuse spatial understanding with multimodal processing. The question in SQA can be divided into Situation (such as “Can the agent reach the warehouse from the current position?") and Non-Situation (such as “How many buildings are in the scene?"), based on whether the query includes the agent’s situation.

2) Classification. SQA can be categorized into two levels based on how the information required to answer a query is represented and acquired.

•Passive SQA. The first level, referred to as Passive SQA. In this task, the agent is a passive observer, relying on static information to answer questions without needing to explore or acquire new data through its own actions. It can be further divided into the following two sub-categories based on the difference in the scope of information: Road-level QA and City-level QA. The first is referred to as Road-level QA, which corresponds to road-level scene understanding, where the model analyzes a single, independent snapshot of the environment (e.g., a street-view image or a single frame of lidar data) to answer a question. Recent progress in autonomous driving research has stimulated the development of numerous SQA datasets designed to enhance road-level understanding capabilities, such as Nuscenes-QA [118], NuInstruct [176], NuPrompt [177], DriveLM [120], and VLAAD [119]. After the data fusion, researchers employ cross-attention or the multi-layer perceptron mechanisms to achieve deep interaction between vision and language [118,120,177]. However, road-level QA focuses on instance-level queries and limited data in the roadside, which leads to an insufficient assessment of broader city scene comprehension and complex reasoning abilities.

To overcome the limitations of road-level Question Answering (QA), the field has progressed towards City-level QA, a paradigm that grants agents access to a comprehensive, prior model of an entire city for macroscopic spatial reasoning [74,178,179,180,181]. A challenge in this domain is maintaining information fidelity during the compression of vast urban data. To mitigate these challenges, researchers have primarily adopted two strategies. The first approach, hierarchical urban modeling, transforms vast and unstructured urban scenes into structured and queryable formats using a Relational Database or Graph Database, as discussed in Sec Section 3.2. For instance, GeoProg3D [181] and Sg-CityU [74] construct a structured tree or graph to abstract complex spatial information into objects and their interrelations, which simplifies spatial reasoning. SOBA [178] and EarthVQANet [182] involve using semantic segmentation to decompose large scenes into individual object units for analysis. The second strategy is to augment the compressed scene representation by integrating external information, such as using the geographic information based on the Spatio-Temporal Database, as discussed in Sec Section 3.2. This approach compensates for details lost during compression. OpenCity3D [180] and CityBench [179] align perception models with real Geographic Information Systems (like OpenStreetMap) to provide precise geographic coordinates and place names.

•Active Embodied QA. To move beyond the cognitive bottlenecks of city-level QA, caused by the information loss and lack of dynamics in static models, Active Embodied QA (AEQA) shifts the agent from a passive analyst to an active explorer to gather high-fidelity, real-time information. Existing works includes EmbodiedCity [123] and CityEQA [183]. These datasets introduce tasks requiring an embodied agent to actively navigate and explore complex urban environments to answer open-vocabulary questions, assessing integrated navigation, sensing, and reasoning skills. The core advantage of AEQA is to transform an agent from a passive receiver of static information into an active explorer of the physical world, overcoming perceptual limitations through active action, and to locate and resolve ambiguity in complex environments. In road-level QA and city-level QA, a model is limited by the quality and viewpoint. For example, due to lighting effects, a black car appeared gray from a static viewpoint, leading to an incorrect judgment by agents. However, the agent in AEQA could actively adjust its observation pose by moving to the car’s side, which reduced the impact of the lighting and allowed it to obtain the object’s true attribute [183].

5.2. Vision-Language Navigation

1) Definition. Vision-and-Language Navigation (VLN) is a multimodal task requiring an embodied agent to navigate a realistic environment based on natural language instructions [184]. The core components involve the agent, the environment it perceives and acts within, and the language instruction guiding its movement. The VLN task necessitates a model or agent, denoted by $\mathcal{F}$, designed to process two inputs: the scene representation S perceived from the environment, and a series of natural language instructions $Q=\{q_1,q_2,...,q_T\}$. The objective is to generate a sequence of actions $A=\{a_1,a_2,\dots ,a_T\}$ that directs the agent through the environment to fulfill the goal specified by the instruction Q. This process can be represented as the mapping:

$$\begin{array}{c}\hfill A=\mathcal{F}(S,Q).\end{array}$$

(5)

2) Classification. VLN tasks can be categorized based on their operational platform and observational perspective required of the agent $\mathcal{F}$. These tasks fall into two main paradigms: Terrain-View Navigation and Aerial-View Navigation.

•Terrain-View Navigation. It focuses on an agent’s ability to perform navigation from a first-person, terrain-level perspective (such as that of a pedestrian or vehicle). They primarily utilize environments from static datasets, like street-view panoramas and predefined navigation graphs, to evaluate foundational skills like landmark recognition and path adherence. Datasets such as Touchdown [48] and map2seq [185] rely on simulated scenarios and fixed paths, which limit the agent’s adaptive decision-making in complex, unpredictable real-world situations. Due to the semantic gap between instructions and first-person perception, researchers explore introducing auxiliary semantic data in vector databases and route and navigation data in time-series format to mitigate this gap. Research efforts expand training sets by automatically generating actions and instructions from unlabeled videos (e.g., VLN-video [186]), or by leveraging large language models to synthesize diverse auxiliary data, such as navigation rationales and landmark descriptions (e.g., FLAME [187], NavAgent [188]). Regarding multimodal fusion, strategies have evolved from early methods like direct feature vector concatenation (e.g., RconCAT [189]) and novel style-transfer fusion (e.g., VLN-Trans [190]), to utilizing large language models as a universal interface that converts all visual and historical information into natural language prompts for fusion and decision-making (e.g., Velma [188]).

•Aerial-View Navigation. It focuses on an agent (typically a drone) performing navigation using a third-person, top-down “bird’s-eye" view. The core of this approach is leveraging external prior knowledge, such as geographic maps and top-down satellite imagery, to provide a global context for the agent’s navigation plan. This Hybrid Spatio-Temporal Architecture directly tackles the pain point of navigating vast and unfamiliar environments, which is unavailable from a limited, first-person perspective. The primary challenge in this paradigm is aligning the provided language instructions to the specific spatial and visual features of the exo-viewpoint. These tasks includes AerialVLN [55], OpenUAV [191], CityNav [192]. For instance, CityNav [192] incorporates an internal 2D spatial map representing landmarks mentioned in the instructions, which has been shown to markedly enhance navigation performance at a city scale. UAV-VLA [193] utilizes satellite imagery as a primary information domain for mission planning. In the image, the agent decomposes the high-level instruction and navigates to the destination. AVDN [194] is built within a simulator that uses top-down satellite images to represent the drone’s visual observations. This gives both the agent and the human commander a bird’s-eye view of the environment, simplifying the navigation challenge by providing inherent global context.

Figure 9. Definition of Vision-Language Navigation for Urban Embodied Agents.

Figure 9. Definition of Vision-Language Navigation for Urban Embodied Agents.

5.3. Human-Agent Collaboration

1) Definition. Human-Agent Collaboration (HAC) in urban environments enables a hybrid team of intelligent agents and human participants to work together towards a common purpose. The core task is to design a system function $\mathcal{F}$ that can process a collective goal (G), system-level input (I), and sensing information from the urban scene (S) to produce a coherent final output (O). This collaborative process is defined by the interaction of several key components. The system itself is composed of a heterogeneous set of n participants, $P=\{p_1,p_2,...,p_n\}$, and the communication channels C. P includes both embodied agents and human participants. Each embodied agent operates based on its internal model $m_i$ and goal $g_i$, while a human participant acts based on their expertise and assigned role $r_i$. The overall system task can be formally expressed as:

$$\begin{array}{c}\hfill O=\mathcal{F}(G,S,I|P,C).\end{array}$$

(6)

HAC necessitates the ability to decompose a high-level collective goal (G) into actionable sub-tasks and to coordinate the actions of both agent and human participants ($p_i\in P$) through structured communication channels (C) to achieve a unified result within the complex urban environment [195,196]. In contrast to AI systems, Human-Agent Collaboration (HAC) leverages human strengths to build more efficient, robust, and trustworthy systems[197,198,199].

2) Classification. Urban Human-Agent Collaboration (HAC) tasks can be categorized based on their primary operational focus. There are two types of HAC tasks: Environment-centric HAC tasks, which aim to optimize the urban system, and Human-centric HAC tasks, which focus on assisting or interacting with individuals or groups within the city.

•Environment-centric Human-Agent Collaboration. In this category, the primary objective is to integrate AI agents into the urban fabric to enhance system-wide efficiency, planning, and management. Agents act as decision-support tools or autonomous managers for urban infrastructure distribution [200,201,202,203,204]. These tasks often involve large-scale simulation, data analysis, and long-term optimization, with humans setting high-level goals and overseeing the system. For instance, some multi-agent frameworks try to employ a hierarchical architecture to process task data, effectively allocating responsibilities between robots and humans to balance safety and efficiency [205].

Figure 10. Definition of Human-Agent Collaboration for Urban Embodied Agents.

Figure 10. Definition of Human-Agent Collaboration for Urban Embodied Agents.

•Human-centric Human-Agent Collaboration. This category emphasizes the direct interaction and coordination between humans and agents to accomplish specific tasks within the urban environment and spatio-temporal data. Achieving this requires the data architecture capable of fusing inputs, ranging from human intent to real-time sensor streams, and leveraging specialized storage paradigms to support complex reasoning and action. The challenge in this task is bridging the gap between high-level, often ambiguous human intent and the low-level, concrete actions of agents. Try to address this challenges, the researchers try to build the agents with two capabilities: grounding human commands in the physical world and managing dynamic task execution over time. First, the agent can interpret the human’s command by fusing the natural language instruction with both real-time environmental sensing and external world knowledge [206]. A command like “find a safe place to land" requires the system to semantically link the abstract concept of `safety’ to visual and spatial features from its sensors. This process relies on a Vector Database to perform semantic search (connecting language to visual patterns) and Graph or Relational Databases to query structured world knowledge (e.g., GIS data identifying open areas, no-fly zones). This fusion of linguistic, perceptual, and knowledge-based data allows the agent to transform an abstract goal into a concrete, localized target [207,208,209,210].

5.4. Discussion

The progression of tasks from passive scene understanding to active interaction reveals that future breakthroughs will depend on building more comprehensive world models. Future world models must learn universal principles, not just memorize patterns. This will allow a navigation agent trained on a US grid-style road network to successfully adapt to the unfamiliar roundabouts of Europe. Besides, the world models will be embedded with social intelligence. For tasks like HAC, future agents will learn to understand unwritten social norms (e.g., driving etiquette, personal space), moving beyond following traffic laws to become more intuitive and effective collaborators in society.

6. Social Impact

The urban environment is a socio-physical environment that integrates social behavior with physical scene [211,212]. This section aims to explore the profound potential of the UrbanEA to drive positive societal change, focusing on the social impacts it can deliver across the following domains.

6.0.1. Transportation 

Modern urban transportation faces congestion and pollution due to the increasing number of vehicles, while complex road networks and unforeseen incidents continually threaten road safety [213,214]. Meanwhile, lagging public transportation planning leads to inefficient service, and the aging of critical infrastructure is difficult to maintain due to a lack of effective monitoring [215]. For instance, in the traffic flow management [216,217,218], the agent fuses real-time data from the terrain-level domain and the drone domain to perceive traffic flow dynamics, enabling adaptive traffic signal control to alleviate urban congestion, reduce carbon emissions, and improve the commuting experience [219].

6.0.2. Energy 

In the context of the energy transition, despite the potential of clean energy sources like solar, planning for and maximizing their deployment in complex urban environments remains a challenge. Traditional energy management methods are macroscopic and static, incapable of performing fine-grained, dynamic management and optimization for energy-consuming units within a city [220,221,222]. For instance, the agent uses detailed 3D building models to calculate the solar conditions and shadow occlusions for every rooftop, generating a city-wide “solar map" [223,224,225].

6.0.3. Climate Change 

The high-density buildings and surfaces of cities exacerbate problems like the “Urban Heat Island" (UHI) effect. The challenge for cities in adapting to climate change is the lack of tools capable of accurately simulating these risks. Macroscopic climate models are insufficient for guiding specific adaptations at the street and community levels. The agent fuses satellite, drone, and terrain data to accurately simulate the formation and distribution of the urban heat island effect [226]

6.0.4. Healthy Care 

Factors within cities, such as air and noise pollution, the uneven distribution of green spaces, and “pedestrian-unfriendly" designs, invisibly harm public health and increase the risk of chronic diseases [227]. Through its multi-dimensional perception and analytical capabilities, the UrbanEA can quantify and visualize these health determinants, integrating public health goals into every aspect of urban planning. The agent integrates data from various urban sensors to generate dynamic, visual maps of air and noise pollution. This provides citizens with “healthy route" suggestions and helps environmental agencies pinpoint pollution sources, thereby improving the city’s overall living environment quality [228,229].

7. Outlook and Discussion

In this section, we summarize these challenges and suggest potentially feasible research directions, organizing them into methodological, systemic, and societal challenges.

7.1. Method-Level Challenges

7.1.1. Robust Fusion with Imbalanced Multi-domain Data

This is a challenge that directly extends the discussion of the Quality & Density Gap in Section III. Future UrbanEA must operate reliably in the real world, where data domains, quality, and quantity are imbalanced. This imbalance manifests in several ways: viewpoint imbalance (e.g., massive volumes of terrain-level street views vs. limited aerial drone imagery), quality imbalance (e.g., high-precision professional LiDAR data vs. noisy crowd-sourced images), and temporal imbalance (e.g., static historical map data vs. sparse real-time sensor streams) [230].

Future research will explore new fusion architectures that not only fuse this heterogeneous data but also enable robust inference and decision-making when critical data is missing or of low quality, preventing the model from being overwhelmed by an abundance of poor data.

7.1.2. Continual Learning and Incremental City Updates

The majority of current UrbanEAs perform one-shot understanding on a static dataset, generating a static snapshot of a city. However, a city is a constantly evolving entity [3,231]. A future challenge is to evolve the agent’s cognition from static snapshots to a dynamic world model capable of continual learning and incremental updates.

This requires solving critical problems such as catastrophic forgetting (forgetting old scenes when learning new ones), model drift, and efficiently managing never-ending data streams. Achieving this goal would elevate the solution to the Spatio-temporal Gap from handling minute-long sequences to managing year-long urban evolution.

7.2. System-Level Challenges

7.2.1. High-Fidelity Simulators and the Sim-to-Real Loop

While existing urban simulators like CARLA [16] and MetaUrban [201] are powerful, a sim-to-real gap persists, both in terms of perceptual realism (e.g., simulation of lighting, sensor noise) and behavioral realism (e.g., pedestrian and vehicle driving habits).

A future direction is to leverage the UrbanEA itself to advance simulator development. By constructing ultra-high-fidelity digital twins from real-world data, we can extract physical materials, lighting properties, and behavioral patterns of traffic agents to build the next generation of more realistic simulators. This forms a “Real-to-Sim-to-Real” loop: real-world data improves the simulator, the improved simulator trains and validates better agents, and those agents then better perceive and influence the real world.

7.2.2. Multi-Agent Collaboration and Swarm Intelligence

The city of the future will be a massive distributed system composed of thousands of independent agents with only local sensing (e.g., autonomous vehicles, drones, infrastructure sensors). A challenge, and the key to the emergence of swarm intelligence, is enabling these agents to collaborate efficiently and safely [232,233].

Future research must address communication bottlenecks, decentralized decision-making consistency, and collaboration among heterogeneous agents. In this vision, the unified world model built by the urban perception can serve as a “digital bedrock", providing an authoritative and consistent environmental representation for all other micro-agents to query and interact with, thus enabling the leap from single-agent intelligence to large-scale swarm intelligence.

7.3. Societal-Level Challenges

7.3.1. Causal Reasoning by Fusing Social Knowledge

This represents the leap across the Semantic Gap. Current fusion operates at the level of physical sensing (geometry, appearance), whereas a future agent must be able to fuse non-physical, abstract social knowledge [234].

By integrating information like social media trends, news events, event schedules, and weather alerts, the agent can transition from answering `“what is happening” to explaining “why it is happening”. This fusion of sensing with abstraction will transform the UrbanEAs from a powerful descriptive tool into an explanatory and predictive tool with rudimentary causal reasoning, providing the true intelligence required for the advanced tasks outlined in Section VI.

7.3.2. Fairness and Data Bias

The challenge of imbalanced multi-domain data transcends a technical concern, posing an issue of social equity. Data perception often mirrors existing socio-economic disparities, leading to the over-representation of affluent areas and the under-representation of marginalized communities [235,236]. An agent trained on such data would develop a biased worldview; it will learn to equate data-richness with importance, effectively rendering data-poor areas invisible [237,238,239].

Future research will create fusion methods that can be audited. It must focus on developing fairness-aware fusion algorithms. These methods must not only audit and quantify data-driven disparities but also actively correct for them. The goal is to ensure the agent’s decisions promote allocative equity, ensuring that urban services and resources (discussed in Section VI) are distributed justly, even when the input data itself is unjust [240].

8. Conclusions

In this paper, we comprehensively survey the emerging field of data lifecycle for UrbanEA. We systematically review the existing work on this entire data lifecycle, including data perception, data management, data fusion, and downstream task applications. We compare mainstream simulators and data management architectures in terms of what they contain and how they are constructed. We analyze the mainstream multi-domain data fusion strategies, highlighting the three core challenges in this field (environmental variability, scale limitation, and interaction complexity) and the four major Gaps in multi-domain data fusion. Finally, we point out potential research opportunities with existing tasks, and list several promising future directions for the field, such as continual learning, causal fusion, and large-scale multi-agent collaboration.