Active Vision for Social Navigation

1. Introduction

A central limitation of the social navigation pipeline is often the narrow instantaneous field of view (FOV) of a forward-facing camera. In cluttered, occluded, or dynamic scenes, pedestrians can enter the robot’s path from oblique angles without being observed until they are already dangerously close, reducing the effectiveness of downstream prediction and increasing the likelihood of high-risk encounters. Active vision addresses this limitation by treating viewpoint selection as a policy in which the robot learns to choose where to look while simultaneously controlling locomotion. Concretely, active vision improves social navigation with a discrete gaze state over a small set of yaw-offset views, enabling proactive sensing behaviors such as checking blind spots and following human trajectories that reduce perceptual surprise and improve social navigation in unstructured environments.

This article introduces a new reinforcement learning framework: Active Vision for Social Navigation. Our proposed method augments a perception–prediction–control pipeline with 1) a discrete view-selection mechanism, 2) joint reinforcement learning over locomotion and gaze actions, and 3) a spatiotemporal attention mechanism to predict near-future human occupancy. Each control step consists of three stages: selecting a gaze window index (pan) together with a wheel-motion command; rectifying the selected view and computing compact perceptual channels as the state observation; and providing the RL agent with a fused low-dimensional observation that includes both geometry and semantic cues alongside predicted human occupancies.

Learning viewpoint control and navigation control simultaneously is substantially harder than learning either in isolation, and much of the active-vision literature explicitly avoids this fully coupled formulation. Prior work typically reduces complexity by factorizing sensing and task behavior into separate policies or asynchronous loops, thereby shortening credit-assignment paths and stabilizing learning. Shang and Ryoo formulate active vision with separate sensory and motor policies to address limited observability and indirect supervision for viewpoint control [1]. Dass et al. show that separating information-seeking from task execution makes learning more tractable by assigning distinct roles to context acquisition and downstream decision making [2], while Wang et al. explicitly decouple observation and action in an asynchronous active vision–action framework [3]. Even self-supervised approaches such as [4] sidestep the coupling by first rewarding camera behaviors that improve predictability, rather than requiring task reward to supervise sensing choices directly.

Our formulation learns a single joint policy over the product action space (navigation × view angle), forcing the agent and world model to solve coupled credit assignment. This is challenging for three reasons: gaze actions have delayed utility since a pan action improves future observation quality rather than yielding immediate reward; gradients compete, as early navigation mistakes dominate returns while the signal for beneficial gaze shifts is weak and indirect; and view transitions expand the learned dynamics, changing the observation distribution even when the robot does not move. Despite this difficulty, joint optimization is well-motivated here because viewpoint control is not an auxiliary behavior but a task-dependent component of navigation itself, where the robot looks directly determines what it can infer about nearby humans and traversable space, which in turn shapes the optimal motion decision. Factored approaches [1,2,3] risk biasing the sensing strategy toward broadly informative views rather than those specifically useful for socially safe locomotion, whereas a joint policy can learn task-specific sensorimotor coordination such as briefly redirecting gaze to resolve an ambiguous human interaction before choosing a safer path. This long-horizon coupling is precisely what model-based RL is designed to capture through imagined trajectories over a learned latent world model [5].

For this to work, the world model must learn to predict how viewpoint changes alter subsequent observations. Because gaze and motor commands are jointly sampled and modeled, the latent world model associates pan actions with their resulting visual consequences, when the agent pans toward a pedestrian, the world model links that action to the ensuing observation change and any downstream proxemic penalty. Imagined rollouts can therefore assign higher value to action sequences that proactively redirect gaze toward anticipated human occupancy, producing look-before-you-drive behavior through temporal credit assignment.

Section 2 provides an overview of related work on social navigation. Our proposed architecture is described in Section 3. Section 4 presents a comparison of our method against several competitive benchmarks in three social navigation scenarios. Our results show that active vision produces a clear statistical difference in performance at achieving a higher number of navigation goals with fewer near collisions.

2. Related Work

Social navigation, the ability for robots to navigate safely and predictably in environments shared with humans—has evolved significantly with advances in perception, learning, and planning. Early work on social robot navigation emphasized proxemics and safety-aware planning [6,7], while recent trends leverage deep learning for trajectory forecasting and policy learning. Social navigation requires autonomous robots to reason not only about geometric constraints but also about human social behaviors, such as maintaining interpersonal distance and yielding right-of-way. Hall’s proxemics theory formalized the interpersonal zones that robots must respect [8]; this article evaluates the performance of the RL agent using Hall’s suggested proxemic zones for personal and social distances. Early planners extended the Dynamic World Approach (DWA) and velocity-obstacle schemes with proxemic costs to remain inside socially acceptable regions [9,10,11,12]. These rule-based methods guarantee safety but struggle with the diversity of human behaviors in dense crowds.

End-to-end learning replaces hand-tuned costs with policies optimized directly for social compliance. Tai et al. used GAIL to learn depth-only navigation that implicitly captures human preferences [13]. DRL-VO merges RL with velocity-obstacle geometry for faster convergence in cluttered scenes [14], while uncertainty-aware RL penalizes high-variance actions to curb the “freezing” phenomenon [15]. Reward shaping based on heuristics or social norms further steers policies toward courteous behavior [16]. Our proposed method is unique in its inclusion of active vision to prevent perceptual surprise and its ability to learn joint policies for sensing and action.

3. Materials and Methods

Our system architecture is a multi-process pipeline that fuses human intent prediction with terrain-aware reinforcement learning. For each new camera frame, a YOLOv11 detector identifies current pedestrian locations while a spatiotemporal attention mechanism predicts future human occupancy patterns from temporal frame sequences. Concurrently, monocular depth estimation extracts geometric scene structure from the latest RGB frame. These three information streams—current pedestrian masks overlaid on grayscale imagery, attention-based trajectory predictions, and depth maps—are fused into a compact 96×96×3 observation tensor that encodes both social and geometric constraints. This multimodal representation serves as input to a DreamerV3 [5] reinforcement learning agent that learns socially compliant navigation policies through latent world modeling, enabling real-time decision making that respects both human proxemics and terrain traversability in unstructured environments. Figure 1 shows the active vision pipeline in which a single wide-angle fisheye camera provides five discrete view frames. The RL agent learns a joint policy with motor commands and view selection.

3.1. Datasets

We employ the SiT Social-Interactive Trajectory (SiT) [17] corpus augmented with sequences collected in the DuNE–Gazebo [18] simulator to ensure coverage of both real and synthetic unstructured scenes, totaling just over 100k images from multiple video sequences. For every video sequence, frames are first subsampled at 10 Hz and assembled into fixed–length windows that respect the temporal offsets in ${\mathcal{T}}_{\mathrm{past}}=\{0,-0.5,-1.0,-1.5,-2.0\}\mathrm{s}$ and a prediction horizon ${\tau }_f=+2.0\mathrm{s}$. Each window thus contains $T=5$ RGB frames (current + 4 previous) ${\left\{{\mathbf{I}}_{t+k}\right\}}_{k\in {\mathcal{T}}_{\mathrm{past}}}$. A lightweight YOLOv11n model detects pedestrians in all frames; detections are linked across time via a distance–based tracker to yield short trajectories. From the centroids ${\mathbf{p}}_i=(x_i,y_i)$ of pedestrians in the future frame we generate a heatmap

$$H(x,y)=\underset{i}{max}exp\left[-\frac{{\parallel {\mathbf{p}}_i-(x,y)\parallel }_2^2}{2{\sigma }^2}\right],\sigma =16\mathrm{px},$$

(1)

which is normalized to $[0,1]$. Simultaneously we rasterize pedestrian YOLO center boxes into binary masks ${\mathbf{M}}_{t+k}$ and compute monocular depth ${\mathbf{D}}_{t+k}$ with Depth-Anything V2 [19]. All tensors are resized to $320\times 320$, stored as half-precision mem-maps, and streamed in constant-time batches during training, eliminating GPU out-of-memory crashes when scaling to the full ( ≈  107,000) image dataset.

3.2. Spatiotemporal Attention Mechanism

We predict near–future human occupancy as a dense heatmap from a short RGB history and per–frame person masks. At each step we form a sequence ${\left\{({\mathbf{I}}_{t-k},{\mathbf{M}}_{t-k})\right\}}_{k=0}^{T-1}$ of T RGB frames and their binary masks (from YOLO detections). Each pair is encoded by lightweight CNNs with shared structure: strided convolutions downsample from $320\times 320$ to $80\times 80$ and $40\times 40$, producing a bottleneck feature ${\mathbf{F}}_{t-k}\in {\mathbb{R}}^{40\times 40\times C}$ and skip maps at $80\times 80$ and $40\times 40$ for decoding. We then concatenate RGB and mask features channel-wise, stack them across the T timesteps, and reshape to a token sequence of length $T·40·40$:

$$\mathbf{Z}\in {\mathbb{R}}^{B\times (T·40·40)\times C}.$$

(2)

A linear layer projects $\mathbf{Z}$ to an embedding dimension D (configurable; defaults provided in code), followed by LayerNorm and the addition of a learned spatiotemporal positional embedding of the same shape. This absolute embedding lets the model encode both spatial indices and relative time implicitly.

We apply a single SelfAttention block with 8 heads over the token axis. The attention is the standard scaled dot-product:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V,$$

(3)

implemented inside multi-head projections so each head attends over the $D/h$-dimensional subspace. The attended sequence is added residually and normalized, then passed through a linear projection and dropout. We then reshape back to $[B,T,40,40,D]$ and perform temporal mean pooling to obtain a single $40\times 40$ latent per batch.

Decoding starts from the pooled $40\times 40$ latent. We concatenate the $40\times 40$ skip feature via a $1\times 1$ projection and upsample with transposed convolutions to $80\times 80$ and finally $320\times 320$, concatenating the $80\times 80$ skip along the way. A final $3\times 3$ convolution with sigmoid produces the occupancy heatmap $\widehat{H}\in {[0,1]}^{320\times 320}$.

Training targets H are built from future YOLO detections by rasterizing each person into an elliptical Gaussian centered at the detection centroid, with per-axis standard deviations scaled by the bounding-box size. The model is optimized with a balanced focal+Dice objective:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{focal}}(\widehat{H},H)+{\mathcal{L}}_{\mathrm{dice}}(\widehat{H},H),$$

(4)

where the focal term uses the usual $(\alpha ,\gamma )$ parameters and the Dice term promotes overlap with soft targets (both implemented in code). For monitoring, we additionally report RMSE over the heatmap.

We train with Adam (Optax) and gradient clipping (global ${\ell }_2$ norm $1.0$). The learning rate follows a warmup+cosine schedule (linear warmup then decay), with default mini-batches of 8 sequences and sequence length $T=5$ (all configurable). The model and optimizer are instantiated in JAX/Flax; training loops include on-the-fly validation and optional TensorBoard logging. Checkpoints save the full parameter PyTree and config for resumption.

3.3. Reinforcement Learning

We build on DreamerV3 [5], which learns a compact latent-space dynamics model from pixels and proprioception, and then performs ’imagined’ rollouts in that latent space to train the policy and estimate long-horizon returns to optimize an actor–critic policy before executing in the environment. Specifically, DreamerV3 learns a recurrent state-space model (RSSM) whose latent evolves stochastically $z_t\sim p_{\varphi }(z_t\mid z_{t-1},a_{t-1})$ while reconstructing observations through a decoder $o_t\sim p_{\varphi }(o_t\mid h_t,z_t)$. Let $h_t$ denote the deterministic recurrent state (e.g., GRU hidden), $z_t$ the stochastic latent, and $s_t[h_t,z_t]$ the concatenated latent consumed by the actor, critic, and decoders. Actor and critic are then trained on imagined sequences ${\left\{s_t\right\}}_{t=1}^T$, enabling sample-efficient control from high-dimensional inputs and reducing sensitivity to pixel noise. In our navigation setting, a fused encoder (RGB + attention + depth) supplies the world model with traversability and goal cues; the learned dynamics support long-horizon, goal-directed planning that captures both skid-steer kinematics and terrain structure.

The attention-based prediction module injects explicit social priors into long-horizon planning. Let ${\mathcal{O}}_{t-\tau :t}$ denote the past $\tau$ seconds of observations and ${\widehat{\mathbf{p}}}_{t+\delta }=f_{\theta }\left({\mathcal{O}}_{t-\tau :t}\right)$ the predicted pedestrian positions $\delta$ seconds ahead. These forecasts are fused into the agent’s representation (as an additional observation channel and latent feature), so DreamerV3’s imagined rollouts evaluate candidate actions under anticipated human occupancy rather than relying solely on raw pixels. Practically, this biases both the learned world model and the agent toward trajectories that steer around predicted human locations and sustain comfortable separation.

3.3.1. RL Agent Reward Design

The reward function is designed to balance efficient goal-directed navigation with socially compliant behavior in dynamic, unstructured environments. At each timestep t, the agent receives a composite reward $R_t$ that encourages progress toward the navigation target while discouraging unsafe or inefficient behaviors. The reward is computed as:

$$R_t=\alpha \Delta d_t+\beta \left(1-{\displaystyle \frac{|{\theta }_t|}{\pi }}\right)+\gamma v_t-\delta \left|{\omega }_t\right|-ϵ-\sum _{i=1}^{N}P\left(d_i\right)+\kappa \Delta d_t{\mathbf{1}}_{\mathrm{aligned}}+R_{\mathrm{goal}}{\mathbf{1}}_{\mathrm{success}}$$

(5)

where $\Delta d_t=d_{t-1}-d_t$ represents the decrease in Euclidean distance to the goal (positive for progress), ${\theta }_t\in [-\pi ,\pi ]$ is the heading difference between the robot’s current orientation and the target direction, $v_t$ is the linear velocity, and ${\omega }_t$ is the angular velocity. Social safety is incorporated through a proxemic penalty function $P\left(d_i\right)$ applied to each of N nearby human actors at distance $d_i$. Based on Hall’s proxemic zones [8], the penalty grows in discrete stages:

$$P\left(d\right)=\left\{\begin{array}{cc}25.0\hfill & \mathrm{if}d<0.5\mathrm{m}(\mathrm{intimate}\mathrm{zone})\hfill \\ 8.0\hfill & \mathrm{if}0.5\mathrm{m}\le d<0.8\mathrm{m}(\mathrm{personal}\mathrm{zone})\hfill \\ 2.0\hfill & \mathrm{if}0.8\mathrm{m}\le d<1.2\mathrm{m}(\mathrm{social}\mathrm{zone})\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(6)

The indicator function ${\mathbf{1}}_{\mathrm{aligned}}$ provides an alignment bonus when the robot makes forward progress ($\Delta d_t>0$) while well-aligned with the target ($|{\theta }_t|<\pi /4$). The goal reward $R_{\mathrm{goal}}=100.0$ is achieved when the robot reaches within 0.3m of the goal . The hyperparameters are set to $\alpha =2.0$ (distance progress), $\beta =0.03$ (heading alignment), $\gamma =0.015$ (velocity), $\delta =1.8$ (spin penalty), $ϵ=0.008$ (time penalty), and $\kappa =0.3$ (alignment bonus). Episodes terminate after 6000 steps. Together, these terms provide a dense and balanced learning signal that guides the agent toward socially aware, efficient navigation through rough terrain populated with dynamic pedestrians.

3.3.2. Environment Configuration

We conduct all experiments in the DUnE simulation testbed [18]. Utilizing the Gazebo Fortress simulation engine, we test in three different environments: Inspection, Construction, and Island (Figure 2). The robot used in testing is the FictionLab Leo Rover (0.985 m wheel-base, differential drive)  [20]. Each scenario is initialized with a random goal position and 2–3 dynamic human actors that follow pre-scripted patrol waypoints within the same area as the random goal points.

3.3.3. Multi-Process Architecture Integration

A separate inference process takes camera images and applies a YOLOv11n-based pedestrian detector to each sampled frame. The resulting detections are converted into binary pedestrian masks, which are fused with the RGB inputs and passed to a spatiotemporal attention model. This model predicts a heatmap representing the likelihood of future human locations in the scene, leveraging both motion history and semantic context. Concurrently, the most recent RGB frame is processed by a monocular depth estimator (DepthAnythingV2) to obtain a normalized depth map. These three signals—grayscale+current pedestrian mask+predicted pedestrian attention heatmap, and depth image—are resized and fused into a $96\times 96\times 3$ tensor suitable for downstream reinforcement learning. The resulting observation is written to a third shared memory block (rl_observation) using an atomic update protocol that includes a timestamp and validity flag.

The overall architecture ensures isolation between perception and control, enabling asynchronous inference and action cycles. By relying on shared memory rather than message-passing, the system minimizes latency and avoids blocking between modules. Synchronization is enforced via timestamp-based freshness checks and atomic memory layout conventions. This integration strategy supports continuous, low-latency operation in real-time robotics environments, with clear modular boundaries between ROS-based frame acquisition, attention inference, and RL-based navigation.

3.3.4. RL Agent Image Observation Space Design

At each time t the agent receives a 96 pixel × 96 pixel $\times 3$ tensor

$${\mathbf{O}}_t=\left[{\mathbf{I}}_t^{\mathrm{gray}}\parallel {\widehat{H}}_t\parallel {\mathbf{D}}_t\right],$$

(7)

where “∥’’ denotes channel-wise concatenation.

• Channel 1: Grayscale image with masks. The current RGB frame is converted to luminance and normalised to $[-1,1]$. Pixels inside the YOLO pedestrian bounding boxes are overwritten with the constant value $-1$ to emphasize occupied regions.
• Channel 2: Human traversability heat-map${\widehat{H}}_t\in {[0,1]}^{96\times 96}$ predicted by the attention model; values are logit-scaled to $(0,1)$.
• Channel 3: Depth map${\mathbf{D}}_t$ acquired from a simulated Intel Realsense at 30 Hz, clipped to 0–10 m and linearly mapped to $[-1,1]$.

Before concatenation each channel is resized with bilinear interpolation and subjected to per-channel mean–variance normalisation computed over the training set. This fusion encodes both physical obstacles (depth), socially preferred regions (heat-map), and fine-grained appearance cues (grayscale) in a compact representation compatible with convolutional encoders. Figure 3 illustrates the information received by the RL agent.