V-PTP-IC: End-to-End Joint Modeling of Dynamic Scenes and Social Interactions for Pedestrian Trajectory Prediction from Vehicle-Mounted Cameras

1. Introduction

With the rapid advancement of autonomous driving technologies, pedestrian trajectory prediction has become a critical component in ensuring road traffic safety. According to the World Health Organization (WHO), approximately 1.27 million fatalities occur globally each year due to road traffic accidents, with pedestrian-related incidents accounting for over 22% of cases. In urban environments, this proportion increases to 35% [1]. Consequently, achieving high-accuracy pedestrian trajectory prediction is essential not only for mitigating traffic risks but also for providing prior knowledge to path planning and decision-making modules in autonomous driving systems.

Most existing pedestrian trajectory prediction methods are designed for fixed-camera or global surveillance perspectives, where they demonstrate strong performance in static monitoring scenarios [2]. However, extending such approaches to vehicle-mounted cameras introduces several challenges. First, the mobility of vehicle-mounted cameras leads to continuously changing viewpoints, making static coordinate-based modeling inadequate [3]. Second, the complexity of real-world road environments—with vehicles, buildings, and frequent occlusions—places higher demands on both scene understanding and trajectory prediction [4]. Furthermore, pedestrian motion is jointly influenced by spatial layout and temporal evolution, creating strong spatiotemporal coupling effects that necessitate more advanced modeling strategies [5].

Although some studies have employed deep learning to enhance modeling capabilities, most existing approaches still target fixed-view scenarios and struggle to adapt effectively to the dynamic and complex conditions encountered from vehicle-mounted perspectives.

To address these challenges, we propose V-PTP-IC (Vehicle-view Pedestrian Trajectory Prediction with Interaction Considerations), a novel end-to-end framework that integrates social interaction modeling with dynamic scene perception. Specifically, the data preprocessing module processes multi-pedestrian trajectory data, applying SIFT-based static keypoint matching to compensate for camera-induced motion and stabilize trajectories. The scene feature extraction module employs VGG19 to extract dynamic scene features, providing contextual semantics to enhance trajectory prediction. The core of the model, the pedestrian interaction modeling block, adopts LSTM-based networks to capture complex interactions between pedestrians and their dynamic scene. A feature fusion component integrates both trajectory and scene information over time, improving prediction accuracy. Finally, the model prediction module leverages tailored loss functions to optimize performance for both trajectory and scene prediction tasks. The main contributions of this work are as follows:

• We propose V-PTP-IC, a unified end-to-end framework that combines social interaction modeling, dynamic scene feature extraction, and adaptation to vehicle-mounted camera viewpoints for joint trajectory and scene prediction.
• We introduce a SIFT-based static keypoint matching strategy to compensate for camera-induced motion, reducing trajectory jitter and improving stability.
• We develop a dynamic scene perception mechanism employing VGG19 to encode environmental semantics, thereby enhancing prediction accuracy.
• We conduct extensive evaluation on the JAAD in-vehicle dataset, demonstrating that V-PTP-IC achieves a 22.2% improvement in ADE and a 25.8% improvement in FDE over state-of-the-art methods.

The remainder of this paper is organized as follows: Section 2 reviews related work; Section 3 details the methodology of V-PTP-IC; Section 4 presents experimental results; and Section 5 concludes the paper.

2. Related Work

Pedestrian trajectory prediction aims to forecast future movement paths based on historical trajectory data while considering interactions with surrounding environmental elements, such as other pedestrians, vehicles, and static obstacles. With the increasing adoption of visual sensing technologies, existing research can be categorized into two primary perspectives: fixed/global-view and vehicle-mounted-view approaches.

2.1. Fixed/Global-View Methods

Under fixed-camera or global-view settings, pedestrian trajectory prediction approaches can broadly be divided into three groups: physics-based models, traditional machine learning methods, and deep learning models. Physics-based approaches, such as constant velocity (CV), constant acceleration (CA), reasoning-based models [6], expert systems [7], and other handcrafted methods [8], offer computational efficiency by assuming simplified kinematic patterns. However, such methods often fail to capture nonlinear movements and complex social interactions among pedestrians, limiting their applicability in crowded or highly dynamic scene. To address these limitations, extensions based on the Markov Decision Process (MDP) [9,10] have been proposed, enhancing adaptability by explicitly modeling state transitions and decision-making processes.

Traditional machine learning techniques, including Hidden Markov Models (HMMs) [11], Gaussian Processes (GPs) [12], and Support Vector Machines (SVMs) [13], improve prediction accuracy by incorporating probabilistic reasoning and data-driven learning. Nonetheless, these approaches typically rely on handcrafted features and perform effectively only on small-scale datasets, suffering from limited scalability in high-dimensional or complex urban environments.

Deep learning-based models have recently emerged as the dominant paradigm, owing to their capability to automatically learn spatiotemporal dependencies and social interaction patterns from large datasets. Representative examples include Social-LSTM [14], Graph Convolutional Networks (GCNs) [15], and Transformer-based architectures [16], which can capture long-range temporal correlations and pedestrian-to-pedestrian interactions. Building upon these foundations, the SoPhie model [17] integrates Generative Adversarial Networks (GANs) and attention mechanisms to improve trajectory diversity and semantic understanding, while more advanced frameworks such as Social-STGCNN [18] and STARnet [19] exploit graph-structured representations and spatiotemporal convolutional features to further enhance prediction performance. Despite these advancements, most existing methods are tailored for fixed surveillance viewpoints, and their performance deteriorates when applied to vehicle-mounted camera scenarios, where viewpoint dynamics, occlusion, and environmental variability are more pronounced.

2.2. Vehicle-Mounted-View Methods

Research on pedestrian trajectory prediction from the vehicle-mounted camera perspective remains relatively limited, yet this setting introduces substantially greater challenges compared to fixed-camera scenarios [20]. One fundamental difficulty arises from viewpoint variability and image distortion: the continuous motion of vehicle-mounted cameras induces non-stationary coordinate systems and frequent perspective shifts, while wide-angle lenses can result in fisheye distortion. To mitigate these effects, calibration procedures or fisheye-specific camera models have been proposed [21], although these approaches are often sensitive to noise and require precise parameter tuning.

Another major challenge is occlusion and the accurate modeling of multi-pedestrian interactions. Compared with fixed-camera views, vehicle-mounted perspectives are more susceptible to partial visibility caused by moving objects or road infrastructure. Recent efforts, such as Social-STGCNN [18], have applied Graph Neural Networks (GNNs) to explicitly capture interaction dependencies under occlusion. Furthermore, integrating contextual environmental factors—including road geometry, crosswalk placement, and traffic signal states—has been shown to improve semantic understanding and prediction accuracy in real-world driving scenes [22].

Despite these advances, existing studies often address these challenges in isolation, without providing a unified framework that simultaneously handles viewpoint dynamics, occlusion, and environmental influences. This gap motivates our work, in which we introduce V-PTP-IC, a feature-fusion framework that combines trajectory stabilization, social interaction modeling, and dynamic scene feature extraction to achieve robust and accurate pedestrian trajectory prediction in vehicle-mounted camera settings.

3. V-PTP-IC

3.1. Model Overview

The proposed V-PTP-IC framework is designed to perform robust pedestrian trajectory prediction from vehicle-mounted camera perspectives under dynamic driving conditions. As illustrated in Figure 1, the framework integrates object detection and tracking, trajectory stabilization, dynamic scene feature extraction, social interaction modeling, and final trajectory prediction into a unified end-to-end pipeline. For the perception stage, YOLOv10 is applied to detect pedestrians in real time, while Simple Online and Realtime Tracking (SORT) is used to maintain multi-target identity consistency across frames. To address viewpoint shifts and vibration artifacts inherent to vehicle-mounted camera systems, a Scale-Invariant Feature Transform (SIFT)-based stabilization refinement strategy is employed, ensuring spatial alignment of tracked trajectories. Dynamic scene features are then extracted using a VGG19-based convolutional network, providing contextual semantics that are fused with the stabilized trajectories to enhance environmental awareness. For the prediction stage, the Social-LSTM paradigm is extended by embedding dynamic scene features into the interaction modeling process. This joint representation enables simultaneous learning of pedestrian behavioral patterns and environmental influences, thereby improving both the accuracy and robustness of future trajectory predictions.

3.2. Target Detection and Tracking

The detection and tracking module employs a detection-then-tracking framework, in which we apply YOLOv10 [23] for pedestrian detection and use SORT [24] for continuous multi-target tracking. We configure SORT with a maximum tracking age of 30 frames, a minimum confirmation threshold of 5 frames, and an Intersection-over-Union (IoU) threshold of $0.5$, enabling precise tracking in resource-constrained vehicle-mounted camera scenarios.

For each detected pedestrian, we initialize an independent Kalman filter-based tracker with a seven-dimensional state vector:

$$\begin{array}{c}\hfill \mathbf{s}={[x,y,s,r,\dot{x},\dot{y},\dot{s}]}^{\mathsf{T}},\end{array}$$

(1)

where $x={\displaystyle \frac{w_1+w_2}{2}}$ and $y={\displaystyle \frac{h_1+h_2}{2}}$ denote the bounding box center coordinates; $s=(w_2-w_1)\times (h_2-h_1)$ is the bounding box area; and $r={\displaystyle \frac{w_2-w_1}{h_2-h_1}}$ is the aspect ratio. The terms $(\dot{x},\dot{y},\dot{s})$ represent the corresponding velocity components.

We formulate data association via an IoU-based cost matrix:

$$\begin{array}{c}\hfill C_{ij}=1-IoU(d_i,b_j),\end{array}$$

(2)

where $C_{ij}$ indicates the cost of assigning detection $d_i$ to the predicted bounding box $b_j$. The IoU metric is computed as:

$$\begin{array}{c}\hfill IoU(d_i,b_j)={\displaystyle \frac{\mathrm{Area}(d_i\cap b_j)}{\mathrm{Area}(d_i\cup b_j)}}.\end{array}$$

(3)

We adopt a two-stage matching procedure, where preliminary filtering based on the IoU threshold discards low-overlap pairs, followed by Hungarian assignment for optimal matching. Trajectory management follows an initialization–confirmation–termination scheme to maintain stable identities and suppress short-lived noise tracks.

Post-processing involves three refinement strategies:

• Coordinate consistency restoration: We transform all detections back to the original image coordinate space and map them to a fixed-resolution reference frame, eliminating inconsistencies caused by preprocessing operations such as padding or cropping. This ensures that all trajectory segments share a uniform spatial reference.
• Coordinate normalization: To remove resolution-dependent variations, we normalize all bounding box coordinates to the unit interval $[0,1]$ relative to the reference frame size. This produces a dimensionless representation, making data association thresholds invariant to image resolution.
• Linear interpolation for missing frames: We mitigate short-term dropouts, caused by occlusion or missed detections, by interpolating the center positions and scales between two high-confidence bounding boxes. This is applied only when the temporal gap is below a predefined limit to prevent generating spurious trajectories.

3.3. SIFT-Based Trajectory Stabilization

In vehicle-mounted camera systems, we obtain raw pedestrian trajectories from object detection and tracking, and these trajectories often suffer from geometric distortions and jitter caused by camera ego-motion, detection noise, and viewpoint variations. To address these issues, we employ a global motion compensation strategy based on the SIFT [25], which identifies stable background keypoints across the full image frame. Although we perform keypoint search in the entire frame, the majority of stable points are typically located in upper image regions, where static environmental structures—such as tree canopies, lamp posts, and building edges—are abundant. These stationary background features serve as geometric anchors for estimating and compensating camera-induced displacement in the image plane, thereby improving trajectory stability.

Given an input frame $I^{\left(t\right)}$, we first convert it to grayscale to improve feature detection robustness. We then extract SIFT keypoints and descriptors as:

$$\begin{array}{c}\hfill {\mathbf{KP}}^{\left(t\right)}=\left\{{\mathbf{KP}}_1^{\left(t\right)},\dots ,{\mathbf{KP}}_{n_t}^{\left(t\right)}\right\},\end{array}$$

(4)

where $n_t$ denotes the number of detected keypoints.

We obtain temporal correspondence using pyramidal Lucas–Kanade Optical Flow(LK) [26], propagating each ${\mathbf{KP}}_i^{\left(t\right)}$ to frame $(t+1)$:

$$\begin{array}{c}\hfill {\mathbf{P}}_i^{(t+1)}={\mathbf{KP}}_i^{\left(t\right)}+\mathrm{LK}\_\mathrm{Flow}\left(I_{bg}^{\left(t\right)},I_{bg}^{(t+1)},{\mathbf{KP}}_i^{\left(t\right)}\right),\end{array}$$

(5)

where $\mathrm{LK}\_\mathrm{Flow}(·)$ outputs the displacement vector $(u,v)$.

We discard keypoints with apparent motion above a fixed threshold ${\tau }_{\mathrm{static}}$:

$$\begin{array}{c}\hfill {\mathbf{KP}}_{\mathrm{static}}^{\left(t\right)}=\left\{{\mathbf{KP}}_j^{\left(t\right)}|{\parallel {\mathbf{P}}_j^{(t+1)}-{\mathbf{KP}}_j^{\left(t\right)}\parallel }_2<{\tau }_{\mathrm{static}}\right\},\end{array}$$

(6)

where ${\tau }_{\mathrm{static}}=1.0\mathrm{pixel}$.

We match the remaining static keypoints frame-to-frame using the Fast Library for Approximate Nearest Neighbors (FLANN) [27], approximating k-nearest neighbor search in the SIFT descriptor space:

$$\begin{array}{c}\hfill N{N}_k\left({\mathbf{d}}_i^{\left(t\right)}\right)=\underset{{\left\{{\mathbf{d}}_{j_m}^{(t+1)}\right\}}_{m=1}^k}{argmin}{\parallel {\mathbf{d}}_i^{\left(t\right)}-{\mathbf{d}}_{j_m}^{(t+1)}\parallel }_2,\end{array}$$

(7)

where ${\parallel ·\parallel }_2$ denotes the Euclidean norm in 128-dimensional descriptor space.

To reject ambiguous matches, we apply Lowe’s ratio test:

$$\begin{array}{c}\hfill {\displaystyle \frac{d_{i,1}}{d_{i,2}}}<\rho ,\rho =0.65,\end{array}$$

(8)

where $d_{i,1}$ and $d_{i,2}$ are the smallest and second-smallest Euclidean distances.

We use the matched static keypoints ${\mathbf{KP}}_{\mathrm{match}}^{\left(t\right)}$ and ${\mathbf{KP}}_{\mathrm{match}}^{(t+1)}$ to estimate the dominant inter-frame motion via a Random Sample Consensus (RANSAC) [28]-based affine transformation:

$$\begin{array}{c}\hfill {\mathbf{T}}^{(t\to t+1)}=RANSAC\left({\mathbf{KP}}_{\mathrm{match}}^{\left(t\right)},{\mathbf{KP}}_{\mathrm{match}}^{(t+1)}\right).\end{array}$$

(9)

We apply the inverse transformation to each pedestrian point ${\mathbf{P}}_i^{\left(t\right)}=(x_i^{\left(t\right)},y_i^{\left(t\right)})$, defining the compensation offset:

$${\mathrm{offset}}_i^{(t+1)}={\left({\mathbf{T}}^{(t\to t+1)}\left[\begin{array}{c}{\mathbf{P}}_i^{\left(t\right)}\\ 1\end{array}\right]-\left[\begin{array}{c}{\mathbf{P}}_i^{\left(t\right)}\\ 0\end{array}\right]\right)}_{1:2}·\alpha ,$$

(10)

where $\alpha =0.4$ controls compensation strength. The stabilized point is:

$$\begin{array}{c}\hfill {\mathbf{P}}_{\mathrm{stable},i}^{(t+1)}={\mathbf{P}}_i^{\left(t\right)}+{\mathrm{offset}}_i^{(t+1)}.\end{array}$$

(11)

Finally, we apply a constant-velocity Kalman filter to each trajectory. The state vector for pedestrian i at frame t is

$$\begin{array}{c}\hfill {\mathbf{s}}_i^{\left(t\right)}={\left[x_i^{\left(t\right)},y_i^{\left(t\right)},v_{x,i}^{\left(t\right)},v_{y,i}^{\left(t\right)}\right]}^{\mathsf{T}},\end{array}$$

(12)

with prediction step:

$$\begin{array}{c}\hfill {\mathbf{s}}_i^{(t+1|t)}=\mathbf{M}{\mathbf{s}}_i^{\left(t\right)},\end{array}$$

(13)

where the transition matrix is

$$\begin{array}{c}\hfill \mathbf{M}=\left[\begin{array}{cccc}1& 0& \Delta t& 0\\ 0& 1& 0& \Delta t\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],\end{array}$$

(14)

and $\Delta t$ is the inter-frame interval. We update the state using the ego-motion–compensated observation ${\mathbf{z}}_i^{(t+1)}$:

$$\begin{array}{c}\hfill {\mathbf{s}}_i^{(t+1)}={\mathbf{s}}_i^{(t+1|t)}+{\mathbf{K}}^{(t+1)}\left({\mathbf{z}}_i^{(t+1)}-\mathbf{H}{\mathbf{s}}_i^{(t+1|t)}\right),\end{array}$$

(15)

with observation matrix

$$\begin{array}{c}\hfill \mathbf{H}=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right].\end{array}$$

(16)

We compute the Kalman gain ${\mathbf{K}}^{(t+1)}$ in the standard form to minimize posterior covariance. The resulting stabilized trajectories ${\mathbf{Traj}}_{\mathrm{stable}}\in {\mathbb{R}}^{T\times N\times 2}$ preserve the original format while significantly reducing jitter and improving temporal coherence, thus providing a reliable foundation for downstream social interaction modeling and long-horizon prediction.

3.4. Scene Feature Extraction

To improve trajectory prediction accuracy, we incorporate a scene feature extraction module into the proposed model to capture dynamic scene features—high-level environmental semantics that characterize contextual constraints in vehicle-mounted camera scenarios. By embedding these features into the prediction pipeline, our model can jointly learn pedestrian motion patterns and environment-dependent behavioral cues.

We resize each region of interest (ROI), obtained from YOLOv10 pedestrian detections or scene-level cropping, to $224\times 224$ pixels and normalize it via mean subtraction using ImageNet training-set statistics. This preprocessing ensures compatibility with the ImageNet-pretrained VGG19 [29] architecture.

The hierarchical layers of VGG19 produce multi-level feature representations, ranging from low-level texture patterns to high-level semantic abstractions. In this work, we select the output of the final convolutional layer because it retains spatial context while providing semantically rich descriptors with acceptable computational cost. Let ${\mathbf{I}}_{\mathrm{scene}}$ denote the input scene image; we compute its feature representation as:

$$\begin{array}{c}\hfill {\mathbf{F}}_{\mathrm{scene}}=VGG19\left({\mathbf{I}}_{\mathrm{scene}}\right),\end{array}$$

(17)

where ${\mathbf{F}}_{\mathrm{scene}}\in {\mathbb{R}}^{C\times H\times W}$ encodes abstract scene semantics, with C, H, and W denoting the number of channels, height, and width of the feature map, respectively.

We integrate the extracted scene features with the stabilized pedestrian trajectories ${\mathbf{Traj}}_{\mathrm{stable}}$ from the trajectory stabilization module. We implement feature fusion as a channel-wise concatenation:

$$\begin{array}{c}\hfill {\mathbf{F}}_{\mathrm{fusion}}=Fuse({\mathbf{F}}_{\mathrm{scene}},{\mathbf{Traj}}_{\mathrm{stable}}),\end{array}$$

(18)

where $Fuse(·)$ denotes concatenation along the feature channel dimension.

By explicitly modeling scene elements such as traffic signal states, crosswalk geometry, and neighboring vehicle movement, this module enables the network to capture critical environmental factors that influence pedestrian motion. The resulting fused representation provides a richer context for downstream social interaction modeling and long-horizon trajectory prediction in complex, real-world traffic conditions.

3.5. Pedestrian Interaction Modeling and Joint Trajectory Prediction

3.5.1. Pedestrian Interaction Modeling

As a core component of the proposed framework, the social pooling module implements the social interaction modeling mechanism, explicitly capturing inter-pedestrian influences in dynamic, multi-agent environments. By computing relative positions between a target pedestrian and neighboring pedestrians within a defined spatial range, and aggregating their corresponding latent representations, the module models the mutual influence of agents—a critical factor in complex traffic scenarios where social behavior significantly affects motion dynamics [14,30].

For a target pedestrian i, we define the neighborhood set ${\mathcal{N}}_i$ as:

$$\begin{array}{c}\hfill {\mathcal{N}}_i=\left\{j|{\parallel {\mathbf{P}}_i-{\mathbf{P}}_j\parallel }_2\le R\right\},\end{array}$$

(19)

where ${\mathbf{P}}_i={[x_i,y_i]}^{\mathsf{T}}$ denotes the 2D position vector, and R is the interaction radius controlling the spatial extent of influence.

We obtain the social context for pedestrian i by aggregating the hidden states of all neighbors $j\in {\mathcal{N}}_i$ generated by the LSTM encoder, where ${\mathbf{h}}_j\in {\mathbb{R}}^d$ denotes the d-dimensional hidden representation of pedestrian j:

$$\begin{array}{c}\hfill {\mathbf{h}}_i^{\mathrm{pool}}=MeanPool\left({\mathbf{h}}_j|j\in {\mathcal{N}}_i\right).\end{array}$$

(20)

We transform the pooled social feature ${\mathbf{h}}_i^{\mathrm{pool}}$ via a fully connected layer and nonlinearity:

$$\begin{array}{c}\hfill {\mathbf{h}}_i^{\mathrm{final}}=\sigma \left(\mathbf{W}{\mathbf{h}}_i^{\mathrm{pool}}+\mathbf{b}\right),\end{array}$$

(21)

where $\mathbf{W}\in {\mathbb{R}}^{d^{\prime }\times d}$ and $\mathbf{b}\in {\mathbb{R}}^{d^{\prime }}$ are trainable parameters, and $\sigma (·)$ denotes the element-wise activation function.

We concatenate the enhanced social feature ${\mathbf{h}}_i^{\mathrm{final}}$ with the hidden state of the target pedestrian and feed it into the subsequent LSTM layer for sequential trajectory decoding. This integration allows the model to adapt the effective interaction radius to varying pedestrian densities and spatial configurations, enabling the extraction of richer social behavior patterns.

3.5.2. Joint Scene–Trajectory Prediction

We adopt a joint prediction strategy in the proposed framework that simultaneously estimates future pedestrian trajectories and future dynamic scene features. This dual-branch design consists of a scene prediction branch that models the temporal evolution of the environment, and a trajectory prediction branch that integrates predicted scene features with social interaction features for improved motion forecasting.

In dynamic, vehicle-mounted camera scenarios, environmental changes—such as traffic signal transitions, moving vehicles, and pedestrian flow variations—can significantly influence human motion. To capture these evolving cues, we apply a recurrent neural architecture in the scene prediction branch to forecast the scene feature representation of a future time step before trajectory decoding.

Let ${\mathbf{F}}_t\in {\mathbb{R}}^{C\times H\times W}$ denote the scene feature map at time step t extracted by the scene feature extraction module, and let ${\widehat{\mathbf{F}}}_{t+1}$ denote its prediction for time step $t+1$. We perform temporal modeling using a scene LSTM, which maintains a hidden state ${\mathbf{h}}_t^{\mathrm{scene}}$ and a cell state ${\mathbf{c}}_t^{\mathrm{scene}}$. The LSTM gate operations are defined as:

$$\begin{array}{cc}\hfill {\mathbf{i}}_t& =\sigma \left({\mathbf{W}}_i[{\mathbf{F}}_t,{\mathbf{h}}_{t-1}^{\mathrm{scene}}]+{\mathbf{b}}_i\right),\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill {\mathbf{f}}_t& =\sigma \left({\mathbf{W}}_f[{\mathbf{F}}_t,{\mathbf{h}}_{t-1}^{\mathrm{scene}}]+{\mathbf{b}}_f\right),\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathbf{o}}_t& =\sigma \left({\mathbf{W}}_o[{\mathbf{F}}_t,{\mathbf{h}}_{t-1}^{\mathrm{scene}}]+{\mathbf{b}}_o\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathbf{c}}_t^{\mathrm{scene}}& ={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}^{\mathrm{scene}}+{\mathbf{i}}_t\odot tanh\left({\mathbf{W}}_c[{\mathbf{F}}_t,{\mathbf{h}}_{t-1}^{\mathrm{scene}}]+{\mathbf{b}}_c\right),\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\mathbf{h}}_t^{\mathrm{scene}}& ={\mathbf{o}}_t\odot tanh\left({\mathbf{c}}_t^{\mathrm{scene}}\right), \hfill \end{array}$$

(26)

where $\sigma (·)$ is the sigmoid activation, ⊙ is the Hadamard product, and ${\mathbf{W}}_i,{\mathbf{W}}_f,{\mathbf{W}}_o,{\mathbf{W}}_c$ and ${\mathbf{b}}_i,{\mathbf{b}}_f,{\mathbf{b}}_o,{\mathbf{b}}_c$ are learnable parameters.

In compact form, the LSTM update becomes:

$$\begin{array}{c}\hfill {\mathbf{h}}_t^{\mathrm{scene}},{\mathbf{c}}_t^{\mathrm{scene}}={LSTM}_{\mathrm{scene}}\left({\mathbf{F}}_t,{\mathbf{h}}_{t-1}^{\mathrm{scene}},{\mathbf{c}}_{t-1}^{\mathrm{scene}}\right).\end{array}$$

(27)

We obtain the predicted future scene representation via a projection layer:

$$\begin{array}{c}\hfill {\widehat{\mathbf{F}}}_{t+1}=Projection\left({\mathbf{h}}_t^{\mathrm{scene}}\right),\end{array}$$

(28)

where $Projection(·)$ denotes a fully connected transformation followed by a non-linear activation. The resulting ${\widehat{\mathbf{F}}}_{t+1}$ lies in the same embedding space as ${\mathbf{F}}_t$, enabling direct fusion with other features in the trajectory prediction branch.

We then use ${\widehat{\mathbf{F}}}_{t+1}$ in the trajectory prediction branch, together with social interaction features ${\mathbf{h}}_i^{\mathrm{final}}$, to anticipate future pedestrian positions. Access to both current and anticipated environmental context improves robustness and accuracy in highly dynamic urban driving scenarios. Given the current position ${\mathbf{p}}_t\in {\mathbb{R}}^2$ of pedestrian i, together with its historical trajectory and those of its neighbors, the social pooling mechanism outputs an aggregated interaction feature ${\mathbf{h}}_i^{\mathrm{final}}$. Simultaneously, we encode the motion history of pedestrian i using a trajectory LSTM:

$$\begin{array}{c}\hfill {\mathbf{h}}_t^{\mathrm{traj}},{\mathbf{c}}_t^{\mathrm{traj}}={LSTM}_{\mathrm{traj}}\left({\mathbf{p}}_t,{\mathbf{h}}_{t-1}^{\mathrm{traj}},{\mathbf{c}}_{t-1}^{\mathrm{traj}}\right),\end{array}$$

(29)

where ${\mathbf{h}}_t^{\mathrm{traj}}$ and ${\mathbf{c}}_t^{\mathrm{traj}}$ denote the hidden and cell states at time t.

We form the multi-source context by concatenating ${\mathbf{h}}_t^{\mathrm{traj}}$, ${\mathbf{h}}_i^{\mathrm{final}}$, and ${\widehat{\mathbf{F}}}_{t+1}$:

$$\begin{array}{c}\hfill {\mathbf{h}}_t^{\mathrm{fused}}=Concat\left({\mathbf{h}}_t^{\mathrm{traj}},{\mathbf{h}}_i^{\mathrm{final}},{\widehat{\mathbf{F}}}_{t+1}\right),\end{array}$$

(30)

where $Concat(·)$ denotes channel-wise concatenation.

We map the fused representation to the predicted displacement $\Delta {\mathbf{p}}_t\in {\mathbb{R}}^2$:

$$\begin{array}{c}\hfill \Delta {\mathbf{p}}_t=OutputLayer\left({\mathbf{h}}_t^{\mathrm{fused}}\right),\end{array}$$

(31)

and update the pedestrian position recursively:

$$\begin{array}{c}\hfill {\mathbf{p}}_{t+1}={\mathbf{p}}_t+\Delta {\mathbf{p}}_t.\end{array}$$

(32)

By integrating scene dynamics through ${\widehat{\mathbf{F}}}_{t+1}$ and interaction dynamics via ${\mathbf{h}}_i^{\mathrm{final}}$, our predictor jointly captures agent–agent and agent–environment dependencies, resulting in accurate trajectory forecasts in complex, dynamic, vehicle-mounted camera scenarios.

3.6. Loss Function

We jointly optimize two task-specific objectives in the proposed dual-task framework, aimed at predicting both pedestrian trajectories and future dynamic scene features. This joint learning strategy improves overall prediction performance in complex vehicle-mounted camera scenarios [31,32]. We define the total loss as:

$$\begin{array}{c}\hfill L_{\mathrm{total}}=L_{\mathrm{traj}}+L_{\mathrm{scene}},\end{array}$$

(33)

where $L_{\mathrm{traj}}$ and $L_{\mathrm{scene}}$ represent the trajectory prediction loss and scene feature prediction loss, respectively.

We represent each ground-truth pedestrian trajectory as a sequence ${\left\{{\mathbf{p}}_t^{\left(i\right)}\right\}}_{t=1}^T$, where ${\mathbf{p}}_t^{\left(i\right)}\in {\mathbb{R}}^D$ denotes the D-dimensional position (e.g., $(x,y)$ coordinates) of pedestrian i at time t, and we denote its prediction as ${\widehat{\mathbf{p}}}_t^{\left(i\right)}$. The baseline objective is the mean squared error (MSE) over all pedestrians and time steps:

$$\begin{array}{c}\hfill L_{\mathrm{MSE}}={\displaystyle \frac{1}{T·N}}\sum _{t=1}^T\sum _{i=1}^N{\parallel {\widehat{\mathbf{p}}}_t^{\left(i\right)}-{\mathbf{p}}_t^{\left(i\right)}\parallel }_2^2.\end{array}$$

(34)

To promote diversity in multi-modal predictions, we extend the objective to:

$$\begin{array}{cc}\hfill L_{\mathrm{multimod}}& =\sum _{k=1}^K{\alpha }_k·\mathrm{MSE}\left({\widehat{\mathbf{p}}}^{\left(k\right)},\mathbf{p}\right)+\lambda \sum _{i<j}exp\left(-{\parallel {\widehat{\mathbf{p}}}_t^{\left(i\right)}-{\widehat{\mathbf{p}}}_t^{\left(j\right)}\parallel }_1\right),\hfill \end{array}$$

(35)

where K denotes the number of hypothesis trajectories, ${\alpha }_k$ denotes the confidence weight for the k-th hypothesis, $\lambda$ controls diversity regularization, and ${\parallel ·\parallel }_1$ denotes the Manhattan distance. The first term enforces accuracy for each hypothesis; the second discourages redundant trajectories by penalizing high similarity. We then define the trajectory prediction loss as:

$$\begin{array}{c}\hfill L_{\mathrm{traj}}=L_{\mathrm{multimod}}.\end{array}$$

(36)

Let ${\mathbf{F}}_t\in {\mathbb{R}}^{C\times H\times W}$ denote the ground-truth scene feature map at time t and ${\widehat{\mathbf{F}}}_t$ the predicted feature. We define the reconstruction loss as:

$$\begin{array}{c}\hfill L_{\mathrm{scene}\_\mathrm{MSE}}={\displaystyle \frac{1}{T_{\mathrm{pred}}}}\sum _{t=T_{\mathrm{obs}}+1}^{T_{\mathrm{obs}}+T_{\mathrm{pred}}}{\parallel {\widehat{\mathbf{F}}}_t-{\mathbf{F}}_t\parallel }_2^2.\end{array}$$

(37)

To encourage temporal coherence in predicted features, we incorporate a smoothness term:

$$\begin{array}{c}\hfill L_{\mathrm{scene}\_\mathrm{smooth}}={\displaystyle \frac{1}{T_{\mathrm{pred}}-1}}\sum _{t=T_{\mathrm{obs}}+1}^{T_{\mathrm{obs}}+T_{\mathrm{pred}}-1}{\parallel {\widehat{\mathbf{F}}}_{t+1}-{\widehat{\mathbf{F}}}_t\parallel }_2^2.\end{array}$$

(38)

We then compute the complete scene feature prediction loss as:

$$\begin{array}{c}\hfill L_{\mathrm{scene}}={\lambda }_1{L}_{\mathrm{scene}\_\mathrm{MSE}}+{\lambda }_2{L}_{\mathrm{scene}\_\mathrm{smooth}},\end{array}$$

(39)

where ${\lambda }_1$ and ${\lambda }_2$ are balancing coefficients tuned on the validation set.

This joint objective enables us to optimize spatial–temporal consistency in dynamic scene representation and achieve an accuracy–diversity trade-off in trajectory forecasting, which is crucial for robustness in highly dynamic traffic environments.

4. Experiments and Evaluation

4.1. Dataset and Experimental Setup

Experiments are conducted on the Joint Attention for Autonomous Driving (JAAD) dataset [33], a public benchmark comprising 346 short video clips captured in real-world traffic environments. The dataset covers diverse driving scenarios, including dense urban streets and complex intersections, with all footage recorded from vehicle-mounted cameras, thereby ensuring authentic dynamic perspectives.

For each video frame, YOLOv10 is used to detect and track pedestrians, producing raw position sequences. These tracked trajectories are processed through stabilization, coordinate restoration, and normalization to eliminate scale and alignment inconsistencies. Each input sequence contains observed trajectories over 8 time steps, with the prediction horizon set to the subsequent 12 steps, ensuring consistent temporal resolution across clips. Dynamic scene features are extracted from the corresponding frames using the pretrained VGG19 network, and then fused with stabilized pedestrian trajectories to form the complete model input. The dataset is split into $70\%$ training, $15\%$ validation, and $15\%$ testing subsets. The training set is used for parameter learning, the validation set for hyperparameter tuning and model selection, and the test set exclusively for generalization evaluation. All splits preserve the distribution of environmental contexts and pedestrian densities to prevent bias. Model optimization is performed using stochastic gradient descent (SGD) with an initial learning rate of $0.001$ over 200 epochs. The learning rate remains fixed during training, and network parameters are updated to minimize the combined trajectory and scene prediction loss. The implementation follows a standardized training pipeline, with fixed random seeds and consistent preprocessing to ensure reproducibility. Evaluation metrics and results are reported in the following sections.

4.2. Evaluation Metrics

To comprehensively evaluate the proposed model in pedestrian trajectory forecasting, we adopt widely used quantitative metrics that capture both short-term and long-term prediction accuracy in complex, dynamic vehicle-mounted camera scenarios. The primary metrics are ADE and FDE. Lower values in both metrics indicate higher predictive accuracy.

4.2.1. Average Displacement Error (ADE)

ADE measures the mean Euclidean distance between the predicted and ground-truth trajectories over the entire prediction horizon. For each time step $t\in \{1,\dots ,T\}$, the L2 norm of the positional error is computed and averaged across all T steps:

$$\begin{array}{c}\hfill \mathrm{ADE}={\displaystyle \frac{1}{T}}\sum _{t=1}^T{∥{\widehat{\mathbf{p}}}_t-{\mathbf{p}}_t∥}_2,\end{array}$$

(40)

where ${\widehat{\mathbf{p}}}_t\in {\mathbb{R}}^D$ and ${\mathbf{p}}_t\in {\mathbb{R}}^D$ denote the predicted and ground-truth positions at time t, respectively, and $D=2$ for $(x,y)$ coordinates. Smaller ADE values reflect the proficiency of the model in maintaining high positional accuracy across multiple predicted steps.

4.2.2. Final Displacement Error (FDE)

FDE evaluates the positional error at the final prediction step, placing greater emphasis on long-term forecasting accuracy. It is defined as:

$$\begin{array}{c}\hfill \mathrm{FDE}={∥{\widehat{\mathbf{p}}}_T-{\mathbf{p}}_T∥}_2,\end{array}$$

(41)

where T is the last prediction time step. Lower FDE values indicate the capability of the model to precisely anticipate the endpoint of the trajectory, an aspect particularly important for decision-making in autonomous driving scenarios.

4.2.3. Collision Rate

The collision rate quantifies the proportion of predicted trajectories that intersect with other pedestrians or static obstacles, based on a spatial proximity threshold [34]. Formally, a collision is counted at time step t if the L2 distance between any two predicted pedestrians is below a predefined safety distance ${\tau }_{\mathrm{col}}$. This metric is particularly important in crowded scenes, as it reflects the capability of the model to generate socially compliant and physically feasible multi-agent predictions.

4.2.4. Trajectory Smoothness

Trajectory smoothness evaluates the physical plausibility and naturalness of predicted motion by penalizing abrupt changes in velocity or acceleration [35]. We assess smoothness through three complementary derivatives of the pedestrian position sequence ${\left\{{\mathbf{p}}_t\right\}}_{t=1}^T$:

• Velocity Smoothness:

  $$\begin{array}{c}\hfill S_{\mathrm{vel}}=\sqrt{{\displaystyle \frac{1}{T-1}}{\displaystyle \sum _{t=1}^{T-1}}{\parallel {\mathbf{v}}_t\parallel }_2^2},\end{array}$$

  (42)

  where ${\mathbf{v}}_t={\mathbf{p}}_{t+1}-{\mathbf{p}}_t$ represents the velocity vector at time t. Lower values indicate more uniform motion.
• Acceleration Smoothness:

  $$\begin{array}{c}\hfill S_{\mathrm{acc}}=\sqrt{{\displaystyle \frac{1}{T-2}}{\displaystyle \sum _{t=1}^{T-2}}{\parallel {\mathbf{a}}_t\parallel }_2^2},\end{array}$$

  (43)

  where ${\mathbf{a}}_t={\mathbf{v}}_{t+1}-{\mathbf{v}}_t$ is the acceleration vector. Smaller values correspond to smoother velocity changes.
• Jerk Smoothness:

  $$\begin{array}{c}\hfill S_{\mathrm{jerk}}=\sqrt{{\displaystyle \frac{1}{T-3}}{\displaystyle \sum _{t=1}^{T-3}}{\parallel {\mathbf{j}}_t\parallel }_2^2},\end{array}$$

  (44)

  where ${\mathbf{j}}_t={\mathbf{a}}_{t+1}-{\mathbf{a}}_t$ is the jerk vector. This term measures the stability of acceleration changes; lower values indicate more physically plausible motions.

These smoothness measures help ensure that predicted trajectories not only match the ground-truth positions (ADE/FDE) but are also kinematically realistic, which is essential for deployment in real-world autonomous navigation.

4.3. Trajectory Stabilization Visualization

Figure 2 presents a qualitative comparison between raw and stabilized pedestrian trajectories. The original trajectory (red) exhibits noticeable high-frequency oscillations and abrupt jumps, which are significantly mitigated after SIFT-based stabilization (blue), yielding a temporally coherent path.

Figure 3 illustrates the matching of static background keypoints between adjacent frames. These matches serve as geometric anchors for estimating and compensating camera-induced displacements, thus directly enabling the trajectory smoothing effect shown in Figure 2.

4.4. Comparative Experiments

To assess the effectiveness of the proposed V-PTP-IC framework—built upon Social-LSTM augmented with dynamic scene feature fusion—we conduct comparative experiments against representative pedestrian trajectory prediction baselines. All baseline models are trained and evaluated under identical experimental settings, using the same trajectory and scene feature preprocessing pipeline, dataset splits, and optimization parameters, ensuring fairness in comparison.

The baselines are as follows:

• Standard LSTM: A fundamental Long Short-Term Memory network that independently models each pedestrian’s trajectory without incorporating social interactions or environmental context. Serves as a minimal baseline for sequential forecasting.
• LSTM Encoder–Decoder: An architecture employing an LSTM-based encoder to capture historical trajectory features and an LSTM-based decoder to predict future positions. This model improves upon Standard LSTM in sequence handling but still lacks explicit interaction or scene modeling.
• Social-LSTM: An extension of LSTM that incorporates a social pooling mechanism to model inter-pedestrian influences, thereby improving accuracy in multi-agent scenarios. This serves as a direct reference point for evaluating the impact of integrating dynamic scene features into social interaction modeling.
• Transformer: A self-attention-based sequence model capable of capturing global temporal dependencies across trajectory sequences. Unlike recurrent architectures, it efficiently models relationships between any two time steps, showing strong performance in long-range motion forecasting tasks.

Table 1 reports the quantitative results on the JAAD test set, with comparisons conducted using ADE, FDE, and training time. These results provide both accuracy and efficiency perspectives for evaluating the benefits of the proposed framework.

4.5. Ablation Studies

We conduct ablation experiments to quantify the contribution of each core component in the proposed V-PTP-IC framework: the social pooling module, dynamic scene feature fusion, and SIFT-based trajectory stabilization. The goal is to separately evaluate the effect of removing or replacing each module and to investigate possible interactions between components.

4.5.1. Ablation on Model Components

Table 2 presents the quantitative results for various model configurations, comparing $\mathrm{ADE}$, $\mathrm{FDE}$, and trajectory smoothness scores. The “Benchmark Model" here denotes the base trajectory LSTM without any of the three components.

Table 3 further summarizes the relative $\mathrm{ADE}$/$\mathrm{FDE}$ improvements (in %) over the benchmark model across different component configurations.

Results indicate several important trends: when applied in isolation, the social pooling module leads to a significant performance drop ($-18.28\%$ ADE, $-25.22\%$ FDE), likely due to the introduction of noisy interaction cues in sparse or low-quality detection scenarios. By contrast, using only dynamic scene features yields moderate gains ($+4.76\%$ ADE, $+4.55\%$ FDE), as scene-level environmental context improves the plausibility of motion forecasts. The SIFT-based stabilization module alone achieves a notable ADE gain ($+12.09\%$) by compensating for ego-motion and detection-induced jitter, though its effect on long-horizon FDE is limited ($+1.37\%$).

Combining scene features with social pooling mitigates interaction noise and substantially boosts FDE performance ($+9.16\%$ ADE, $+16.32\%$ FDE), indicating that environmental cues support more reliable interaction modeling. Pairing social pooling with stabilization ($+17.95\%$ ADE, $+0.33\%$ FDE) suggests that stabilizing trajectories improves the extraction of social interaction patterns, though benefits are less pronounced in long-term predictions. Integrating scene features with stabilization achieves balanced improvements in both metrics ($+9.04\%$ ADE, $+8.39\%$ FDE), enhancing the physical realism and smoothness of predicted paths.

The complete V-PTP-IC model, which unifies all three components, delivers the highest gains ($+22.22\%$ ADE, $+25.88\%$ FDE), confirming that balanced fusion of dynamic scene understanding, social interaction modeling, and stabilization yields the optimal trade-off between accuracy, diversity, and trajectory smoothness.

4.5.2. Comparison of Scene Feature Backbones

Given the critical role of scene features in V-PTP-IC, we further compare two popular CNN backbones: VGG19 and ResNet50, both pretrained on ImageNet. Figure 4 shows the training loss comparisons.

We observe that although ResNet50 exhibits faster early-stage convergence, VGG19 achieves lower final validation loss. This may be attributed to the capability of VGG19 for capturing high-level semantic features of static structures (e.g., buildings, road layouts) that remain stable in dynamic driving scenes, leading to better generalization when data is limited.

4.5.3. Qualitative Visualization

Figure 5 shows multi-modal trajectory prediction results from V-PTP-IC. Observed trajectories are in blue, the ground truth in green, the most probable predicted mode in red, and lighter curves represent alternative predicted modes. The main predicted mode (red) aligns with the ground truth (green), confirming accurate path forecasting. Alternative modes exhibit diverse yet reasonable variations that comply with scene constraints, reflecting the ability of the model to represent motion uncertainty.

These results are consistent with quantitative evaluations and verify that combining SIFT-based stabilization, VGG19-based scene features, and social interaction features leads to predictions that are both accurate and smooth. In particular, SIFT-based stabilization reduces jitter from ego-motion and detection noise; VGG19-based scene features provide rich contextual cues from the environment; and social interaction modeling captures pedestrian–pedestrian influence. Together, they produce trajectories that are precise, stable, and socially compliant in complex vehicle-mounted camera scenarios.

5. Conclusions

This paper has addressed the challenges of viewpoint dynamism, scene complexity, and spatiotemporal coupling in pedestrian trajectory prediction from vehicle-mounted camera perspectives through an end-to-end joint feature fusion framework. The proposed V-PTP-IC method integrates two components: a dynamic scene understanding module, including SIFT-based background keypoint matching for ego-motion compensation and trajectory stabilization together with VGG19-based scene feature extraction for capturing both static and dynamic environmental context; and a social interaction modeling module built upon a Social-LSTM network for learning spatiotemporal dependencies among pedestrians and producing socially compliant motion forecasts. By unifying these two modalities with a balanced feature fusion strategy, the framework fully leverages both dynamic scenes and pedestrian interactions to produce accurate and robust trajectory forecasts in complex driving scenarios. Experiments on the JAAD dataset demonstrate that V-PTP-IC achieves 22.2% and 25.8% improvements in ADE and FDE, respectively, over mainstream baselines, validating the effectiveness of the proposed joint modeling approach.

Future work will focus on enhancing real-time performance through lightweight architecture design, exploring advanced sequence modeling such as Transformers to capture long-range spatiotemporal dependencies in both scene and interaction domains, and extending the framework to more diverse real-world driving conditions—including adverse weather and challenging illumination—to further improve generalization and practical applicability.