Towards Better Predictive Models: The Role of Density in Pedestrian Trajectory Predictions

1. Introduction

The challenge of predicting pedestrian trajectories has emerged as a pivotal challenge in recent years. This surge in interest is largely attributed to the profound implications it holds for autonomous vehicle navigation [1], service robot deployment [2], and the strategic planning of infrastructure and mass gatherings [3]. In addressing these intricate challenges, researchers have traditionally employed physics-based (PB) models to simulate and understand pedestrian behavior. These models have been instrumental in dissecting collective phenomena and enhancing our understanding of pedestrian dynamics, particularly in high-density contexts relevant to crowd management and evacuation strategies [3]. However, the landscape of pedestrian trajectory prediction has witnessed a paradigm shift over the last decade with the advent and integration of deep learning (DL) algorithms [4]. Despite the opaqueness of these models in terms of interpretability, their superiority in mirroring observed trajectories has been markedly pronounced, especially when juxtaposed with their PB counterparts [5]. Nonetheless, it is important to acknowledge that the domains of applicability for PB and DL models do not entirely overlap. While PB models excel in the realm of high-density simulations, providing insights into collective behavior, DL algorithms predominantly thrive in low-density environments where individual pedestrian movements are characterized by a greater degree of freedom and intricate long-range interactions [4]. This paper introduces a novel, real-world pedestrian trajectory dataset, gathered during the Festival of Lights in Lyon. Field pedestrian trajectory datasets are typically gathered from low-density situations. In contrast, this dataset captures the nuanced dynamics of pedestrian movements across a large spectrum of density levels. It ranges from sparse crowds observed during show moments to the densely packed throngs seen after the event. Utilizing this dataset, we train DL algorithms, including Long Short-Term Memory (LSTM) networks and Generative Adversarial Networks (GAN). Our methodology is underscored by a novel approach: we harness situational classification predicated on crowd density to refine our models’ learning process. This two-stage process, which initially classifies the scene based on density and then predicts the trajectories, not only bolsters the efficiency of our models but also substantially elevates the precision of trajectory predictions. This improvement is demonstrated by comparative analyses with traditional DL algorithms and PB models.

Another challenge of trajectory prediction that we face in this paper is overlapping and colliding of predicted trajectories [6]. To tackle that problem, we integrate a time-to-collision (TTC) [7] term into the loss function of the algorithms. A parameter, $\lambda$, is utilized to modulate the TTC’s influence on the training. Our empirical research uncovers a significant relationship between the optimal $\lambda$ values and the density levels, highlighting the intricacies of pedestrian behavior across different densities.

The remainder of this paper is organized as follows: first, we review related studies in Section 2. Then, our novel dataset is presented in Section 3 and the methodology for the empirical work is proposed in Section 4. The results are shown in Section 5. The last Section 6 includes a discussion of the results and an outlook on future works.

2. Related work

The domain of pedestrian trajectory prediction is multifaceted, drawing insights from various disciplines and methodologies. The two main stream are the physics-based models and data-based algorithms [4]. PB models have been the cornerstone of understanding pedestrian dynamics, especially in high-density scenarios. The Social Force model (SF), introduced by Helbing and Molnar [8], exemplifies this approach, simulating pedestrian movement by balancing attractive and repulsive forces. Other famous PB models are the Optimal Reciprocal Collision Avoidance (ORCA) from Van den Berg et al. [9] or the cellular automata model from Burstedde et al. [10]. However, these models are not without their challenges, particularly when it comes to encapsulating the full range of crowd behavior [4]. For more PB models see reviews like [11,12,13].

In pursuit of addressing these limitations, the research frontier has gradually shifted towards data-driven methodologies. Notably, the past decade has witnessed a burgeoning interest in DL approaches. Pioneering works like the Social LSTM from Alahi et al. [5] introduce the use of Recurrent Neural Networks (RNN), specifically LSTM networks, in conjunction with a novel concept known as Social Pooling. This innovative approach incorporates neighbouring information, thereby enriching the model’s contextual understanding. This social concept was further enhanced by Gupta et al. [14] through Social GAN, where the generative adversarial framework allowed for the generation of multiple plausible future paths, addressing the inherent uncertainty in human movement. Alternative methods employed for social predictions include attention mechanisms [15], graph-based approaches [16], and the utilization of relative coordinates [17]. Additionally, deep learning architectures such as Convolutional Neural Networks [18,19] and Transformers [20] have been applied in trajectory prediction tasks.

A pivotal aspect of this paper is the innovative classification of trajectory scenes based on crowd density prior to prediction. To the best of our knowledge, this approach is a novel paradigm, potentially owing to the scarcity of high-density, real-world pedestrian trajectory datasets. Xue et al. [21] predict pedestrian destinations using bidirectional LSTM classification. This involves an additional classification stage to distinguish between possible destinations of pedestrians. They classify the route manually into four distinct categories. In another paper from the authors [22] the classification is done based on a clustering algorithm. Kothari et al. [23] categorize pedestrian trajectories based on the nature of interactions observed, identifying behaviors such as collision avoidance, leader-follower dynamics, and grouping behavior. An alternative methodology involves classifying trajectories based on individual pedestrian characteristics. Papathanasopoulou et al. [24] concentrate on attributes such as age, gender, height, and speed to inform their classification. A second cornerstone of this work is the seamless integration of a PB concept, TTC, into the loss function of DL algorithms. This synthesis of PB principles and DL models is not an isolated endeavor. Alahi et al. [5] and Khadka et al. [25] utilized simulated data from PB models for training DL algorithms. Antonucci et al. [26] embedded a PB model directly into the DL architecture. Furthermore, the works of Silvestri et al. [27] and Kothari et al. [28] stand out for their use of PB principles within the loss function to eliminate unrealistic predictions.

3. The Dataset

With a growing interest in data-based methods the significance of pedestrian trajectory data has been elevated in recent research. This area has seen a proliferation of datasets published by researchers, which can be categorized into field data and experimental data obtained in laboratory conditions. In the field studies, real-world settings are employed where individuals, unaware of their participation in a study, navigate through various scenarios. Famous field datasets are the ETH [29] and UCY [30] datasets, which are widely used in the machine learning community. Originating from surveillance videos, these datasets capture pedestrians scene of low density (0.1-0.5 $ped/m^2$). Other field datasets are the Stanford Drone Dataset [31], the Grand Central Station Dataset [32], and the Edinburgh Informatics Forum Dataset [33]. None of these have densities above 0.2 $ped/m^2$. In the following we will present a field dataset with pedestrian densities between 0.2-2.2 $ped/m^2$.

The data was collected at Lyon´s Festival of Lights. The event running for four days from 7 pm to 11 pm attracts millions (2 million in 2022) of visitors each year. Key attractions are light shows at Place des Terreaux and Place Saint-Jean. We have installed cameras at the Place des Terreaux to film the area which is represented by the red rectangle in Figure 1.

In Figure 1a, the entirety of Place des Terreaux is depicted. The red box on the right-hand side delineates our designated tracking region. Figure 1b offers an aerial perspective of this same area. This designated zone measures 9 meters in length and 6.5 meters in width. On average, we concurrently tracked 55 pedestrians, resulting in a mean density of 0.95 $ped/m^2$. The distribution of pedestrian density exhibited significant variability. During the light show, the majority of pedestrians congregated in the central area of the square, remaining largely stationary. Consequently, the pedestrian density within our tracking zone was relatively low. However, when the show concluded—which consistently lasts approximately 9 minutes—the crowd dynamics shifted dramatically as most individuals sought to exit towards another event. In this transition phase, the density within our tracking corridor surged, often exceeding 120 pedestrians moving simultaneously. For video calibration, we meticulously established nine calibration points, ensuring the precise tracking of pedestrian trajectories using the PeTrack software [34]. Throughout our study, we recorded 5195 individual trajectories, which averaged a duration of 12.38 seconds and a mean velocity of 0.62 $m/s$. For the training and testing of the algorithms we need trajectories of a minimal length of 7 seconds (see 4.1). Because many trajectories are more than 14 seconds long, they can be used more than once. All in all we get 7450 trajectories for training and testing.

4. Methodology

4.1. Overview

For predicting pedestrian trajectories we have $i^{th}$ pedestrian in a scene represented by image coordinates$(x_t^i,y_t^i)$ for each time instant $t=k·dt$, with $k\in \mathbb{N}$ und $dt=1/3$ s the time step. The observed positions from $t=T_1$ to $t=T_{obs}$ is taken as input and the aim is to predict future trajectories from $t=T_{obs}+dt$ to $t=T_{pred}$. Every scene involves a primary pedestrian and his neigbours over the timespan $T_1$ to $T_{pred}$. A neigbours is a pedestrian whose position at $T_1$ is closer to the position of the primary pedestrian than a radius R. Our dataset has a framerate of three observation for each second. We choose input trajectories of 9 observations (3 sec.) and want to predict 12 timesteps (4 sec.).

• The predicted trajectory of all primary pedestrians are evaluated on two commonly utilized Euclidean distance metric and a collision metric. In the first distance-based metrics, called average displacement error (ADE) [29], the distance between the predicted trajectory and the ground truth trajectory is measured at any time step t

$$\mathbf{ADE}=\frac{1}{NT}\sum _{i=1}^N\sum _{t=1}^T\parallel {\widehat{x}}_i\left(t\right)-x_i\left(t\right)\parallel .$$

(1)

$x_i\left(t\right)$ is the actual position of the $i^{th}$ pedestrian at time t while ${\widehat{x}}_i\left(t\right)$ is the predicted position. The Euclidean distance is denoted as $\parallel ·\parallel$. The second distance-based metric, called final displacement error (FDE) [30] displays the distance between the final point $t=T_{pred}$ of the predicted trajectory and the ground truth trajectory

$$\mathbf{FDE}=\frac{1}{N}\sum _{i=1}^N\parallel {\widehat{x}}_i\left(T\right)-x_i\left(T\right)\parallel .$$

(2)

These distance-based metrics are widely used in pedestrian trajectory predictions for their effectiveness in quantifying the goodness-of-fit. However, repulsive forces, which are pivotal in shaping interactions between pedestrians, are not taken into account [35]. Consequently, these metrics do not account for potential overlaps or collisions between pedestrians. Therefore the collision metric is used to enhance the evaluating process

$${\displaystyle \mathbf{COL}=\frac{1}{\left|S\right|}\sum _{\widehat{Y}\in S}COL\left(\widehat{Y}\right),}$$

(3)

with

$$COL\left(\widehat{Y}\right)=min\left(1,\sum _{t=1}^T\sum _{i=1}^N\sum _{j>i}^N\left[\left|\right|{\widehat{x}}_i\left(t\right)-{\widehat{x}}_j\left(t\right)\left|\right|\le 2R\right]\right).$$

(4)

S includes all scenes in the test set, $\widehat{Y}$ represents a scene prediction containing N agents, and ${\widehat{y}}_i$ is the prediction of agent i over the prediction time of T, while $[·]$ is the Iverson bracket

$$\left[P\right]=\left\{\begin{array}{cc}1\hfill & \mathrm{if}P\mathrm{is}\mathrm{true},\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(5)

This metric counts a prediction as a collision when a predicted pedestrian trajectory intersects with neighboring trajectories, thus indicating the proportion of predictions where collisions occur. A vital factor in this calculation is the chosen pedestrian size R. An increase in R will likewise increase the number of collisions.

4.2. Prediction approaches

In the subsequent trajectory predictions, various trajectory prediction approaches ranging from traditional PB models to modern DL algorithms are chooses as benchmarks for comparison with our two-stage approach. We present the results of the Constant Velocity model (CV) and SF model [8] as well as the results of a Vanilla LSTM, the Social LSTM (SLSTM) [5] and the Social GAN [14]. The position and velocity of the $i^{th}$ pedestrian are denoted as $x_i\in {\mathbb{R}}^2$ and $v_i\in {\mathbb{R}}^2$, respectively. For a system of N pedestrians, the position and velocity vectors, $x=(x_1,\dots ,x_N)$ and $v=(v_1,\dots ,v_N)$, have dimensions of $2N$. All variables, including $x\left(t\right)$ and $x_i\left(t\right)$, are functions of time t.

4.2.1. Constant Velocity Model

The CV model assumes pedestrian velocities remain unchanged over time. It serves as a baseline for more complex models. The future position of a pedestrian is predicted as:

$$x_i(t+t_p)=x_i\left(t\right)+t_p{v}_i\left(t\right),\forall t_p\in [0,T_p].$$

(6)

4.2.2. Social Force Model

Introduced by Helbing and Molnar [8], the SF model treats pedestrians as particles influenced by forces. Within this framework, the model calculates acceleration based on the cumulative effect of three distinct forces, as delineated in equation 7

$$m_i\frac{d{v}_i}{dt}=m_i\frac{v_i^0-v_i}{\tau }+\sum _{j\ne i}\nabla U(x_j-x_i)+\sum _W\nabla V(x_W-x_i)$$

(7)

Here, $m_i$, $v_i$, and $v_i^0$ signify the mass, current velocity, and desired velocity of pedestrian i. The term $\nabla U(x_j-x_i)$ represents the repulsive force from other pedestrians, while $\nabla V(x_W-x_i)$ indicates the repulsive force from obstacles. The potential functions $U\left(d\right)$ and $V\left(d\right)$ are given by:

$$U\left(d\right)=AB{e}^{-\left|d\right|/B},A,B>0\mathrm{and}V\left(d\right)=A^{\prime }{B}^{\prime }{e}^{-\left|d\right|/B^{\prime }},A^{\prime },B^{\prime }>0$$

(8)

The first term of Equation 7 signifies the driving force experienced by the $i^{th}$ pedestrian. This force propels the individual towards their desired speed and direction within a relaxation time $\tau >0$. The second term encapsulates the summation of social forces, originating from the repulsive effects as pedestrians endeavour to maintain a comfortable distance from one another. The third term accounts for the aggregate interaction forces between pedestrian i and various obstacles.

• Whereas the CV model has no parameter, the SF model has three parameters, preferred velocity, interaction potential, and reaction time, that can be optimized to get accurate predictions.

4.2.3. Vanilla LSTM

LSTM networks, a class of RNN designed to learn long-term dependencies, have proven effective in handling sequential data, particularly for time series prediction tasks. Introduced by Hochreiter and Schmidhuber [36], LSTMs address the vanishing and exploding gradient problems common in traditional RNNs, making them suitable for complex sequence modeling tasks such as trajectory prediction. The vanilla LSTM model considers historical trajectories to predict future positions.

$$x_i(t+t_p)=x_i\left(t\right)+\mathrm{LSTM}\left(t_p,\left(x_i(t-t_o),t_o\in [0,T_o]\right)\right),\forall t_p\in [0,T_p].$$

(9)

Here, $x_i(t+t_p)$ predicts the future trajectory of a pedestrian i at time $t+t_p$, based on its past positions $x_i(t-t_o)$, over an observation window $t_o\in [0,T_o]$.

4.2.4. Social LSTM

LSTM networks have demonstrated effective performance in sequence learning tasks. One such task, the prediction of pedestrian trajectories, presents the challenges, that the trajectory of a pedestrian can be significantly influenced by the trajectories of surrounding pedestrians. The number of these neighboring influences can fluctuate widely, especially in densely crowded environments [37].

Enhancing the LSTM framework, the SLSTM by Alahi et al. [5] incorporates a social pooling layer, enabling the model to consider the influence of neighboring pedestrians explicitly. This is a key distinction from the Vanilla LSTM, reflecting the model’s capacity to capture social interactions:

$$x_i(t+t_p)=x_i\left(t\right)+\mathrm{SLSTM}\left(i,t_p,\left(x(t-t_o),t_o\in [0,T_o]\right)\right),\forall t_p\in [0,T_p].$$

(10)

In this formulation, the inclusion of the index i and the collective pedestrian state x emphasizes the model’s attention to the surrounding pedestrians’ trajectories, making it adept at handling complex social behaviors in dense scenarios.

4.2.5. Social GAN

Another approach we take into account is the Social GAN (SGAN) introduced by Gupta et al. [14]. This model extends traditional approaches by incorporating GANs to predict future trajectories. GANs, conceptualized by Goodfellow et al. [38], consist of two competing networks: a Generator, which generates data samples, and a Discriminator, which evaluates the authenticity of the samples against real data. SGAN leverages this architecture to generate plausible future trajectories of pedestrians, addressing the complex dynamics of pedestrian movement in crowded spaces. A key feature of the SGAN model is its pooling mechanism, which processes the relative positions of pedestrians to each other. This mechanism is crucial for understanding the social interactions and dependencies among individuals in crowded environments

$$x_i(t+t_p)=x_i\left(t\right)+\mathrm{SGAN}\left(i,t_p,\left(x(t-t_o),t_o\in [0,T_o]\right)\right),\forall t_p\in [0,T_p].$$

(11)

4.3. Two-stage process

The foundation of our innovative classification framework lies in its capacity to predict trajectories across varying density levels, marking a departure from traditional models that typically utilize a single algorithm to process a wide array of scenarios within a dataset. Our strategy entails segmenting the dataset according to the density of each scene, thereby generating distinct subsets. At the inception of our methodology, we establish well-defined criteria for classification. This process is underpinned by two distinct methodologies: a statistical analysis and a review of existing literature. The results of this clustering process are depicted in Figure 2. These figures visually represent each dataset item as a point measurement each second, with the left side of Figure 2 illustrating points based on their average density

$$\rho \left(t\right)=\frac{N\left(t\right)}{A},$$

(12)

and average velocity

$$\overline{v}\left(t\right)=\frac{1}{N\left(t\right)}\sum _{i\in \mathcal{S}\left(t\right)}{v}_i\left(t\right),$$

(13)

and the right side based on their average density and flow, the flow being the product of the density by the average velocity

$$J\left(t\right)=\rho \left(t\right)\overline{v}\left(t\right).$$

(14)

In these illustrations, the use of varied colors signifies the identification of distinct clusters. Additionally, to corroborate these findings, a hierarchical cluster analysis was undertaken, which yielded analogous outcomes.

We can see clear vertical colour switch’s of the points at densities around 0.7, 1.1, and 1.6 $ped/m^2$. Remarkably, without presetting the number of clusters or explicitly focusing on density levels, the K-Means algorithm autonomously reveals density-dependent clustering. The delineation of clusters and their boundary values align closely with those identified in the literature. Stefan Holl [39] delineates critical density thresholds for various infrastructures, signifying points at which pedestrian behavior undergoes significant changes. According to Holl, densities below 0.7 $ped/m^2$ indicate a free flow state, densities below 1.3 $ped/m^2$ represent a bound flow, and values above 1.3 $ped/m^2$ are indicative of congested flow. In our model, we refine these categories by slightly narrowing the bound flow range and subdividing the congested flow category into two distinct segments.

Figure 12 illustrates the procedural steps undertaken to evaluate our proposed methodology and within the sizes of the clusters, which are taken from Figure 2.

Figure 3. Schemata of our two-stage prediction approach

Figure 3. Schemata of our two-stage prediction approach

New trajectory scene are given to our framework, where the initial step involves calculating the scene’s density using (12) in individuals per square meter ($ped/m^2$), N represents the total number of pedestrians observed within the scene, and A is the scene’s total area in square meters.

The density categorization is as follows: scenes with a density below 0.7 $ped/m^2$ are labelled as lowD; densities ranging from 0.7 to 1.2 $ped/m^2$ are classified as mediumD; densities between 1.2 and 1.6 $ped/m^2$ are designated as highD; and densities exceeding 1.6 $ped/m^2$ are identified as veryHD. Following classification, the scene is bifurcated into two segments: the initial segment spans 9 timesteps and serves as input for one of the four specialized Sub-LSTMs, while the subsequent segment, encompassing 12 timesteps, is utilized to appraise the LSTMs’ performance through the computation of error metrics ADE, FDE and COL.

• Figure 4a to 4d provide illustrative examples of each density level encountered in our dataset.

In Figure 4a, the scene exhibits very low density, with pedestrian movement primarily from two directions, leading to numerous interactions and avoidance behaviors. This is characteristic of our lowD data. Conversely, Figure 4b showcases a moderately higher density, yet still affords space for interactions, avoidance, and bidirectional pedestrian flow. In Figure 4c, representing highD data, the dynamics of pedestrian movement markedly differ from those observed in lowD and mediumD scenes, with movement predominantly unidirectional from the top, indicating a tendency to follow the pedestrian ahead. This pattern is even more pronounced in Figure 4d, where the flow from the top is so dense that passage from the bottom becomes challenging, leading pedestrians to follow the leader with limited freedom of movement and space. These observed behavioral differences underpin our classification rationale.

4.4. Collision weight

Predictions of pedestrian trajectories presents the challenge to predict trajectory paths that do not collide with neighbours. Accurately measuring these collisions is challenging due to the shapes of pedestrians, which can vary from person to person. Traditionally, collisions are defined by the overlap of the radii of two pedestrians, as delineated in Equations 3 to 5. However, this method proves sub optimal for inclusion as a penalty function within the loss function of DL algorithms. Analysis of pedestrian trajectory data frequently reveals instances where collisions are not genuine but rather instances of grouping behavior, with individuals walking closely, sometimes shoulder-to-shoulder. It is not these interactions we aim to deter, but rather scenarios in which individuals move directly towards one another without any attempt to avoid collision—behaviors that are unrealistic and undesirable.

• To address this, we adopt the TTC concept, a widely recognized principle in the study of pedestrian dynamics [7]. Implementing this variable in the loss function of an DL algorithm would reduce predicted situations, where pedestrians walk straight towards each other without avoidance mechanism. Integrating TTC into a DL algorithm’s loss function significantly mitigates predictions where pedestrians are on a direct collision course without any avoidance mechanisms. The TTC term calculates the time until two pedestrians would collide if they continue moving at their current velocities, a concept validated by Karamouzas et al. [7]. The relative position and velocity between the pedestrian i and j can be denoted by $p_{ij}=(x_i-x_j,y_i-y_j)$ and $v_{ij}=(v_{x_i}-v_{x_j},v_{y_i}-v_{y_j})$, respectively. A collision between pedestrian i and pedestrian j occurs if a ray, originating from $(x_i,y_i)$ and extending in the direction of $v_{ij}$, intersects the circle centred at $(x_j,y_j)$ with a radius of $R_i+R_j$ at some time ${\tau }_{ij}$ in the future. This condition can be mathematically represented as $\left|\right|p_{ij}+v_{ij}.t{\left|\right|}^2<{(R_i+R_j)}^2$ where $\left|\right|.\left|\right|$ denotes Euclidean norm. Solving this quadratic inequality for t yields ${\tau }_{ij}$ as the smallest positive root:

$${\tau }_{ij}=\frac{-p_{ij}·v_{ij}-\sqrt{{(p_{ij}·v_{ij})}^2-\left|\right|v_{ij}{\left|\right|}^2(||p_{ij}{\left|\right|}^2-{(r_i+r_j)}^2)}}{\left|\right|v_{ij}{\left|\right|}^2}$$

(15)

A collision is imminent when ${\tau }_{ij}=0$, whereas a large positive value for ${\tau }_{ij}$ indicates no collision risk. To implement ${\tau }_{ij}$ into the loss function we have to use an sigmoid function f that has high values, if ${\tau }_{ij}$ is low and vice versa:

$$f\left(\tau \right)=\frac{1}{1+e^{s(\tau -\delta )}},$$

(16)

where $s=10$ and $\delta =0.4$ are slope and threshold parameters, respectively. This function is then integrated into the loss function, traditionally focused solely on minimizing the ADE. The revised loss function combines ADE with TTC loss, optimized through minimization:

$${\mathbf{L}}_i=\sum _{t=1}^T\parallel x_i\left(t\right)-{\widehat{x}}_i\left(t\right)\parallel +\lambda \sum _{t=1}^{T}f(\underset{j\ne i}{min}\{{\tau }_{ij}\}),$$

(17)

where $\lambda >0$ modulates the influence of the TTC component in their loss function. The calculation of ${\tau }_{ij}$ considers all nearby pedestrians to the primary pedestrian, employing the minimum ${\tau }_{ij}$ to identify and mitigate the most critical potential collision scenario in the model.

4.5. Implementation details

The algorithms are implemented in the commonly accepted configurations of related contributions [23]. All computations are performed using the PyTorch framework. The learning rate is set to 0.001 and an ADAM optimizer is utilizied. The batch size is set to 8 and training is carried out for 15 epochs, if not the early stop mechanism interrupt. This is the case, when the validations error starts to rise for three epochs. For validation and testing, a hold-out validation strategy is adopted by allocating 15% of the dataset for each validation and testing, while the remaining data serves as the training set. For capturing pedestrian interactions, we choose a circles with a radius of $R=4.5$ m surrounding the primary pedestrian.

5. Results

We will unveil the predictive outcomes of our dataset using two distinct yet synergistic methods. Initially, we will showcase the performance of our two-stage prediction framework, comparing it with contemporary state-of-the-art methodologies. Subsequently, we will demonstrate the seamless integration of our two-stage process with the incorporation of the TTC term into the loss function, illustrating its efficacy in mitigating collision instances.

5.1. Two-Stage Predictions

The results of our predictions will be presented in Table 1. As described in Section 4.3 we evaluated the predictions on the different density levels lowD, mediumD, highD, and veryHD. For every approach we measure ADE, FDE and COL metrics.

The initial insight gleaned from Table 1 reveals a clear trend: as density increases, the COL metric rises while the ADE/FDE diminish. This pattern emerges because higher densities naturally lead to reduced distances between individuals, consequently resulting in increased overlaps among agents. Additionally, it’s observed that velocities decrease as density intensifies, leading to trajectories that are shorter in spatial extent. This reduction in travel distance directly contributes to the observed decrease in both ADE and FDE metrics at higher densities. Furthermore, it is clear that the DB algorithm outperform the traditional models CV and SF in terms of ADE/FDE. In terms of COL metric SF performs very well.

• In the last three rows of Table 1, we present the effectiveness of our two-stage approach and its combination with the TTC term. The results clearly show a significant improvement in the algorithm’s precision, attributed to the strategy of classification before prediction. Our enhanced two-stage SLSTM model consistently outperforms the traditional SLSTM across all evaluated datasets, demonstrating superior performance in terms of ADE, FDE, and COL metrics. Similarly, our adapted SGAN model shows marked improvements over the standard SGAN in three out of four datasets with respect to ADE. Integrating the TTC term further enhances the SLSTM results, notably in reducing collisions. A more detailed discussion on this enhancement is provided in the subsequent Section 5.2.

5.2. Collision weight

In this study, we propose to integrate TTC in the loss function with the two-stage approach outlined in Section 4.3. As described in Equation 17 the collision part in the loss function can be adjusted by a parameter $\lambda$ [35]. If $\lambda$ is high, the impact of the TTC term is high compared to the impact of ADE and vice versa. In the following diagram, the impact of different values of $\lambda$ on the prediction accuracy of the SLSTM algorithm is displayed. In Figure 5a for lowD data in Figure 5b for mediumD data, in Figure 5c for highD data, and in Figure 5d for veryHD data. The first observation in each Figures is for $\lambda =0$, which means, that it is equivalent to the value of our two-stage SLSTM in Table 1.

Across all figures, a consistent pattern emerges: increasing the collision weight $\lambda$ generally results in fewer collisions. Optimal predictions occur at a specific $\lambda$ value, where ADE and FDE are equivalent to or lower than those at $\lambda =0$. Each dataset exhibits a maximum effective $\lambda$ value beyond which ADE sharply increases. In the lowD dataset (Figure 5a), the ideal $\lambda$ is 0.08, reducing ADE by 19% and collisions by 6%. Values slightly higher than 0.08 are still beneficial, yielding fewer collisions and enhanced avoidance behavior, but $\lambda$ values exceeding 0.16 lead to a significant increase in ADE. In the mediumD dataset (Figure 5b), the optimal $\lambda$ is 0.04, reducing ADE and collisions by 3% and 40%, respectively. Here, a $\lambda$ value above 0.06 results in increased ADE, although a $\lambda$ of 0.1 reduces the collision metric by 75%. In the highD dataset, improvements in ADE are marginal too, only notable at a $\lambda$ of 0.02. However, the collision metric significantly decreases, by up to 48%, at a $\lambda$ of 0.08. Conversely, in the very high-density (veryHD) dataset, increasing collision weight initially results in a rise in ADE, with no subsequent decrease. While the collision metric decreases by 37% at $\lambda =0.08$ values, there’s no improvements for ADE. These empirical observations lead to the insight, that pedestrian behavior at different densities is very different and need different parameter configurations. At lower densities our TTC term can improve overall accuracy (ADE and COL) in higher densities we can only reduce COL, by taking higher ADE into account.

6. Conclusions

Pedestrian behavior is inherently complex, exhibiting a wide variety of patterns across different contexts. This paper introduced a novel pedestrian trajectory dataset, characterized by its diversity in situational contexts, including varying densities and motivations. Our analysis of the data reveals variations in pedestrian behaviors correlating with the density of the scene. To address these variations, we propose a novel two-stage classification and prediction process. This approach first classifies scenes based on density and then applies the suitable model for predicting behavior within that specific density context. Implementing this framework enhanced the prediction accuracy of two famous DL algorithms, Social LSTM and Social GAN.

• Further, we integrated a TTC based term into the loss function of the SLSTM to improve avoidance behaviors, consequently reducing potential collisions. Our empirical studies indicate that the effectiveness of the TTC-based term varies with density; it significantly benefits scenarios of low density by correlating higher TTC values with reduced collision incidents. However, the outcomes in high-density situations were more ambiguous, suggesting a nuanced impact of density on the efficacy of this approach. This observation could be attributed to the nuanced dynamics of pedestrian behavior across different densities. Specifically, in environments with lower densities, pedestrians tend to navigate more through avoidance and interactions, making TTC particularly relevant. Conversely, in higher density settings, pedestrian movement is more characterized by forced leader-follower dynamics, diminishing the prominence of TTC in explaining behavior. This study underscores the complexity of pedestrian behavior, which varies significantly under different environmental conditions. It highlights the necessity of adopting a flexible modeling approach to accurately predict pedestrian trajectories in diverse settings.

This research opens several avenues for future investigation in the field of pedestrian trajectory prediction, especially concerning heterogeneous datasets characterized by variable densities. Current methodologies typically rely on a one-size-fits-all model for behavior prediction across all conditions. We advocate for the development and application of multiple specialized models, each tailored to different scene characteristics, with scene density being a pivotal factor. While focusing on density has proven to be a successful strategy, exploring additional factors could yield further improvements. For instance, our methodology utilized an estimate of overall scene density. However, pedestrians do not take global densities for there decision-making into account, but rather local densities. Wirth et al. [40] demonstrate that pedestrian decisions are primarily influenced by their visual neighborhood. Future studies should investigate the impact of assessing local density variations within a scene, which could be particularly beneficial in environments exhibiting a wide range of density levels. This direction could unlock new dimensions of accuracy and reliability in trajectory prediction models.

Moreover, incorporating the TTC concept into the loss function has shown promise in enhancing prediction accuracy at lower density levels. Future research should explore alternative loss functions, particularly for high-density scenarios, where the traditional ADE based approaches may not suffice. Investigating other metrics that could more effectively capture the complexities of high-density pedestrian behavior is crucial for advancing the field.