HTIM: A Hybrid Deep Learning Approach For Pedestrian Trajectory Imputation

1. Introduction

Pedestrian trajectory unpredictability has presented a formidable challenge for autonomous vehicles, necessitating sophisticated trajectory imputation techniques. The erratic and dynamic nature of pedestrian movement makes it challenging to accurately predict their future paths. Factors such as sudden changes in direction, varying speeds, and unexpected interactions with the environment contribute to this unpredictability. Bearing this unpredictability in mind, autonomous vehicles have transformed transportation by utilizing state-of-the-art technologies like artificial intelligence, sensor fusion, and machine learning to navigate safely and independently. Central to their operation is the ability to efficiently plan and follow trajectories while adapting to changing environments. In this context, a trajectory refers to the path a entity takes over time, including its spatial coordinates, velocity, and orientation. Mathematically, a trajectory $X\left(t\right)=\left(x\right(t),y(t),v(t),\theta (t\left)\right)$ describes the vehicle’s state in a specific coordinate system where $\left(x\right(t),y(t\left)\right)$ denotes its position, $v\left(t\right)$ represents its velocity, and $\theta \left(t\right)$ indicates its heading at time t. Trajectory imputation is a critical aspect of motion prediction, focusing on filling in missing points within incomplete trajectories to create a comprehensive representation of an object’s movement. Essentially, trajectory imputation involves using existing trajectory data to interpolate or extrapolate missing segments, compensating for factors like sensor errors, occlusions, or data loss. By integrating predictive models with historical trajectory information, trajectory imputation facilitates the reconstruction of a coherent and uninterrupted trajectory, vital for applications such as autonomous navigation and activity recognition. The primary objective of trajectory imputation is to estimate the missing points inside the incomplete trajectory to yield a complete trajectory represnted as following: ${\mathit{T}}_{\mathbf{complete}}=\{(x_1,y_1,t_1),(x_2,y_2,t_2)\dots (x_n,y_n,t_n)\}$ and

Figure 1. Humans are unpredictable, a critical security concern for autonomous vehicles, Image Courtesy: coursera.org.

Figure 1. Humans are unpredictable, a critical security concern for autonomous vehicles, Image Courtesy: coursera.org.

$$\begin{array}{cc}\hfill {\mathit{T}}_{\mathbf{incomplete}}=\{& (x_1,y_1,t_1),(x_2,y_2,t_2)\dots \hfill \\ \hfill & (x_{n-m-1},y_{n-m-1},t_{n-m-1}),\hfill \\ \hfill & (x_{n-m},y_{n-m},t_{n-m}),\hfill \\ \hfill & (x_{n-m+p},y_{n-m+p},t_{n-m+p})\dots \hfill \\ \hfill & (x_n,y_n,t_n)\}\hfill \end{array}$$

where $p-1$ represents the number of missing points to be imputed.

2. Related Work

3. Gap Analysis

4. Dataset Observation and Prepossessing

4.1. Dataset Observation

Various trajectory datasets have existed for road users, but very few have met the needs of autonomous driving research. Examples include BIWI Walking Pedestrians [25], Stanford Drone [26],CITR and DUT [27], highD [28], Interaction [29] and Ko-PER [30]. Most lack sufficient data or public road scenarios. The inD dataset [31] is a new and unique dataset of naturalistic vehicle trajectories recorded at German intersections using a drone. The dataset includes more than 11,500 road users consisting of vehicles, bicyclists and pedestrians at four different locations(Braunschweig, Frankfurt, Lindau, Wurzburg) with varying traffic density and complexity. Table has shown its comparison with other datasets.

The inD dataset surpasses other datasets in capturing naturalistic road user behavior due to its strategic design principles:

• Preserving naturalistic behavior: By avoiding visible sensors resembling traffic surveillance cameras, the inD dataset ensures road users are not impacted by the measurement method.
• Having satisfactory size: The inclusion of trajectories from thousands of road users provides the necessary size and variety for robust data-driven algorithms.
• Differentiating recording locations and times: The inD dataset captures measurements from multiple sites and diverse times of day, including public roads, ensuring coverage of different road layouts, densities and traffic rules for enhanced signifance in automated driving scenarios.
• Detecting and tracking all road user types: Unlike datasets that limit themselves to specific categories, the inD dataset tracks all road users, recognizes the vital importance of capturing interactions across diverse user types.
• Tracking road users with top accuracy: The inD dataset guarantees trajectories with a positioning inaccuracy of less than 0.1 meters, regardless of road user type, ensuring precision in capturing the intricacies of user movement.
• Including infrastructure details: With the precise recording of road layouts and local traffic rules, the inD dataset acknowledges the dependency of road user behavior on these factors, providing comprehensive information within the dataset.

Table 3. Overview of Trajectory Data and Characteristics. 

Table 3. Overview of Trajectory Data and Characteristics. 

Title	Location	# Trajectories	# Locations	Road User Types	Data Frequency	Method
BIWI ETH	University building entry	360	1	pedestrian	2.5 Hz	stat. sensor
Interaction	Urban intersection	18642	4	vehicles	10 Hz	drone/cam.
Ko-PER	Urban intersection	350	1	pedestrian, bicycle, car,truck	25 Hz	stat. sensor
Crowds UCY/ Zara	Campus, urban street	909	3	pedestrian	2.5 Hz	stat. sensor
Stanford Drone	Campus	10240	8	pedestrian, bicycle, car, skateboard, cart, bus	25 Hz	drone
BIWI	Hotel sidewalk, hotel entry	389	1	pedestrian	2.5 Hz	stat. sensor
VRU Trajectory	Urban intersection	3278	1	pedestrian, bicycle	25 Hz	stat. sensor
DUT	Campus	1862	2	pedestrian, vehicles	23.98 Hz	drone
CITR	Designed experiment	340	1	pedestrian, golf-cart	29.97 Hz	drone
inD	Urban intersection	13599	4	pedestrian, bicycle, car, truck, bus	25 Hz	drone

5. Methodology

5.1. LSTM Encoder-GRU Decoder Architecture with Teacher Forcing

The LSTM-GRU (Gated Recurrent Unit) based Encoder-Decoder is a sequence-to-sequence model commonly used in various natural language processing and sequence generation tasks. As the name suggests, it consists of two main components, Encoder and Decoder. We picked LSTM for the Encoder part and GRU for the Decoder after thorough experimentation.

5.1.1. LSTM Encoder

The LSTM Encoder, a crucial component of sequence-to-sequence models, analyzes input sequences to capture their temporal dependencies. For an input sequence $X=\{x_1,x_2...x_T\}$ of length T, the encoder produces a hidden state sequence $H^{\mathrm{enc}}=\{h_1^{\mathrm{e}},h_2^{\mathrm{e}}...h_T^{\mathrm{e}}\}$.

The LSTM encoder updates its hidden states by the following equations:

$$\begin{array}{cc}\hfill i_t& =\sigma ({\mathit{W}}_{\mathit{ii}}{x}_t+{\mathit{b}}_{\mathit{ii}}+{\mathit{W}}_{\mathit{hi}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{hi}})\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill f_t& =\sigma ({\mathit{W}}_{\mathit{if}}{x}_t+{\mathit{b}}_{\mathit{if}}+{\mathit{W}}_{\mathit{hf}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{hf}})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill g_t& =tanh({\mathit{W}}_{\mathit{ig}}{x}_t+{\mathit{b}}_{\mathit{ig}}+{\mathit{W}}_{\mathit{hg}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{hg}})\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill o_t& =\sigma ({\mathit{W}}_{\mathit{io}}{x}_t+{\mathit{b}}_{\mathit{io}}+{\mathit{W}}_{\mathit{ho}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{ho}})\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill c_t& =f_t\odot c_{t-1}+i_t\odot g_t\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill h_t^{\mathrm{e}}& =o_t\odot tanh\left(c_t\right)\hfill \end{array}$$

(7)

where: $\sigma$ denotes the sigmoid function, ⊙ represents element-wise multiplication, $x_t$ stands for the input at time step t, $h_t^{\mathrm{e}}$ represents the hidden state at time step t, $i_t$, $f_t$, $g_t$, and $o_t$ denote the input, forget, cell, and output gates at time step t respectively, and W and b represent the weight and bias matrices for different gates. $c_t$ represents the cell state at time step t.The LSTM’s unusual architecture, comprising input, forget, cell, and output gates has allowed it to successfully capture long-range relationships in sequential data, making it well-suited for jobs requiring memory retention over longer periods. The LSTM encoder is renowned for its ability to alleviate the vanishing gradient problem commonly encountered in traditional RNNs. This is achieved through its sophisticated gating mechanisms, which regulate the flow of information across time steps. The input gate $i_t$ controls the influx of fresh information into the cell state $c_t$, while the forget gate $f_t$ regulates the confinement of former cell state information. The output gate $o_t$ governs the flow of information from the cell state to the hidden state, ensuring that relevant information is propagated forward while irrelevant information is suppressed.

5.1.2. GRU Decoder with Teacher Forcing

The GRU Decoder with Teacher Forcing is responsible for generating an output sequence based on the hidden states obtained from the encoder. Given the hidden state sequence $H^{\mathrm{enc}}$ and an input sequence $Y=\{y_1,y_2\dots y_U\}$ of length U, the decoder computes an output sequence $Y^{\prime }=\{y_1^{\prime },y_2^{\prime }\dots y_U^{\prime }\}$.

The GRU decoder updates its hidden states and produces the output using the following equations, incorporating the concept of teacher forcing:

$$\begin{array}{cc}\hfill h_t^{\mathrm{d}}& =\mathrm{GRU}(y_t,h_{t-1}^{\mathrm{d}})\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill r_t& =\sigma (W_{yr}{y}_t+U_{hr}{h}_{t-1}^{\mathrm{d}}+b_r)\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill z_t& =\sigma (W_{yz}{y}_t+U_{hz}{h}_{t-1}^{\mathrm{d}}+b_z)\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\tilde{h}}_t& =tanh(W_{y\tilde{h}}{y}_t+r_t\odot \left(U_{h\tilde{h}}{h}_{t-1}^{\mathrm{d}}\right)+b_{\tilde{h}})\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill h_t^{\mathrm{d}}& =(1-z_t)\odot h_{t-1}^{\mathrm{d}}+z_t\odot {\tilde{h}}_t\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill y_t^{\prime }& =\mathrm{Regressor}\left(h_t^{\mathrm{d}}\right)\hfill \end{array}$$

(13)

where GRU represents the operation of a Gated Recurrent Unit, $\sigma$ denotes the sigmoid function, ⊙ represents element-wise multiplication, $y_t$ is the input at time step t, $h_t^{\mathrm{d}}$ represents the hidden state at time step t, $r_t$ and $z_t$ denote the reset and update gates respectively, ${\tilde{h}}_t$ is the candidate hidden state, W and U are weight matrices, and b are bias vectors. Regressor is a function that maps the hidden state to the output space.

The GRU Decoder with Teacher Forcing employs the use of teacher forcing, a technique where the decoder receives the true values of the previous time steps as inputs during training instead of its predictions. This technique facilitates learning by providing more accurate guidance to the model during training. The probability of using teacher forcing at each time step can be controlled by a parameter $\alpha \in [0,1]$, where $\alpha =1$ indicates full teacher forcing (always using true values), and $\alpha =0$ indicates no teacher forcing (always using model predictions). The probability of using teacher forcing at time step t can be expressed as:

$$P\left(\mathrm{teacher}\mathrm{forcing}\mathrm{at}t\right)=\alpha$$

Thus, the input $y_t$ to the decoder is determined by whether teacher forcing is applied or not:

$$y_t=\left\{\begin{array}{cc}{y}_{t-1}\hfill & \mathrm{with}\mathrm{probability}\alpha \hfill \\ y_{t-1}^{\prime }\hfill & \mathrm{with}\mathrm{probability}1-\alpha \hfill \end{array}\right.$$

where $y_{t-1}^{\prime }$ represents the predicted output of the decoder at time step $t-1$.

Figure 6. Encoder-Decoder Architecture.

Figure 6. Encoder-Decoder Architecture.

5.1.3. Justification for Using GRU

(1)

Sequential Modeling: Because each point in a trajectory depends on the ones before it, trajectory data is inherently sequential. GRUs excel at sequential modeling as they maintain a hidden state that evolves, allowing them to capture dependencies in sequential data.

(2)

Memory Cells with Gates: GRUs are equipped with memory cells and gating mechanisms, enabling them to reset and update their internal state dynamically. The reset gate allows the model to choose which historical data to disregard, while the update gate determines which fresh data to incorporate. This flexibility mitigates the vanishing gradient issue and facilitates the capture of long-term interdependencies.

(3)

Efficient Training: GRUs are designed for computational efficiency, making them suitable for handling lengthy sequences often encountered in trajectory data. The efficiency of GRUs may contribute to quicker convergence during training.

(4)

Parameter Efficiency: Compared to LSTM networks, GRUs have fewer parameters, enhancing their parameter efficiency. This characteristic is particularly beneficial when working with trajectory data that may have limited samples. The reduced number of parameters helps prevent overfitting and may lead to better generalization for new trajectories.

6. Experimental Results and Discussion

6.5. Graph Representations of Predicted Trajectory and Ground Truth Trajectory

In this subsection, we present a visual contrast between the predicted trajectory and the baseline trajectory. Figure 14 illustrates the trajectories plotted on a graph, with velocity values indicated. The x-axis represents the horizontal position, while the y-axis represents the vertical position. The blue line denotes the ground truth trajectory, while the red line represents the generated trajectory. Additionally, velocity values are annotated along the trajectories, providing insights into the speed variations throughout the movement. Arrows indicating the direction of movement are also included to enhance the visualization. This comparison allows for a qualitative assessment of the model’s performance in predicting trajectories and provides valuable insights into the dynamics of the system under study.

Figure 14. Comparison of generated and ground truth trajectories with velocity values.

Figure 14. Comparison of generated and ground truth trajectories with velocity values.

7. Conclusion

References

1. Kalman, R.E. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering 1960, 82, 35–45, [https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/82/1/35/5518977/35_1.pdf]. [Google Scholar] [CrossRef]
2. Dellaert, F.; Fox, D.; Burgard, W.; Thrun, S. Monte Carlo localization for mobile robots. In Proceedings of the 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C); 1999; Vol. 2, pp. 1322–1328. [Google Scholar] [CrossRef]
3. Cristianini, N.; Ricci, E. Support Vector Machines. In Encyclopedia of Algorithms; Kao, M.Y., Ed.; Springer US: Boston, MA, 2008; pp. 928–932. [Google Scholar] [CrossRef]
4. Goodfellow, I.J.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016; https://www.deeplearningbook.org. [Google Scholar]
5. Graves, A.; Fernández, S.; Schmidhuber, J. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. In Proceedings of the Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005, Berlin, Heidelberg, 2005; Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S., Eds.; pp. 799–804. [Google Scholar]
6. Liao, Y.; Lin, R.; Zhang, R.; Wu, G. Attention-based LSTM (AttLSTM) neural network for Seismic Response Modeling of Bridges. Computers & Structures 2023, 275, 106915. [Google Scholar] [CrossRef]
7. Ullah, M.; Ullah, H.; Khan, S.D.; Cheikh, F.A. Stacked Lstm Network for Human Activity Recognition Using Smartphone Data. In Proceedings of the 2019 8th European Workshop on Visual Information Processing (EUVIP); 2019; pp. 175–180. [Google Scholar] [CrossRef]
8. Shi, X.; Chen, Z.; Wang, H.; Yeung, D.Y.; kin Wong, W.; chun Woo, W. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting, 2015, [arXiv:cs.CV/1506.04214].
9. Lea, C.; Vidal, R.; Reiter, A.; Hager, G.D. Temporal Convolutional Networks: A Unified Approach to Action Segmentation. In Proceedings of the Computer Vision – ECCV 2016 Workshops; Hua, G.; Jégou, H., Eds., Cham; 2016; pp. 47–54. [Google Scholar]
10. Deng, B.; Yan, J.; Lin, D. Peephole: Predicting Network Performance Before Training, 2017, [arXiv:cs.LG/1712.03351].
11. Cao, W.; Wang, D.; Li, J.; Zhou, H.; Li, L.; Li, Y. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems 2018, 31. [Google Scholar]
12. Kreindler, D.; Lumsden, C.J. The effects of the irregular sample and missing data in time series analysis. Nonlinear dynamics, psychology, and life sciences 2006, 10, 187–214. [Google Scholar] [PubMed]
13. Nawaz, A.; Huang, Z.; Wang, S.; Akbar, A.; AlSalman, H.; Gumaei, A. GPS trajectory completion using end-to-end bidirectional convolutional recurrent encoder-decoder architecture with attention mechanism. Sensors 2020, 20, 5143. [Google Scholar] [CrossRef] [PubMed]
14. Zheng, Y. GPS Trajectories with transportation mode labels, 2010.
15. Ma, J.; Yang, C.; Mao, S.; Zhang, J.; Periaswamy, S.C.; Patton, J. Human Trajectory Completion with Transformers. In Proceedings of the ICC 2022-IEEE International Conference on Communications. IEEE; 2022; pp. 3346–3351. [Google Scholar]
16. Sikeridis, D.; Papapanagiotou, I.; Devetsikiotis, M. CRAWDAD unm/blebeacon, 2022. [CrossRef]
17. Fujii, R.; Vongkulbhisal, J.; Hachiuma, R.; Saito, H. A two-block rnn-based trajectory prediction from incomplete trajectory. IEEE Access 2021, 9, 56140–56151. [Google Scholar] [CrossRef]
18. Pellegrini, S.; Ess, A.; Schindler, K.; van Gool, L. You’ll never walk alone: Modeling social behavior for multi-target tracking. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. IEEE; 2009; pp. 261–268. [Google Scholar]
19. Lerner, A.; Chrysanthou, Y.; Lischinski, D. Crowds by example. In Proceedings of the ACM Transactions on Graphics (TOG). ACM, Vol. 26; 2007; pp. 1–10. [Google Scholar]
20. Al-Molegi, A.; Jabreel, M.; Martínez-Ballesté, A. Move, Attend and Predict: An attention-based neural model for people’s movement prediction. Pattern Recognition Letters 2018, 112, 34–40. [Google Scholar] [CrossRef]
21. Yang, D.; Zhang, D.; Yu, Z.; Yu, Z. Fine-grained preference-aware location search leveraging crowdsourced digital footprints from LBSNs. In Proceedings of the Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 2013, pp. 479–488.
22. Wang, Z.; Zhang, S.; Yu, J.J. Reconstruction of Missing Trajectory Data: A Deep Learning Approach. In Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC); 2020; pp. 1–6. [Google Scholar] [CrossRef]
23. Soydaner, D. Attention mechanism in neural networks: where it comes and where it goes. Neural Computing and Applications 2022, 34, 13371–13385. [Google Scholar] [CrossRef]
24. Xue Yang, Jin Chen, Q.G.H.G.W.X. Enhanced Spatial–Temporal Savitzky–Golay Method for Reconstructing High-Quality NDVI Time Series: Reduced Sensitivity to Quality Flags and Improved Computational Efficiency. IEEE Transactions on Geoscience and Remote Sensing 2022.
25. Pellegrini, S.; Ess, A.; Schindler, K.; van Gool, L. You’ll never walk alone: Modeling social behavior for multi-target tracking. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision; 2009; pp. 261–268. [Google Scholar]
26. Robicquet, A.; Sadeghian, A.; Alahi, A.; Savarese, S. Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes. In Proceedings of the Computer Vision – ECCV 2016; Leibe, B.; Matas, J.; Sebe, N.; Welling, M., Eds., Cham; 2016; pp. 549–565. [Google Scholar]
27. Yang, D.; Li, L.; Redmill, K.; Ozguner, U. Top-view Trajectories: A Pedestrian Dataset of Vehicle-Crowd Interaction from Controlled Experiments and Crowded Campus. 06 2019, pp. 899–904. [CrossRef]
28. Krajewski, R.; Bock, J.; Kloeker, L.; Eckstein, L. The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC); 2018; pp. 2118–2125. [Google Scholar] [CrossRef]
29. Zhan, W.; Sun, L.; Wang, D.; Shi, H.; Clausse, A.; Naumann, M.; Kümmerle, J.; Königshof, H.; Stiller, C.; de La Fortelle, A.; et al. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps. CoRR, 1910. [Google Scholar]
30. Strigel, E.; Meissner, D.; Seeliger, F.; Wilking, B.; Dietmayer, K. The Ko-PER intersection laserscanner and video dataset. 2014 17th IEEE International Conference on Intelligent Transportation Systems, ITSC 2014; 2014; pp. 1900–1901. [Google Scholar] [CrossRef]
31. Bock, J.; Krajewski, R.; Moers, T.; Runde, S.; Vater, L.; Eckstein, L. The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections. In Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV); 2020; pp. 1929–1934. [Google Scholar] [CrossRef]
32. https://github.com/adhocmaster/drone-dataset-tools [Accessed on March 07, 2024].
33. Muktadir, G.M.; Ikram, Z.; Whitehead, J. Pedestrian Crossing Dataset Extraction From InD Dataset, 2023. [CrossRef]