Bi-PredRNN: An Enhanced PredRNN++ with Bi-Directional Network for Spatiotemporal Sequence Prediction

1. Introduction

Spatiotemporal data integrates spatial information with temporal progression and forms multidimensional datasets [1,2,3,4]. The interpretation of spatiotemporal data necessitates the efficient utilization of spatial data, which is inherently more intricate and diverse than traditional data while incorporating both spatial and temporal aspects. To effectively analyze and interpret the complexities of spatiotemporal data, predictive analytics [5] is employed to forecast future events by analyzing current and historical data, thereby assisting users in identifying upcoming trends, behaviors, and outcomes [6,7]. In the context of spatiotemporal sequence prediction, which has emerged as a critical challenge in various domains from weather forecasting to video understanding, let $X=\{x_1,x_2,\cdots ,x_T\}$ be a sequence of spatiotemporal data of length T, where $x_t\in {\mathbb{R}}^{h\times w\times c}$ represents a spatial state at current time step t ($1\le t\le T$) with height $h\in \mathbb{N}$, width $w\in \mathbb{N}$, and $c\in \mathbb{N}$ channels. Given the prediction horizon $k\in \mathbb{N}$, the goal is to estimate future frames $\{x_{T+1},x_{T+2},\cdots ,x_{T+k}\}$ based on the observed sequence [8].

Traditional approaches to this problem have evolved through several key developments in deep learning architectures. Recurrent Neural Networks (RNNs) [9] initially showed promise in handling sequential data, but struggled with preserving long-term dependencies due to the vanishing gradient problem. This fundamental limitation occurs during backpropagation through time, where the gradient signal is repeatedly multiplied by weights and activation derivatives less than 1. This repeated multiplication leads to an exponential decay. As a result, the network fails to effectively learn from early timesteps that may contain critical information for future predictions, as the gradient becomes too weak to meaningfully update the corresponding weights [10].

Long Short-Term Memory (LSTM) networks [11] addressed this limitation through their gating mechanisms, enabling better preservation of temporal information. Further advancement came with Spatiotemporal LSTM (ST-LSTM) [12], which introduced dedicated memory cells for capturing both spatial and temporal features simultaneously. However, these architectures still face significant challenges in modeling complex spatiotemporal dependencies. While they successfully preserve high-level features, they often struggle to maintain fine-grained spatial details over long sequences [4]. Additionally, their unidirectional processing nature limits their ability to capture comprehensive temporal context, as they only consider past information when making predictions.

The introduction of PredRNN [13] marked a significant advancement in addressing these limitations. This architecture implements a dual memory mechanism: local memory cells for preserving detailed spatial information, and a global memory flow for capturing high-level temporal dynamics. PredRNN++ [14] further enhanced this approach by introducing gradient highway units to facilitate better gradient flow during training. PredRNN-v2 [15] added memory decoupling and reverse scheduled sampling, improving the model’s ability to learn from longer temporal contexts.

Despite these improvements, PredRNN-based models still exhibit limitations in long-term prediction tasks due to their inherent unidirectional temporal processing. They process information solely from past to future, potentially missing important temporal dependencies that could be captured by considering future context. Recent research in sequence modeling has shown that bi-directional processing [16] can significantly enhance model performance by enabling information flow in both temporal directions.

Building upon these insights, we propose Bi-PredRNN, an enhanced architecture that extends the PredRNN framework with bi-directional processing capabilities. Our model has two key contributions:

• A bi-directional memory mechanism that processes temporal information in both forward and backward directions, enabling more comprehensive temporal context modeling
• A dynamic window size adaptation mechanism that automatically adjusts the temporal receptive field based on motion complexity in the input sequence

Through extensive experiments on multiple datasets, including Moving MNIST, KTH Action, and climate data, we demonstrate that Bi-PredRNN achieves superior prediction accuracy compared to existing approaches. The rest of this paper details our methodology, experimental validation, and analysis of results. Section 2 reviews related work in spatiotemporal sequence prediction, Section 3 describes our proposed architecture, Section 4 presents the experimental setup and results, and Section 5 concludes with discussions and future directions.

2. Related Works

2.1. Overview of Spatiotemporal Sequence Prediction

Spatiotemporal sequence prediction involves predicting future data points by analyzing both spatial and temporal dependencies in a sequence of data. This type of prediction is crucial for applications such as video processing, weather forecasting, and traffic flow prediction. Challenges arise from the need to capture two aspects: spatial correlations that represent relationships between different locations, and temporal dynamics that describe the evolution of these relationships. The spatial aspect of prediction involves understanding relationships between different locations or features in a given space. Yu et al. [17] formalized this as a graph structure where nodes represent spatial locations and edges capture their interactions in their seminal work on traffic forecasting. This graph-based framework has become fundamental in modern approaches to spatiotemporal modeling. For grid-structured data, Shi et al. [4] introduced the ConvLSTM architecture that effectively captures spatiotemporal correlations in regular grids, establishing a foundation for applications like precipitation forecasting. Temporal dependencies capture how patterns evolve over time, often exhibiting both short-term and long-term characteristics, requiring models to capture multiple temporal scales simultaneously.

2.2. Spatiotemporal Convolutional Neural Networks

Spatiotemporal Convolutional Neural Networks (CNN) extend traditional 2D CNNs to capture temporal dynamics in video data. Early architectures focused on direct 3D convolutions for video understanding tasks [18,19], particularly in action recognition and temporal feature extraction [20]. These architectures have evolved into more complex designs that break down spatiotemporal features using factorized convolutions [21,22]. Notably, the probabilistic perspective introduces frameworks where temporal and spatial features are processed through separate 2D and 1D convolutional pathways [23], enhancing both computational efficiency and interpretability of learned representations.

Convolutional LSTM (ConvLSTM) architectures represent a significant advancement, fundamentally redesigning the LSTM cell structure to accommodate convolutional operations [4]. The authors combined graph convolutional layers composed of spatiotemporal convolutional blocks with a convolutional sequence learning layer, utilizing convolutional structure along the time axis to fully leverage spatial information. Recent advances have also focused on memory-efficient implementations and real-time processing capabilities, introducing innovations in sparse convolution techniques [24] and adaptive temporal sampling strategies [25].

2.3. Spatiotemporal Long Short-Term Memory

Long Short-Term Memory (LSTM) networks have evolved beyond their original sequence modeling capabilities to address the complex challenges of spatiotemporal prediction [11]. The integration of spatial features with LSTM’s temporal modeling has led to several innovative architectures specifically designed for spatiotemporal forecasting tasks. ST-LSTM [12] represents a fundamental architecture in this evolution, designed to process both spatial and temporal features simultaneously. By incorporating dedicated processing paths for spatial and temporal information streams, ST-LSTM effectively captured spatiotemporal dependencies for sequence prediction tasks. Several architectures have emerged combining the advantages of both ST-LSTM and ConvLSTM approaches. The Memory In Memory (MIM) network [26] introduced a more sophisticated memory mechanism that can learn higher-order non-stationary transitions in spatiotemporal data, making it particularly effective for complex prediction tasks involving multiple temporal frequencies.

As shown in Figure 1, i represents the input gate, f is the forget gate, c is the cell state, g is the candidate cell state, o is the output gate, and h is the hidden state. In Conv LSTM, the spatiotemporal memory state $M_t$ serves as its feature, while in PredRNN, M represents the spatial memory and C denotes the temporal memory.

2.4. Multi Modal Prediction & Attention Models

The landscape of spatiotemporal prediction has evolved significantly beyond traditional recurrent architectures, incorporating more sophisticated approaches to handle the increasing complexity of real-world data. While RNN-based models process information sequentially, modern architectures leverage attention mechanisms that enable direct modeling of relationships between any elements in the input sequence, regardless of their temporal distance. This shift has been particularly important in handling multi-modal data, where information from different sources must be integrated coherently. Additionally, the emergence of graph-based approaches has provided a natural framework for modeling complex spatial interactions, while transformer architectures have demonstrated remarkable capability in capturing long-range dependencies through their self-attention mechanisms. These architectural innovations have been crucial in addressing key challenges in spatiotemporal prediction, including the need to handle missing data, integrate multiple data modalities, and model complex interaction patterns.

Recent advancements in spatiotemporal prediction have seen promising results in incorporating multi-modal data and handling complex interaction patterns. Junan Chen et al. [27] proposed a spatiotemporal graph neural network that explicitly models vehicle interactions for trajectory prediction. Their approach combines attention mechanisms with graph neural networks to capture both spatial relationships and temporal dependencies between vehicles in complex traffic scenarios. The integration of self-supervised learning techniques has also shown promise in improving spatiotemporal forecasting. Weyn et al. [28] introduced a novel multi-modality framework that leverages self-supervised learning to capture correlations across different data modalities. Their approach addresses the challenge of learning from multiple data sources while maintaining computational efficiency, particularly valuable for applications involving diverse sensor inputs. Furthermore, environmental monitoring applications have driven innovations in handling missing data in spatiotemporal sequences. Kumar et al. [29] developed SERT, a transformer-based architecture specifically designed for environmental sensor data with missing values. Their model combines spatial and temporal attention mechanisms with specialized embedding techniques to handle irregular sampling and missing observations, demonstrating robust performance in real-world environmental monitoring scenarios.

2.5. Limitations of Current Approaches

Recent advances in spatiotemporal prediction models have demonstrated significant progress in capturing complex temporal dependencies and spatial relationships. However, several fundamental limitations persist in current approaches and reduce their effectiveness in real-world applications. This section analyzes these limitations and presents our research position addressing these challenges.

A primary challenge in RNN-based architectures is the persistent issue of vanishing gradients during backpropagation through time (BPTT) [30]. While various techniques such as gradient clipping and careful initialization have been proposed [31], these solutions provide only partial remedies and often fail in scenarios requiring long-term dependency modeling [10]. This limitation becomes particularly apparent in applications requiring extensive temporal context, such as long-term weather forecasting or traffic prediction.

The integration of spatial and temporal features presents another significant challenge. Traditional Fully Connected LSTM architectures, by treating spatial information as flat vectors, inherently lose crucial spatial relationships that are essential for accurate prediction [4]. Although ConvLSTM architectures have attempted to address this by introducing spatial operations, they still struggle with maintaining long-range spatial dependencies effectively [20]. The fundamental trade-off between preserving spatial features and maintaining temporal memory remains suboptimal, particularly in scenarios involving complex spatial transformations [14].

Data efficiency and scalability concerns significantly impact the practical deployment of current models. Transformer-based approaches, despite their powerful representation capabilities, exhibit considerable data hunger due to their limited inductive biases [32]. The quadratic scaling of self-attention mechanisms with sequence length imposes practical challenges on their application to long sequences [33]. Furthermore, the extensive pre-training requirements make these models particularly resource-intensive for specialized domains. This makes them inaccessible to researchers with limited computational resources [34].

Model adaptability represents another critical limitation. Current approaches, such as 3D CNN with DropPath (also known as Drop Connect, is a regularization technique used in deep neural networks where randomly selected paths or connections between layers are temporarily set to zero during training.), typically require complete retraining for new datasets [35] This lack of transfer learning capabilities not only increases computational overhead but also limits the models’ practical utility in real-world scenarios where data distributions may shift over time [36]. The challenge of domain adaptation becomes particularly acute in specialized architectures, where the model’s performance can degrade significantly when applied to slightly different domains.

3. Method

In this section, we present a comprehensive description of our proposed bi-directional Predictive Recurrent Neural Network (Bi-PredRNN). Our methodology introduces two principal contributions to the field of spatiotemporal prediction. First, we propose a novel bi-directional memory network architecture that systematically integrates both antecedent and subsequent temporal information in the feature representation process. This approach enables more robust modeling of long-term dependencies through detailed temporal context utilization. Second, we demonstrate the model’s versatility across diverse data representations, successfully applying it to both regular grid-structured data (e.g., 2D dynamics) and complex irregular spatiotemporal patterns (e.g., climate datasets). This cross-domain applicability validates the generalization of our approach across varying data representations and temporal scales.

3.1. Bi-Directional Memory

Our model integrates bi-directional LSTM processing into long-term memory learning to overcome the limited temporal context modeling of unidirectional approaches. The introduction of bi-directional processing represents a significant paradigm shift in spatiotemporal prediction for several key reasons [16,37,38]. By solely leveraging past information, traditional unidirectional models create an inherent information bottleneck that limits their predictive capabilities. As illustrated in Figure 2, our bi-directional approach enables more comprehensive context utilization through dual paths of information flow, significantly mitigating the vanishing gradient problem through multiple gradient propagation routes.

The dual memory approach in Bi-PredRNN creates a more robust feature representation mechanism by capturing both forward and backward temporal dependencies simultaneously. The forward path maintains causal relationships essential for prediction, while the backward path incorporates future context that enables more accurate long-term forecasting [13,14]. This comprehensive approach to temporal modeling allows the network to leverage a more complete set of spatiotemporal relationships.

Our model implements several key contributions to enhance its effectiveness. First, the dual memory integration mechanism maintains spatial coherence through specialized memory cells while enabling efficient information flow between the forward and backward paths. Second, the enhanced feature capture mechanism preserves spatial relationships through structured memory organization, ensuring that critical spatial information is not lost during temporal processing. Finally, our adaptive processing framework dynamically adjusts feature importance through learned attention mechanisms, balancing computational efficiency with model expressiveness.

3.2. Theoretical Framework of Bi-Directional Memory Flow

Let $X=\{x_1,x_2,\cdots ,x_T\}$ be a spatiotemporal sequence, where $x_t\in {\mathbb{R}}^{H\times W\times C}$ represents a spatial state at time $t\in \{1,\cdots ,T\}$ with height H, width W, and channels C. The bi-directional memory flow operates by maintaining two complementary hidden states: a forward hidden state $\overrightarrow{h_t}$ that processes the sequence from $t=1$ to T, and a backward hidden state $\overleftarrow{h_t}$ that processes from $t=T$ to 1. As illustrated in Figure 2, these bi-directional paths are implemented through cascaded memory cells with convolution operations and layer normalization.

The core principle of this architecture is to integrate both causal and predictive patterns through bi-directional processing. At each time-step t, the complete representation $h_t$ is derived from these bi-directional states:

$$\begin{array}{cc}\hfill \overrightarrow{h_t}& =\psi (\overrightarrow{h_{t-1}},x_t,{\theta }_{\overrightarrow{f}})\hfill \\ \hfill \overleftarrow{h_t}& =\psi (\overleftarrow{h_{t+1}},x_t,{\theta }_{\overleftarrow{b}})\hfill \\ \hfill h_t& =\varphi (\overrightarrow{h_t},\overleftarrow{h_t}),\hfill \end{array}$$

where $\psi$ denotes the memory update function, ${\theta }_{\overrightarrow{f}}$ and ${\theta }_{\overleftarrow{b}}$ are the learnable parameters for forward and backward processes respectively, and $\varphi$ represents the fusion function that integrates the bi-directional information. The memory update function incorporates both spatial and temporal dependencies through a CausalLSTM [14] architecture:

$$\begin{array}{cc}\hfill i_t& =\sigma (W_{xi}{x}_t+W_{hi}\overrightarrow{h_{t-1}}+b_i)\hfill \\ \hfill f_t& =\sigma (W_{xf}{x}_t+W_{hf}\overrightarrow{h_{t-1}}+b_f)\hfill \\ \hfill c_t& =f_t\odot c_{t-1}+i_t\odot tanh(W_{xc}{x}_t+W_{hc}\overrightarrow{h_{t-1}}+b_c)\hfill \\ \hfill o_t& =\sigma (W_{xo}{x}_t+W_{ho}\overrightarrow{h_{t-1}}+b_o)\hfill \\ \hfill \overrightarrow{h_t}& =o_t\odot tanh\left(c_t\right),\hfill \end{array}$$

where $\sigma \left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$ is the sigmoid function, ⊙ denotes element-wise multiplication, and W and b are learnable parameters.

3.3. Training Strategy and Loss Functions

The training of Bi-PredRNN employs a carefully designed loss function that addresses multiple aspects of spatiotemporal prediction quality. The total loss is composed of three main components:

$${\mathcal{L}}_{total}={\lambda }_1{\mathcal{L}}_{reconstruction}+{\lambda }_2{\mathcal{L}}_{temporal}+{\lambda }_3{\mathcal{L}}_{consistency}$$

(1)

The first term, reconstruction loss, ensures accurate frame-level predictions by minimizing the mean squared error between predicted and ground truth frames:

$${\mathcal{L}}_{reconstruction}={\displaystyle \frac{1}{T}}\sum {t=1}^T{|x_t-{\widehat{x}}_t|}_2^2,$$

(2)

where $x_t$ and ${\widehat{x}}_t$ denote the ground truth and predicted frames at time step t, respectively. This loss directly measures the pixel-wise accuracy of predictions, encouraging the model to generate visually accurate frames.

The second term introduces temporal coherence by penalizing inconsistencies in frame-to-frame transitions:

$${\mathcal{L}}_{temporal}={\displaystyle \frac{1}{T-1}}\sum {t=1}^{T-1}{|\Delta x_t-\Delta {\widehat{x}}_t|}_2^2,$$

(3)

where $\Delta x_t=x_{t+1}-x_t$ represents frame-to-frame differences. This loss ensures smooth temporal transitions by matching the dynamics between consecutive frames in both predicted and ground truth sequences, reducing temporal artifacts such as jittering or sudden changes.

The third term, bi-directional consistency loss, enforces agreement between forward and backward predictions:

$${\mathcal{L}}_{consistency}={\displaystyle \frac{1}{T}}\sum {t=1}^T{|{\overrightarrow{h}}_t-{\overleftarrow{h}}_t|}_2^2,$$

(4)

where ${\overrightarrow{h}}_t$ and ${\overleftarrow{h}}_t$ represent the hidden states from forward and backward passes respectively. This loss term encourages the model to learn consistent representations regardless of sequence direction, leading to more robust predictions.

The weighting coefficients ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are set to 1.0, 0.5, and 0.2 respectively, based on empirical validation. These weights balance the contribution of each loss component, with higher weight on reconstruction accuracy while maintaining temporal coherence and bi-directional consistency.

3.4. Multi-Layer Implementation

The theoretical framework is implemented through a K-layer architecture where each layer $k\in \{1,...,K\}$ maintains its hidden state $h_t^k$ and cell state $c_t^k$. Although deeper architectures generally enable the capture of more sophisticated features, we adopt the widely-used 4-layer configuration (K=4) that has been empirically validated across various spatiotemporal prediction tasks. As illustrated in Figure 3, the network processes information in both forward and backward directions with cascaded memory flow across time steps.

The network first processes information in the forward direction. For time-step t from 1 to T:

$$\begin{array}{cc}\hfill \mathrm{Layer}\mathrm{k}=1\text{:}& \overrightarrow{h_t^1},\overrightarrow{c_t^1},\overrightarrow{M_t^1}=\mathrm{CausalLSTM}(x_t,\overrightarrow{h_{t-1}^1},\overrightarrow{c_{t-1}^1},\overrightarrow{M_{t-1}^K})\hfill \\ \hfill \mathrm{Layer}\mathrm{k}\text{>}1\text{:}& \overrightarrow{h_t^k},\overrightarrow{c_t^k},\overrightarrow{M_t^k}=\mathrm{CausalLSTM}(\overrightarrow{h_t^{k-1}},\overrightarrow{h_{t-1}^k},\overrightarrow{c_{t-1}^k},\overrightarrow{M_t^{k-1}})\hfill \end{array}$$

Following the forward pass, the network processes information in the backward direction. For time-step t from T to 1:

$$\begin{array}{cc}\hfill \mathrm{Layer}\mathrm{k}=1\text{:}& \overleftarrow{h_t^1},\overleftarrow{c_t^1},\overleftarrow{M_t^1}=\mathrm{CausalLSTM}(x_t,\overleftarrow{h_{t+1}^1},\overleftarrow{c_{t+1}^1},\overleftarrow{M_{t+1}^K})\hfill \\ \hfill \mathrm{Layer}\mathrm{k}\text{>}1\text{:}& \overleftarrow{h_t^k},\overleftarrow{c_t^k},\overleftarrow{M_t^k}=\mathrm{CausalLSTM}(\overleftarrow{h_t^{k-1}},\overleftarrow{h_{t+1}^k},\overleftarrow{c_{t+1}^k},\overleftarrow{M_t^{k-1}})\hfill \end{array}$$

The hidden states from forward and backward passes are first concatenated:

$$\begin{array}{cc}\hfill h_t^{combined}& =[h_t^K;\overleftarrow{h_t^K}]\hfill \end{array}$$

The final prediction for the next time step is then generated through a dimensionality reduction layer:

$$\begin{array}{cc}\hfill x_{t+1}& =\mathrm{Conv}2\mathrm{D}\left(h_t^{combined}\right),\hfill \end{array}$$

where Conv2D is a 1×1 convolutional layer that reduces the channel dimension to match the input dimension. This multi-layer structure enables sophisticated spatiotemporal pattern modeling through hierarchical feature extraction and bi-directional information flow. Note that in both passes, the spatial memory from the highest layer ($\overrightarrow{M_t^4}$ and $\overleftarrow{M_t^4}$) flows to the lowest layer of the next time step ($\overrightarrow{M_{t-1}^4}\to \overrightarrow{M_t^1}$ for forward pass, $\overleftarrow{M_{t+1}^4}\to \overleftarrow{M_t^1}$ for backward pass). This cascade memory flow allows the model to maintain long-term spatial dependencies across time steps while processing information bi-directionally.

3.5. Dynamic Window Size Mechanism

We propose a dynamic window size mechanism that adaptively adjusts the temporal processing window based on the motion between frames. This approach aims to balance computational efficiency with prediction accuracy by varying the temporal context according to the observed motion patterns. The window size configuration is governed by three key parameters: minimum window size ($w_{min}=3$), maximum window size ($w_{max}=11$), and EMA smoothing coefficient ($\alpha =0.8$). The minimum window size ensures basic temporal dependency modeling while maintaining computational efficiency for low-motion sequences. The maximum window size is empirically determined to capture extended temporal dependencies in high-motion sequences while maintaining reasonable computational costs. The EMA smoothing coefficient prevents rapid oscillations in window size, ensuring temporal consistency in predictions.

3.5.1. Optical Flow Based Motion Estimation

The motion estimation is performed using a lightweight convolutional neural network that consists of three layers:

$$\begin{array}{cc}\hfill f_1& =\mathrm{ReLU}\left(\mathrm{Conv}(X_{t,t+1};7\times 7,2\to 64)\right)\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill f_2& =\mathrm{ReLU}\left(\mathrm{Conv}(f_1;5\times 5,64\to 32)\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill f_{\mathrm{flow}}& =\mathrm{Conv}(f_2;3\times 3,32\to 2),\hfill \end{array}$$

(7)

where $X_{t,t+1}\in {\mathbb{R}}^{H\times W\times 2}$ represents the concatenated input of two consecutive frames, and $\mathrm{Conv}(x;k,c_{in}\to c_{out})$ denotes a convolutional operation with kernel size k and channel dimensions $c_{in}\to c_{out}$. The network progressively processes the spatial information while maintaining the ability to capture motion patterns at different scales through its hierarchical structure. The motion magnitude $s_{\mathrm{motion}}$ is computed as the average magnitude of optical flow vectors across all spatial locations:

$$s_{\mathrm{motion}}={\displaystyle \frac{1}{H\times W}}\sum _{i=1}^H\sum _{j=1}^W\sqrt{{\left(f_{\mathrm{flow}}^x(i,j)\right)}^2+{\left(f_{\mathrm{flow}}^y(i,j)\right)}^2},$$

(8)

where $f_{\mathrm{flow}}^x(i,j)$ and $f_{\mathrm{flow}}^y(i,j)$ represent the horizontal and vertical components of the optical flow at spatial position $(i,j)$, respectively, and $H,W$ are the height and width of the feature map.

3.5.2. Adaptive Window Size Selection

The window size is determined through a three-step process:

Base Window Size is:

$$w_{\mathrm{raw}}=w_{\min }+(w_{\max }-w_{\min })·\sigma \left(s_{\mathrm{motion}}\right),$$

(9)

where $\sigma$ is the sigmoid function that maps the motion magnitude to a value between 0 and 1. Temporal Smoothing using Exponential Moving Average (EMA) is

$$w_{\mathrm{smooth}}\left(t\right)=\alpha w_{\mathrm{smooth}}(t-1)+(1-\alpha )w_{\mathrm{raw}}\left(t\right),$$

(10)

where $w_{\mathrm{smooth}}\left(t\right)$ and $w_{\mathrm{smooth}}(t-1)$ represent the smoothed window sizes at the current and previous time steps, respectively. Window Size Parity Adjustment is

$$w_{\mathrm{final}}=\left\{\begin{array}{cc}{w}_{\mathrm{smooth}}\hfill & \mathrm{if}{w}_{\mathrm{smooth}}\mathrm{is}\mathrm{odd}\hfill \\ w_{\mathrm{smooth}}+1\hfill & \mathrm{if}{w}_{\mathrm{smooth}}\mathrm{is}\mathrm{even}\hfill \end{array}\right.$$

(11)

The final window size determines the temporal context for both forward and backward passes:

$$\begin{array}{cc}\hfill t_{\mathrm{start}}& =max(0,t-\lfloor w_{\mathrm{final}}/2\rfloor )\hfill \\ \hfill t_{\mathrm{end}}& =min(T-1,t+\lfloor w_{\mathrm{final}}/2\rfloor ),\hfill \end{array}$$

(12)

where T is the total number of frames in the sequence.

4. Experiments

We evaluated the performance of our proposed method using the following datasets: Moving MNIST, KTH Action dataset, ToSNA (North Atlantic) SST dataset, and the Global Climate dataset. The first two datasets are widely-used benchmarks for sequence prediction tasks, while the latter two provide real-world climate data to validate our model’s applicability to environmental forecasting problems. All models were implemented using PyTorch framework [39] and trained until convergence using the Adam optimizer. We employed an L2 loss (mean squared error, MSE) function for all experiments and tuned the hyperparameters with the learning rate initialized at ${10}^{-3}$.

4.1. Moving MNIST Dataset

For the Moving MNIST dataset [40], we evaluate our model’s predictive performance by generating 10 future frames given 10 input frames. The dataset consists of sequences where digits move and bounce within a 64 × 64 pixel frame. The input data is structured with clips array of shape $(2,10000,2)$ where 2 represents the number of moving digits, 10000 represents the number of data samples, and 2 represents $x,y$ coordinates of each digit, dims array of shape $(1,3)$ where 1 is the batch size and 3 represents spatial dimensions, and input_raw_data array of shape $(200000,1,64,64)$ where 200000 is the total number of frames, 1 represents the channel number (grayscale), and $64\times 64$ represents the height and width of the image in pixels. The raw input data has a mean pixel value of 0.047 and a standard deviation of 0.194, indicating a sparse presence of digits in the frames. We split the dataset into 10,000 training sequences and 3,000 validation sequences, with digits in the validation set drawn from different portions of the MNIST dataset than those used in training. Additionally, we assess the model’s generalization capability by testing the models trained on two-digit sequences against sequences containing three moving digits.

4.2. KTH Dataset

For the KTH dataset [41], we evaluate our model’s predictive performance by generating 10 future frames given 10 input frames. The dataset consists of sequences where human subjects perform six types of actions (walking, jogging, running, boxing, handwaving, and handclapping) in four different scenarios (outdoors, outdoors with scale variation, outdoors with different clothes, and indoors). Each sequence is recorded at 25fps and downsampled to $160\times 120$ pixel frames. The input data is structured with input_raw_data array of shape $(N,1,120,160)$, where N represents the total number of frames across all sequences, 1 represents the grayscale channel, and $120\times 160$ represents the height and width of the image in pixels. The raw input data has been preprocessed to normalize pixel values between 0 and 1, with background pixels typically having values close to 0 and foreground (human subjects) showing higher intensity values.

We organize the dataset into training, validation, and test sets following a person-based splitting strategy, where sequences from 16 subjects are used for training, 4 for validation, and 5 for testing. This ensures that the model learns to generalize across different individuals rather than memorizing specific subjects’ movements. Each action sequence is temporally segmented into multiple subsequences of 20 frames (10 for input, 10 for prediction) with a sliding window approach, resulting in approximately 7,000 training sequences. The validation and test sets contain approximately 1,800 and 2,200 sequences respectively. Additionally, we assess the model’s robustness by evaluating its performance across different scenarios, particularly focusing on its ability to handle variations in clothing, scale, and background conditions.

4.3. Climate Dataset

For the climate datasets, we analyze two distinct but related spatiotemporal datasets derived from the Community Earth System Model (CESM2) pre-industrial control experiment: a global surface air temperature dataset and a North Atlantic sea surface temperature (SST) dataset [42]. These datasets enable the evaluation of our model’s capacity to capture complex climate system dynamics under controlled conditions. A sample of projected worldwide map is described in Figure 4.

The global temperature dataset consists of monthly averaged fields spanning 1200 years, with a spatial resolution of 192 × 288 grid points (latitude × longitude), resulting in a data tensor of shape (14389, 192, 288). The input data has been preprocessed to remove the climatological seasonal cycle, focusing on deviations that represent internal climate system variability. The normalized temperature data shows near-zero mean $4.097\times {10}^{-5}$ with a standard deviation of 0.708, reflecting the successful removal of systematic seasonal variations. The spatial coordinates are represented by latitude arrays (shape: 192, std: 52.233) and longitude arrays (shape: 288, mean: 179.375, std: 103.922), providing complete global coverage.

The North Atlantic SST dataset (ToSNA) focuses on a specific region with dimensions (14389, 70, 125), representing 800 years of monthly data at a nominal 1° horizontal resolution. The data has been carefully curated to maintain scientific validity, with no missing values (NaN count: 0) and a temperature range from -1.80°C to 30.41°C, with a mean of 11.28°C. To isolate internal climate variability, we employ two key preprocessing steps: (1) removing the time-mean SST at each geographical location to eliminate latitudinal variation, and (2) applying a 12-month moving average at each location to remove the seasonal cycle.

We structure our evaluation framework to assess both spatial and temporal prediction capabilities, with a particular focus on the model’s ability to capture internal climate system dynamics when external forcing is held constant. The dataset split maintains temporal coherence while ensuring sufficient data for both training and validation. Additionally, we evaluate the model’s ability to capture various known climate patterns and oscillations, providing a robust test of its capacity to learn complex Earth system dynamics.

4.4. Evaluation Metrics

We utilize three primary metrics—MSE, PSNR, and SSIM—to evaluate the performance of the models. Each metric provides a unique perspective on assessing the similarity between the predicted and ground truth frames.

MSE (Mean Squared Error) measures the average squared difference between the predicted and ground truth frames. A lower MSE indicates that the predicted frame is closer to the actual frame. MSE is defined as follows:

$$\mathrm{MSE}={\displaystyle \frac{1}{mn}}\sum _{i=1}^m\sum _{j=1}^n{\left(I_{ij}-{\widehat{I}}_{ij}\right)}^2,$$

where $I_{ij}$ represents the pixel value of the ground truth frame, ${\widehat{I}}_{ij}$ is the pixel value of the predicted frame, and m and n denote the height and width of the frame, respectively.

PSNR (Peak Signal-to-Noise Ratio) is derived from MSE and represents the ratio between the maximum possible power of a signal and the power of noise, in a logarithmic scale. A higher PSNR indicates that the predicted frame has a higher similarity to the ground truth frame. PSNR is defined as:

$$\mathrm{PSNR}=10·{log}_{10}\left({\displaystyle \frac{MA{X}^2}{\mathrm{MSE}}}\right),$$

where $MAX$ is the maximum possible pixel value of the image (e.g., $MAX=255$ for 8-bit images). PSNR complements MSE by emphasizing the perceptual quality of the image.

SSIM (Structural Similarity Index Measure [43]) measures the structural similarity between the predicted and ground truth frames by considering luminance, contrast, and structural information. SSIM ranges from 0 to 1, where a value closer to 1 indicates higher structural similarity. SSIM is calculated as follows:

$$\mathrm{SSIM}(I,\widehat{I})={\displaystyle \frac{(2{\mu }_I{\mu }_{\widehat{I}}+C_1)(2{\sigma }_{I\widehat{I}}+C_2)}{({\mu }_I^2+{\mu }_{\widehat{I}}^2+C_1)({\sigma }_I^2+{\sigma }_{\widehat{I}}^2+C_2)}},$$

where ${\mu }_I$ and ${\mu }_{\widehat{I}}$ represent the mean pixel values of the ground truth frame I and the predicted frame $\widehat{I}$, ${\sigma }_I^2$ and ${\sigma }_{\widehat{I}}^2$ denote the variances of the ground truth and predicted frames, respectively, ${\sigma }_{I\widehat{I}}$ indicates the covariance between I and $\widehat{I}$, and $C_1$ and $C_2$ are small constants used to stabilize the division.

SSIM is particularly effective for assessing image structural similarity, often correlating better with human perceptual quality than PSNR or MSE alone. Each of these metrics provides a different lens through which the quality of the predicted frames can be analyzed and helps to comprehensively assess how well the proposed model approximates the ground truth frames.

5. Results

We evaluated our bi-directional approach with dynamic window size adaptation using SSIM and MSE. Table 1 presents a comparative analysis of our model against state-of-the-art approaches. SSIM provides a comprehensive assessment of structural similarity between predicted and ground truth frames, ranging from -1 to 1, with higher scores indicating better prediction quality. Table 1 presents a comparative analysis of our model against state-of-the-art approaches. Notably, we include the baseline PredRNN model (Wang et al., 2018 [13]) to demonstrate the impact of our bi-directional memory flow and dynamic temporal window mechanism.

Our experimental results show that the adaptive window size mechanism, which ranges from $w_{min}=3$ to $w_{max}=11$ frames, effectively captures both short-term and long-term dependencies in the sequence. The motion-adaptive processing window, governed by our optical flow estimator and EMA smoothing ($\alpha =0.8$), allows the model to allocate computational resources more efficiently based on the temporal dynamics of the input sequence. For the 10-frame prediction task, our model achieves comparable performance to existing approaches, particularly in sequences with varying motion intensities. This suggests that our bi-directional architecture can effectively leverage both past and future context through its dynamic window adaptation.

We extended the prediction horizon from 10 to 30 frames to evaluate the temporal prediction limits of our model. While the model maintains competitive performance in terms of SSIM and MSE metrics for longer sequences, we observe that the prediction quality gradually decreases, particularly in regions with high motion complexity where the dynamic window size approaches its upper limit ($w_{max}=11$). This degradation can be attributed to the accumulation of uncertainty in long-term predictions and the inherent limitations of the temporal window size in capturing extended dependencies. Based on these observations, we focus our subsequent analysis on the 10-frame prediction task, where our model demonstrates more stable and reliable performance.

In Figure 5, we present examples of predicted frames that demonstrate the superior performance of our Bi-directional PredRNN model with dynamic window size adaptation. The quantitative results show that our model achieves better performance across all metrics on the Moving MNIST dataset (MSE: 47.60, PSNR: 31.34, SSIM: 0.90) compared to existing approaches, including PredRNN++ (MSE: 50.70, PSNR: 31.08, SSIM: 0.89).

When evaluating on the KTH action dataset, we observe an interesting pattern in our model’s performance. The basic Bi-directional PredRNN shows slightly degraded performance (SSIM: 0.75) compared to PredRNN++ (SSIM: 0.87), which can be attributed to the increased complexity of human motion patterns and the potential over-fitting of the bi-directional architecture to local temporal dependencies. However, with the introduction of our dynamic window size adaptation mechanism, the model significantly improves its performance (MSE: 91.28, PSNR: 29.25, SSIM: 0.88), surpassing PredRNN++ in PSNR metrics and achieving comparable SSIM scores.

This performance improvement can be attributed to the synergistic effect of bi-directional processing and dynamic window size adaptation: while the bi-directional architecture provides comprehensive temporal context understanding, the dynamic window size mechanism prevents over-reliance on either local or global temporal features, resulting in more balanced and accurate predictions.

For the ToSNA dataset, our model shows steady improvement over baseline approaches, with the dynamic window size adaptation mechanism further enhancing prediction accuracy (MSE: 48.25, PSNR: 27.29, SSIM: 0.84) compared to the conventional ConvLSTM (MSE: 60.30, PSNR: 26.32, SSIM: 0.78). Regarding the climate dataset, while our model shows promising results (MSE: 37.25, PSNR: 28.60, SSIM: 0.82) and outperforms existing approaches, it’s important to note some methodological considerations. Unlike traditional validation approaches, the climate dataset was processed sequentially due to its temporal nature, with later time periods serving as test data. This temporal split, while practical for climate data analysis, may introduce potential risks of overfitting since the validation set wasn’t randomly sampled from the entire dataset. The strong temporal correlations inherent in climate data, combined with this sequential validation approach, suggest that the reported performance metrics should be interpreted with some caution. Future work might benefit from exploring alternative validation strategies or incorporating additional regularization techniques specifically designed for temporally correlated data.

This performance improvement across datasets can be attributed to the synergistic effect of bi-directional processing and dynamic window size adaptation: while the bi-directional architecture provides comprehensive temporal context understanding, the dynamic window size mechanism prevents over-reliance on either local or global temporal features, resulting in more balanced and accurate predictions.

5.1. Qualitative Works

In Figure 5, we analyze the qualitative performance of our Bi-directional PredRNN model with dynamic window size adaptation across both Moving MNIST and KTH action datasets. For Moving MNIST sequences, our model maintains sharp digit boundaries and preserves distinctive numerical features (e.g., the curve in ’0’, ’8’) without distortion. The model accurately predicts the trajectory changes when digits collide, and it demonstrates physically plausible reflection patterns at frame boundaries.

The KTH action sequences show our model’s capability to handle complex human motion patterns. The dynamic window size adaptation mechanism is particularly effective for the capture of natural human movements because it automatically adjusts the temporal receptive field based on motion complexity. The results show smooth and continuous predictions of coordinated body movements, from the synchronized motion of arms and legs during walking to precise finger movements in hand-waving actions. Our model consistently maintains temporal coherence and preserves subtle details such as clothing folds and shadow interactions under various lighting conditions.

6. Conclusions

We have tackled key challenges in spatiotemporal sequence prediction, focusing on the difficulties of capturing complex temporal dependencies and preserving spatial coherence in long sequences. Traditional models often struggle with these tasks due to the vanishing gradient problem and limitations in maintaining spatial information over time. Bi-PredRNN extends PredRNN by incorporating bi-directional processing capabilities to address these challenges.

Our primary contribution is a bi-directional memory network architecture that processes temporal information in both forward and backward directions, with specialized memory cells to ensure spatial coherence. We have also introduced a dynamic window size mechanism, which adapts temporal context based on optical flow estimation and EMA smoothing, allowing for more flexible temporal processing. Experimental results on datasets such as Moving MNIST, KTH Action, and climate data demonstrate that our model achieves improved prediction accuracy and more stable long-term dependencies compared to traditional unidirectional models.

Despite these advancements, we recognize limitations in spatiotemporal prediction due to inherent temporal ambiguity. While bi-directional processing enriches temporal context, it can introduce uncertainty when future states significantly influence past interpretations. This motivates the development of advanced temporal attention mechanism that dynamically adjust the weight of past and future contexts. Hybrid architectures, which adapt between unidirectional and bi-directional processing based on the sequence’s temporal features, may also help. Finally, adding domain-specific knowledge could reduce temporal ambiguity in specialized applications.