A Transformer Encoder Approach for Indoor Localization Reconstruction from an IMU and GPS Based Sensor

1. Introduction

A widely used approach for 3D localization is the Inertial Measurement Unit (IMU) [1], which typically includes accelerometers, gyroscopes, and occasionally magnetometers. IMUs provide 3D acceleration and orientation data. However, due to inherent noise, IMU-based localization (which requires double integration of acceleration) is prone to significant temporal drift [2], making reliable positioning feasible for only brief periods. To address this, IMU data are often combined with Global Positioning System (GPS) data through a Kalman Filter (KF) [3].

In outdoor environments, combining IMU and GPS data with a KF enables accurate localization. However, in indoor settings, this approach is limited by GPS signal issues, such as weak signal or signal loss. Since the sensor’s localization depends heavily on GPS data, its accuracy declines sharply indoors.

Deep learning-based methods have been developed to provide real-time localization during GPS outages by leveraging information from past data. Although these methods can also offer localization for applications that do not require real-time processing, they have a limitation: they cannot utilize future data. Thus, these approaches lead to suboptimal localization when both past and future data can be used.

In this paper, we present a novel deep learning approach using a Transformer-based bidirectional encoder to reconstruct indoor localization during postprocessing. This approach employs raw IMU and GPS data, along with the KF output, to predict velocity. Position is then obtained by bidirectional integration of this velocity data. We also incorporate position interpolation to ensure smooth transitions between recorded and predicted positions during GPS outages.

The proposed approach is tested using the Alogo Move Pro sensor, a device specifically designed for equestrian applications. This device uses an Inertial Navigation System (INS) that combines IMU and GPS data through a KF, enabling 3D localization of a horse during training and competition. Based on this localization, the device generates various metrics such as stride analysis, power, and maximum jump height. Riders typically review session data after training or competition, allowing for postprocessing and enabling to leverage information from both past and future data when applying localization reconstruction.

The sensor’s outdoor accuracy has been validated [4]. However, in indoor environments, the sensor experiences GPS outages, compromising the precision required for effective analysis of horse behaviors. GPS outages in the sensor occur in three main cases: at the start, during, or at the end of a session. Additionally, GPS outages duration can be categorized into short outages (10 to 20 seconds) and medium outages (30 seconds to 2 minutes).

2. Related Works

Many researchers have explored ways to reduce noise in IMU data by investigating alternative fusion methods that do not rely on GPS data. Several factors may drive the choice to use only IMU data, including the cost of adding a GPS receiver and the impracticality of GPS in certain contexts, such as indoor localization within buildings. Sun et al. [5] proposed a two-stage pipeline to address this challenge: first predicting device orientation and then estimating position based on IMU data from a smartphone. Their approach utilized a Long Short-Term Memory (LSTM) network in the first stage and a bidirectional LSTM network in the second. Chen et al. [6] introduced a method that segments IMU data into windows, applying an LSTM network to learn the relationship between raw acceleration data and the polar delta vector, from which localization is derived. Brossard et al. [7] took a different approach, presenting a Convolutional Neural Network (CNN) to correct gyroscope noise in raw IMU signals.

In another study [8], the authors proposed an extended KF based on a 1D version of ResNet18 to fuse IMU data for localization estimation. A more lightweight adaptation of this approach, compatible with standard smartphones, was developed in [9]. Similarly, Wang et al. [10] proposed a ResNet18-inspired network for fusing IMU data. Cioffi et al. [11], focusing on drones, adopted a Temporal Convolutional Network (TCN), citing studies showing that TCNs perform comparably to Recurrent Neural Networks (RNNs) for temporal sequences, with the added advantages of easier training and deployment on robotic platforms. Rao et al. [12] introduced a contextual transformer network with a spatial encoder and a temporal decoder to fuse IMU data for velocity prediction, which allowed localization via velocity integration. Wang et al. [13] achieved high localization accuracy from IMU data using a network with convolution blocks, a bidirectional LSTM, and a Multi-Layer Perceptron (MLP), incorporating an attention mechanism.

Other studies have aimed to improve the fusion of IMU and GPS data, which is traditionally managed through a KF. Hosseinyalamdary [14] presented a deep KF using an LSTM to fuse IMU and GPS data, achieving better localization than the conventional KF. Wu et al. [15] replaced the KF with a TCN for position and velocity prediction, adopting a multitask strategy to reduce prediction errors. Lastly, Kang et al. [16] developed a model combining a CNN with a Gated Recurrent Unit (GRU) to predict pedestrian velocity.

This literature review indicates that extensive efforts have been made to either predict localization from IMU data alone or enhance GPS and IMU data fusion. However, the problem of GPS signal loss has received limited attention, with most studies addressing GPS outages in real-time applications. As a result, these real-time methods cannot leverage the benefits of a postprocessing bidirectional perspective. While such methods may work for applications not requiring real-time localization, they yield suboptimal results. To address this gap, we propose a deep learning approach for localization reconstruction in postprocessing, utilizing both past and future data to improve localization accuracy.

3. Materials

3.1. Data

To develop our deep learning approach, we collected data from the Alogo Move Pro device during outdoor sessions. The data capture a variety of contexts, including training sessions, equestrian trails, and competition courses. In total, 53 sessions were recorded, amounting to 29 hours and 4 minutes of data. The shortest session lasted 27 seconds, while the longest extended to 49 minutes and 57 seconds. The average session duration was 32 minutes and 54 seconds, with a standard deviation of 12 minutes and 48 seconds. The sessions were divided into three datasets: training (64%), validation (18%), and testing (18%).

Each session consists of 100 timestamps per second, and for each timestamp, the device provides the following data:

• A^X_IMU, A^Y_IMU, A^Z_IMU: Acceleration measured by the IMU
• R^A_IMU, R^B_IMU, R^C_IMU: Rate of rotation measured by the IMU
• M^X_IMU, M^Y_IMU, M^Z_IMU: Magnetic field measured by the IMU
• P^X_GPS, P^Y_GPS, P^Z_GPS: Position measured by the GPS
• V^X_GPS, V^Y_GPS, V^Z_GPS: Velocity measured by the GPS
• E_GPS: Position accuracy estimation provided by the GPS (PDOP)
• P^X_FUSION, P^Y_FUSION, P^Z_FUSION: Position computed by the sensor fusion algorithm
• V^X_FUSION, V^Y_FUSION, V^Z_FUSION: Velocity computed by the sensor fusion algorithm
• O^A_FUSION, O^B_FUSION, O^C_FUSION: Orientation computed by the sensor fusion algorithm
• E_FUSION: Position accuracy estimation provided by the sensor algorithm

GPS data are refreshed at a frequency of 5 Hz, while IMU and sensor fusion algorithm data are refreshed at 100 Hz. Table A1 presents various statistical details about these features across the entire dataset.

The P_FUSION, V_FUSION and O_FUSION values are computed by the sensor fusion algorithm using IMU data (A_IMU, R_IMU, and M_IMU) and GPS data (P_GPS and V_GPS). During GPS outages or periods of weak GPS signal, the accuracy of P_GPS and V_GPS declines, which affects P_FUSION, V_FUSION and O_FUSION due to error propagation. In this study, our primary goal is to accurately reconstruct P_FUSION during these GPS outage phases.

3.2. Testing Subdatasets

As outlined in the introduction, there are three primary cases of GPS outages: at the beginning of the session, during the session, and at the end of the session. Additionally, GPS outages can be categorized by duration into two types: short outages (10 to 20 seconds) and medium outages (30 seconds to 2 minutes). To simulate these scenarios, we created three masking schemas to generate testing subdatasets from the original testing dataset:

• Schema 1: The beginning of the session is masked
• Schema 2: A section within the session is masked, while the beginning and end of the session remain unmasked (at least 2 seconds are left unmasked at both the start and end).
• Schema 3: The end of the session is masked

The lengths of the masked sections (denoted as L_mask) and the positions of the mask within the session (denoted as P_mask) for schema 2 were randomly sampled from a uniform distribution. To expand the testing subdatasets, each session was duplicated multiple times, with each duplicate receiving a different mask sampled from the same distribution (see Figure 1).

For each schema, we created two subdatasets: one simulating short GPS outages (10 s < L_mask < 20 s) and another simulating medium GPS outages (30 s < L_mask < 120 s). The masking process was implemented by setting the following features to null values, preventing the model from accessing this information during localization reconstruction:

• GPS features: P^X_GPS, P^Y_GPS, P^Z_GPS, V^X_GPS, V^Y_GPS, V^Z_GPS, E_GPS
• Sensor fusion algorithm output features: P^X_FUSION, P^Y_FUSION, P^Z_FUSION, V^X_FUSION, V^Y_FUSION, V^Z_FUSION, O^A_FUSION, O^B_FUSION, O^C_FUSION, E_FUSION

3.3. Preprocessing

Since the sensor is used across multiple countries, the GPS and sensor fusion algorithm positions (P_GPS and P_FUSION) vary significantly in range. To address this, we converted all positions to relative positions. For the training and validation datasets, as well as the testing subdatasets based on schemas 2 and 3, relative positions were calculated using each session’s first position. For testing subdatasets based on schema 1, relative positions were calculated based on the first position after the masked section.

Next, we applied normalization [17] to scale the features to the range [0, 1]. Additionally, we created a separate velocity vector (a duplicate of V^X_FUSION, V^Y_FUSION, V^Z_FUSION) to serve as the target for training our neural network, with normalization applied in the range [-1, 1].

4. Approach

4.1. Flow

To reconstruct the localization of a horse during a GPS outage, we propose a six-step process, illustrated in Figure 2.

Extracting the GPS outage section: The GPS outage section is isolated from the session to create a window of size​ L_seq, containing all sensor features (listed in Section 3.1. Data) at a frequency of 100 Hz. For short GPS outages (up to 20 seconds), we define L_seq = 2400 (24 seconds). For medium GPS outages (up to 120 seconds), L_seq = 12400 (124 seconds). This ensures that the window includes at least 4 seconds of data not recorded during a GPS outage. For outages at the beginning or end of a session, the valid GPS data will be positioned at one extremity of the window. For outages occurring mid-session, valid data will be available at both sides, allowing the network access to past and/or future data to inform its predictions.

Masking GPS outage timestamps: Timestamps within the GPS outage are selected, and features dependent on GPS are masked to signify the outage to the neural network. Masking is applied by replacing values with -1.

Neural network processing: The processed window is fed into the neural network, which outputs velocity predictions across three axes. Tests indicated that directly predicting positions resulted in poor performance. As illustrated in Figure 3, velocity distributions are consistently centered around 0 across sessions, while position distributions are highly variable due to differences in session types and distances covered. Therefore, predicting velocities—constrained within a typical range—is more stable than predicting positions, which may vary extensively.

Integrating velocities: Positions are derived by integrating the predicted velocities. For outages at the beginning or end of a session, positions are computed unidirectionally. For mid-session outages, a bidirectional integration (forward and backward) is performed, and the results are averaged to distribute position errors more evenly across the GPS outage section.

Applying interpolation (only for GPS outage in the middle of a session): Because bidirectional integration can create discontinuities with the rest of the session, an interpolation step is applied at the boundaries of the GPS outage. To achieve smooth junctions, we calculate the duration of data to be interpolated at each extremity of the GPS outage section, denoted as t_interp ​(illustrated in Figure 4). Two quadratic interpolation functions are fitted—one at each boundary. The first function is fitted using data from the 20 ms before the outage and the 20 ms after t_interp ​seconds at the beginning of the outage. The second function uses data from the 20 ms preceding t_interp ​seconds at the end of the outage and the 20 ms following the outage. These functions are then applied to the first and last t_interp ​seconds of the predicted positions. The process is illustrated in Figure 5.

Merging results: The final integrated (and interpolated, if applied) positions are merged back into the session.

As we might note, our approach is based on a Transformer network, which poses training challenges due to high computational demands and GPU memory requirements. Given the $O\left(L^2\right)$ complexity of Transformers concerning input sequence length, we also propose a lighter variant of our approach that processes data at a frequency of 10 Hz instead of 100 Hz. In this version, the network outputs the average velocities over 10 timestamps rather than per timestamp.

4.2. Network Architecture

Our proposed network architecture is inspired by the BERT model [18]. The network consists of multiple Transformer encoder layers stacked sequentially, followed by a feed-forward layer. To improve precision, the model uses a learnable positional encoding that is updated during training. The architecture is shown in Figure 2. We define d_model as the dimension of the embedding, N_head the number of head in the multi head attentions, d_hidden the size of the inner layer of the feed forwards and N_encoder the number of encoder layers.

The network takes as input a sequence $\mathrm{X}\in {\mathbb{R}}^{L_{seq}\times d_{features}}$ where $d_{\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{s}}=d_{imu}+d_{gps}+d_{fusion}=9+7+10=26$is the dimension of the input sequence and L_seq is the length of the sequence as defined in section 4.1. Flow. Initially, the input sequence is projected into the embedding space ${\mathbb{R}}^{L_{seq}\times d_{model}}$:

$$F_{proj}\left(X\right)=X{W}_{emb}+b_{emb}$$

where $W_{emb}\in {\mathbb{R}}^{d_{features}\times d_{model}}$ and $b_{emb}\in {\mathbb{R}}^{d_{model}}$Then, the learnable positional encoding is added to the projected input:

$$PE\left(X\right)=X+b_{pe}$$

Where $b_{pe}\in {\mathbb{R}}^{{L_{seq}\times d}_{model}}$is the learnable positional encoding.

After the input projection and positional encoding, the sequence is processed through N_encoder​ encoder layers. Each encoder layer has an identical structure but unique weights and biases. Each layer consists of a residual multi-head attention mechanism followed by a residual feed-forward network.

The residual multi-head attention mechanism, as described by Vaswani et al. in the original Transformer paper [19], is defined as follows:

$$\begin{array}{}hea{d}_i\left(X\right)=softmax\left({\displaystyle \frac{X{W}_i^Q{\left(X{W}_i^K\right)}^T}{\sqrt{\frac{d_{model}}{N_{head}}}}}\right)X{W}_i^V\\ MultiHeadAttention\left(X\right)=Concat\left(hea{d}_1\left(X\right),hea{d}_2\left(X\right),\dots ,{head}_{N_{head}}\left(X\right)\right)W^O\\ X^{\mathrm{\text{'}}}=LayerNorm(X+MultiHeadAttention\left(X\right))\end{array}$$

where $W_i^Q\in {\mathbb{R}}^{d_{model}\times {\displaystyle \frac{d_{model}}{N_{head}}}}$, $W_i^K\in {\mathbb{R}}^{d_{model}\times {\displaystyle \frac{d_{model}}{N_{head}}}}$, $W_i^V\in {\mathbb{R}}^{d_{model}\times {\displaystyle \frac{d_{model}}{N_{head}}}}$, and $W^O\in {\mathbb{R}}^{{d_{model}\times d}_{model}}$The residual feed-forward network, as described by Vaswani et al. in [19], is given by:

$$\begin{gathered} FF\left(X\right)=ReLU\left(X{W}_1+b_1\right)W_2+b_2 \\ {X}^{\mathrm{\text{'}}}=LayerNorm(X+FF\left(X\right)) \end{gathered}$$

where $W_1\in {\mathbb{R}}^{d_{model}\times d_{hidden}}$, $b_1\in {\mathbb{R}}^{d_{hidden}}$, $W_2\in {\mathbb{R}}^{d_{hidden}\times d_{model}}$, and $b_2\in {\mathbb{R}}^{d_{model}}$After processing through the N_encoder encoder layers, the sequence passes through a final feed-forward network that projects it into ${\mathbb{R}}^{L_{seq}\times 3}$, representing the velocities along the x, y, and z axes for the entire sequence:

$$\begin{array}{}F{F}_{final}\left(X\right)=ReLU\left(X{W}_1^{\mathrm{\text{'}}}+b_1^{\mathrm{\text{'}}}\right)W_2^{\mathrm{\text{'}}}+b_2^{\mathrm{\text{'}}}\\ Y=LayerNorm\left(FF\left(X\right)\right)\end{array}$$

where $W_1^{\mathrm{\text{'}}}\in {\mathbb{R}}^{d_{model}\times d_{hidden}}$, $b_1^{\mathrm{\text{'}}}\in {\mathbb{R}}^{d_{hidden}}$, $W_2^{\mathrm{\text{'}}}\in {\mathbb{R}}^{d_{hidden}\times 3}$, and $b_2^{\mathrm{\text{'}}}\in {\mathbb{R}}^3$.

4.3. Network Training

The network was trained for up to 500 epochs, with early stopping applied if no improvement was observed for 20 consecutive epochs [20].

To train and validate the network, sequences of length L_seq ​were extracted from the training and validation sessions. For small GPS outage sections (10 to 20 seconds), L_seq was set to 2400 (or 240 for the lighter network version). For medium GPS outage sections (30 seconds to 2 minutes), L_seq ​was set to 12400 (or 1240 for the lighter version). To extract sequences, we sampled 1/5000^th of the timestamps in training sessions as sequence starting points, while 1/1000^th of timestamps were sampled in validation sessions. During training, the sampled timestamps were rotated through the dataset every 20 epochs, while validation timestamps remained fixed. This process is illustrated in Figure 6.

Each sequence was assigned a mask of size L_mask, where L_mask was sampled from a uniform distribution. For small GPS outage training, $L_{mask}~U(1000,2000)$ (for the lighter version: $L_{mask}~U(100,200)$), and for medium GPS outage training, $L_{mask}~U(3000,12000)$ (for the lighter version: $L_{mask}~U(300,1200)$). We explored several masking strategies (where P_mask is the first index of the mask on the sequence):

• Strategy 1: Mask applied at the start of the sequence, $P_{mask}=0$
• Strategy 2: Mask applied in the center of the sequence, $P_{mask}={\displaystyle \frac{L_{seq}-L_{mask}}{2}}$
• Strategy 3: Mask applied at the end of the sequence, $P_{mask}=L_{seq}-L_{mask}$
• Strategy 4: Mask applied at a random position, $P_{mask}~U(0,L_{seq}-L_{mask})$

Strategies 1, 2, and 3 were tailored for schemas 1, 2, and 3, respectively, while strategy 4 was intended for use across all schemas. During training, the mask length (and position for strategy 4) was regenerated at each epoch, while validation masks remained constant. Masking was performed by setting GPS and sensor fusion features to -1:

• GPS features: P^X_GPS, P^Y_GPS, P^Z_GPS, V^X_GPS, V^Y_GPS, V^Z_GPS, E_GPS
• Sensor fusion algorithm output features: P^X_FUSION, P^Y_FUSION, P^Z_FUSION, V^X_FUSION, V^Y_FUSION, V^Z_FUSION, O^A_FUSION, O^B_FUSION, O^C_FUSION, E_FUSION

The network was trained to reconstruct the velocity using the Adam optimizer with weight decay [21]. To emphasize minimizing large prediction errors, we selected the Mean Square Error (MSE) loss function. Additionally, we hypothesized that encouraging the network to predict velocity in both the masked and unmasked sections could improve results by promoting continuity. Therefore, we constructed a dual-objective function that considers both masked and unmasked sections:

$$\begin{array}{}{\mathcal{L}}_{masked}={\displaystyle \frac{1}{L_{mask}}}\sum 1_{i\in M}\mathrm{*}{\left(y_i-y_i^{pred}\right)}^2\\ {\mathcal{L}}_{unmasked}={\displaystyle \frac{1}{{L_{seq}-L}_{mask}}}\sum 1_{i\notin M}\mathrm{*}{\left(y_i-y_i^{pred}\right)}^2\\ \mathcal{L}=\lambda \mathrm{*}{\mathcal{L}}_{masked}+{\left(1-\lambda \right)\mathcal{*}\mathcal{L}}_{unmasked}\end{array}$$

where $M$is the masked part of the sequence and $\lambda \in [0;1[$ is a hyperparameter, $y_i$ and $y_i^{pred}$ are respectively the ground truth velocity and the predicted velocity of the i^th timestamp of the sequence.

4.4. Hyperparameter Tuning

Hyperparameter tuning was conducted using Bayesian search [22]. The hyperparameter space is detailed in Table 1.

4.5. Evaluation

To evaluate our approach, we defined three metrics:

• The Absolute Trajectory Error (ATE): Measures the discrepancy between the ground truth and the predicted trajectories. This metric is sensitive to outliers. Therefore, it is common that this metric increases with prediction duration and length.
• The Relative Trajectory Error (RTE): Quantifies the relative error in the distance between the ground truth and predicted start/end points.
• The Relative Distance Error (RDE): Calculates the relative error between the total predicted and ground truth distances.

$$\begin{array}{}\mathrm{A}\mathrm{T}\mathrm{E}={\displaystyle \frac{1}{\mathrm{N}}}\times {\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}}|\mathrm{P}\left(\mathrm{n}\right)-\widehat{\mathrm{P}}\left(\mathrm{n}\right)|\\ \mathrm{R}\mathrm{T}\mathrm{E}={\displaystyle \frac{|\mathrm{P}\left(\mathrm{N}\right)-\mathrm{P}\left(0\right)|-|\widehat{\mathrm{P}}\left(\mathrm{N}\right)-\widehat{\mathrm{P}}\left(0\right)|}{|\mathrm{P}\left(\mathrm{N}\right)-\mathrm{P}\left(0\right)|}}\\ \mathrm{R}\mathrm{D}\mathrm{E}=1-{\displaystyle \frac{{\sum }_{\mathrm{n}=1}^{\mathrm{N}}|\widehat{\mathrm{P}}\left(\mathrm{n}\right)-\widehat{\mathrm{P}}\left(\mathrm{n}-1\right)|}{{\sum }_{\mathrm{n}=1}^{\mathrm{N}}|\mathrm{P}\left(\mathrm{n}\right)-\mathrm{P}\left(\mathrm{n}-1\right)|}}\end{array}$$

where:

• $\mathrm{P}\left(\mathrm{t}\right)$ is the ground truth position in 3D at instant t: $\left(\mathrm{x}\right(\mathrm{t});\mathrm{y}(\mathrm{t});\mathrm{z}(\mathrm{t}\left)\right)$
• $\widehat{\mathrm{P}}\left(\mathrm{t}\right)$ is the predicted position in 3D at instant t: $\left(\widehat{\mathrm{x}}\left(\mathrm{t}\right);\widehat{\mathrm{y}}\left(\mathrm{t}\right);\widehat{\mathrm{z}}\left(\mathrm{t}\right)\right)$
• $|{\mathrm{P}}_1\left(\mathrm{n}\right)-{\mathrm{P}}_2\left(\mathrm{n}\right)|$ is the Euclidean norm between ${\mathrm{P}}_1\left(\mathrm{n}\right)$ and ${\mathrm{P}}_2\left(\mathrm{n}\right)$

5. Results

Our approach achieves an ATE of approximately 3 meters for small GPS outages across the entire testing dataset. For medium GPS outages, the ATE is around 12 meters. This means that, on average, the predicted positions for small GPS outages are within 3 meters of the actual position of the horse, and for medium GPS outages, within 12 meters. In comparison, state-of-the-art methods on pedestrian datasets, which feature shorter distances, lower velocities (up to 10 times slower), and reduced acceleration, report ATE values mostly between 4 and 9 meters for GPS outages of several seconds to minutes [5,9,10,12].

For the RTE, our approach yields values ranging from 0.2 to 0.7. This implies that the ratio of the distance between the predicted end point and the predicted start point relative to the distance between the actual end point and start point falls between 0.2 and 0.7. In contrast, state-of-the-art methods report RTE values ranging from 0.9 to 6 for one-minute GPS outages in pedestrian contexts [5,9,10,12].

In terms of RDE, our approach also performs well, averaging 0.3 regardless of GPS outage duration, highlighting stable performance across various outage lengths. A summary of these results is provided in Table 2 and Table 3.

Regarding the hyperparameters space, we determined an optimal subspace and the best hyperparameters set, available in Table 4. A more detailed illustration of the results of the hyperparameters search is available in Figure B1.

Another crucial aspect for real-world applications is quantifying the processing time of the deep neural network, which represents the majority of our approach’s computation time, and assessing memory consumption.

With the best-performing hyperparameter configuration, the network can make an inference on a CPU in 160 ms for small GPS outages and 5.01 s for medium GPS outages. For applications requiring more rapid responses, the lighter network version completes predictions on a CPU in 4.14 ms for small GPS outages and 49 ms for medium GPS outages. On a GPU, prediction times decrease significantly: 3.05 ms (1.76 ms for the lighter version) for small GPS outages and 57.8 ms (1.80 ms for the lighter version) for medium GPS outages. GPU memory consumption is relatively low, with most modern GPUs having ample memory for this task.

Using the best hyperparameter configuration, training on a CPU is impractically slow. However, on a GPU, even a basic model with a few GB of RAM is sufficient. Table 5 provides detailed specifications of the network with the optimal hyperparameters.

For applications requiring hyperparameter searches in the explored space, we recommend a high-performance GPU, as larger networks may demand up to several dozen GB of RAM. Additional information on processing time and memory usage for various network sizes is available in Appendix C.

As shown in Figure 7, our approach accurately predicts the velocity trends and amplitudes, while avoiding adherence to velocity noise, demonstrating the network’s robustness. For position predictions, there is a close alignment with the ground truth signal. Although minor deviations exist at certain points, they appear to be well-balanced by the positive and negative errors within the velocity signals, with occasional accumulations being compensated over time.

From a top-view perspective (illustrated in Figure 8, as seen by riders in the sensor application), the predicted positions align well, and the interpolation step following position integration creates a seamless transition between the GPS outage section and the rest of the session.

6. Discussion

In this research, we compared fixed masking strategies (strategies 1, 2, and 3) with a random masking strategy (strategy 4) to train the network. Each fixed strategy is designed for a specific schema, while the random strategy is applicable across all three schemas. On average, both fixed and random strategies produced nearly identical results in most cases. Depending on the schema, either a fixed or random strategy yielded the best results (see Figure 9). Thus, while training all four strategies achieves highly accurate results, training only the random strategy can achieve good results with a quarter of the computational cost.

Regarding input frequency (100 Hz for the base version, 10 Hz for the lighter version), we observed similar results for both, with better outcomes in some cases at 10 Hz (see Figure 9). This may seem surprising at first, but is logical: a lower frequency reduces precision in velocity predictions (predicting the mean over 100 ms rather than every 10 ms). However, velocity errors are well-distributed (balanced between positive and negative errors) over the 100 ms period, so after integration, they cancel out, preventing drift. The only noticeable errors are short-lived (within 100 ms) and correlate with the duration before positive and negative velocity errors balance out. Additionally, the lighter network’s reduced input sequence length allows for faster training, freeing up resources to explore more hyperparameter sets and increasing the chance of finding an optimal configuration. Moreover, even with a GPU with 25 GB of RAM, some configurations remain untested on the base version due to excessive memory requirements, especially for sequences up to 2 minutes in length.

We computed hyperparameter importance (shown in Figure 10). For model architecture, the number of encoder layers N_encoder​ and the embedding dimension d_model​ were the most impactful hyperparameters. In training configuration, the learning rate was the most influential, followed by batch size and weight decay.

Despite these promising results, several areas of our approach warrant further exploration.

Validation in indoor environments: Our approach, designed to predict velocity during GPS outages, was trained on sequences extracted from outdoor sessions. However, horses can exhibit different behaviors (in stride, jump, velocity, and acceleration) between indoor and outdoor environments. Thus, results based on outdoor data do not guarantee accuracy indoors. To validate our approach for indoor use, data would need to be collected in indoor settings, with performance evaluated on masked sequences from those sessions. If indoor performance proves less accurate, retraining the network with both outdoor and indoor data may be necessary. However, because accurate ground truth cannot be obtained in indoor settings with our sensor due to GPS outages, recording such sessions would require a more sophisticated setup not reliant on GPS and IMU. For example, a combination of UWB and LiDAR could provide reliable ground truth in indoor environments. We did not conduct this experiment due to the high cost of such setups, which can reach hundreds of thousands of euros.

Computational requirements for longer GPS outages: The computational resources required for training and predicting velocity during GPS outages increase exponentially with input sequence length. We introduced a lighter version of our approach, which achieved similar accuracy by reducing the input frequency by a factor of ten. However, even with this adjustment, handling large GPS outages (e.g., 5 to 10 minutes) is challenging and demands substantial computational infrastructure. Preliminary attempts to reduce input frequency further (by a factor of hundred) compromised accuracy. For longer GPS outage duration, future work could explore advanced attention mechanisms that replicate the performance of standard Transformer attention while reducing computational costs.

Validation across different devices and use cases: We validated our approach on a dataset of horse positions recorded using the Alogo Move Pro device. Future work could expand validation to different devices and various types of motion (e.g., pedestrians, cyclists, racing dogs, drone) to assess the generalizability of the approach.