A Robust Method for Validating Orientation Sensors Using a Robot Arm as a High-Precision Reference

1. Introduction

Orientation sensors, whether standalone or integrated into pose-tracking systems, are extensively used in drones, mobile robots, medical applications [1], and immersive visualization devices [2]. These sensors provide absolute measurements, differential rotations, or a combination of both. For instance, Inertial Navigation Systems integrate data from a gyroscope, accelerometer, and magnetometer, while 3D SLAM applications can leverage IMU data alongside odometry for precise localization and mapping.

Sensor systems with stable external reference like motion capture cameras, VR tracking devices and electromagnetic trackers provides reliable absolute orientation with moderate noise level. However, the increasingly popular MEMS-based and the so-called inside-out vision-based systems are less robust against noise, bias, drift and delay. For example, concerning MEMS sensors, Shi et al. pointed out that environmental factors, particularly ambient temperature, significantly affect the bias and drift of these sensors, which can lead to decreased orientation estimation accuracy [3]. Another analysis by Suvorkin et al. utilized the Allan deviation slope method to evaluate noise characteristics across different grades of sensors. This analysis provides insights into the stability and reliability of IMU measurements over time, which is essential for applications requiring high precision [4]. Understanding noise characteristics is crucial for interpreting IMU data accurately and ensuring its validity in practical applications.

Inside-out tracking systems, such as those used in Microsoft’s HoloLens or HTC Vive, rely on onboard cameras to track the environment and determine orientation. While these systems offer the advantage of not requiring external infrastructure, they are highly sensitive to environmental conditions, such as lighting and occlusions. A study by Zhang et al. found that inside-out tracking systems could experience substantial errors in orientation estimation when subjected to rapid movements or changes in lighting conditions [5]. Additionally, the latency introduced by image processing can lead to noticeable delays in orientation updates, impacting user experience. Similarly, a paper by Niehorster et al. on the HTC Vive’s inside-out tracking system highlighted that while the precision of tracking measurements is high, the system can experience significant offsets when tracking is temporarily lost, affecting overall accuracy [6]. Other studies using the HTV Vive tracker gives similar conslusion [7,8]. This illustrates the challenges faced by systems that rely on internal measurements compared to those that utilize stable external references.

These imperfections underlines the importance of validation methodologies that allows for precise and automatic comparisons of various sensors under multiple conditions in a repeatable way. The validation of orientation sensors relies heavily on the establishment of ground truth data. Ground truth serves as a reference against which the performance of the sensors can be assessed. Various methods for obtaining ground truth have been documented in the literature, each with its unique advantages and applications.

One of the most reliable forms of ground truth for validating IMU orientation sensors is the use of motion capture systems. These systems provide high-precision data regarding the position and orientation of subjects in three-dimensional space. For instance, Morrow et al. conducted a validation study where they compared the kinematic data obtained from a commercially available IMU system against a standard optical motion capture system [9]. Inside-out tracking systems have been validated in similar setup [7,10]. This comparative approach is widely recognized for its effectiveness if a motion capture laboratory is available.

Another approach to obtain reliable reference is the use of mechanical fixtures. The use of fixtures in experimental setups allows for controlled conditions that can significantly enhance the reliability of sensor validation processes. Bliley et al. [11] described a validation method for wearable motion-sensing devices that involved a mechanical fixture. Their approach utilized a platform rotated by a servo motor, which allowed for precise control over the orientation of the sensors being tested. This method ensured that the sensors were subjected to known and repeatable movements, facilitating accurate comparisons between the sensor outputs and the expected values. Additionally, Eastman et al. [12] highlighted the importance of using fixtures in their work on 6DOF (six degrees of freedom) object pose ground truth measurements. They employed an aluminum pose fixture along with a laser tracker to establish a reliable reference for validating sensor outputs. This combination of a rigid fixture and precise measurement tools allowed for high accuracy in determining the orientation and position of objects. Similarly, Herickhoff et al. [13], utilized a lightweight plastic fixture to hold an IMU sensor. The adaptability of such fixtures makes them valuable tools in various applications, including medical imaging and motion analysis.

These published methods are lack of the ability of providing a programmatic way to generate test motions and then analyze the results in a tractable manner. Using a robot arm as a ground truth reference would serves with important benefits: The robot arm can provide precise and repeatable movements, allowing for accurate comparisons between the sensor outputs and the known positions and orientations of the arm. This motivation has been supported by studies from the literature of the field. Schmidt and Kraft noted that accurate ground truth trajectories can be obtained when a camera is mounted on an industrial robot arm, which allows for reliable evaluations of visual SLAM (Simultaneous Localization and Mapping) algorithms [14]. Papers such as [15,16,17] exploits the benefits of robot arms in generating motion patterns for testing IMU-s. One prominent approach by Hoslip et al. uses an industrial robot to compare the performance of multiple IMU sensors simultaneously [18]. Even this methods utilizes various robotic manipulators a systematic approach to mathematically handle the geometric relations between the robot’s and the sensor’s references, has not been published.

Authors faced this unsolved issue during the experimental validation of a generic direction and orientation tracking method based on IMU Sensor, that has been published in [19]. In our former work, we proposed a mathematical framework for calibration and tracking of objects with with one functional direction (continuous rotational symmetryc case) and multiple functional directions (nonsymmetric case). This previous work has inspired the the current study which proposes a method for the validation of orientation sensors with respect to an external reference. The proposed method is agnostic to the external (reference) measurement system, but it is discussed considering a robotic arm as a robust integrated actuation and sensing device and the robotics terminology is used throughout the discussion.

In our setup, a commercial robot arm is employed for validation, providing a precise orientation reference and enabling automated testing capabilities Figure 1. This approach not only ensures high accuracy but also facilitates systematic and repeatable validation processes.

The validation of orientation sensors using any test setup introduces the challenge of dealing with different reference and target frames. The sensor initializes its own unknown reference frame and measures the orientation of a frame fixed to the sensor relative to this reference. The analogous frames of an industrial manipulator are called base frame and TCP frame respectively (see Figure 2). We can suppose that the industrial robot’s internal measurement system provides reliable position and accurate orientation information regarding the base to TCP transformation [20,21].

This paper proposes a method to approximate the rotation matrices between the base-reference and TCP-sensor frames. The computation involves two main steps: first, deriving an initial estimate using geometric principles, followed by a local optimization to minimize the impact of measurement noise and errors..

To demonstrate the method’s effectiveness, experiments were conducted using a UR16e robot [22] with two distinct sensors: the widely-used ICM20948 IMU sensor [23] and the HTC VIVE headset’s inside-out tracking system [6,24,25,26].

The paper is organized as follows: Section 2 defines the problem and introduces the relevant notations. Section 3 elaborates on the proposed calibration method. Section 4 presents experimental results obtained with various sensors and provides a comparative analysis. Section 5 outlines future research activities and potential practical applications. Finally, Section 6 summarizes the findings. The Appendix A includes details on vector and quaternion operations, along with their respective notations, as used throughout the paper. Original measurement data associated with the content of the paper can be found at [27].

2. Problem Description

In the investigated measurement setup, the ${\mathbf{q}}^{(base,TCP)}\left(t\right)$ orientation can be obtained from the state of the manipulator, while the sensor provides its ${\mathbf{q}}^{(ref,sensor)}\left(t\right)$ orientation with respect to its own reference frame, see Figure 2.

In ideal case, there exist ${\mathbf{q}}^{(base,ref)}$ and ${\mathbf{q}}^{(TCP,sensor)}$ orientations such that

$${\mathbf{q}}^{(base,TCP)}\left(t\right)·{\mathbf{q}}^{(TCP,sensor)}={\mathbf{q}}^{(base,ref)}·{\mathbf{q}}^{(ref,sensor)}\left(t\right)$$

(1)

for all $t\in [0,t_{meas}]$. However, in practical scenarios, there is an error resulting from the measurement noises and the drift of the sensor.

In the followings, the ${\mathbf{q}}^{(base,TCP)}\left(t\right)$ value (read from the robot controller) will be considered as ground truth. The computation of ${\mathbf{q}}^{(TCP,sensor)}$ and ${\mathbf{q}}^{(base,ref)}$ orientations from some initial data will be referred to as calibration.

Considering the whole recorded dataset, the angle error of Eq. (1) will be analyzed to obtain the measurement noise and the drift of the sensor.

3. Calibration Method

The calibration is performed using solely the first $0\le t<t_{cal}$ values of the measured data. This approach enables the analysis of sensor drift in the subsequent measurements for $t\ge t_{cal}$.

The method requires at least three values from the $t<t_{cal}$ portion of the measured data, each rotated by an angle $60[deg]\le \phi \le 120[deg]$ relative to the others. Denote the time of these measurements as $t_A$, $t_B$, and $t_C$.

The calibration process consists of two stages: first, obtaining a good initial guess using the measurements at $t_A$, $t_B$, and $t_C$. Next, refining this initial estimate with all measurements for $t<t_{cal}$. The following subsections provide a detailed description of these steps.

3.1. Initial Guess

Consider the sensor pose at $t_A$, $t_B$ and $t_C$. The axis of rotation between two orientations gives a well-defined and easily obtainable common direction for both the sensor and the TCP.

For this, compute the direction of axes as

$$[{\mathbf{t}}_{AB}^{\left(TCP\right)},{\phi }_{AB}]==\mathrm{axis}\_\mathrm{angle}\left({\mathbf{q}}^{(base,TCP)}{\left(t_A\right)}^{-1}{\mathbf{q}}^{(base,TCP)}\left(t_B\right)\right),$$

(2)

if the computation results in a negative ${\phi }_{AB}$ angle, flip the direction of ${\mathbf{t}}_{AB}^{\left(TCP\right)}$. (The computation requires that ${\phi }_{AB}$, ${\phi }_{BC}$ and angle of ${\mathbf{t}}_{AB}$ and ${\mathbf{t}}_{BC}$ to be as close to the right angle as possible.) Furthermore, vectors ${\mathbf{t}}_{BC}^{\left(TCP\right)}$, ${\mathbf{t}}_{AB}^{\left(sensor\right)}$ and ${\mathbf{t}}_{BC}^{\left(sensor\right)}$ can be computed similarly.

Then, these values can be obtained by $base$ and $ref$ frames as

$$\begin{array}{cc}\hfill & {\mathbf{t}}_{AB}^{\left(base\right)}={\mathbf{q}}^{(base,TCP)}\left(t_B\right)·{\mathbf{t}}_{AB}^{\left(TCP\right)},\hfill \\ \hfill & {\mathbf{t}}_{AB}^{\left(ref\right)}={\mathbf{q}}^{(ref,sensor)}\left(t_B\right)·{\mathbf{t}}_{AB}^{\left(sensor\right)},\hfill \\ \hfill & {\mathbf{t}}_{BC}^{\left(base\right)}={\mathbf{q}}^{(base,TCP)}\left(t_B\right)·{\mathbf{t}}_{BC}^{\left(TCP\right)},\hfill \\ \hfill & {\mathbf{t}}_{BC}^{\left(ref\right)}={\mathbf{q}}^{(ref,sensor)}\left(t_B\right)·{\mathbf{t}}_{BC}^{\left(sensor\right)}.\hfill \end{array}$$

Now the ${\mathbf{t}}_{AB}$ and ${\mathbf{t}}_{BC}$ directions are known by all frames; the following orthogonal directions can be defined based on them:

$$\begin{array}{cc}\hfill & {\mathbf{i}}_1={({\mathbf{t}}_{AB}+{\mathbf{t}}_{BC})}_{norm},\hfill \\ \hfill & {\mathbf{i}}_2={({\mathbf{t}}_{AB}\times {\mathbf{t}}_{BC})}_{norm},\hfill \\ \hfill & {\mathbf{i}}_3={\mathbf{i}}_1\times {\mathbf{i}}_2,\hfill \end{array}$$

(3)

that constructs a frame denoted by i. The method and the resulting frame are illustrated in Figure 3.

Determining these base vectors from the vectors given by the $ref$ frame, the rotation matrix between the frames $ref$ and i can be described as

$${\mathbf{R}}^{(ref,i)}=\left[\begin{array}{ccc}{\mathbf{i}}_1& {\mathbf{i}}_2& {\mathbf{i}}_3\end{array}\right].$$

(4)

It can also be computed for frames $base$, $TCP$ and $sensor$ as ${\mathbf{R}}^{(base,i)}$, ${\mathbf{R}}^{(TCP,i)}$ and ${\mathbf{R}}^{(sensor,i)}$.

Then the initial guess for quaternions ${\mathbf{q}}^{(base,ref)}$ and ${\mathbf{q}}^{(TCP,sensor)}$ can be computed as

$$\begin{array}{cc}\hfill & {\mathbf{q}}_0^{(base,ref)}=\mathrm{rotm}2\mathrm{quat}({\mathbf{R}}^{(base,i)}·{\left({\mathbf{R}}^{(ref,i)}\right)}^T),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & {\mathbf{q}}_0^{(TCP,sensor)}=\mathrm{rotm}2\mathrm{quat}({\mathbf{R}}^{(TCP,i)}·{\left({\mathbf{R}}^{(sensor,i)}\right)}^T).\hfill \end{array}$$

(6)

3.2. Local Optimization

The previous step provided a good calibration for the three selected measurements. The local optimization step refines this calibration by taking into account all measurements for $0\le t<t_{cal}$.

The correction of the orientations is defined with Roll-Pitch-Yaw variables ($\mathbf{x}$)

$$\begin{array}{c}{\mathbf{q}}_{RPY}\left(\mathbf{x}\right)=\left(cos(x_1/2)+\mathfrak{i}sin(x_1/2)\right)·\hfill \\ \hfill ·\left(cos(x_2/2)+\mathfrak{j}sin(x_2/2)\right)·\left(cos(x_3/2)+\mathfrak{k}sin(x_3/2)\right),\end{array}$$

(7)

where small angles are assumed due to the good initial guess.

With this correction term, the quaternions can be defined as

$$\begin{array}{cc}\hfill & {\mathbf{q}}^{(base,ref)}\left(\mathbf{x}\right)={\mathbf{q}}_{RPY}\left(\mathbf{x}\right)·{\mathbf{q}}_0^{(base,ref)},\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & {\mathbf{q}}^{(TCP,sensor)}\left(\mathbf{y}\right)={\mathbf{q}}_{RPY}\left(\mathbf{y}\right)·{\mathbf{q}}_0^{(TCP,sensor)}.\hfill \end{array}$$

(9)

According to the parameters $\mathbf{x}$ and $\mathbf{y}$, the measurements can be evaluated as

$$\begin{array}{cc}\hfill & {\mathbf{q}}^{(base,sensorbyrobot)}(t,\mathbf{x})={\mathbf{q}}^{(base,ref)}\left(\mathbf{x}\right){\mathbf{q}}^{(ref,sensor)}\left(t\right),\hfill \\ \hfill & {\mathbf{q}}^{(base,sensorbysensor)}(t,\mathbf{y})={\mathbf{q}}^{(base,TCP)}\left(t\right){\mathbf{q}}^{(TCP,sensor)}\left(\mathbf{y}\right),\hfill \end{array}$$

then the error of the measurement can be computed as the angle of a quaternion

$$ϵ(t,\mathbf{x},\mathbf{y})=\mathrm{angle}\left({\mathbf{q}}^{(sensorbysensor,sensorbyrobot)}(t,\mathbf{x},\mathbf{y})\right),$$

(10)

where

$$\begin{array}{c}{\mathbf{q}}^{(sensorbysensor,sensorbyrobot)}(t,\mathbf{x},\mathbf{y})=\hfill \\ \hfill ={\mathbf{q}}^{(base,sensorbysensor)}{(t,\mathbf{y})}^{-1}·{\mathbf{q}}^{(base,sensorbyrobot)}(t,\mathbf{x}).\end{array}$$

Based on this derivation, this optimization problem can be written as

$$\begin{array}{c}\hfill \underset{\mathbf{x}\in {\mathbb{R}}^3,\mathbf{y}\in {\mathbb{R}}^3}{\mathrm{minimize}}\sum _{t<t_{cal}}ϵ{(t,\mathbf{x},\mathbf{y})}^2,\end{array}$$

(11)

where a Nelder-Mead optimization can be used initialized from $\mathbf{x}=\mathbf{0}$, $\mathbf{y}=\mathbf{0}$.

From the resulted $\mathbf{x}$, $\mathbf{y}$ vectors, the error of a measurement can be written as

$$ϵ\left(t\right)=ϵ(t,\mathbf{x},\mathbf{y}).$$

(12)

4. Experimental Demonstration

This section presents examples of validating various orientation sensors using a UR16e manipulator [22] with the proposed method. The results provide a straightforward comparison of the devices’ performance. We investigated two sensors, a MEMS-based IMU sensor and the HTC VIVE VR headset equipped with 6 DoF tracking capabilities (so-called inside-out tracking).

4.1. ICM20948 IMU with Disabled Magnetometer

The IMU sensor ICM20948 [23] used in the experiment integrates gyroscope, magnetometer, and accelerometer units in a single package. For this measurement, the magnetometer was turned off because of the disturbing magnetic field of the manipulator. It was found that the micro-vibrations of the manipulator can influence the performance of the sensor to a considerable extent causing noise and drift in the output signal. For this reason, the sensor was attached to the manipulator in silicon bedding instead of using a rigid mounting fixture.

Furthermore, the effect of these vibrations was also diminished by applying a higher robot speed (2-4[cm]/step). Figure 4 shows the path of the manipulator. After an initial motion phase, it performs five full circles. The figure also shows the orientation in each measurement point with red line segments in the x direction and blue segments in the z direction.

First, three measurements (samples recorded at $t_A$, $t_B$, and $t_C$) must be chosen as close to perpendicular as possible.

It was obtained by performing the following optimization:

$$\begin{array}{cc}\hfill \underset{t_A,t_B,t_C<t_{cal}}{\mathrm{minimize}}& max\left(\right|{\delta }_{AB}|,|{\delta }_{BC}|,|{\delta }_{\mathbf{t}}\left|\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& {\delta }_{AB}=\pi /2-{\varphi }_{AB},\hfill \\ \hfill & {\delta }_{BC}=\pi /2-{\varphi }_{AB},\hfill \\ \hfill & {\delta }_{\mathbf{t}}=\pi /2-\mathrm{angle}({\mathbf{t}}_{AB},{\mathbf{t}}_{BC}),\hfill \end{array}$$

where in optimal case $\delta =max\left(\right|{\delta }_{AB}|,|{\delta }_{BC}|,|{\delta }_{\mathbf{t}}\left|\right)=0$, so the angles are perpendicular.

On the initial phase of the path, the procedure automatically looks for three measured orientations that are inclined at least 60 degrees and less than 120 degrees relative to each other (In this case, their indices were $i_A=1$, $i_B=374$ and $i_C=457$). The angle difference between the orientations are ${\phi }_{AB}=89.65\left[deg\right]$ and ${\phi }_{BC}=89.29\left[deg\right]$. Furthermore, the angle of ${\mathbf{t}}_{AB}$ and ${\mathbf{t}}_{BC}$ is $89.68\left[deg\right]$, because of the initial path segment, $\delta$ is almost zero.

With the results of calibration, the $ϵ\left(t\right)$ error of the orientation of the IMU sensor (considering the robot as a reference) can be computed, and it is plotted to Figure 5 with blue. The histogram of the error is plotted in Figure 6. They show well that the error is mostly below $1.5\left[deg\right]$, and by plotting the magnitude of rotations between the measurements (by red in Figure 5), it is easy to see that significant errors are only measured during fast rotations. These errors are partially caused by the unavoidable time difference between signal processing in the robot and the sensors that result in a certain time delay between the parallel orientation samples.

4.2. HTC VIVE (IMU and SLAM Sensor-Fusion Using 6 Cameras)

The HTC VIVE Cosmos headset uses six cameras and an IMU unit for its SLAM algorithm and can be considered a cutting-edge inside-out tracking technology. The headset was fixed to the flange of the manipulator and was moved on the path plotted in Figure 4, without the initial segment. Because it was not sensitive to vibrations of slow robot motions, two rounds were recorded at different speeds.

In this case, an automatic search was also applied to obtain orientation samples with $60\left[deg\right]\le \phi \le 120\left[deg\right]$ angle between them. The indices of the chosen measurements were $i_A=1$, $i_B=228$ and $i_C=1462$, the angles between them are ${\phi }_{AB}=30.03\left[deg\right]$ and ${\phi }_{BC}=36.29\left[deg\right]$. Furthermore, the angle of ${\mathbf{t}}_{AB}$ and ${\mathbf{t}}_{BC}$ is $30.01\left[deg\right]$, so in this case $\delta =60\left[deg\right]$.

After performing the calibration method described in Section 3, the computation results in $ϵ\left(t\right)$ errors plotted in Figure 7 with blue, and in histogram of Figure 8. The figures show that the error is mostly below $0.6\left[deg\right]$. The rotation angle between the measurements is also plotted in Figure 7 with red, and it shows well that the larger differences between the sensors are measured only if the device is rotating.

5. Future Work

Concerning the further research two directions are considered. The first goal is performing experimental work on the validation of standalone 6DoF tracking sensors, like [28] on a similar robotic setup extending the investigation to positioning accuracy. The other research direction aims to apply the presented approach for the geometric calibration of manipulators as well as the real-time monitoring of the consistency between the joint trajectory and the Cartesian trajectory of robot manipulator with known kinematic model. Such a monitoring method can be beneficially applied in mission critical medical robotic systems e.g., the CyberKnife [29].

6. Conclusion

The paper proposed a generic method to validate orientation sensors using an independent reference of higher accuracy. The method has two main steps. First, a good initial guess is computed via simple geometric computation, which is followed by a refinement via local optimization starting from the previously computed initial guess. The method is presented in a generic manner enabling the application with any kind of programmable manipulator and the associated orientation measurement instrumentation. Commercial robot arms are compact and reliable devices suitable for this purpose. The main benefits of the proposed method includes the repeatable excitation pattern that allows for the comparison of various orientation sensors under the same conditions regarding the orientation trajectory, external disturbances, visual environment, etc. The method is illustrated by validating an IMU (ICM20948) and a complex inside-out tracking system (HTC VIVE Cosmos) an UR 16e manipulator. The experiments has been shown that concerning the investigated sensors, significant orientation errors only occur at larger angular velocities.