Reprojection Error Analysis and Algorithm Optimization of Hand-Eye Calibration for Manipulator System

1. Introduction

2. Coordinate systems definitions and hand-eye calibration equation

2.1. Coordinate systems definitions

In this paper, the camera is installed in an eye-in-hand manner in the robotic arm visual grasping system. The origin of the base coordinate system of the robotic arm, the end coordinate system of the robotic arm, the camera coordinate system, and the AR marker coordinate system are defined as $O_b$、$O_e$、$O_c$、$O_m$, respectively. The coordinate system and its conversion relationship in the hand-eye calibration process are shown in Figure 1.

Let ${}_m{}^{c }T$be the transformation matrix of the AR marker coordinate system relative to the camera coordinate system. According to Figure 1, the transformation relationship between the AR marker coordinate system and the base coordinate system of the manipulator ${}_m{}^{b }T$ is:

(1)

2.2. Hand-eye Calibration equation

Formula 1,${}_{c }{}^{e }T$ is fixed, which is the hand-eye transformation matrix to be solved. If the position of the AR marker relative to the base coordinate system of the manipulator is unchanged, ${}_m{}^{b }T$is also fixed. For a certain state$S_i | i\in \mathbb{N}$ of the manipulator, ${{}_m{}^{c }T}_i$ can be calculated by using the size of the AR marker, the corner coordinates and the camera's internal parameters. Therefore, for a certain state$S_{i }$of the manipulator, the above Equation can be expressed as:

(2)

There are two fixed unknown matrices ${}_m{}^{b }T$ and ${}_{c }{}^{e }T$ in the above formula. In order to solve these two unknown matrices, it is necessary to control the manipulator to move to two different states, and the position of the AR marker should be kept unchanged during the movement. Using these two different states, the following equations are listed:

(3)

According to the above Equation, the following can be obtained:

(4)

The above Equation can be converted to:

(5)

Furthermore, the problem of solving the hand-eye transformation matrix${}_{c }{}^{e }T$is transformed into the problem of solving the homogeneous Equation AX=XB, where A, X and B are 4 × 4 homogeneous transformation matrices.

To further solve the homogeneous equation AX = XB, the homogeneous transformation matrix in Equation (5) is written in the form of a rotation matrix and translation vector:

(6)

By expanding the above formula, the equations to be solved can be obtained:

(7)

In the above Equation, $R_A$、$R_B$、$t_A$、$t_B$ can be measured, and $I$ is the unit matrix. There are many solutions to get rotation matrix $R$ and translation vector$\mathrm{t}$from this equation set, such as the Tsai-Lenz algorithm, Horaud algorithm, Andreff algorithm and Daniilidis algorithm. Next, we will carry out simulation experiments, and find an error analysis method to analyze and optimize the above four hand-eye calibration algorithms.

3. Reprojection error analysis method of calibration algorithms

3.1. Hand-eye calibration algorithm simulation experiments

In this paper, ROS and Gazebo simulation platforms are used to build the simulation environment to test the performance of the above four hand-eye calibration algorithms [17,20,24,25]. The true values of ${}_{c }{}^{e }T$ and ${}_m{}^{b }T$ in simulation experiments are shown in Table 1, and the calculation results of each calibration algorithm are shown in Table 2.

In order to quantitatively evaluate the performance of each calibration algorithm, the translation error ${err}_t$ of the hand-eye transformation matrix is defined as the two norms of the difference between the calculated value${\mathrm{t}}_{\mathrm{c}}$of the translation vector and the true value $t_r$. The translation error is measured by the Euclidean distance:

(8)

Similarly, the rotation matrix R is first converted to an Euler angle E, and the Euler angle is expressed in vector form as $E={[roll, pitch, yaw]}^T$, then the rotation error${err}_{\mathrm{R}}$of the hand-eye conversion matrix can be defined as:

(9)

According to the real values of the parameters in the simulation environment, the statistical results of the translation error and rotation error of the hand-eye transformation matrix calculated by each calibration algorithm are shown in Figure 2. It can be seen from the statistical figure that the translation error of the hand-eye conversion matrix calculated by the Tsai-Lenz and Andreff algorithms is significantly lower than the other two algorithms, but the rotation error of the hand-eye conversion matrix calculated by the Tsai-Lenz algorithm is slightly higher than the other algorithms. Overall, the calibration accuracy of the Tsai-Lenz and Andreff algorithm is relatively high in the simulation environment.

3.2. Heuristic error analysis

It can be seen from Figure 2 that each algorithm still has a certain optimization space. The following is a heuristic error analysis of the simulation results. In the eye-in-hand calibration process, the position of the calibration object relative to the base coordinate system of the manipulator is fixed. Therefore, in theory, ${}_m{}^{b }T$ should be a fixed value, but according to the coordinate transformation relationship shown in Figure 1, after calculating the hand-eye transformation matrix ${}_{c }{}^{e }T$,${}_m{}^{b }T$ can be calculated by the following formula:

(10)

The results are shown in Figure 3. It can be seen that the fluctuation range of the corresponding data of the Tsai-Lenz algorithm is small, which is in line with expectations. However, the Andreff algorithm has a large fluctuation range of corresponding data, which is inconsistent with expectations. Therefore, it is unreasonable to judge the error of the hand-eye conversion matrix ${}_{c }{}^{e }T$ by the fluctuation degree of data in set C, because the above conjecture actually ignores the influence of the error of ${}_m{}^{c }T$ on the calculation result of ${}_m{}^{b }T$.

3.3. Reprojection error analysis

From the results of heuristic error analysis, it can be seen that ${}_c{}^{e }T$ and ${}_m{}^{c }T$ may have some errors, so it is unreasonable to calculate ${}_m{}^{b }T$ by Equation 10. Since the position of the AR marker is fixed during the calibration process, the following error analysis process considers that ${}_m{}^{b }T$ is known and fixed.

According to the coordinate transformation relationship shown in Figure 1, the pose representation of the AR marker in the camera coordinate system can be obtained:

(11)

If the AR marker coordinate system $\left\{O_{marker}\right\}$is defined as the world coordinate system, then ${}_w{}^{c }T={}_m{}^{c }T$. According to the pinhole camera imaging model, the conversion relationship between the coordinates$\left(X_w,Y_w,Z_w\right)$and pixel coordinates $(u,v)$in the AR marker coordinate system and the z-axis coordinate${ Z}_c$in the camera coordinate system can be obtained as follows:

(12)

On the basis of the above definition, the coordinates of the origin$O_{marker }$of the AR marker coordinate system in the world coordinate system $\left(X_w,Y_w,Z_w\right)=\left(X_m,Y_m,Z_m\right)=\left(\mathrm{0,0},0\right)$, so the pixel coordinates $\left(u,v\right)$ of the AR marker center point can be calculated by the following formula:

(13)

In the above formula, $M$is the inherent property of the camera, ${}_e{}^{c }T$is the hand-eye conversion matrix to be calibrated, ${}_b{}^{e }T$ can be calculated by the forward kinematics equation of the manipulator, and ${}_m{}^{b }T$ is known and fixed.

Since the translation part of the homogeneous transformation matrix ${}_m{}^{b }T$reflects the coordinates of the AR marker center point in the base coordinate system of the manipulator, the function of Equation 13 is actually to remap the coordinates of the AR marker center point into the pixel coordinate system. For a certain position$P_i,i\in \mathbb{N}$, that the manipulator moves to during the calibration process, the AR mark image captured by the camera is denoted as${\mathrm{i}\mathrm{m}\mathrm{g}}_{\mathrm{i}}$, and the pixel coordinate after the reprojection of the AR mark center point in${img}_i$is denoted as $q_i={[u_i,v_i]}^T$, then:

(14)

Since the real pixel coordinate $Q_i={[U_i,V_i]}^T$ of the AR marker center point in ${\mathrm{i}\mathrm{m}\mathrm{g}}_{\mathrm{i}}$ can be obtained by corner detection or manual labelling, the reprojection error ${err\_proj}_i$ corresponding to${img}_i$can be defined as the Euclidean distance between the real pixel coordinates of the AR marker center point and the reprojection coordinates:

(15)

If the manipulator moves to N positions during the hand-eye calibration process, the average reprojection error can be defined as:

(16)

According to Equation (15) and Equation (16), the reprojection error of each group of simulation experiment data is calculated. The results are shown in Figure 4. The horizontal line in the figure reflects the average reprojection error of each calibration algorithm. It can be seen that the average reprojection error corresponding to the calculation results of Tsai-Lenz and Andreff algorithms is small, and the fluctuation of the reprojection error of each group of data is relatively small. In addition, from the previous analysis results, the Euclidean distance errors of these two algorithms are relatively small, which proves that the reprojection error can reflect the accuracy of the calibration results to a certain extent. In general, the smaller the reprojection error, the higher the accuracy of the calibration results. In the process of hand-eye calibration of the real manipulator, because the real value of the hand-eye transformation matrix cannot be obtained, the Euclidean distance error of calibration results cannot be calculated, and the reprojection error can be used as the evaluation standard of calibration accuracy.

3.4. Optimization calibration algorithm by minimizing reprojection error analysis

From the statistical results of the Euclidean distance error of each algorithm in Figure 1, it can be seen that even if the hand-eye calibration is carried out in the simulation environment, the translation error of the hand-eye conversion matrix calculated by different calibration algorithms is also quite different, and both are greater than 2mm, which indicates that each calibration algorithm still has a large optimization space.

In the process of hand-eye calibration in the simulation environment, the only error source is the pose calculation error of the AR marker. However, when using the above four common algorithms for calibration, it is considered that the calculated ${}_m{}^{c }T$is error-free, resulting in a certain error in the calibration results of each algorithm. In other words, the conventional hand-eye calibration algorithm pays more attention to versatility, and does not use the prior knowledge that the position of the AR marker is fixed in the calibration process, so it is difficult to obtain high-precision calibration results. The definition of the reprojection error of the hand-eye calibration results makes full use of this prior knowledge. According to the previous analysis, the smaller the average reprojection error is, the higher the accuracy of the calibration results. Therefore, by minimizing the reprojection error, the accuracy of the hand-eye calibration results may be improved.

Based on the above analysis, the following exploratory experiments are carried out by controlling variables to test whether smaller average reprojection errors can be obtained by adjusting the parameters in the calibrated hand-eye transformation matrix. In the experiment, the ${}_{ m}{}^{b }T$ used to calculate the reprojection error takes the real value in Table 1, and the translation parameters calibrated by the Tsai-Lenz algorithm in Table 2 are taken as the initial values. The three translation parameters $\mathrm{x}$、$\mathrm{y}$ and $\mathrm{z}$ are adjusted with the step length of 0.001m, and the adjusted hand-eye transformation matrix is substituted into Equation 15 and Equation 16 to calculate the average reprojection error of each group of samples in the simulation experiment. The experimental results are shown in Figure 5. The purple dotted line reflects the parameter value of the minimum point, and the black dotted line reflects the average reprojection error of the minimum point.

From Figure 5, it can be seen that by adjusting the translation parameters separately, the average reprojection error can indeed be reduced to a certain extent, and the translation parameters calibrated by the Tsai-Lenz algorithm are used as the initial values, which can reduce the search space of the parameters and help quickly find the translation parameters corresponding to the lowest point of the reprojection error. However, when$\mathrm{z}=$0.03858m, the average reprojection error gets the minimum, but this value obviously deviates from the real value ${\mathrm{z}}_{\mathrm{r}}=$0.0345m (in Table 2). Therefore, by adjusting x, y, z alone, it is not guaranteed that the translation parameters with higher accuracy can be obtained.

Next, we take 0.001 m as the step length, and adjust x, y, z parameters at the same time. The change rule of reprojection error is shown in Figure 6. The color of the data points in the figure reflects the size of reprojection error. It can be seen that the translation matrix that minimizes the average reprojection error is$t_m={[-0.07059, 0.00015, 0.03558]}^T$, and the corresponding average reprojection error ${err\_proj}_{avg}=$ 0.69861. According to Equation 8, the corresponding translation error ${\mathrm{e}\mathrm{r}\mathrm{r}}_{\mathrm{t}}=$ 0.00165 can be calculated. It can be seen from Figure 2 and Figure 4 that the translation error ${err}_t=$ 0.0022 and the average reprojection error ${err\_proj}_{avg}=$ 2.96867 of the calibration results are calculated by the Tsai-Lenz algorithm. It can be seen that there is a set of translation parameters $t_b=\left(x_b,y_b,z_b\right)$，that can minimize the average reprojection error, and the translation error of this set of parameters${ err}_t$is less than the parameters calibrated by the Tsai-Lenz algorithm. In other words, the accuracy of hand-eye calibration results can be improved by simultaneously adjusting the three parameters $\mathrm{x},\mathrm{y}$and$\mathrm{z}$to minimize the reprojection error.

In the calibration process, after a certain position $P_i$of the manipulator is determined, the four parameters $U_i, V_i, {{}_{b }{}^{e }T}_i, {}_m{}^{b }T$in Equation 15 are determined accordingly. Therefore, ${}_{e }{}^{c }T={{}_{c }{}^{e }T}^{-1}$ determines the size of the reprojection $err\_pro{j}_i$. According to Equation 16, if N positions are determined, then the magnitude of the average reprojection error $err\_pro{j}_{avg}$ is uniquely determined by ${}_{ c }{}^{e }T$, so the mapping function $\mathrm{f}$ of the hand-eye transformation matrix ${}_{c }{}^{e }T$ to the average reprojection error $err\_pro{j}_{avg}$ can be defined as:

(17)

Furthermore, ${}_{c }{}^{e }T$ can be uniquely determined by the translation parameter $t=\left(x,y,z\right)$ and the rotation parameter$r=\left(roll,pitch,yaw\right)$, so the mapping function $F$ of the translation and rotation parameters to the average reprojection error $err\_pro{j}_{avg}$can be defined:

(18)

Based on the above definition, this paper transforms the optimization problem of the hand-eye conversion matrix into the problem of finding the minimum value of the objective function$\mathrm{F}$, and optimizes the hand-eye conversion matrix from the perspective of minimizing the reprojection error.

Next, with a step size of 0.0001m, the three parameters $x$、$y$ and $z$ are adjusted to search for the translation parameter that minimizes the function $\mathrm{F}$. The optimal translation parameter$t_b=$ (-0.07058, 0.00039, 0.03483), and the corresponding average reprojection error ${err\_proj}_{avg}$is 0.36084, which is 87.845 % lower than that of the Tsai-Lenz algorithm. The translation error${ err}_t$is 0.0007836 m, which is 64.382 % lower than that of the Tsai-Lenz algorithm.

In the above optimization process, only the translation parameters in the hand-eye conversion matrix are adjusted. Because after many simulation calibration experiments, it is found that the translation parameters have a greater influence on the accuracy of the calibration results than the rotation parameters, and the number of translation parameters is less. In general, only adjusting the translation parameters can obtain ideal calibration results. Of course, if the translation and rotation parameters are adjusted at the same time, higher precision calibration results can be obtained theoretically, but this process will be very time-consuming.

4. Hand-eye calibration algorithm experiment

5. Summary

References

1. Ge, Pengxiang, et al. "Multivision Sensor Extrinsic Calibration Method With Non-Overlapping Fields of View Using Encoded 1D Target." IEEE Sensors Journal 22.13 (2022): 13519-13528. [CrossRef]
2. Bu, Lingbin, et al. "Concentric circle grids for camera calibration with considering lens distortion." Optics and Lasers in Engineering 140 (2021): 106527. [CrossRef]
3. Chen, Xiangcheng, et al. "Camera calibration with global LBP-coded phase-shifting wedge grating arrays." Optics and Lasers in Engineering 136 (2021): 106314. [CrossRef]
4. Zhang, Zhengyou. "A flexible new technique for camera calibration." IEEE Transactions on pattern analysis and machine intelligence 22.11 (2000): 1330-1334. [CrossRef]
5. Hartley, Richard I. "Self-calibration of stationary cameras." International journal of computer vision 22 (1997): 5-23. [CrossRef]
6. Li, Mengxiang. "Camera calibration of a head-eye system for active vision." Computer Vision—ECCV'94: Third European Conference on Computer Vision Stockholm, Sweden, May 2–6, 1994 Proceedings, Volume I 3. Springer Berlin Heidelberg, 1994. [CrossRef]
7. Hartley, Richard I. "Projective reconstruction and invariants from multiple images." IEEE Transactions on Pattern Analysis and Machine Intelligence 16.10 (1994): 1036-1041. [CrossRef]
8. Pollefeys, Marc, Luc Van Gool, and Andre Oosterlinck. "The modulus constraint: a new constraint self-calibration." Proceedings of 13th International Conference on Pattern Recognition. Vol. 1. IEEE, 1996. [CrossRef]
9. Triggs, Bill. "Autocalibration and the absolute quadric." Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 1997. [CrossRef]
10. Heyden, Anders, and K. Astrom. "Flexible calibration: Minimal cases for auto-calibration." Proceedings of the Seventh IEEE International Conference on Computer Vision. Vol. 1. IEEE, 1999. [CrossRef]
11. Pollefeys, Marc, Reinhard Koch, and Luc Van Gool. "Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters." International journal of computer vision 32.1 (1999): 7-25. [CrossRef]
12. Pollefeys, Marc. Self-calibration and metric 3D reconstruction from uncalibrated image sequences. Diss. PhD thesis, ESAT-PSI, KU Leuven, 1999.
13. Hartley, Richard I., et al. "Camera calibration and the search for infinity." Proceedings of the Seventh IEEE International Conference on Computer Vision. Vol. 1. IEEE, 1999. [CrossRef]
14. Wenyu Chen, Jia Du, Wei Xiong, Yue Wang, Shueching Chia, Bingbing Liu, Jierong Cheng, Ying Gu. "A Noise-Tolerant Algorithm for Robot-Sensor Calibration Using a Planar Disk of Arbitrary 3-D Orientation," in IEEE Transactions on Automation Science and Engineering, vol. 15, no. 1, pp. 251-263, Jan. 2018. [CrossRef]
15. Mingyang Li, Zhijiang Du, Xiaoxing Ma, Wei Dong, Yongzhuo Gao. A robot hand-eye calibration method of line laser sensor based on 3D reconstruction. Robotics and Computer-Integrated Manufacturing. Volume 71, October 2021, 102136. [CrossRef]
16. Yuan Zhang, Zhicheng Qiu, and Xianmin Zhang. Calibration method for hand-eye system with rotation and translation couplings. Applied Optics, Vol. 58, Issue 20, pp. 5375-5387, 2019. [CrossRef]
17. Andreff, Nicolas, Radu Horaud, and Bernard Espiau. "On-line hand-eye calibration." Second International Conference on 3-D Digital Imaging and Modeling (Cat. No. PR00062). IEEE, 1999.
18. Tabb, Amy, and Khalil M. Ahmad Yousef. "Solving the robot-world hand-eye (s) calibration problem with iterative methods." Machine Vision and Applications 28.5-6 (2017): 569-590. [CrossRef]
19. Jianfeng Jiang, Xiao Luo, Shijie Xu, Qingsheng Luo, Minghao Li. "Hand-Eye Calibration of EOD Robot by Solving the AXB = YCZD Problem," in IEEE Access, vol. 10, pp. 3415-3429, 2022. [CrossRef]
20. Tsai, Roger Y., and Reimar K. Lenz. "A new technique for fully autonomous and efficient 3 d robotics hand/eye calibration." IEEE Transactions on robotics and automation 5.3 (1989): 345-358. [CrossRef]
21. Yan Liu, Qinglin Wang, Yuan Li. A method for hand-eye calibration of a robot vision measuring system. 2015 10th Asian Control Conference (ASCC), Kota Kinabalu, Malaysia, 2015, pp. 1-6. [CrossRef]
22. Yanbiao Zou, Xiangzhi Chen. "Hand–eye calibration of arc welding robot and laser vision sensor through semidefinite programming", Industrial Robot, Vol. 45 No. 5, pp. 597-610, 2019. [CrossRef]
23. Shichao Deng, Feng Mei, Long Yang, Chenguang Liang, Yingliang Jiang, Gaoxing Yu, Yihai Chen. "Research on the Hand-eye calibration Method Based on Monocular Robot". Journal of Physics: Conference Series, Volume 1820, 2021 International Conference on Mechanical Engineering, Intelligent Manufacturing and Automation Technology (MEMAT), January 15-17, 2021, Guilin, China. [CrossRef]
24. Horaud, Radu, and Fadi Dornaika. "Hand-eye calibration." The international journal of robotics research 14.3 (1995): 195-210. [CrossRef]
25. Daniilidis, Konstantinos. "Hand-eye calibration using dual quaternions." The International Journal of Robotics Research 18.3 (1999): 286-298. [CrossRef]