Reprojection Error Analysis and Algorithm Optimization of Hand-Eye Calibration for Manipulator System

1. Introduction

Robotic grasping is a crucial function of robots, and it involves identifying and locating the target object through a visual sensor. To ensure the accuracy of the visual information and achieve coordinated hand-eye motion, it is essential to analyze the manipulator's hand-eye visual calibration problem and improve its accuracy. Hand-eye calibration establishes the transformation matrix between the camera's pixel coordinate system and the manipulator's spatial coordinate system. By transforming pixel coordinates into the manipulator's coordinate system, the robot can calculate the necessary motor movements to reach the target position and control the manipulator. Hand-eye calibration can be divided into eye-to-hand and eye-in-hand, depending on the camera's installation position. This paper focuses on experiments using the eye-in-hand installation.

Traditional calibration methods build calibration models based on pre-set imaging scenes, and select optimal algorithms, including reference-based, active vision and self-calibration methods, to calculate camera parameters based on scene geometric constraints. Based on the reference object camera calibration method, the corner points of the target image are extracted as control points, a system of equations corresponding to the pixels and spatial coordinates is constructed, and then the optimization algorithm is used to calculate the parameters, in which the shape and size of the reference object and other information are known. The reference object is divided into one-dimensional straight line [1], two-dimensional plane calibration plate, three-dimensional solid block, etc. according to the spatial dimension. Due to its simple production and controllable accuracy, flat calibration plates are often used as targets in industrial applications instead of calibration blocks. Commonly used calibration plate patterns include checkerboards, solid circles [2], concentric rings, etc. In recent years, various new templates have been proposed [3]. The Zhang Zhengyou calibration method [4] based on the chessboard calibration board is a classic representative of this type of method. This method has strong imaging constraints, simple calibration process, and high algorithm robustness. This type of method has strong imaging constraints, simple calibration process, and high algorithm robustness, but the production and maintenance cost of high-precision reference objects is high, and it is lost in situations where it is impossible to carry reference objects significance. The camera calibration method based on active vision obtains multiple images by accurately controlling special movements such as pure rotation and translation of the camera or target, and uses controllable quantitative motion constraints to determine the internal and external information of the camera. It is an important branch of self calibration methods. Typical methods include calibration based on pure rotational motion [5], calibration based on three orthogonal translation motions, calibration based on plane orthogonal motion, and calibration based on infinite plane homography matrix, hand eye calibration [6], self calibration method based on projection reconstruction, etc. Active visual calibration technology can linearly solve camera internal parameters, with strong algorithm robustness. However, strict requirements for control equipment limit its use and promotion. The camera self-calibration method does not need to set up a reference object or control the precise movement of the camera. It only uses the geometric consistency constraints of corresponding points in multiple frames of images [7,8] to solve the camera basic matrix, and does not rely on scene structure and motion information, including directly solving the Kruppa equation, methods based on absolute quadratic curves and absolute quadric surfaces [9,10], Pollefeys module constrained calibration [11,12], and hierarchical stepwise calibration methods under variable internal parameters [13] etc.

The focus of this paper is the hand-eye calibration algorithm. Since 1989, first proposed the problem of hand-eye calibration, domestic and foreign scholars have done a lot of research, such as Chen et al. proposed a noise-tolerant algorithm for robot sensor calibration using a planar disk in any three-dimensional direction [14], Li et al. proposed a hand-eye calibration method for linear laser sensors based on three-dimensional reconstruction [15] and Zhang et al. proposed a calibration method of hand-eye system with rotation and translation coupling [16]. The solution of the hand-eye calibration problems can be divided into two categories according to the solution order of the calibration matrix. One is to solve the rotation part and translation vector of the matrix at the same time. A typical algorithm is Andreff 's closed-loop solution of hand-eye calibration equation based on matrix direct product for calibration of small-scale moving measurement scenarios [17]; Tabb proposed a hand-eye calibration algorithm based on an iterative optimization method and solved it with a nonlinear optimizer [18]; Jiang et al. proposed a method to calibrate the hand-eye of EOD robot by solving the AXB = YCZD problem [19]. The other is to solve the rotation matrix of the calibration matrix first and then the translation vector of the calibration matrix. The most common one is the method proposed by Tsai and Lenz [20], which introduces the rotation axis-rotation angle system to describe the rotation motion; Liu et al. proposed a hand-eye calibration method for robot vision measurement system [21]; Zou et al. performed hand eye calibration on arc welding robots and laser vision sensors through semi definite programming [22]; Deng et al. proposed a hand-eye calibration method based on monocular robot [23]; But in fact, the above algorithm still has some space for optimization.

In the hand-eye calibration process of the real manipulator, the actual error of the calibration result cannot be calculated because the true value of the hand-eye transformation matrix cannot be obtained. A new error analysis method is needed. Therefore, a reprojection error analysis method is proposed. The contribution of this paper is to find that the only source of error in the process of hand-eye calibration in the simulation environment is the position and attitude calculation error of the AR marker. However, when using common algorithms for calibration, it is considered that the calculated pose is error-free, resulting in a certain error in the calibration results of each algorithm. The method of this paper innovatively utilizes the prior knowledge that the position marked by AR is fixed in the calibration process to conduct error analysis. First, the coordinates of the center point of the AR mark are re-projected to the pixel coordinate system, and then compared with the real pixel coordinates of the center point of the AR mark obtained by corner detection or manual labeling in order to obtain the Euclidean distance between the two coordinates, which is the basis for error analysis.

2. Coordinate systems definitions and hand-eye calibration equation

2.1. Coordinate systems definitions

In this paper, the camera is installed in an eye-in-hand manner in the robotic arm visual grasping system. The origin of the base coordinate system of the robotic arm, the end coordinate system of the robotic arm, the camera coordinate system, and the AR marker coordinate system are defined as $O_b$、$O_e$、$O_c$、$O_m$, respectively. The coordinate system and its conversion relationship in the hand-eye calibration process are shown in Figure 1.

Let ${}_m{}^{c }T$be the transformation matrix of the AR marker coordinate system relative to the camera coordinate system. According to Figure 1, the transformation relationship between the AR marker coordinate system and the base coordinate system of the manipulator ${}_m{}^{b }T$ is:

(1)

2.2. Hand-eye Calibration equation

Formula 1,${}_{c }{}^{e }T$ is fixed, which is the hand-eye transformation matrix to be solved. If the position of the AR marker relative to the base coordinate system of the manipulator is unchanged, ${}_m{}^{b }T$is also fixed. For a certain state$S_i | i\in \mathbb{N}$ of the manipulator, ${{}_m{}^{c }T}_i$ can be calculated by using the size of the AR marker, the corner coordinates and the camera's internal parameters. Therefore, for a certain state$S_{i }$of the manipulator, the above Equation can be expressed as:

(2)

There are two fixed unknown matrices ${}_m{}^{b }T$ and ${}_{c }{}^{e }T$ in the above formula. In order to solve these two unknown matrices, it is necessary to control the manipulator to move to two different states, and the position of the AR marker should be kept unchanged during the movement. Using these two different states, the following equations are listed:

(3)

According to the above Equation, the following can be obtained:

(4)

The above Equation can be converted to:

(5)

Furthermore, the problem of solving the hand-eye transformation matrix${}_{c }{}^{e }T$is transformed into the problem of solving the homogeneous Equation AX=XB, where A, X and B are 4 × 4 homogeneous transformation matrices.

To further solve the homogeneous equation AX = XB, the homogeneous transformation matrix in Equation (5) is written in the form of a rotation matrix and translation vector:

(6)

By expanding the above formula, the equations to be solved can be obtained:

(7)

In the above Equation, $R_A$、$R_B$、$t_A$、$t_B$ can be measured, and $I$ is the unit matrix. There are many solutions to get rotation matrix $R$ and translation vector$\mathrm{t}$from this equation set, such as the Tsai-Lenz algorithm, Horaud algorithm, Andreff algorithm and Daniilidis algorithm. Next, we will carry out simulation experiments, and find an error analysis method to analyze and optimize the above four hand-eye calibration algorithms.

3. Reprojection error analysis method of calibration algorithms

3.1. Hand-eye calibration algorithm simulation experiments

In this paper, ROS and Gazebo simulation platforms are used to build the simulation environment to test the performance of the above four hand-eye calibration algorithms [17,20,24,25]. The true values of ${}_{c }{}^{e }T$ and ${}_m{}^{b }T$ in simulation experiments are shown in Table 1, and the calculation results of each calibration algorithm are shown in Table 2.

In order to quantitatively evaluate the performance of each calibration algorithm, the translation error ${err}_t$ of the hand-eye transformation matrix is defined as the two norms of the difference between the calculated value${\mathrm{t}}_{\mathrm{c}}$of the translation vector and the true value $t_r$. The translation error is measured by the Euclidean distance:

(8)

Similarly, the rotation matrix R is first converted to an Euler angle E, and the Euler angle is expressed in vector form as $E={[roll, pitch, yaw]}^T$, then the rotation error${err}_{\mathrm{R}}$of the hand-eye conversion matrix can be defined as:

(9)

According to the real values of the parameters in the simulation environment, the statistical results of the translation error and rotation error of the hand-eye transformation matrix calculated by each calibration algorithm are shown in Figure 2. It can be seen from the statistical figure that the translation error of the hand-eye conversion matrix calculated by the Tsai-Lenz and Andreff algorithms is significantly lower than the other two algorithms, but the rotation error of the hand-eye conversion matrix calculated by the Tsai-Lenz algorithm is slightly higher than the other algorithms. Overall, the calibration accuracy of the Tsai-Lenz and Andreff algorithm is relatively high in the simulation environment.

3.2. Heuristic error analysis

It can be seen from Figure 2 that each algorithm still has a certain optimization space. The following is a heuristic error analysis of the simulation results. In the eye-in-hand calibration process, the position of the calibration object relative to the base coordinate system of the manipulator is fixed. Therefore, in theory, ${}_m{}^{b }T$ should be a fixed value, but according to the coordinate transformation relationship shown in Figure 1, after calculating the hand-eye transformation matrix ${}_{c }{}^{e }T$,${}_m{}^{b }T$ can be calculated by the following formula:

(10)

The results are shown in Figure 3. It can be seen that the fluctuation range of the corresponding data of the Tsai-Lenz algorithm is small, which is in line with expectations. However, the Andreff algorithm has a large fluctuation range of corresponding data, which is inconsistent with expectations. Therefore, it is unreasonable to judge the error of the hand-eye conversion matrix ${}_{c }{}^{e }T$ by the fluctuation degree of data in set C, because the above conjecture actually ignores the influence of the error of ${}_m{}^{c }T$ on the calculation result of ${}_m{}^{b }T$.

3.3. Reprojection error analysis

From the results of heuristic error analysis, it can be seen that ${}_c{}^{e }T$ and ${}_m{}^{c }T$ may have some errors, so it is unreasonable to calculate ${}_m{}^{b }T$ by Equation 10. Since the position of the AR marker is fixed during the calibration process, the following error analysis process considers that ${}_m{}^{b }T$ is known and fixed.

According to the coordinate transformation relationship shown in Figure 1, the pose representation of the AR marker in the camera coordinate system can be obtained:

(11)

If the AR marker coordinate system $\left\{O_{marker}\right\}$is defined as the world coordinate system, then ${}_w{}^{c }T={}_m{}^{c }T$. According to the pinhole camera imaging model, the conversion relationship between the coordinates$\left(X_w,Y_w,Z_w\right)$and pixel coordinates $(u,v)$in the AR marker coordinate system and the z-axis coordinate${ Z}_c$in the camera coordinate system can be obtained as follows:

(12)

On the basis of the above definition, the coordinates of the origin$O_{marker }$of the AR marker coordinate system in the world coordinate system $\left(X_w,Y_w,Z_w\right)=\left(X_m,Y_m,Z_m\right)=\left(\mathrm{0,0},0\right)$, so the pixel coordinates $\left(u,v\right)$ of the AR marker center point can be calculated by the following formula:

(13)

In the above formula, $M$is the inherent property of the camera, ${}_e{}^{c }T$is the hand-eye conversion matrix to be calibrated, ${}_b{}^{e }T$ can be calculated by the forward kinematics equation of the manipulator, and ${}_m{}^{b }T$ is known and fixed.

Since the translation part of the homogeneous transformation matrix ${}_m{}^{b }T$reflects the coordinates of the AR marker center point in the base coordinate system of the manipulator, the function of Equation 13 is actually to remap the coordinates of the AR marker center point into the pixel coordinate system. For a certain position$P_i,i\in \mathbb{N}$, that the manipulator moves to during the calibration process, the AR mark image captured by the camera is denoted as${\mathrm{i}\mathrm{m}\mathrm{g}}_{\mathrm{i}}$, and the pixel coordinate after the reprojection of the AR mark center point in${img}_i$is denoted as $q_i={[u_i,v_i]}^T$, then:

(14)

Since the real pixel coordinate $Q_i={[U_i,V_i]}^T$ of the AR marker center point in ${\mathrm{i}\mathrm{m}\mathrm{g}}_{\mathrm{i}}$ can be obtained by corner detection or manual labelling, the reprojection error ${err\_proj}_i$ corresponding to${img}_i$can be defined as the Euclidean distance between the real pixel coordinates of the AR marker center point and the reprojection coordinates:

(15)

If the manipulator moves to N positions during the hand-eye calibration process, the average reprojection error can be defined as:

(16)

According to Equation (15) and Equation (16), the reprojection error of each group of simulation experiment data is calculated. The results are shown in Figure 4. The horizontal line in the figure reflects the average reprojection error of each calibration algorithm. It can be seen that the average reprojection error corresponding to the calculation results of Tsai-Lenz and Andreff algorithms is small, and the fluctuation of the reprojection error of each group of data is relatively small. In addition, from the previous analysis results, the Euclidean distance errors of these two algorithms are relatively small, which proves that the reprojection error can reflect the accuracy of the calibration results to a certain extent. In general, the smaller the reprojection error, the higher the accuracy of the calibration results. In the process of hand-eye calibration of the real manipulator, because the real value of the hand-eye transformation matrix cannot be obtained, the Euclidean distance error of calibration results cannot be calculated, and the reprojection error can be used as the evaluation standard of calibration accuracy.

3.4. Optimization calibration algorithm by minimizing reprojection error analysis

From the statistical results of the Euclidean distance error of each algorithm in Figure 1, it can be seen that even if the hand-eye calibration is carried out in the simulation environment, the translation error of the hand-eye conversion matrix calculated by different calibration algorithms is also quite different, and both are greater than 2mm, which indicates that each calibration algorithm still has a large optimization space.

In the process of hand-eye calibration in the simulation environment, the only error source is the pose calculation error of the AR marker. However, when using the above four common algorithms for calibration, it is considered that the calculated ${}_m{}^{c }T$is error-free, resulting in a certain error in the calibration results of each algorithm. In other words, the conventional hand-eye calibration algorithm pays more attention to versatility, and does not use the prior knowledge that the position of the AR marker is fixed in the calibration process, so it is difficult to obtain high-precision calibration results. The definition of the reprojection error of the hand-eye calibration results makes full use of this prior knowledge. According to the previous analysis, the smaller the average reprojection error is, the higher the accuracy of the calibration results. Therefore, by minimizing the reprojection error, the accuracy of the hand-eye calibration results may be improved.

Based on the above analysis, the following exploratory experiments are carried out by controlling variables to test whether smaller average reprojection errors can be obtained by adjusting the parameters in the calibrated hand-eye transformation matrix. In the experiment, the ${}_{ m}{}^{b }T$ used to calculate the reprojection error takes the real value in Table 1, and the translation parameters calibrated by the Tsai-Lenz algorithm in Table 2 are taken as the initial values. The three translation parameters $\mathrm{x}$、$\mathrm{y}$ and $\mathrm{z}$ are adjusted with the step length of 0.001m, and the adjusted hand-eye transformation matrix is substituted into Equation 15 and Equation 16 to calculate the average reprojection error of each group of samples in the simulation experiment. The experimental results are shown in Figure 5. The purple dotted line reflects the parameter value of the minimum point, and the black dotted line reflects the average reprojection error of the minimum point.

From Figure 5, it can be seen that by adjusting the translation parameters separately, the average reprojection error can indeed be reduced to a certain extent, and the translation parameters calibrated by the Tsai-Lenz algorithm are used as the initial values, which can reduce the search space of the parameters and help quickly find the translation parameters corresponding to the lowest point of the reprojection error. However, when$\mathrm{z}=$0.03858m, the average reprojection error gets the minimum, but this value obviously deviates from the real value ${\mathrm{z}}_{\mathrm{r}}=$0.0345m (in Table 2). Therefore, by adjusting x, y, z alone, it is not guaranteed that the translation parameters with higher accuracy can be obtained.

Next, we take 0.001 m as the step length, and adjust x, y, z parameters at the same time. The change rule of reprojection error is shown in Figure 6. The color of the data points in the figure reflects the size of reprojection error. It can be seen that the translation matrix that minimizes the average reprojection error is$t_m={[-0.07059, 0.00015, 0.03558]}^T$, and the corresponding average reprojection error ${err\_proj}_{avg}=$ 0.69861. According to Equation 8, the corresponding translation error ${\mathrm{e}\mathrm{r}\mathrm{r}}_{\mathrm{t}}=$ 0.00165 can be calculated. It can be seen from Figure 2 and Figure 4 that the translation error ${err}_t=$ 0.0022 and the average reprojection error ${err\_proj}_{avg}=$ 2.96867 of the calibration results are calculated by the Tsai-Lenz algorithm. It can be seen that there is a set of translation parameters $t_b=\left(x_b,y_b,z_b\right)$，that can minimize the average reprojection error, and the translation error of this set of parameters${ err}_t$is less than the parameters calibrated by the Tsai-Lenz algorithm. In other words, the accuracy of hand-eye calibration results can be improved by simultaneously adjusting the three parameters $\mathrm{x},\mathrm{y}$and$\mathrm{z}$to minimize the reprojection error.

In the calibration process, after a certain position $P_i$of the manipulator is determined, the four parameters $U_i, V_i, {{}_{b }{}^{e }T}_i, {}_m{}^{b }T$in Equation 15 are determined accordingly. Therefore, ${}_{e }{}^{c }T={{}_{c }{}^{e }T}^{-1}$ determines the size of the reprojection $err\_pro{j}_i$. According to Equation 16, if N positions are determined, then the magnitude of the average reprojection error $err\_pro{j}_{avg}$ is uniquely determined by ${}_{ c }{}^{e }T$, so the mapping function $\mathrm{f}$ of the hand-eye transformation matrix ${}_{c }{}^{e }T$ to the average reprojection error $err\_pro{j}_{avg}$ can be defined as:

(17)

Furthermore, ${}_{c }{}^{e }T$ can be uniquely determined by the translation parameter $t=\left(x,y,z\right)$ and the rotation parameter$r=\left(roll,pitch,yaw\right)$, so the mapping function $F$ of the translation and rotation parameters to the average reprojection error $err\_pro{j}_{avg}$can be defined:

(18)

Based on the above definition, this paper transforms the optimization problem of the hand-eye conversion matrix into the problem of finding the minimum value of the objective function$\mathrm{F}$, and optimizes the hand-eye conversion matrix from the perspective of minimizing the reprojection error.

Next, with a step size of 0.0001m, the three parameters $x$、$y$ and $z$ are adjusted to search for the translation parameter that minimizes the function $\mathrm{F}$. The optimal translation parameter$t_b=$ (-0.07058, 0.00039, 0.03483), and the corresponding average reprojection error ${err\_proj}_{avg}$is 0.36084, which is 87.845 % lower than that of the Tsai-Lenz algorithm. The translation error${ err}_t$is 0.0007836 m, which is 64.382 % lower than that of the Tsai-Lenz algorithm.

In the above optimization process, only the translation parameters in the hand-eye conversion matrix are adjusted. Because after many simulation calibration experiments, it is found that the translation parameters have a greater influence on the accuracy of the calibration results than the rotation parameters, and the number of translation parameters is less. In general, only adjusting the translation parameters can obtain ideal calibration results. Of course, if the translation and rotation parameters are adjusted at the same time, higher precision calibration results can be obtained theoretically, but this process will be very time-consuming.

4. Hand-eye calibration algorithm experiment

4.1. Calibration process and results

Under the premise that the internal parameters of the camera have been calibrated, the hand-eye calibration experiment is performed in a real environment. The experimental configuration is shown in Figure 7a, and the AR mark is placed on the workbench in front of the robotic arm. In order to obtain the real position of the AR marker, an auxiliary calibration tool is installed at the end of the manipulator, and then the manipulator is manually controlled to align the top of the calibration tool with the center point of the AR marker (as shown in Figure 7b). Finally, the forward kinematics equation of the manipulator is used to calculate the translation matrix of the center point of the AR marker relative to the base coordinate system of the manipulator. The translation matrix is t_r=[0.53514m, 0.00406m, 0.25409m]^T

According to the experimental results and error analysis results in the simulation environment, the optimized hand-eye calibration process is as follows:

• Using the auxiliary calibration tool, the real translation matrix$t_r$of the AR marker coordinate system relative to the base coordinate system of the manipulator is obtained, and then the position of the AR marker is kept unchanged.
• The manipulator is controlled to move to 20 different states where the corner information of the AR marker can be detected, and the corresponding 20 groups of coordinate system transformation data are collected and recorded.
• The mean value ${{}_m{}^{b }T}_{avg}$of ${}_m{}^{b }T$is calculated by using the coordinate transformation data of each group, and the translation matrix in ${{}_m{}^{b }T}_{avg}$ is replaced by $t_r$to obtain ${{}_m{}^{b }T}_{proj}$ for calculating the reprojection error.
• The Tsai-Lenz algorithm is used to calculate the initial value of the hand-eye transformation matrix ${{}_{c }{}^{e }T}_{init}$, and then its translation parameters are automatically adjusted to minimize the average reprojection error. The optimized hand-eye transformation matrix${ {}_{c }{}^{e }T}_{optimized}$.

Since it is difficult to obtain the true value of the rotation parameter in ${}_m{}^{b }T$ in the real environment, the average value is taken when calculating the reprojection error in the above calibration process. According to the above process, the hand-eye calibration experiment is carried out in the real environment, and the hand-eye conversion matrix ${}_c{}^{e}T$ is calculated by using the four traditional algorithms and the optimized algorithms mentioned above respectively. The results are shown in Table 3.

4.2. Reprojection error analysis

In order to evaluate the performance difference between the traditional algorithm and the optimized algorithm in the real environment, the reprojection errors corresponding to each hand-eye transformation matrix in Table 3 are calculated by using the coordinate transformation data of each group. The results are shown in Figure 8, where the horizontal line reflects the average reprojection error of each method. Because the difference between the hand-eye conversion matrix calculated by the Tsai-Lenz and Horaud algorithms is very small, only the reprojection error corresponding to the Tsai-Lenz algorithm is drawn in the figure. It can be seen that in the real environment, except for the Andreff algorithm, the performance of the other traditional algorithms is close. It is worth mentioning that the average reprojection error of the optimized algorithm is reduced by 44.43% compared with the Tsai-Lenz algorithm.

4.3. Visual positioning error analysis

In the real scene, the calibration results of each algorithm are used to test the positioning accuracy of the manipulator visual positioning. During the test, the AR marker is moved several times, and the state of the manipulator is adjusted to ensure that the corner information of the AR marker can be detected. Then the position pc of the center point of the AR marker in the base coordinate system of the manipulator is calculated by Equation 2. Finally, manually control the manipulator and use the auxiliary calibration tool to obtain the real position pr of the AR marker center point.

The positioning accuracy of the manipulator in the real scene is quantitatively evaluated, and the visual positioning error is defined as the two norms of the difference between the calculated value pc and the real value pr of the AR mark center point position.

In the actual test process, 10 sets of data are collected, and the hand-eye conversion matrix calculated by each algorithm is used for the visual positioning of the manipulator. The error statistics are shown in Figure 9. The horizontal line in the figure reflects the average visual positioning error corresponding to each algorithm. It can be seen that the optimized hand-eye calibration method can significantly reduce the visual positioning error of the manipulator. Compared with the traditional Tsai-Lenz algorithm, the average visual positioning error is reduced by 50.63%.

5. Summary

When using common algorithms for hand-eye calibration, it is believed that the calculated ${}_m{}^{c }T$ is error-free, which leads to certain errors in the calibration results of each algorithm. In this paper, the coordinates of the center point of the AR mark are re-projected into the pixel coordinate system, and then compared with the real pixel coordinates of the center point of the AR mark obtained by corner detection or manual labeling, in order to obtain the Euclidean distance between the two coordinates as the basis for error analysis. The method in this paper can reduce the hand-eye calibration error and improve the operation accuracy of the manipulator based on vision.