Multilayer Perceptron-based Error Compensation for Automatic On-the-Fly Camera Orientation Estimation using a Single Vanishing Point from Road Lane

1. Introduction

With the advancements in vision sensor technology, the utilization of vision sensors in advanced driving-assistant systems (ADAS) has become prevalent. The organic incorporation of vision-based sensing technologies such as lane-keeping systems (LKS) and forward collision warnings (FCW) into self-driving scenarios has begun to improve. Most functions are based on vision systems, such as cameras, to obtain the visual context of human perception. Although this pipeline approach of streamlining vision-based inputs to obtain plausible outcomes is common in the ADAS-based industry for accomplishing multiple tasks, such as object detection and segmentation. This is continuously widening the sensing capabilities of self-driving systems [2]. Lane detection is a basic task of an ADAS to identify lane boundaries and extend these to attain LKS and warning systems. Commercial products such as open pilots can be considered successful cases of combining object detection and semantic segmentation into a product with vertical coverage from sensing to control. The simulator also contributes significantly to the regulation and simulation of control systems [3].

Several aspects of the camera, such as its intrinsic and extrinsic parameters (including its position and orientation), are typically used to transform points in the camera coordinate system into a world coordinate system. Conventional camera calibration methods require a calibration target (typically a 2D plane of known size and shape, such as a checkerboard) to determine the intrinsic parameters of the camera. This target is placed in the camera field-of-view (FOV), and a series of images with different angles and distances are acquired. Therefore, geometric transformation techniques can estimate the intrinsic and extrinsic parameters. However, the position and orientation of cameras are still not in practice, particularly when an autonomous vehicle travels through different environments. Therefore, a method for updating camera parameters during real-time online camera calibration is required [4]. The camera parameters can remain updated by updating the intrinsic and extrinsic parameters in real-time. This would ensure a more accurate and reliable autonomous driving system [5].

Camera calibration is also a sub-problem in computer vision. It mainly analyzes known scenes from a sensor from various perspectives and uses these to calibrate the sensor. For example, in the FCW, a practical calculation of the distance between the vehicle and the object ensures a safe distance for the vehicle. This can be affected by the terrain and weather conditions. Single-camera distance estimation assumes that the object and vehicle are on the same plane and projects a 3D object onto a 2D image using calibrated intrinsic and extrinsic parameters of the camera [6]. Because the intrinsic parameters would have been established at the time of production, these do not vary under normal conditions. Meanwhile, the extrinsic parameters represent the relationship between the target coordinate system and the camera coordinate system. The extrinsic parameters are generally updated according to the vehicle’s motion.

There are two methods to calibrate the camera automatically using targetless methods: 1) identify the vanishing point (VP) to estimate the angles (orientations) and 2) determine the area of the object to estimate the yaw and pitch (as described in [7]). Hold et al. used periodic dashed lanes to estimate the initial extrinsic parameters [8]. Paula et al. developed a model to estimate using the VP [9]. Lee et al. estimated the pitch and yaw using a new method in conjunction with VPs [10]. Jang et al. estimated the three angles using three VPs [11]. Guo et al. [12] also developed indoor applications using VPs. These methods accumulate errors in the camera orientation estimation and do not consider error compensation factors.

The contributions of this study are as follows:

• A working pipeline for the multilayer perceptron (MLP)-based adaptive error correction automatic on-the-fly camera orientation estimation algorithm with a single VP from a road lane is proposed (see Figure 1).
• A stable camera on-the-fly orientation estimation variant is proposed. It uses a Kalman filter that can estimate the angular pitch and yaw with respect to the road lane.
• The residual error of using VP to estimate the camera orientation is compensated, and several related compensation modules are compared.

The remainder of the paper is organized as follows. The related work is presented in Section 2. This is followed by the proposed method, including the rotation estimation, system of estimation, and evaluation metrics in Section 3.1. The experimental results are presented in Section 4. The performance of the proposed system and ablation study are discussed in Section 5. Section 6 discusses the limitations of the present study and future work. Section 7 which concludes this study.

2. Related work

Fundamental camera calibration can be categorized into two types: intrinsic (focal length, camera center, and distortions) [13] and extrinsic, which involves mapping rotations and translations from the camera to the world coordinates [14,15] or other sensor coordinates [16,17]. This study focuses on the orientation between the camera and world coordinates. To obtain rotations and translations for online camera calibration, targets such as registered objects [18,19], road marker lanes [20], and objects with apparent appearances [21]are commonly used. However, these methods can not cover the case regarding varying extrinsic parameters. Paula et al. proposed another model to estimate the pitch, yaw, and height for the target-less calibration methods. The generated case was used for extrinsic parameter estimation. Lee et al. proposed an online extrinsic camera calibration method that estimates pitch, yaw, roll angles, and camera height from road surface observations by using a combination of VP estimation and lane width prior. Guo et al. used a more detailed version to estimate the absolute orientation angle. It included the pitch and yaw. They used a method similar to that of Lee et al. and analyzed the failure cases. Although they utilized specific applications considering auto-calibration, their approach had significant errors. These methods cannot prevent projection errors even with an appropriate VP. Jang et al. proposed an online extrinsic calibration approach for estimating camera orientation using motion vectors and line structures in an urban driving environment from three VPs using 3-line RANSAC [22] on a Gaussian sphere. However, the method required more reference lines in the image, and the algorithm was easy to get failed results. To solve this problem, the concept of the reprojection root-mean-squared error [23] for pixels was used. It is a metric used to express the calibration error. This method is a camera-setup-independent error metric used to measure the performance of the calibration algorithm (our model) while omitting extrinsic influences.

For the calibration system, both Lee et al. and Jang et al. proposed stabilization systems using an extended Kalman filter (EKF) to solve the nonlinear system. However, Jang et al. needs more reference lines to get several sets of candidate VPs from 3-line RANSAC on a Gaussian sphere. Therefore, a considerably long time is required to complete the process. The VPs also fail because the selected lines intersect at infinity. Lee et al. has residual error based on the distance from VP and the center point(CP) of the image.

This study proposes an MLP-based error compensation method for automatic on-the-fly camera orientation estimation with a single VP from the road lane and steady-state systems. The method considers the commonly available scenes around a self-driving vehicle. It estimates the pitch and yaw with the correction part using an image.

3. Orientation estimation

It is established that parallel lines do not intersect in the world coordinate system. Thus, we can conclude that the VPs are located at an infinite distance from each other. If we consider the forward direction of a car parallel to the lane lines, we can determine that the lane lines intersect at z = infinity. The road reference frame denotes This point as (X, Y, Z). Here, Z corresponds to the forward axis of the car. It should be noted that the road reference frame is fixed to the vehicle, as shown in Figure 2. Assumptions regarding the alignment of the vehicle with the lane and lane straightness are crucial. This is because the VP, the intersection point of the lane lines in the image, can provide information regarding camera mounting. Specifically, it reveals the orientation of the camera relative to a vehicle. However, if this assumption is not satisfied, the VP would only provide information regarding the orientation of the vehicle in relation to the lane lines and not that regarding the camera orientation.

3.1. Camera Orientation estimation

The camera projection equation with $P_i$ as a point in the image coordinates and $P_r$ as a point in the real-world coordinates is as follows:

$$s{P}_i=K\left[R\right|T]P_r$$

(1)

where

$$P_i={(u,v,1)}^T$$

(2)

$$P_r={(X,Y,Z,1)}^T$$

(3)

where K is the camera intrinsic parameter, $R|T$ are extrinsic parameters, the R is the rotation matrix and T is the translation matrix. Because this is an equation in homogeneous coordinates, we can multiply both sides by a scalar value s and absorb this value into Z on the left-hand side of the equation:

$$s\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)=K\ast R|T\left(\begin{array}{c}X/Z\\ Y/Z\\ 1\\ 1/Z\end{array}\right)$$

(4)

The z-axis value of the VP in the world coordinates is infinite. Thus, (4) becomes

$$s\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)=K\ast R|T\left(\begin{array}{c}0\\ 0\\ 1\\ 0\end{array}\right)$$

(5)

The rotation matrix is subject to the pitch, yaw, and roll angles and follows the common case. It is considered to be zero when the camera is mounted on the windshield. The rotation metrics are as follows:

$$R=\left(\begin{array}{ccc}{r}_{11}& r_{21}& r_{31}\\ r_{12}& r_{22}& r_{32}\\ r_{13}& r_{23}& r_{33}\end{array}\right)$$

(6)

As previously assumed for the roll degree, the rotation matrix represents the yaw and pitch as follows:

$$R=\left(\begin{array}{ccc}cos\left(\alpha \right)& sin\left(\alpha \right)sin\left(\beta \right)& sin\left(\alpha \right)cos\left(\beta \right)\\ 0& cos\left(\beta \right)& -sin\left(\beta \right)\\ -sin\left(\alpha \right)& cos\left(\alpha \right)sin\left(\beta \right)& cos\left(\alpha \right)cos\left(\beta \right)\end{array}\right)$$

(7)

As shown in (5), when (6) is substituted into (5), only the third column of information remains. Therefore, the following formula can be used for the two angles:

$$\left(\begin{array}{c}\alpha \\ \beta \end{array}\right)=\left(\begin{array}{c}atan2(r_{31},r_{33})\\ -arcsin\left(r_{32}\right)\end{array}\right)$$

(8)

where $\alpha$ is yaw and $\beta$ is pitch radian angle.

Following this function, the reprojection formula can calculate the VPs from the pitch and yaw. In this way, the error ($e_p,e_y$) distribution with angle under different camera FOV can be obtained (see Figure 3 and Figure 4). The function that obtains the error is calculated as follows:

$$\left(\begin{array}{c}{e}_p\\ e_y\end{array}\right)=\sum _{i=1}^n(f(u,v)-(\alpha ,\beta ))$$

(9)

where the function f(u,v) is defined as follows:

$${(\widehat{\mathbf{ff}},\widehat{\mathbf{fi}})}^T=f{(u,v)}^T$$

(10)

where $\widehat{\alpha }$,$\widehat{\beta }$ represent the estimated yaw and pitch angle.

3.2. Error compensation of orientation estimation

As mentioned in the previous section, the residual of this model is determined by the camera pose, that is the position of the VP. Since the pitch angle is a sin model, the residual distribution of pitch has a sin relationship with the distance between VP and CP, while yaw is called a tan model. Therefore, the residual distribution of yaw has a tan relationship with the distance between VP and CP. That is, the residual pitch is expressed by From the center of the picture to the edge, a curve approximates a quadratic function, and the residual of yaw appears as an even function about the zero point.

MLP [24]is a conventional method for determining nonlinear coefficients. Considering that the complexity of the nonlinear relationship is moderate and that the correlation is quasi-linear, this study used a shallow network with a rectified linear unit (ReLU) activation function (see Figure 5).

The MLP model is for the VP to estimate the pitch and yaw errors. The two inputs are the distances from the image’s center point to the VP. These are described as $D_u$ and $D_v$ in the image coordinates. The two outputs of the model are the pitch $E_p$ and yaw $E_y$ errors for error compensation.

After the MLP model, the error compensation part would function as follows:

$$H(D_u,D_v)={(\widehat{\mathbf{ff}},\widehat{\mathbf{fi}})}^T-MLP{(D_u,D_v)}^T$$

(11)

$$(D_u,D_v)=\overrightarrow{vp}-\overrightarrow{cp}$$

(12)

where the MLP model which has trained is:

$${(E_p,E_y)}^T=MLP{(D_u,D_v)}^T$$

(13)

3.3. Multilayer Perceptron

In this section, we utilize an MLP model as the core component for the error compensation of the angle estimation from the VP. The MLP architecture comprises an input layer, at least one hidden layer, and an output layer. This neural network design aims to generate a specific flow of information between the layers to learn complex representations of the input data.

The MLP network used in this study consists of five linear layers, each with a different number of neurons. The first hidden layer receives two input features and maps these to a 64-dimensional feature space. The subsequent layers increase the feature space to 128 dimensions, gradually decrease it to 64 and 32 dimensions, and produce a 2-dimensional output. The layers are connected through linear transformations (weights and biases) and activation functions such as ReLU. The number of neurons in each layer enables the network to learn higher-level features and abstractions to fit the error distribution.

4. Experimental Result

This section presents the settings and scenarios used in this study. First, to ensure the authenticity of the algorithm verification environment while considering the accuracy and operability of the ground truth(GT) data, the current city simulator was selected (see Table 1). The table outlines the main features and functions of certain highly effective emulators, such as the image resolution, FOV, camera model, camera pose, map generation engine, and minimum hardware requirements. Among these, MORAI has a relatively high accuracy. This map of South Korea was selected as the experimental simulator. An experiment was conducted on IONIQ5, which is used for self-driving research.

4.1. Experimental Environment: Simulator

Table 1 shows the differences. Here, the platforms offer a range of image resolutions. Most of these support a maximum of 1,920 × 1,920 pixels. The FOV for these platforms also varies. A few provide a more comprehensive range of up to ${180}^{\circ }$ and ${200}^{\circ }$. The camera models utilized by these platforms can be ideal or physical. Specific platforms support both types. The camera poses are indicated by the rotation R and translation T values. These demonstrate the precision in the simulation environment.

From the survey of the simulators, the setting of the MORAI simulator was more suitable for this study. Moreover, it supports the city map of South Korea. The experimental setup of the emulator used in this study is shown in Figure 6. The map was that of Sangam-dong in Seoul, South Korea. The car model was the Kia Niro(EV) 2020 version. The camera model was the default one that supports undistorted image information with adjustable FOV and image resolution. The sensor position and orientation were based on the car coordinates.

To evaluate the stability of the pitch and yaw angle estimation and verify the performance of angle prediction, one angle was fixed, and the other was varied. The scenarios of the experiments are designed as follows in Figure 7:

• Scenario 1: For the algorithm to follow the pitch motion with a $0.1^{\circ }$ variation, it modifies the pitch angle continuously while the yaw angle is fixed to $0^{\circ }$.
• Scenario 2: For the algorithm to follow the yaw motion with a $0.1^{\circ }$ variation, the yaw angle is modified continuously while the pitch angle is fixed to $0^{\circ }$.
• Scenario 3: For the system to converge in certain frames to correct the pitch angle with a ${10}^{\circ }$ variation, the pitch angle is modified by ${10}^{\circ }$ in each frame while the yaw angle is fixed to $0^{\circ }$.
• Scenario 4: For the system to converge in certain frames to correct the yaw angle with a ${10}^{\circ }$ variation, the yaw angle is modified by ${10}^{\circ }$ in each frame while the pitch angle is fixed to $0^{\circ }$.
• Scenario 5: For the system to converge in certain frames to correct the angle with a ${10}^{\circ }$ variation, the pitch and yaw angle is modified by $1^{\circ }$ one of them in each frame.

4.2. Experimental Environment: Real Car

To better verify the algorithm in the real world, experiments related to real data were conducted on IONIQ5 vehicles equipped with a GMSL camera (SEKONIX). The camera was located above the windshield. There were two cameras with FOVs of ${60}^{\circ }$ and ${120}^{\circ }$. In this study, the intrinsic parameters and image undistortion of the camera with ${60}^{\circ }$ were preconfigured fully. Therefore, a camera with a FOV of ${60}^{\circ }$ was used to verify this experiment (see Figure 8). This study used linear lane lines that exist at a short distance from the camera (within about 20m), such as cornerstones or obstacles along the outer edge of the road.

4.3. Result

For each scenario, the table provides the average error (AE), minimum error (MINE), maximum error (MAXE), and standard deviation (STDEV) for all the control algorithms. This enabled the assessment of their performance in terms of accuracy and stability.

To ensure the validity and efficiency of the proposed method, in the system design, a KF is added at the end of the VP and angle estimation process to filter out false VP detections or miscalculations. The performance of Paula et al. consistently exhibits larger error than the method proposed by Lee et al.; therefore, its results are often not visible in the graphs.

Table 2 presents a quantitative analysis of the pitch angle estimation with a fixed yaw angle. As shown in the table, the proposed method outperformed the methods of Paula et al. and Lee et al. regarding the AE, MINE, MAXE, and STDEV. This indicated that the proposed method is more accurate and stable for pitch angle estimation. This is crucial for robotics and computer vision applications. The angle estimation and error curve results are shown in Figure 9.

Table 3 presents a quantitative analysis of the yaw angle estimation with a fixed pitch angle. Compared with the method of Paula et al., the proposed method demonstrated superior performance in the MINE, MAXE, and STDEV. Although the method of Lee et al. performed marginally better in terms of the AE, the proposed method maintained a balanced performance across all the metrics. This demonstrated its effectiveness for yaw angle estimation. The results of the angle estimation and error curves are shown in Figure 10.

Table 4 and Table 5 focuses on the step scenarios with pitch angle estimation under a fixed yaw angle and yaw angle estimation under a fixed pitch angle. The proposed method exhibited advantages regarding specific error metrics such as the MINE and MAXE in both cases. Although the proposed method did not outperform Lee et al. in all the categories, its overall performance remained competitive and stable. This further emphasized the robustness and reliability of the proposed method in various scenarios. The system convergence interval was approximately 30 frames. The angles estimation and error curve result are shown in Figure 11 and Figure 12.

It can be observed from Figure 13 that the proposed method is very close to the lines of the GT and, therefore, has a high accuracy in estimating the pitch angle. The Lee et al. and Paula et al. methods seem to have larger error, especially in some frames, where the errors are pretty significant. In Figure 14, the proposed method again shows very close characteristics to the GT, especially in the error plot; the error of the method is relatively small. However, the yaw angle error of the Lee et al. method is significant in most frames, and although the Paula et al. method seems to be close to GT in some frames, the error is also large in other frames.

Table 6 presents a quantitative analysis of the orientation using three estimation methods to pitch and yaw for real car datasets. The performances of the methods of Paula et al. and Lee et al. and that of the proposed method were compared in terms of the AE, steady-state error (SSE), and STDEV for both pitch and yaw angles.

As observed from the table, the proposed method demonstrates superior performance in terms of the AE and STDEV for both pitch and yaw angles compared with the other two methods. The proposed method achieved the lowest AE for pitch and yaw estimation. This indicated its higher accuracy in orientation estimation for real car datasets. Furthermore, the proposed method displayed the lowest STDEV for yaw estimation and a competitive STDEV for pitch estimation. This illustrated the stability and reliability of the method.

5. Ablation study

This study introduces a compensation module based on the foundational mathematical model proposed by Lee et al. Furthermore, an ablation study was conducted to evaluate the efficacy of various compensation modules. The following methods were analyzed: the original model proposed by Lee et al. (designated as "No process"), Linear Regression (designated as "Linear"), Plane Function Compensation (designated as "Function"), and MLP.

Figure 15 to Figure 18 elucidate the estimation performance of pitch and yaw angles across cameras with different field-of-views (FOVs), specifically 60°, 90°, 120°, and 150°. This exploration covers multiple pivotal aspects:

• Pitch Analysis: The MLP method consistently outperformed other techniques in estimating pitch angles across all FOV ranges. Relative to GT, the estimations from MLP consistently exhibited remarkable accuracy. Conversely, the "No process" method presented substantial deviations from true values.
• Yaw Analysis: Several methods achieved commendable precision for yaw estimations across most FOVs. Nonetheless, the "Linear" encountered marginal error increments at specific angles, while the MLP’s accuracy remained relatively invariant.
• FOV Assessment: Across varied FOVs, the MLP method stood out for its pitch and yaw angle estimations accuracy. While the linear and other methods demonstrated efficacy within certain angular ranges, they exhibited noticeable deviations under particular conditions.

Figure 15. Compensate pitch performance comparison with different camera FOV.

Figure 15. Compensate pitch performance comparison with different camera FOV.

Figure 16. Compensate yaw performance comparison with different camera FOV.

Figure 16. Compensate yaw performance comparison with different camera FOV.

Table 7 conveys a comprehensive quantitative evaluation of the ablation studies across different FOVs. This table contrasts the performance metrics of the four methodologies on a simulated dataset. Key insights from this analysis include:

• Estimation error for Pitch: Across all FOVs, MLP consistently registered the most minor error. Notably, at a 150° FOV, its performance superiority was markedly evident. The Linear Regression’s error trajectory steeply ascended with FOV increments. Notably, the error magnitude for the "Function" surged most significantly with FOV enlargement.
• Estimation error for Yaw: Astonishingly, both Linear Regression and Function Compensation methods yielded zero error across all FOVs, epitomizing impeccable estimations. Similarly, MLP’s performance mirrored this perfection. In comparison, the "No process" method, while competent, did exhibit a marginal error increase as the FOV expanded.
• Processing Time Assessment: The "Function" consistently achieved the swiftest processing times across all FOVs, signifying optimal resource efficiency. In contrast, the "Linear" generally demanded more prolonged processing intervals. MLP and the "No process" methods displayed commendable consistency in processing durations across all FOVs.

Figure 17. Compensate pitch error comparison with different camera FOV.

Figure 17. Compensate pitch error comparison with different camera FOV.

Figure 18. Compensate yaw error comparison with different camera FOV.

Figure 18. Compensate yaw error comparison with different camera FOV.

While the MLP methodology showcased exemplary precision in pitch and yaw estimations, its consistency in processing durations substantiated its robustness. "Function," while unparalleled in processing speed, witnessed occasional challenges in estimation accuracy, especially under expanded FOVs. This analysis underscores the intricate balance between speed, resource allocation, and estimation accuracy, suggesting the potential superiority of MLP in wide-ranging practical applications.

6. Discussion

The integration of a compensation module into the foundational mathematical model introduced by Lee et al. marks a significant advancement in enhancing the estimation performance of pitch and yaw angles. This evolution of compensation methods, as evidenced in the ablation study, has profound implications for the accuracy and efficiency of angle estimations. Findings, as represented in Figure 15 to Figure 18, findings underscore the undeniable influence of FOVs on estimation outcomes. The MLP method emerges as a frontrunner, demonstrating consistently superior accuracy across all FOVs.

An intriguing aspect of the study is the Yaw Analysis, which indicates near-perfect estimations by several methods, notably the Linear Regression and Function Compensation. Such impeccable results necessitate further exploration into the factors contributing to yaw’s high accuracy compared to pitch. Such a line of inquiry could be instrumental for future research endeavors. Efficiency analyses shed light on a pivotal trade-off between accuracy and processing time. "Function," while impressively speedy, occasionally falters in accuracy, especially under broader FOVs. The MLP method, while not being the fastest, presents a balanced profile of consistent processing times and commendable accuracy, solidifying its potential in applications where precision is of utmost importance.

The MLP-based technique for camera orientation estimation is promising. However, assumptions like vehicle alignment with the lane and consistent roll angle can lead to inaccuracies. The MLP model may not always capture the intricate relationship between the vanishing point and error distribution. The lack of camera height estimation is a limitation, especially for camera calibration in autonomous vehicles. Future research should focus on improving calibration parameters and refining the error model for various scenarios.

7. Conclusion

This study introduces an MLP-based error compensation method for camera orientation estimation using lane lines and a vanishing point. The aim is to identify an algorithm for Automatic On-the-Fly Camera Orientation Estimation and assess its accuracy. An ablation study compared various compensation modules, including No processed, Linear Regression, Function and MLP. Results showed that the MLP compensation improved the original algorithm’s accuracy, especially in estimating pitch and yaw angles. Future research could expand the calibration approach to include the full rotation matrix and camera elevation. With its potential in autonomous vehicles and driver-assistance systems, the method offers promise for real-time camera orientation tasks.