A Novel Feedforward Youla Parameterization Method for Avoiding Local Minima in Stereo Image Based Visual Servoing Control

1. Introduction

Determining the accurate pose of the camera is a fundamental problem in robot manipulations, as it provides the spatial transformation needed to map 3D world points to 2D image coordinates. The task involving camera pose estimation is essential for various applications, such as augmented reality [1], 3D reconstruction [2], SLAM [3], and autonomous navigation [4]. This becomes especially critical when robots operate in unstructured, fast-changing, and dynamic environments, performing tasks such as human-robot interaction, accident recognition and avoidance, and eye-in-hand visual servoing. In such scenarios, accurate camera pose estimation ensures that visual data is readily available for effective robotic control [5].

A classic approach to estimating the pose of a calibrated camera is solving the Perspective-n-Point (PnP) problem [6], which establishes a mathematical relationship between a set of n 3D points in the world and their corresponding 2D projections in an image. To uniquely determine the pose of a monocular camera in space, it is a Perspective-4-Point (P4P) problem, where exactly 4 known 3D points and their corresponding 2D image projections are used. Bujnak et al. [7] generalize four solutions for P3P problem while giving a single unique solution existed for P4P problem in a fully calibrated camera scenario. To increase accuracy, modern PnP approaches considers more than three 2D-3D correspondences. Among PnP solutions, EPnP (Efficient PnP) method finds the optimal estimation of pose from a linear system that expresses each reference point as a weighted sum of four virtual control points [8]. Another advanced approach, SQPnP (Sparse Quadratic PnP) formulates the problem as a sparse quadratic optimization, achieving enhanced accuracy by minimizing a sparse cost function [9].

In recent years, many other methods have been developed to show improved accuracy than PnP based methods. For instance, Alkhatib et al. [10] utilize Structure from Motion (SfM) to estimate a camera’s pose by extracting and matching key features across various images taken from different viewpoints to establish correspondences. Moreover, Wang et al. [11] introduce visual odometry into camera’s pose estimations based on the movement between consecutive frames. In addition, recent advancements in deep learning have led to the development of models, such as Convolutional Neural Networks, specifically tailored for camera pose estimation [12,13,14]. However, these advanced methods often come with significant computational costs, requiring multiple images from different perspectives for accurate estimation. In contrast, PnP-based approaches offer a balance between accuracy and efficiency, as they can estimate camera pose from a single image, making them highly suitable for real-time applications such as navigation and scene understanding.

In image-based visual servoing (IBVS) [15], the primary goal is to control a robot’s motion using visual feedback. Accurate real-time camera pose estimation is crucial for making informed control decisions, particularly in eye-in-hand (EIH) configurations [16,17], where a camera is mounted directly on a robot manipulator. In this setup, robot motion directly induces camera motion, making precise pose estimation essential. Due to its computational efficiency, PnP-based approaches remain widely applied in real-world IBVS tasks [18,19,20]. The PnP process begins by establishing correspondences between 3D feature points and their 2D projections in the camera image. The PnP algorithm then computes the camera pose from these correspondences, translating the geometric relationship into a format that the IBVS controller can use. By detecting spatial discrepancies between the current and desired camera poses, the robot can adjust its movements accordingly.

However, PnP-based IBVS presents challenges for visual control in robotics. One key issue is that IBVS often results in an overdetermined system, where the number of visual features exceeds the number of joint variables available for adjustment. For example, at least four 2D-3D correspondences are needed for a unique pose solution [6], but a camera’s full six-degree-of-freedom (6-DOF) pose means that a 6-DOF robot may need to align itself with eight or more observed features. In traditional IBVS [15], the interaction matrix (or image Jacobian) defines the relationship between feature changes and joint velocities. When the system is overdetermined, this matrix contains more constraints than joint variables, leading to redundant information. Research [15] suggests that this redundancy may cause the camera to converge to local minima, failing to reach the desired pose. Although local asymptotic stability is always ensured in IBVS, global asymptotic stability cannot be guaranteed when the system is overdetermined.

Many studies have explored solutions to mitigate the local minimum problem in IBVS. One approach, proposed by Nicholas et al. [21], introduces a switched control method, where the system alternates between different controllers to escape local minima and avoid singularities in the image Jacobian. Another strategy, developed by Chaumette et al. [22], utilizes a 2-1/2-D visual servoing technique, which combines image-based and position-based features. This integration allows the camera to navigate around local minima during motion execution. Roque et al. [23] implement a model predictive control (MPC) approach, optimizing the quadrotor’s trajectory to enhance robustness against local minima by predicting and adjusting control inputs in real time.

While these methods achieve significant improvements in most scenarios, they also introduce computational challenges compared to traditional IBVS. The switched control method requires different control strategies tailored to specific dynamics, increasing the complexity of the overall control architecture [24]. The 2-1/2-D visual servoing method demands real-time processing of both visual and positional data, which can impose significant computational loads and limit performance in dynamic environments [25]. MPC approaches introduce additional computational overhead by requiring complex optimization at every time step, making real-time implementation costly [26].

In this paper, we focus on the PnP framework for determining and controlling the pose of a stereo camera within an image-based visual servoing (IBVS) architecture. In traditional IBVS, depth information between objects and the image plane is crucial for developing the interaction matrix. However, with a monocular camera, depth can only be estimated or approximated using various algorithms [15], and inaccurate depth estimation may lead to system instability. In contrast, a stereo camera system can directly measure depth through disparity between two image planes, enhancing system stability.

A key novelty of this paper is providing a systematic proof that stereo camera pose determination in IBVS can be formulated as a P3P (Perspective-3-Point) problem, which, to the best of our knowledge, has not been explored in previous research. Since three corresponding points, totaling nine coordinates, are used to control the six DoFs camera pose, the IBVS control system for a stereo camera is overdetermined, leading to the potential issue of local minima during control maneuvers. While existing approaches can effectively address local minima, they often introduce excessive computational overhead, making them impractical for high-speed real-world applications.

To address this challenge, we propose a feedforward-feedback control architecture. The feedback component follows a cascaded control loop based on the traditional IBVS framework [15], where the inner loop handles robot joint rotation, and the outer loop generates joint angle targets based on visual data. One key improvement in this work is the incorporation of both kinematics and dynamics during the model development stage. Enhancing model fidelity in the control design improves pose estimation precision and enhances system stability, particularly for high-speed tasks. Both control loops are designed using Youla parameterization [27], a robust control technique that enhances resistance to external disturbances. The feedforward controller takes target joint configurations, which are associated with the desired camera pose as inputs, ensuring a fast system response while avoiding local minima traps. Simulation results presented in this paper demonstrate that the proposed control system effectively moves the stereo camera to its desired pose accurately and efficiently, making it well-suited for high-speed robotic applications.

2. System Configuration

An eye-in-hand robotic system has been developed to precisely control the pose of a stereo camera system, as illustrated in Figure 1. The robotic manipulator is equipped with six revolute joints, allowing unrestricted movement of the camera across six degrees of freedom (DoFs)—three for positioning and three for orientation. Assume a set of fiducial markers is placed in the workspace, with their coordinates fixed and predefined in an inertial frame. Utilizing the Hough transform [28] in computer vision, these markers can be detected and localized by identifying their centers in images captured by the stereo camera system. The control system within the robotic manipulator aligns the camera to its desired pose by matching the detected 2D features in the current frame with target 2D features. Throughout this process, it is assumed that all fiducial markers remain within the camera’s field of view. As depicted in Figure 1, multiple Cartesian coordinate systems are illustrated. The base frame {O} serves as an inertial reference fixed to the bottom of the robot manipulator, while the camera frame {C} is a body-fixed frame attached to the robot’s end-effector.

3. Proof of P3P for the Stereo Camera System

Given its intrinsic parameters and a set of n correspondences between its 3D points and 2D projections to determine the camera’s pose is known as perspective-n-point (PnP) problem. This well-known work [7] has proved that at least four correspondences are required to uniquely determine the pose of a monocular camera, a situation referred to as the P4P problem.

To illustrate, consider the P3P case for a monocular camera. Let points A, B, and C exist in space, with $O_1$, $O_2$ and $O_3$representing different perspective centers. The angles ∠ AOB, ∠ AOC and ∠ BOC remain the same across all three perspectives. Given a fixed focal length, the image coordinates of points A, B, and C will be identical when observed from these three perspectives. In other words, it is impossible to uniquely identify the camera’s pose based solely on the image coordinates of three points.

Figure 2. P3P case of a monocular camera.

Figure 2. P3P case of a monocular camera.

The PnP problem with a stereo camera has not been thoroughly addressed in prior research. A stereo camera can detect three image coordinates of a 3D point in space. This paper proposes that a complete solution to the PnP problem for a stereo camera can be framed as a P3P problem. Below is the complete proof of this proposition.

Proof:

For a stereo system, if all intrinsic parameters are fixed and given, we can readily compute the 3D coordinates of an object point given the image coordinates of that point. This provides a unique mapping from the image coordinates of a point to its corresponding 3D coordinates in a Cartesian frame. The orientation and position of the camera system uniquely define the origin and axis orientations of this Cartesian coordinate system in space. Consequently, PnP problem can be framed as follows: given n points with their 3D coordinates measured in an unknown Cartesian coordinate system in space, what is the minimum number n required to accurately determine the position and orientation of the 3D Cartesian coordinate frame established in that space?

1) P1P problem with the stereo camera:

If we know the coordinates of a single point in space, defined by a 3D Cartesian coordinate system, an infinite number of corresponding coordinate systems can be established. Any such coordinate system can have its origin placed on the surface of a sphere centered at this point, with a radius $R=\sqrt{{X^C}^2+{Y^C}^2+{Z^C}^2}$, where ${\left[X^C, Y^C, Z^C\right]}^T$ are the coordinates measured by the Cartesian system (see Figure 3).

2) P2P problem with the stereo camera:

When the coordinates of two points in space are known, an infinite number of corresponding coordinate systems can be established. Any valid coordinate system can have its origin positioned on a circle centered at point O with a radius $R$ as illustrated in Figure 4. This circle is constrained by the triangle formed by points $O_C$, $P_1$, and $P_2$, where the sides of the triangle are defined by the lengths $R_1, R_2$ and $D$. Specifically, $R_1=\sqrt{{X_1^C}^2+{Y_1^C}^2+{Z_1^C}^2}, R_2=\sqrt{{X_2^C}^2+{Y_2^C}^2+{Z_2^C}^2},$ $D=\sqrt{{(X}_1^C-{X_2^C)}^2+(Y_1^C-{Y_2^C)}^2+(Z_1^C-{Z_2^C)}^2}.$The radius of the circle R corresponds to the height of the base D of the triangle. The center of the circle O is located at the intersection of the height and the base.

3) P3P problem with the stereo camera:

When three points in space are known, and the lines connecting these points are not collinear, we can uniquely establish one coordinate system. As illustrated in the figure below, three non-collinear points define a plane in space, which has a uniquely defined normal unit vector $\stackrel{⃑}{n}$. Given the coordinates of the three points, we can calculate vectors as follows: the vector $\overrightarrow{P_1{P}_2}$ = ($X_2^C-X_1^C,Y_2^C-Y_1^C,Z_2^C-Z_1^C$), and the vector $\overrightarrow{P_1{P}_3}$ = ($X_3^C-X_1^C,Y_3^C-Y_1^C,Z_3^C-Z_1^C$). The unit vector $\stackrel{⃑}{n}$ which is perpendicular to the plane formed by these three points, can be expressed as:

$$\stackrel{⃑}{n}= {\displaystyle \frac{\overrightarrow{P_1{P}_2}\times \overrightarrow{P_1{P}_3}}{\left|\overrightarrow{P_1{P}_2}\times \overrightarrow{P_1{P}_3}\right|}}$$

(1)

Here, × denotes the cross product.

The angles between $\stackrel{⃑}{n}$ and XYZ axes of the coordinate system can be expressed as follows:

$$\cos \left({\theta }_x\right)=\stackrel{⃑}{n}·\overrightarrow{i}$$

(2)

$$\cos \left({\theta }_y\right)=\stackrel{⃑}{n}·\overrightarrow{j}$$

(3)

$$\cos \left({\theta }_z\right)=\stackrel{⃑}{n}·\overrightarrow{k}$$

(4)

Where ${\theta }_x, {\theta }_y$ and ${\theta }_z$are the angles between $\stackrel{⃑}{n}$ and the unit vectors in the X, Y, and Z directions, denoted as $\overrightarrow{i}$, $\overrightarrow{j},$and $\overrightarrow{k}$ respectively. Therefore, with the direction $\stackrel{⃑}{n}$ fixed in space, the orientations of each axis of the coordinate system can be computed uniquely.

According to the P1P problem, the origin of the coordinate system must lie on the surface of a sphere centered at $P_1$with radius $=\sqrt{{X_1^C}^2+{Y_1^C}^2+{Z_1^C}^2}$ as depicted in Figure 3. Each coordinate system established with a different origin point on the surface of this sphere results in a unique configuration of the axis orientations. Therefore, as the orientations of the axes are defined in space, the position of the frame (or the position of the origin) is also uniquely defined.

In conclusion, the P3P problem is sufficient to solve the PnP problem for a stereo camera system.

Figure 5. P3P Problem with a Stereo Camera System.

Figure 5. P3P Problem with a Stereo Camera System.

Prove Concluded

This proposition indicates that to uniquely determine the full 6 DoFs of the stereo camera, at least three points (or nine 2D features) are required to match in the image-based visual servoing control.

4. Model Development

4.1. Stereo Camera Model

Depth between the objects to the camera plane is either approximated or estimated in the IBVS for generating the interaction matrix [15]. Using a stereo camera system in IBVS eliminates the inaccuracies associated with monocular depth estimation, as it directly measures depth by leveraging the disparity between the left and right images.

The stereo camera model is illustrated in Figure 6. A stereo camera consists of two lenses separated by a fixed baseline b. Each lens has a focal length $F$ (measured in mm) which is the distance from the image plane to the focal point. Assuming the camera is calibrated, the intrinsic parameters: b, $F$ is accurately estimated. A scene point $I$ is measured in the 3D coordinate frame {C} centered at the middle of the baseline with its coordinates as ${[X^C, Y^C,Z^C]}^T$. The stereo camera model maps the 3D coordinates of this point to its 2D coordinates projected on the left and right image plane as ${[u_l,v]}^T$ and ${[u_r,v]}^T$, respectively. The full camera projection map, incorporating both intrinsic and extrinsic parameters, is given by:

$$s·p_{image}=K·\left[R\right|T]{·P}_C$$

(5)

Where $p_{image}$ are the image coordinates of the point and $P_C$ are the 3D coordinates measured in the camera frame {C}. $s$ is the scale factor that ensures correct projection between 2D and 3D features. $K$ is the intrinsic matrix with a size of 3X3, and the mathematical expression is presented as:

$$K= \left[\begin{array}{ccc} F& k& u_0\\ 0& F& v_0\\ 0& 0& 1\end{array}\right]$$

(6)

Where $k$ is the skew factor, which represents the angle between the image axes (u and v axis). $u_0$ and $v_0$ are coordinates offsets in image planes.

In Equation (5), R is the rotational matrix from camera frame {C} to each image coordinate frame, and T is the translation matrix from camera frame {C} to each camera lens center. Since there is no rotation between the camera frame {C} and image frames but only a translation along the $X_C$ axis occurs, the transformation matrices for the left and right image planes are expressed as$:$

$${[R|T]}_{Left}= \left[\begin{array}{ccc}\begin{array}{ccc} 1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}& |& \begin{array}{c}\text{-}b/2\\ 0\\ 0\end{array}\end{array}\right]$$

(7)

$${[R|T]}_{Right}=\left[\begin{array}{ccc}\begin{array}{ccc} 1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}& |& \begin{array}{c}\mathrm{b}/2\\ 0\\ 0\end{array}\end{array}\right]$$

(8)

Assume the u and v axis are perfectly perpendicular (take k = 0), and there are no offsets in the image coordinates (take $u_0=v_0=0$) for both lens. Also, set factor $s=Z_I^C$accounts for perspective depth scaling. The projection equations for the left and right image planes can be rewritten in homogeneous coordinates as:

$$Z_I^C·\left[\begin{array}{c}{u}_l\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc} F& 0& 0\\ 0& F& 0\\ 0& 0& 1\end{array}\right]·\left[\begin{array}{ccc}\begin{array}{ccc} 1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}& |& \begin{array}{c}b/2\\ 0\\ 0\end{array}\end{array}\right]·\left[\begin{array}{c}\begin{array}{c}{X}_I^C\\ Y_I^C\end{array}\\ \begin{array}{c}{Z}_I^C\\ 1\end{array}\end{array}\right]$$

(9)

$$Z_I^C·\left[\begin{array}{c}{u}_r\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc} F& 0& 0\\ 0& F& 0\\ 0& 0& 1\end{array}\right]·\left[\begin{array}{ccc}\begin{array}{ccc} 1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}& |& \begin{array}{c}\text{-}\mathrm{b}/2\\ 0\\ 0\end{array}\end{array}\right]·\left[\begin{array}{c}\begin{array}{c}{X}_I^C\\ Y_I^C\end{array}\\ \begin{array}{c}{Z}_I^C\\ 1\end{array}\end{array}\right]$$

(10)

Equations (9) and (10) establish the mathematical relationship between the 3D coordinates of a point in the camera frame {C} and its 2D projections on the left and right image planes. The pixel value along the $v$ -axis remains the same for both images. As a result, a scene point’s 3D coordinates can be mapped to a set of three image coordinates in the stereo camera system, expressed as:

$$\mathrm{Stereo}\text{-}\mathrm{camera} \mathrm{mapping} M \text{：}{P}^C={[X^C, Y^C,Z^C]}^T\to p_{image}={[u_l,u_r,v]}^T$$

(11)

The mapping function $M$ is nonlinear and depends on the stereo camera parameters $P_{a\_camera}$, specifically b, and $F$.

4.2. Robot Manipulator Kinematic Model

A widely used method for defining and generating reference frames in robotic applications is the Denavit-Hartenberg (D-H) convention [29]. In this approach, each robotic link is associated with a Cartesian coordinate frame ${ O}_i{X}_i{Y}_i{Z}_i$. According to the D-H convention, the homogeneous transformation matrix $A_i^{i-1}$, which represents the transformation from frame $i-1$ to frame $i$, can be decomposed into a sequence of four fundamental transformations:

$$\begin{gathered} A_i^{i-1}= {Rot}_{z,q_i}{Trans}_{z,d_i}{Trans}_{x,a_i}{Rot}_{x,{\alpha }_i} \\ \text{Expanding} \text{the} \text{transformation} \text{into} \text{its} \text{matrix} \text{form:} \\ {A}_i^{i-1}=\left[\begin{array}{cccc}{c}_{q_i}& -s_{q_i}& 0& 0\\ s_{q_i}& c_{q_i}& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& d_i\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& a_i\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& c_{{\alpha }_i}& {-s}_{{\alpha }_i}& 0\\ 0& s_{{\alpha }_i}& c_{{\alpha }_i}& 0\\ 0& 0& 0& 1\end{array}\right] \end{gathered}$$

(12)

$$=\left[\begin{array}{cccc}{c}_{q_i}& -s_{q_i}{c}_{{\alpha }_i}& s_{q_i}{s}_{{\alpha }_i}& a_i{c}_{q_i}\\ s_{q_i}& c_{q_i}{c}_{{\alpha }_i}& -c_{q_i}{s}_{{\alpha }_i}& a_i{s}_{q_i}\\ 0& s_{{\alpha }_i}& c_{{\alpha }_i}& d_i\\ 0& 0& 0& 1\end{array}\right]$$

(13)

$$\mathrm{Note}: c_{{\theta }_i}\equiv \mathrm{c}\mathrm{o}\mathrm{s}\left(q_i\right), c_{{\alpha }_i}\equiv \mathrm{c}\mathrm{o}\mathrm{s}\left({\alpha }_i\right), s_{{\theta }_i}\equiv \mathrm{s}\mathrm{i}\mathrm{n}\left(q_i\right), s_{{\alpha }_i}\equiv \mathrm{s}\mathrm{i}\mathrm{n}\left({\alpha }_i\right)$$

(14)

The parameters $q_i$, $a_i$, ${\alpha }_i$ and $d_i$ define the link and joint characteristics of the robot. Here, $a_i$ is the link length, $q_i$ is the joint rotational angle, ${\alpha }_i$ is the twist angle, and $d_i$ is the offset between consecutive links. The values for these parameters are determined following the procedure outlined in [29].

To compute the transformation from the end-effector frame $O_6{X}_6{Y}_6{Z}_6$ (denoted as {E}) to the base frame $O_0{X}_0{Y}_0{Z}_0$ (denoted as {O}), we multiply the individual transformations along the kinematic chain:

$$T_6^0= A_1^0{A}_2^1{A}_3^2{A}_4^3{A}_5^4{A}_6^5$$

(15)

Furthermore, the transformation matrix from the base frame {O} to the end-effector frame {E} can be derived by taking the inverse of $T_6^0$:

$$T_0^6=({T_6^0)}^{-1}$$

(16)

If a point $P^O$is defined in the base frame, its coordinates in the end-effector frame $P^E$ can be found using:

$$P^E= T_0^6 P^O$$

(17)

Assuming that the camera remains static relative to the end-effector, we introduce a constant transformation matrix $T_E^C$ that maps points from the end-effector frame {E} to the camera frame {C}. For a stereo camera system, this camera frame is located at the center of the stereo baseline, as shown in Figure 6. The coordinates of a point in space, measured in the base frame, can then be expressed in the camera frame as:

$$P^C= {T_E^{C}T}_0^6 P^O$$

(18)

Equation (18) describes how a given 3D point in the base frame $P^O$= [$X^O, Y^O,Z^O$] is mapped to the camera frame $P^{ C}$= [$X^C, Y^C,Z^C$] using transformation:

$$\mathrm{Robot} \mathrm{manipulator} \mathrm{mapping} \mathcal{H}\text{：}{P}^O={[X^O, Y^O,Z^O]}^T\to P^C={[X^C, Y^C,Z^C]}^T$$

(19)

The mapping function $\mathcal{H}$ is nonlinear and depends on the current joint angle of robot $q=\left[q_i\right|\mathrm{i}\mathsf{ϵ}1,2,3,4,5,6],$robot geometric parameter $P_{a\_robot}$= [$a_i$, ${\alpha }_i$, $d_i$ |i ϵ 1,2,3,4,5,6], and constant transformation matrix $T_E^C.$

4.3. Eye-in-Hand Kinematic Model

By combining the stereo camera mapping $M$ from Equation (11) and the robot manipulator mapping $\mathcal{H}$ from Equation (19), we define a nonlinear transformation 𝓕, which maps any point measured in the base frame {O} to its image coordinates as captured by the stereo camera. This transformation is expressed as:

$$\mathrm{Eye}\text{-}\mathrm{in}\text{-}\mathrm{hand} \mathrm{mapping} \mathcal{F} \text{：}{P}^O={[X^O, Y^O,Z^O]}^T\to p_{image}={[u_l,u_r,v]}^T$$

(20)

The Mapping $\mathcal{F}$ is nonlinear and depends on variables current joint angles: $q$ and parameters $P_a$, which includes stereo camera parameter $P_{a\_camera}$, robot geometric parameter $P_{a\_robot},$ and transformation matrix $T_E^C$. In other words:

$$\mathrm{For} \mathrm{any} \mathrm{time} t\ge 0,p_{image}=\mathcal{F}(q\left(t\right), Pa, P^O)$$

(21)

$$Pa=[P_{a_{camera}},P_{a_{robot}},T_E^C]$$

(22)

4.4. Robot Inverse Kinematic Model

Inverse kinematics determines the joint angles required to achieve a given camera pose relative to the inertial frame. The camera pose in the inertial frame can be expressed as a 4X4 matrix:

$${Pose}^O= \left[\begin{array}{cc}\begin{array}{cc}{n}_x& s_x\\ n_y& s_y\end{array}& \begin{array}{cc}{a}_x& d_x\\ a_y& d_y\end{array}\\ \begin{array}{cc}{n}_z& s_z\\ 0& 0\end{array}& \begin{array}{cc}{a}_z& d_z\\ 0& 1\end{array}\end{array}\right]$$

(23)

Here, the vectors ${[n_x,n_y,n_z]}^T$, ${[s_x,s_y,s_z]}^T$and ${[a_x,a_y,a_z]}^T$represent the camera’s directional vectors for Yaw, Pitch, and Roll, respectively, in the base frame {C}. Additionally, the vector ${[d_x,d_y,d_z]}^T$ denotes the absolute position of the camera center in the base frame {C}.

The camera pose in the camera frame {C} is straightforward as it can be expressed as another 4X4 matrix:

$${Pose}^C= \left[\begin{array}{cc}\begin{array}{cc}1& 0\\ 0& 1\end{array}& \begin{array}{cc}0& 0\\ 0& 0\end{array}\\ \begin{array}{cc}0& 0\\ 0& 0\end{array}& \begin{array}{cc}1& 0\\ 0& 1\end{array}\end{array}\right]$$

(24)

The nonlinear inverse kinematics problem involves solving for the joint angles $q$ that satisfy the equation:

$${Pose}^C={T_E^C·T}_0^6\left(q\right)·{Pose}^O$$

(25)

where $T_E^C$ is the transformation from the end-effector frame to the camera frame, and $T_0^6\left(q\right)$ represents the transformation from the base frame to the end-effector frame, which is a function of the joint angles $q$.

The formulas for computing each joint angle are derived from the geometric parameters of the robot. The results of the inverse kinematics calculations for the ABB IRB 4600 elbow manipulator [30], used for simulations in this paper, are summarized in Appendix A.

4.5. Robot Dynamic Model

Dynamic models are included in the inner joint control loop, which will be discussed in section 6. Without derivation, the dynamic model of a serial of 6-link rigid, non-redundant, fully actuated robot manipulator can be written as [31]:

$$\left(D\left(q\right)+J\right)\ddot{q}+(C\left(q,\dot{q}\right)+{\displaystyle \frac{B}{r}})\dot{q}+\mathrm{g}\left(q\right)=u$$

(26)

Where $q ϵ {\mathbb{R}}^{6X1}$ is the vector of joint positions, and $u ϵ {\mathbb{R}}^{6X1}$ is the vector of electrical power input from DC motors inside joints, $D\left(q\right)$ $ϵ {\mathbb{R}}^{6X6}$ is the symmetric positive defined matrix, $C\left(\mathit{q},\dot{\mathit{q}}\right) ϵ {\mathbb{R}}^{6X6}$is the vector of centripetal and Coriolis effects, $g\left(\mathit{q}\right) ϵ {\mathbb{R}}^{6X1}$ is the vector of gravitational torques, $Jϵ {\mathbb{R}}^{6X6}$ is a diagonal matrix expressing the sum of actuator and gear inertias, $B$ $ϵ {\mathbb{R}}^{6X1}$is the damping factor, $r$ $ϵ {\mathbb{R}}^{6X1}$is the gear ratio.

5. Control Policy Diagram

Figure 7 illustrates the overall control system architecture, designed to guide the robot manipulator so that the camera reaches its desired pose, $\overline{{pose}^O}ϵ {\mathbb{R}}^{4X4}$, in the world space. To achieve this, three fiducial markers are placed within the camera’s field of view, with their coordinates in the inertial frame pre-determined and represented as $P^Oϵ {\mathbb{R}}^{9X1}$. Using the robot’s inverse kinematics and the eye-in-hand kinematic model, the expected image coordinates of these fiducial markers, when viewed from the desired camera pose, are computed as $\overline{p_{image}}ϵ {\mathbb{R}}^{9X1}$. These computed image coordinates serve as reference targets in the feedback control loop.

The IBVS framework is implemented within a cascaded feedback loop. In the outer control loop, the camera controller processes the visual feedback error, $e_p$, which represents the difference between the image coordinates of the fiducial points at the current and desired camera poses. Based on this error, the outer loop generates reference joint angles, $q_{feedback}ϵ {\mathbb{R}}^{6X1}$, to correct the robot’s configuration.

The inner control loop, shown in Figure 8, incorporates the robot’s dynamic model to regulate the joint angles, ensuring they align with the commanded reference angles, $q_{ref}ϵ{\mathbb{R}}^{6X1}$. However, due to the limitations of low-fidelity and inexpensive joint encoders, as well as inherent dynamic errors such as joint compliance, high frequency noises and low frequency model disturbances are introduced into the system. All sources of errors from the joint control loop are collectively modeled as an input disturbance, $d_{q_T}$ $ϵ{\mathbb{R}}^{6X1}$, which affects the outer control loop.

The feedforward control loop operates as an open-loop system, quickly bringing the camera as close as possible to its target pose, despite the presence of input disturbances. The feedforward controller outputs a reference joint angle command, ${ q}_{feedforward} ϵ{\mathbb{R}}^{6X1}$, which is sent to the inner loop to facilitate rapid convergence to the desired configuration.

6. Controller Designs

6.1. Inner Joint Angle Control Loop

As shown in Figure 8, the primary goal of the inner joint controller is to stabilize the manipulator dynamics, which is expressed as a nonlinear Equation (26).

Simplify Equation (5.1) as follows:

$$M\left(q\right)\ddot{q}+h\left(\mathrm{q}, \dot{\mathrm{q}}\right)=\mathrm{u}$$

(27)

With:

$$M\left(q\right)= D\left(q\right)+J$$

(28)

$$h\left(\mathrm{q}, \dot{\mathrm{q}}\right)=(C\left(q,\dot{q}\right)+{\displaystyle \frac{B}{r}})\dot{q}+\mathrm{g}\left(q\right)$$

(29)

Then, transform the control input as following:

$$u= M\left(q\right)v+h\left(\mathrm{q}, \dot{\mathrm{q}}\right)$$

(30)

where $v$ $ϵ{\mathbb{R}}^{6X1}$is a virtual input. Then, substitute for $u$ in Equation (27) using Equation (30), and since $M\left(q\right)ϵ {\mathbb{R}}^{6X6}$ is invertible, we will have a reduced system equation as follows:

$$\ddot{q}=v$$

(31)

This transformation is feedback linearization technique with the new system equation given in Equation (31). This equation represents 6 uncoupled double integrators. The overall feedback linearization method is illustrated in Figure 9. In this control block diagram, the joint angle $q$ $ϵ{\mathbb{R}}^{6X1}$are forced to follow the target joint angle $q_R ϵ{\mathbb{R}}^{6X1}$. The Nonlinear interface transforms the linear virtual control input $v$ $ϵ{\mathbb{R}}^{6X1}$to the nonlinear control input $u ϵ{\mathbb{R}}^{6X1}$ by using Equation (30). The output of the manipulator dynamic model, the joint angles, $q ϵ{\mathbb{R}}^{6X1},$ and their first derivatives, $\dot{q} ϵ{\mathbb{R}}^{6X1}$, are utilized to calculate $M\left(q\right)$ $ϵ {\mathbb{R}}^{6X6}$and $h\left(\mathrm{q}, \dot{\mathrm{q}}\right)$ ${ ϵ\mathbb{R}}^{6X1}$in the Nonlinear interface. The linear joint controller is designed using Youla parameterization technique [27] to control the nominally linear system in Equation (31).

The design of a linear Youla controller with nominally linear plant is presented next.

Since the transfer functions between all inputs to outputs in Equation (31) are the same and decoupled, it is valid to first design a Single Input and Single Output (SISO) controller and use the multiple of the same controller for a six-dimension to obtain the Multiple Input and Multiple Output (MIMO) version. In other words, first design a controller $G_{C_{SISO}}^{Inner}$that satisfies:

$$v_{SISO}={\ddot{q}}_{SISO}$$

(32)

where $v_{SISO}$ is a single input to a nominally linear system and ${\ddot{q}}_{SISO}$is the second order derivative of a joint angle. The controller in Figure 9 can be then written as:

$$G_{C_{MIMO}}^{Inner}=G_{C_{SISO}}^{Inner}·I_{6X6}$$

(33)

where $I_{6X6}$ is a $6\times 6$ identity matrix. The transfer function of the SISO nominally linear system from Equation (31) is:

$$G_{p_{SISO}}^{Inner}={\displaystyle \frac{1}{s^2}}$$

(34)

Note that $G_{p_{SISO}}^{Inner}$has two Bounded Input Bounded Output (BIBO) unstable poles at origin. To ensure internal stability of the feedback loop, the closed loop transfer function, $T_{SISO}$, should meet the interpolation conditions [32]:

$$T_{SISO}^{inner} \left(s=0\right)=1$$

(35)

$${\displaystyle \frac{d{T}_{SISO}^{inner}}{ds}}{|}_{s=0}=0$$

(36)

Use the following relationship to compute a Youla transfer function: $Y_{SISO}$ as:

$$T_{SISO}^{inner}= Y_{SISO}^{inner}{G}_{p_{SISO}}^{Inner}$$

(37)

The $T_{SISO}^{inner}$ is designed so that it satisfies the conditions in Equations (35) and (36). The sensitivity transfer function, $S_{SISO}^{inner}$, is then calculated as follows:

$$S_{SISO}^{inner}=1-T_{SISO}^{inner}$$

(38)

Without providing the design details, the closed-loop transfer function can be in the following form to satisfy the interpolation conditions:

$$T_{SISO}^{inner}= {\displaystyle \frac{(3{\tau }_{in}s+1)}{{({\tau }_{in}s+1)}^3}}$$

(39)

Where ${\tau }_{in}$specifies the pole and zero locations and represents the bandwidth of the control system. ${\tau }_{in}$can be tuned so that the response can be fast with less-overshoot.

The next step is to derive $G_{C_{SISO}}^{Inner}$ from relationships between the closed-loop transfer function, $T_{SISO}^{inner}$, the sensitivity transfer function, $S_{SISO}^{inner},$ and the Youla transfer function, $Y_{SISO}^{inner},$ in Equations (40)–(42):

$$Y_{SISO}^{inner}=T_{SISO}^{inner}{G_{p_{SISO}}^{Inner}}^{-1}={\displaystyle \frac{s^2(3{{\tau }_{in}}^{2}s+1)}{{({\tau }_{in}s+1)}^3}}$$

(40)

$$S_{SISO}^{inner}=1-T_{SISO}^{inner}={\displaystyle \frac{s^2({{\tau }_{in}}^{3}s+3{{\tau }_{in}}^2)}{{({\tau }_{in}s+1)}^3}}$$

(41)

$$G_{C_{SISO}}^{Inner}=Y_{SISO}^{inner}{S_{SISO}^{inner}}^{-1}={\displaystyle \frac{3{{\tau }_{in}}^{2}s+1}{{{\tau }_{in}}^{3}s+3{{\tau }_{in}}^2}}$$

(42)

From Equation (33), a MIMO controller can be computed as follows:

$$G_{C_{MIMO}}^{Inner}= {\displaystyle \frac{3{{\tau }_{in}}^{2}s+1}{{{\tau }_{in}}^{3}s+3{{\tau }_{in}}^2}} ·I_{6\times 6}$$

(43)

Equation (43) provides the expression of the inner joint controller and the closed loop of the inner loop can be expressed as:

$$T_{MIMO}^{inner}= {\displaystyle \frac{(3{\tau }_{in}s+1)}{{({\tau }_{in}s+1)}^3}} ·I_{6\times 6}$$

(44)

6.2. Feedforward Control Loop

The feedforward loop is an open loop, disturbed by input disturbances as shown in Figure 10. Feedforward controller is the inverse process of Inner-loop Joint control loop $T_{MIMO}^{inner},$ whose closed-loop transfer function is given in Equation (44).

Therefore, the feedforward controller can be designed as

$$T_{forward}={\displaystyle \frac{1}{T_{inner-closed}}}{\displaystyle \frac{1}{({{\tau }_{forward}s+1)}^2}}={\displaystyle \frac{{({\tau }_{in}s+1)}^3}{(3{\tau }_{in}s+1)}}{\displaystyle \frac{1}{({{\tau }_{forward}s+1)}^2}}·I_{6\times 6}$$

(45)

The double poles s = -1/${\tau }_{forward}$ are added to make $T_{forward}$ proper. Choose ${\tau }_{forward}$ so that the added double poles are 10 times larger than the bandwidth of the original improper $T_{forward}.$ In other words, ${\tau }_{forward}$ is chosen as

$${\tau }_{forward}=0.1{\tau }_{in}$$

(46)

6.3. Outer Feedback Control Loop

In control system block diagram (Figure 7), the linear closed loop transfer function has been developed in equation (44) and the eye-in-hand kinematic model is a nonlinear map defined in Equations (21) and (22).

When disturbed joint angles $\check{q}$are inputs to the eye-in-hand model, the outputs are expressed as $\widehat{p_{image}.}$ Revise Equation (22) accordingly, we have:

$$\widehat{p_{image}}=\mathcal{F}(\check{q}\left(t\right), Pa, P^O)$$

(47)

Where $P^Oϵ {\mathbb{R}}^{9X1},$ are fiducial markers’ coordinates in inertial frame, and $Pa$ are constant parameters, which consist of camera intrinsic parameters, robot geometric parameters, and transformation matrix between the end-effector to the camera.

By choosing a set of linearized points ${\breve{q}}^0ϵ {\mathbb{R}}^{6X1}$, the model expressed in Equation (47) can be linearized with those points in Jacobian matrix form as:

$$\widehat{p_{image}}=J\left({\check{q}}^0,Pa,P^O\right)\check{q}\left(t\right)+\mathcal{F}({\check{q}}^0,Pa,P^O)$$

(48)

Where $J\left({\check{q}}^0,Pa,P^O\right)ϵ{\mathbb{R}}^{6X6}$ is the Jacobian matrix of $\mathcal{F}(\check{q}\left(t\right),Pa,P^O)$ evaluated as $\check{q}={\check{q}}^0.$ Assuming $C_1=J\left({\check{q}}^0,Pa,P^O\right)$, $C_2=\mathcal{F}(\check{q}\left(t\right),Pa,P^O)$ , therefore, Equation (48) can be rewritten as:

$$\widehat{p_{image}}=C_1\check{q}\left(t\right)+C_2$$

(49)

Let’s define $\widehat{p_{image}}\text{'}= \widehat{p_{image}}-C_2$, then, the overall block diagram of the linearized system is shown in Figure 11.

The linearized plant transfer function is derived as:

$${G_p}_{MIMO\_outer}^{linear}={\displaystyle \frac{\widehat{p_{image}}\text{'}}{q_{feedback}}}=C_1{\displaystyle \frac{\left(3{\tau }_{in}s+1\right)}{{\left({\tau }_{in}s+1\right)}^3}}·I_{6\times 6}$$

(50)

As $C_1$ is coupled, the first step to derive an observer for the multivariable system using model linearization is to find the Smith-McMillan form of the plant [32].

To get Smith-McMillan form, we can decompose ${G_p}_{MIMO\_outer}^{linear}$ $ϵ {\mathbb{R}}^{9X6}$with singular value decomposition (SVD) as:

$${G_p}_{MIMO\_outer}^{linear}=U_L{M}_p{U}_r$$

(51)

where $U_Lϵ{\mathbb{R}}^{9X9}$and $U_r$ $ϵ{\mathbb{R}}^{6X6}$are the left and right unimodular matrices, and $M_p$ $ϵ{\mathbb{R}}^{9X6}$is the Smith-McMillan form of ${G_p}_{MIMO\_outer}^{linear}.$

$M_p$ is a diagonalized transfer function with each nonzero entry equals to a gain multiple the transfer function ${\displaystyle \frac{(3{\tau }_{in}s+1)}{{({\tau }_{in}s+1)}^3}}$ ; For the $i^{th}$row of $M_p$ the entry on the diagonal is:

$$M_p\left(i,i\right)=gain\left(i\right)·{\displaystyle \frac{\left(3{\tau }_{in}s+1\right)}{{\left({\tau }_{in}s+1\right)}^3}}, iϵ(1,2,3,4,5,6)$$

(52)

Where $gain$ $ϵ {\mathbb{R}}^{6X1}$ is a numerical vector.

The design of a Youla controller for each nonzero entry in $M_p$ is trivial in this case as all poles/zeros of the plant transfer function are in the left half-plane, and therefore, they are stable. In this case, the selected decoupled Youla transfer function: $M_Y$can shape the decoupled closed loop transfer function,$M_T$, by manipulating poles and zeros. All poles and zeros in the original plant can be cancelled out and new poles and zeros can be added to shape the closed-loop system. Let’s select a Youla transfer function so that the decoupled closed-loop SISO system behaves like a second order Butterworth filter, such that:

$$M_T= {\displaystyle \frac{{{\omega }_n}^2}{(s^2+2\mathsf{\zeta }{\omega }_{n}s+{{\omega }_n}^2)}}·\left[\begin{array}{cc}{I}_{6\times 6}& {Zero}_{3\times 3}\\ {Zero}_{3\times 3}& {Zero}_{3\times 3}\end{array}\right],$$

(53)

where ${\omega }_n$ is called natural frequency and approximately sets the bandwidth of the closed –loop system. It must be ensured that the bandwidth of the outer-loop is smaller than the inner-loop, i.e., ${1/\omega }_n>{\tau }_{in}$. $\mathsf{\zeta }$ is called the damping ratio, which is another tuning parameter. ${Zero}_{3\times 3}$ is a 3$\times$3 matrix with all entries equaling to zero. Note that the coordinates from the last point cannot be controlled in the feedback loop.

Then we can compute the decoupled diagonalized Youla transfer functions $M_Y ϵ {\mathbb{R}}^{6X9}.$The diagonal entry of $i^{th}$ row is denoted as $M_Y(i,i)$:

$$M_Y\left(i,i\right)={\displaystyle \frac{M_T\left(i,i\right)}{M_p\left(i,i\right)}}={\displaystyle \frac{1}{gain\left(i\right)}}{\displaystyle \frac{{{\omega }_n}^2}{\left(s^2+2\mathsf{\zeta }{\omega }_{n}s+{{\omega }_n}^2\right)}} {\displaystyle \frac{{\left({\tau }_{in}s+1\right)}^3}{\left(3{\tau }_{in}s+1\right)}},, iϵ(1,2,3,4,5,6)$$

(54)

Similar to Equations (40)–(42), the final coupled Youla, closed loop, sensitivity, and observer transfer function matrices are computed as:

$$Y_{MIMO\_outer}^{linear}ϵ {\mathbb{R}}^{6X9}=U_R{M}_Y{U}_L$$

(55)

$${T_y}_{MIMO\_outer}^{linear}ϵ {\mathbb{R}}^{9X9}={G_p}_{MIMO\_outer}^{linear}·Y_{MIMO\_outer}^{linear}$$

(56)

$${S_y}_{MIMO\_outer}^{linear}ϵ {\mathbb{R}}^{9X9}=1-{T_y}_{MIMO\_outer}^{linear}$$

(57)

$${G_C}_{MIMO\_outer}^{linear}ϵ {\mathbb{R}}^{6X9}=Y_{MIMO\_outer}^{linear}·({{S_y}_{MIMO\_outer}^{linear})}^{-1}$$

(58)

The controller developed in the above section is based on the linearization of the combined model at a particular linearized point ${\check{q}}^0.$ This controller can only stabilize at certain range of joint angles around ${\check{q}}^0.$ As current joint angles ${\check{q}}^0$ deviates from $\check{q},$ the error between the estimated linearized system (48) and the true nonlinear system (47) increases.

To tackle this problem, we develop an adaptive controller that is computed online based on linearization of the model at current joint angles. This control process is depicted in Figure 12.

The first step is to estimate current joint angles $\tilde{q}$ from current measured images coordinates $\widehat{p_{image}}.$ The mathematic models of eye-in-hand kinematic model been given in expression is defined in Equation (22). Therefore, the mathematical function of the inverse model can be derived and expressed as:

$$\tilde{q}={\mathcal{F}}^{-1}(\widehat{p_{image}}, Pa, P^O)$$

(59)

Where ${\mathcal{F}}^{-1}$ is the inverse process of Equation (22), which is a combination of coordinate system transformation from the image frame to the end-effector frame and robot inverse kinematics process. Given estimated current angle $\tilde{q}$, we can calculate the Jacobian matrix of the nonlinear model at current time. By obtaining left and right unimodular matrices and Smith-McMillan form from singular value decomposition, the current linear controller ${G_C}_{MIMO\_outer}^{linear}$ can be built by Equations (55)-(58).

7. Simulations Results

To evaluate the performance of our controller design, we simulated two scenarios in MATLAB Simulink using a Zed 2 stereo camera system [33] and an ABB IRB 4600 elbow robotic manipulator [30]. The specifications for the camera system and robot manipulator are summarized in the tables in the appendix. The camera system performs 2D feature estimation of three virtual points in space, with their coordinates in the inertial frame selected as: ${[\text{-}0.5m 0 0]}^T$, ${[0 0 0.5m]}^T$, ${[2m, \text{-}2\mathrm{m}, 0]}^T$.

Many camera noise removal algorithms have been proposed and shown to be effective in practical applications, such as spatial filters [34], wavelet filters [35], and the image averaging technique [36]. Among these denoising methods, there is always a tradeoff between computational efficiency and performance. For this paper, we assume that the images captured by the camera have been preprocessed using one of these methods, and the noise has been almost perfectly attenuated. In other words, the only remaining disturbances in the system are due to unmodeled joint dynamics, such as compliance and flexibility, which are modeled as input disturbances in the controlled system.

Two scenarios were simulated:

• Scenario 1: Without input disturbances.
• Scenario 2: With a 1° step input disturbance added to each joint of the robot arms for the entire simulation time.

In both scenarios, the camera system starts from an initial pose in the inertial frame, denoted as ${Pose}_{initila}^O$, and maneuvers target pose, denoted as ${Pose}_{final}^O$. Table 1 summarizes the initial and final poses for each scenario, along with the corresponding joint configurations.

Figure 13 and Figure 15 present the responses of the six joint angles for the two scenarios, respectively. Figure 14 and Figure 16 show the responses of the nine image coordinates over time for each scenario. In each case, the feedback-only controlled system (left plot) is compared to the feedforward-and-feedback controlled system (right plot). These comparisons focus on overshoot, response time, and target tracking performance.

Both scenarios are simulated with bandwidth of the inner-loop as 100rad/s and the bandwidth of the outer-loop as 10rad/s.

The response plots indicate that both the feedback-only controller and the combined feedforward-and-feedback controller successfully stabilize the system and reach a steady state within three seconds, even in the presence of small input disturbances (Scenario Two). However, the feedback-only controller fails to guide the camera to its desired pose and falls into local minima, as evident from the third point’s coordinates (${ul}_{3,} {ur}_{3,}$ $v_{3,}$), which do not match the target at steady state. This issue arises from the overdetermined nature of the stereo-based visual servoing system, where the number of output constraints (9) exceeds the degrees of freedom (DoFs) available for control (6). As a result, the feedback controller can only match six out of nine image coordinates, leaving the rest coordinates unmatched.

In contrast, the feedforward-and-feedback controller avoids local minima and accurately moves the camera to the target pose. This is because the feedforward component directly controls the robot’s joint angles rather than image features. Since the joint angles (6 DoFs) uniquely correspond to the camera’s pose (6 DoFs), the feedforward controller helps the system reach the global minimum by using the desired joint configurations as inputs.

When comparing performance, the system with the feedforward controller exhibits a shorter transient period (less than 2 seconds) compared to the feedback-only system (less than 3 seconds). However, the feedforward controller can introduce overshoot, particularly in the presence of disturbances. This occurs because feedforward control provides an immediate control action based on desired setpoints, resulting in significant initial actuator input that causes overshoot. Additionally, a feedforward-only system is less robust against disturbances and model uncertainties. Fine-tuning the camera’s movement under these conditions requires a feedback controller.

Therefore, the combination of feedforward and feedback control ensures fast and accurate camera positioning. The feedforward controller enables rapid convergence toward the desired pose, while the feedback controller improves robustness and corrects errors due to disturbances or uncertainties. Together, they work cooperatively to achieve optimal performance.

8. Conclusions

In this article, we first provide a systematic proof of the PnP problem for a stereo camera system and then propose an innovative control policy to address the overdetermination issues in image-based visual servoing (IBVS) control. Results from two simulation scenarios demonstrate that the proposed algorithm successfully brings the camera to the desired pose with high accuracy and speed.

Several existing approaches [21,22,23], mentioned in the introduction, have also addressed the issue of local minima in IBVS. Compared to those methods, the key advantage of our system is its simplicity and ease of implementation. A linear feedforward controller is sufficient to handle the local minimum problem without requiring complex online optimization as in MPC or additional 3D feature measurements as in 2 ½-D visual servoing. The feedback loop is designed following the traditional IBVS structure but incorporates a higher-fidelity dynamic model. The adaptive features in the feedback controller stabilize the system across the entire state space, achieved by combining multiple linear Youla-parameterized controllers. To reduce computational overhead, we can lower the online update frequency or predesign several linear Youla controllers offline and switch between them smoothly using a switching algorithm.

However, the feedforward design introduces challenges, particularly with large overshoots that can cause erratic joint movements. This may increase the risk of accidents and potential damage to the robot. A possible solution is to optimize the controller parameters—such as bandwidth and damping ratios—which can be explored in future work. In addition, feedback and feedforward controllers can be designed simultaneously using $H_{\infty }$ control techniques [37], which optimizes the system’s stability and performance in the presence of disturbances.

In summary, this paper investigates the overdetermination problem in stereo-based IBVS tasks. While future improvements to the algorithm are possible, the proposed control policy has demonstrated significant potential as an accurate and fast solution for real-world eye-in-hand (EIH) visual servoing tasks.