Development of Continuum Robot Arm and Gripper for Harvesting Cherry Tomatoes

Designing and development of agricultural robot is always a challenging issue, because of robot intends to work an unstructured environment and at the same time, it should be safe for the surrounded plants. Therefore, traditional robots cannot meet the high demands of modern challenges, such as working in confined and unstructured workspaces. Based on current issues, we developed a new tomato harvesting wire-driven discrete continuum robot arm with a flexible backbone structure for working in confined and extremely constrained spaces. Moreover, we optimized a tomato detaching process by using newly designed gripper with passive stem cutting function. Moreover, by designing the robot we also developed ripe tomato recognition by using machine learning. This paper explains the proposed continuum robot structure, gripper design, and development of tomato recognition system


Introduction
The tomato is a high-demand fruit globally and consumption is gradually increasing year by year.Manual harvesting of tomatoes is laborious, time-consuming and inefficient, which makes it impractical in large-scale fields.Moreover, the tomato is very soft and prone to bruising, which makes harvesting difficult and grasping process too.Furthermore, one of the main problems in the agriculture sector is rising labour costs and labour shortages because agriculture work is not popular among young people.Therefore, robotization of tomato harvesting could be one of the most efficient solutions to eliminate the above-mentioned problems.However, many technological issues remain to be solved when we consider robotization.This paper proposes one approach for robotization of tomato harvesting.
The main components in designing harvesting robots are a moving platform, a reaching manipulator, a grasping tool and a tomato recognition algorithm.In this research, we propose a manipulating robot arm, a grasping tool and tomato recognition system.
Firstly, the main component of the harvesting manipulator is the arm.Many researchers and engineers have proposed various types of robot arms for harvesting purposes.Many examples use a commercially available rigid link manipulator such as KUKA and Universal Robots [1][2][3].Similarly, Wang et.al.utilized a SCARA type serial manipulator mounted on the top of the mobile robot [4].However, Takaaki proposed hard continuum manipulator with a flexible structure, which has proven to be safe with wide reachability but possessing a low payload capacity [5].Similar harvesting robot arms are proposed by Van et.al.designed and developed a cucumber harvesting robot arm utilizing thermal cutter, and its harvesting successful percentage was 74% [6,7].Thus, proposed continuum manipulator has a unique design and improved rigidity to handle reaching and manipulation of the object.
The second challenging issue in fruit harvesting is the gripper design.The tomato is a fragile and soft fruit, so grasping the tomato should be gentle to avoid any bruising and smashing.Moreover, detaching the tomato is also challenging.For instance, Wang et.al.developed a gripper with a clamp mechanism to cut the stem after grasping the tomato [2].Similarly, Zhao et.al proposed a dual-arm SCARA robot, where one arm holds a tomato and the second arm cuts the stem.This requires additional training for tomato stem detection, and the two arms should work synchronously with precision.Some researchers propose a vacuum mechanism with a sucking function and scissors [8].Furthermore, Hiroaki et.al.proposed a plucking gripper with an infinite rotational joint to automatically detach tomatoes [9], but the success rate of detaching in the real application was only 60%.A similar gripper design has also has been proposed by Root Ai Company and their Virgo robot having a SCARA type arm that detaches tomatoes by twisting them after grasping.[10].Panasonic also presented a commercially available prototype tomato picker robot.The robot spends only 2-3 seconds per tomato and now it is the fastest prototype [11].However, the above-mentioned prototypes cannot detach tomatoes with their sepal, which may cause problems in terms of tomato durability during transportation.Thus, in our proposed prototype design there are two semispherical cups with sharp blades on the edges to cut the stem with sepal in a passive mechanical way.
For harvesting work, a tomato recognition system is also a critical issue.There is much research on tomato recognition systems proposing some methods on discriminating of mature tomatoes from non-mature tomatoes.One of the popular methods of tomato detection uses image data obtained by an RGB stereo camera [12][13][14][15][16][17].Furthermore, the infrared method of tomato detection also could be used, which easily discriminate tomato colour by using reflected light.[18].In order to improve the tomato harvesting process, we employed a machine learning based on a neural network to detect tomatoes and identify ripeness of the tomato.This method could also be used for tomato classification, and it improves recognition accuracy [19][20][21][22][23].
In this research, we present a new tomato harvesting robot system with an arm and a tomato detection system.As a manipulating arm, we use the previously presented discrete wire-driven continuum robot arm TakoBot [24,25].This paper is organized in the following order: design concept, kinematic/kinetic formulation, development of the recognition system and experimental results.

Continuum part design
According to the intended application, the robot should have the following features:  Flexible structure to work in the confined workspace  Rigid structure and decent precision of motion  The payload capacity of over 100 g  Tomato grasping with no damage  Ripe tomato discrimination system  Tomato detaching mechanism  The selectivity of picking of tomato Based on the following requirements for the robot, we proposed continuum manipulator named TakoBot which can work in unstructured workspace and can be safe for the surrounded environment as well.
TakoBot is a discrete hyper-redundant cable-driven continuum robot arm with two actuating sections.TakoBot consists of three main parts: continuum arm, pretension unit and control box.The continuum part divides into two sections five segments each, and each segment consist of two spacer discs whish are interconnected by universal joint and four compression springs for rigidity.In the center of the disc mounted linear bearings which allow to linear shaft slide along the slender part.Moreover, pretension part which is designed to compensate cable tension and provide rigidity to the robot structure.The last part is a control box, which actuates wires (Fig. 1).
As demonstrated in Fig 2, spacer disc can slide along the guide shaft.Mechanically, the disc can travel up to 10 mm only, but such kind sliding does not affect the total manipulator length, such design assists to distribute internal stress along the slender part.Likewise, the design provides smart bending stress distribution for the nearest segments compressional springs.Furthermore, such a design provides sharper bending angle, and compensates a wire tension during the motion and prevents wire derailing from the pulleys.
In TakoBot actuates two sections: first section (end section) and the second section (midsection).Each section contains five segments and section's spring constants are variable, for instance, midsection compressional springs are harder than first section compression springs.Likewise, the approach provides the necessary robot rigidity to improve robot accuracy.

Pretension part
The wire tension in wire-driven continuum robots is a challenging issue.Wire-tension slack becomes more significant when the actuating system is paired, which means when one motor control has two wires (push-pull mode).Furthermore, in the proposed manipulator there are two sections, where the tension problem is more serious, especially when the mid-section actuates the tip-section.The wire tension changes and it loses tension as well.To avoid such a problem, all motors should work simultaneously to keep the required tension; technically such an approach requires additional feedback sensors to control motors.In this manipulator, we proposed a passive device to prevent cable slack during movement in a passive mechanical way.The pretension mechanism can compensate for the cable tension of eight cables simultaneously and does not require any sensors.This enhances device reliability and applicability in severe environments.
The pretension mechanism consists of five parts: a pretension octagon base, an inner part, springs with shafts, rollers and roller holders.The octagon base and inner part are connected by paired linear shafts, and the roller holders slide along the shaft on the mounted the linear bearings.To prevent wire friction, the PtM device is equipped with idler pulleys (Fig. 3).

Actuating unit
TakoBot actuates wires by the linear lead screw attached to the stepping motor with a rated torque of 0.49 N/m.In total, TakoBot utilizes four stepping motors: two motors for the first section and two motors for the second section.Each motor drives two wires using the push and pull principle.The linear lead shaft is attached to the stepping motors.The cables are fixed to the screw housing with wire sleeves.(Fig. 4).To avoid any friction between wires and robot parts, we equipped the robot with additional idlers.

Gripper design
Designing the gripper tool for harvesting tomatoes is a challenging issue because the tomato is a soft and juicy fruit.Grasping the tomato should be gentle to prevent any failures such as overpressure.Moreover, the tool design should consider detaching the tomato from the tomato stem (Fig. 6).Therefore, we designed a gripper tool with a semi-spherical shape for grasping spherical objects such as a tomato.For detaching tomatoes, we added cutting blades on the edges of the cup.This design makes it possible to separate the tomato in a passive way, and it improves the harvesting time (Fig. 5).The gripper cup size is defined by calculating the average size of the cherry tomato.According to our calculations, the diameter of cherry tomatoes is 30-35 mm.Compared with other grippers this prototype can separate in a passive way and does not require sensors to control.A lack of electronics allows the robot to work in a wet and highly humid environment.Furthermore, the proposed gripper can cut the tomato with its sepal which helps to increase the storage time for harvested tomatoes, while other prototypes leave the sepal on the stem.

Tomato recognition system
Mature tomato recognition is also a critical part of the robot.In this research, we used machine learning to train neural networks to distinguish ripe tomatoes from non-ripe ones and from other similar fruits.As a tomato classifier, we utilized a YOLO.We developed three main filters to discriminate for tomatoes.The first filter detects the shape of the object, the next filter for colour or RGB filter, and the third filter is machine learning.This is required because there are many fruits or objects which might be very similar to the tomato, such as an apple or tangerine (Fig. 7).Moreover, using a classification camera should make it possible to calculate the distance between the camera and the tomato.For such a technique, we used a measurement method with a single camera by using the focal length of the camera.The dataset was collected in Almaty Tomato Greenhouse, Kazakhstan.The distance measurement feedback from the camera gives an opportunity to calibrate the gripper tool for perfectly grasping the tomato without any damage (Fig. 8).Here is o-opposite, h-hypotenuse and a-adjacent.The opposite distance is fixed, and the adjacent distance comes with the camera.Based on this formulation, we can calibrate the gripper position for accurate grasping.In figure 9, the camera measures the distance and detects matured tomatoes.This experiment had been conducted in Kawasaki tomato greenhouse, Japan.In this research, we utilized a borescope camera with 2-megapixel resolution.
where,  0 and  0 are an initial position of the base.  (  ) and   (  ) are rotation matrices of ith universal joint that has two rotation angles   and   ,   ( 1 ) is a rotation matrix of the ith disk with a rotation angle  1 along the axial axis and L is a length between neighbouring universal joints (Fig. 10).
Multiplying the H-matrices successively, we get unit vectors and the position vector of the ith coordinate system; where,   is the position of the ith universal joint Ui ( = 1, ⋯ ,  − 1).
The position vector   of the end-point P  and position of sliding plates P i (i=1, ⋯, n-1) of the manipulator are obtained by, where,   is a fixed length between the nth universal joint and the most distal plate.
where,   is an axial length between the ith universal joint and the ith plate, which varies as the plate slides along rods, except   .

Kinetic formulation
TakoBot has two actuating sections: the first section ( distal part) and the second section (proximal part).Each section operates by two motors actuated by paired two cables, so there are 8 cables total.Kinetic formulation describes the motion with force in combination with springs and motor angle.Moreover, in this formulation we also need to consider pretension mechanism formulation in order to calculate wire tension.
The second segment has m units and the first segment has n-m units.
where, a n -a n-1 = a n -a n-1 |a n -a n-1 | , etc. m  is a payload applying at the end-point and g is the gravity acceleration vector.
Equilibrium in moments at Ui , (i=m+1, ⋯, n-1), belonging to the first segment is where.f a , f a ̂, f c , f c ̂ are wire tensions, S a,i , S a ̂,i , S c,i , S c ̂,i , (i=m+1, ⋯, n) are spring tensions of the ith unit."×" means a cross product and "|*|", means the modulus of a vector * .  is the mass of one unit including the plate, the rod and the universal joint (Fig. 13).
Equilibrium in moments at Um , the universal joint located at the most distal position belonging to the second segment is For the second segment, we can derive similar equations as ( 8), ( 9) and ( 10) by replacing {a i , a ̂i, c i , c ̂i} with {b i , b ̂i, d i , d ̂i}, {S a,i , S a ̂,i , S c,i , S c ̂,i } with {S b,i , S b ̂,i , S d,i , S d ̂,i } for i=1, ⋯, m-1 in (8) and for i=1, ⋯, m in ( 9) and (10).
As a result, we obtain 4m equations included by (11), which suffices in number to solve for 4m variables; θ x,i , θ y,i , θ z,i and l i (i=1, ⋯,m) for a given set of wire tensions   ,   ̂,   ,   ̂ (Fig. 11).
where,   is a motor rotation angle to generate a pretension,  is a lead of the screw rod and   is the spring constant of the pretension spring.
where 2̅  and 2̅ ̂  compression length of the pretension spring, which are preset initially.
Substituting Equation ( 12) into Equation ( 13), we have; where   is a motor rotation angle to generate a pretension,  is a lead of the screw rod and   is the spring constant of the pretension spring.
Eq.( 19) provides a variation of motor angles ∆ϕ for a given position and direction variation ∆p n .
Applying the Euler method, we have the following variational equation, ,which is solved by As a candidate of φ , we take φ=k nz 2 , where, k  is the z component of k n : the unit vector of the end-point orienting an axial direction.It means that the axial direction the end-point takes on a horizontal plain as far as possible while keeping a designated position (Fig. 14).

Control
TakoBot's control architecture consists of two main parts: software and hardware (Fig. 15).The work process starts from the software.Firstly it scans ripe tomatoes.Then, after detection of the tomato, the camera measures the distance.Finally, the measured information helps calculate the robot's inverse kinematics.Calculated inverse kinematics sends the information and coordinates of  TakoBot has six motors, one micro-servo motor for the gripper tool and five bipolar stepping motors.Four stepping motors are dedicated to the manipulator, and one motor is for the linear slider.Power consumption is divided into two parts as well.The motor and motor drivers (TMC2208) consume 12V, but the Arduino board and micro-servo motor consumes 6V.
For tomato selection, we made an algorithm which measures the priority value P. First of all, camera detects several tomatoes and to make decision which tomato should come first and following picking order.We designed two dimensional planes where it indicates tomato location in the frame, after it would be measured by horizontal value H and vertical value V, and sum of two values gives priority value P. Tomato selection priority starts from the lowest value to highest.Tomato picking diagram illustrated in the figure 16.

Experiment
As a controller, we used an Arduino uno board and TMC2208 motor drivers for the stepping motors.For continuous work we also equipped the electronic parts with a cooler.TakoBot has four motors and one stepping motor for linear motion along the linear slider, and one micro-servo motor in the end-effector.
For the experiment, we fabricated cherry tomatoes and hung them in front of the manipulator.The task was to reach and grasp the tomato, detach it from the stem and put it into the basket.Therefore, during the experiment we also tested robot manipulability such as reaching to the object from various   According to the conducted experiment observations, TakoBot demonstrated high feasibility in working in a confined workspace as well as reachability.The single arm was enough to perform the given task.
During the experiment we discovered that harvesting time had been increased when obstacles was applied.In such case, the robot slender part spent more time to fit the newly constrained environment and to reach to the object (Fig. 20).Furthermore, the tomato grasping process also takes more time, based on observed results almost half time of the whole harvesting process.However, the success rate of the tomato is pretty high, the manipulator was able to grasp all detected tomatoes, but it spend more time to complete the task.The robot harvesting procedure basically consists of four steps: scanning the tomato (Fig. 18); after detection of the tomato, the robot moves to capture one.The next step after getting close enough is grasping the tomato by the gripper, then detaching it from the stem, and finally putting it in the basket.The newly developed gripper provides smooth detaching of the tomato which increases the harvesting time and simplifies harvesting tasks.Moreover, TakoBot uses only a single arm for all of these procedures which makes the robot compact and cost-effective (Fig. 19).

Conclusion
This paper presented a hyper-redundant wire-driven continuum robot arm for cherry tomato harvesting.Furthermore, the robot design, kinematic/kinetic formulation, tomato recognition system and control architecture of the robot were discussed.As practical evidence, the experiment results were demonstrated.According to the obtained results, the proposed manipulator was able to manipulate the objects up to 200grams and during the harvesting process the robot's slender part reached high stiffness rigidity.Figure 19 shows the captured tomatoes.Such detection wouldn't be possible if the slender shakes.In terms of speed, the proposed manipulator was slower than commercially available prototypes, but the main beneficial point of the robot is reachability and tomato selectivity.The conducted experiment demonstrated manipulator ability in working in an extremely confined environment.Moreover, the proposed gripper simplified the harvesting process by cutting the stem in passive way.
As a future plan, we are planning to use machine learning for manipulator control.According to the obtained data, manipulator spends more time for obstacle avoidance and tomato grasping, so by using machine learning tool, we can improve the harvesting time and to reach the optimal continuum robot shape for fast harvesting.

Figure 3 .
Figure 3. TakoBot a) Pretension mechanism and b) Wire routing schematics and pretension device structure.

Figure 5 .
Figure 5. a) Gripper CAD design and b) Fabricated prototype.

Figure 8 .
Figure 8.The gripper calibration a) CAD view and b) geometric formulation.

Figure 9 .
Figure 9. Testing of the tomato recognition in a real environment.

4 .
Kinematic and Kinetic formulations 4.1.Forward kinematic formulation Coordinate systems are set at every universal joint.The homogeneous coordinate transform matrices: Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 18 December 2019 doi:10.20944/preprints201912.0237.v1 Position vectors of 8 hole a  ,   , ȃ  , ̂ first section and for second section b  , d  , b ̂, d ̂ at the base plate are determined as,

Figure 11 .
Figure 11.eyelet arrangement for end and mid -section discs.

Preprints
(www.preprints.org)| NOT PEER-REVIEWED | Posted: 18 December 2019 doi:10.20944/preprints201912.0237.v1 the tomato to the Arduino board.Thus, the Arduino board sends data to the motors to get the desired position.

Figure 18 .
Figure 18.Tomato harvesting procedure.a) scanning and reaching to the tomato, b) grasping of the tomato, c) separating of the tomato, d) putting to the basket.

PreprintsFigure 19 .
Figure 19.Cherry tomatoes capturing and detection during the harvesting process.