1 Gesture-based control of a robot arm with 2 neuromorphic haptic feedback 3

Research on bidirectional human-machine interfaces will enable the smooth interaction 21 with robotic platforms in contexts ranging from industry to tele-medicine and rescue. This paper 22 introduces a bidirectional communication system to achieve multisensory telepresence during the 23 gestural control of an industrial robotic arm. We complement the gesture-based control by means 24 of a tactile-feedback strategy grounding on a spiking artificial neuron model. Force and motion from 25 the robot are converted in neuromorphic haptic stimuli delivered on the user’s hand through a 26 vibro-tactile glove. Untrained personnel participated in an experimental task benchmarking a pick27 and-place operation. The robot end-effector was used to sequentially press six buttons, illuminated 28 according to a random sequence, and comparing the tasks executed without and with tactile 29 feedback. The results demonstrated the reliability of the hand tracking strategy developed for 30 controlling the robotic arm, and the effectiveness of a neuronal spiking model for encoding hand 31 displacement and exerted forces in order to promote a fluid embodiment of the haptic interface and 32 control strategy. The main contribution of this paper is in presenting a robotic arm under gesture33 based remote control with multisensory telepresence, demonstrating for the first time that a spiking 34 haptic interface can be used to effectively deliver on the skin surface a sequence of stimuli emulating 35 the neural code of the mechanoreceptors beneath. 36


40
In the last decades, research about the development of human-robot interfaces for the remote  tracking [16,17]. Differently from RGB-D cameras, the Leap Motion controller explicitly targets the 65 task of tracking the hand pose. Even though its interaction zone is rather limited, the extracted data 66 are very accurate and it is not necessary to perform image processing tasks to extract the relevant 67 points. Authors in [18] report that, although they were not able to achieve the theoretical accuracy of

82
The importance of tactile sense can be better understood when considering all those individuals 83 who experienced its loss. Without tactile information, actions like using tools, holding objects or 84 motor control tasks can become extremely difficult if not impossible to perform [31]. In all those 85 situations where a fine control of mechanical tools or robotic hands is required, the possibility to 86 deliver information from the environment directly on users' skin via tactile feedback can enhance the 87 performance of executed tasks. The activation of a force feedback real-time channel, coming from the 88 sensing elements on the robot, allows the user to receive the aggregated profiles of exerted forces.

89
Touch feedback allows the user to collect information from the contact points (a force array), and the

92
In this work, we are focused on distant human-robot bidirectional physical interaction. We

114
In this section, we will firstly present the three main subsystems which constitute our 115 experimental setup for the remote robot control with tactile telepresence: 1) a hand tracking device 116 which is used to detect the movements of the hand during the execution of the experimental tasks 117 and to remotely control a robotic arm; 2) a vibrotactile glove, equipped with two piezoelectric

311
The purpose of this experiment is to evaluate whether the hand tracking recognition input,

315
We first evaluated the capability of the hand tracking device in the detection of the hands 316 movements in order to control the robotic arm. We used an intuitive approach in which hand posture 317 and motion were transformed into specific commands to be sent to the robot (see Figure 2 for the 318 correspondence between the hand commands and the robot movements).

319
The hand position and velocity profiles were acquired during the execution of the task in both 320 the experimental conditions. These profiles were then compared with the trajectory of the robotic arm 321 velocity during the execution of the task. The velocity profile commanded via the hand tracking 322 device and the corresponding profile of the robotic arm velocity are overlapped in Figure 5 (A).

323
Results showed how the robotic arm is capable to follow the movements of the hand over the hand 324 tracking device. This confirms that the implemented algorithm for the robot control has a suitable 325 dynamics so to enable the robotic arm to follow the hand trajectory and velocity in a reliable manner.

326
The reliability of the gesture-based control is also detectable from the analysis of the commanded 327 velocity profile versus the corresponding robot velocity (see Figure 5 (B)) within the three-328 dimensional experimental workspace (see Figure 5 (C)).

342
As the force value increases, the rate of spikes delivered to the index fingertip increases, while the 343 absence of spikes means that no contact events are detected (Figure 6 (A, C)). The spiking activity on 344 the hand palm is instead representative of the hand distance with respect to the rest position over the  368 condition (Video S1).

384
We described an intuitive gesture-based system for the remote control of a robotic arm with 385 tactile telepresence, which allows the users to perform a pick-and-place-like industrial task. Tactile

387
Vibrotactile information was generated according to neuronal spiking models and delivered directly 388 on the hand palm, with a rate proportional to the hand displacement over a hand-tracking device,

389
and on the fingertip of the index, with a rate proportional to the contact forces exerted by the robot 390 end-effector.

391
Our system has been tested with untrained volunteers, both in the cases where tactile feedback 392 was or was not provided, on an experimental pipeline aimed at emulating activities that can be 393 typically encountered in an industrial context as well as in a whelm of robot remote control 394 applications.

395
The analysis of experimental data shows that the commands acquired via the hand tracking

401
The marker-less technology of the hand tracking device enabled participants to wear the glove 402 and receive tactile feedback during the experimental task without affecting the tracking performance.

403
Furthermore, since the hand tracking is independent from the variation of anthropometry of the