AI-Powered Camera and Sensors for the Rehabilitation Hand Exoskeleton

1. Introduction:

Hand motor impairments due to conditions such as Motor Neurone Diseases (MND) significantly affect a person's ability to perform daily tasks, leading to a decreased quality of life and independence. Such impairments manifest as muscle weakness in the hand and forearm, making it difficult to perform fine motor tasks like writing, buttoning a shirt, or gripping objects. Traditional rehabilitation methods include task-oriented training and the use of assistive devices, but these approaches often require extensive training and are not universally accessible.

Recent advancements in rehabilitation technology have focused on enhancing therapy through the use of robotic exoskeletons. These devices, such as the Pediatric Whole Hand Exoskeleton (PEXO) [1], offer promising solutions by providing actuated assistance tailored to the needs of children with neuromotor disorders, improving their ability to engage in functional tasks. Advancements in wearable robotics have shown significant potential in enhancing the rehabilitation process. For instance, devices like the SCRIPT passive orthosis and the Gloreha hand robotic rehabilitation system have made strides in this field by focusing on hand and wrist rehabilitation through interactive and dynamic assistance [2,3]. Moreover, recent studies highlight the importance of integrating user-friendly and intuitive control systems to maximize the effectiveness of these rehabilitation tools [4,5,6,7,8,9]. However, many existing systems, including [10], rely on active control methods like electromyography (EMG), which require individualized training and adaptation.

To address these limitations, our research presents a vision-enabled rehabilitation hand exoskeleton designed to assist individuals with hand impairments through a simple, intuitive interface. Unlike EMG-based systems, our prototype integrates a camera and embedded processor to automate hand movements using air pressure, allowing for object manipulation without the need for user training. Additionally, an accelerometer detects characteristic hand gestures to facilitate object release, enhancing the overall usability and accessibility of the device [11].

Given the widespread use of camera-based solutions across different systems [12,13], we have adopted this approach in our design. For projects focused on image analysis, single-board computers like the Jetson Nano [14,15,16,17,18], Google Coral [19,20], and Raspberry Pi [21,22,23] are favored for their compact size and low power consumption. To optimize our system, we have used the Google Coral Dev Board Mini to run image processing and control peripherals because of its smaller size and inclusion of a Tensor Processing Unit (TPU) for faster processing of deep learning models.

In our previous research, we implemented a vision-based system on a prosthetic hand [20]. In this project, we plan to implement a vision-based system for an exoskeleton. By employing vision-based control, this project aims to reduce rehabilitation exoskeletons' cost, weight, and power consumption, making them more feasible for widespread implementation. This approach will simplify the user interface as well as significantly improve the independence and quality of life for individuals with motor neuron syndrome and other hand motor impairments. Figure 1 shows the prototype of AI cameras and sensors for hand exoskeleton with battery.

The contributions of this project are,

• We have implemented a novel vision-based system in an exoskeleton for hand
• An object detection model was trained with a custom dataset collected by the developed system

2. Methodology:

Figure 2 shows the block diagram of the vision-enabled rehabilitation hand exoskeleton. It shows the interconnected components that collectively enable automated hand movements. The system's core is a camera and embedded processor, Coral Dev Board mini [24]. The camera (5MP Coral camera [25]), Time of flight (TOF) distance sensor (VL6180X), and accelerometer are connected to the main processor. The camera captures real-time images, which the embedded processor. The switching circuit controls the air pressure system that drives the exoskeleton's finger movement through a solenoid, facilitating the opening and closing of the hand to grasp objects. Additionally, an accelerometer (ADXL345) is integrated to detect specific hand gestures, allowing for the release of objects.

The flowchart of the vision-enabled rehabilitation hand exoskeleton is shown in Figure 3. The image capture stage involves the camera capturing real-time images of the user's hand and its environment. These images are then processed in the Object Detection phase, where computer vision algorithms analyze the visual data to identify the target object. Once the object is determined, the system moves to the control signal generation stage, where the control unit interprets the processed data and generates the appropriate control signals. These signals are then sent to the Air Pressure System, which adjusts the pressure in the actuators to mimic natural hand movements for grasping the object. During this process, the Gesture Detection stage continuously monitors the user’s hand gestures via an accelerometer, allowing for intuitive commands such as releasing the object. The final Action Execution phase involves the actual movement of the exoskeleton’s hand to grasp or release the object. Figure 4 shows a vertical and horizontal view of the vision-based exoskeleton and an example of detecting and grasping the object.

2.1. Hardware

The main board we used for this exoskeleton is the Google Coral Dev Board mini [24]. The Coral Dev Board Mini is a compact single-board computer designed to make machine learning (ML) inferencing faster. While primarily intended as an evaluation device for the Accelerator Module, which includes the Edge TPU, it also functions as a fully operational embedded system suitable for a wide range of on-device ML projects. This board contains MediaTek 8167s System on Chip (SoC), a quad-core Arm Cortex-A35 processor designed for efficient performance and energy management. The IMG PowerVR GE8300 GPU is integrated within this SoC, ensuring reliable graphics processing. Google Edge TPU coprocessor delivers 4 TOPS (int8) and achieves an impressive efficiency of 2 Tera-Operations Per Second (TOPS) per watt. Complementing these components is 2 GB of LPDDR3 RAM and 8 GB of eMMC flash memory, providing adequate memory and storage capacity for various applications. We took the Hemiplegia Hand [26], which is for training and rehabilitation with preprogrammed functions, and replaced only the gloves with our control system.

This exoskeleton utilizes a 5-megapixel camera module compatible with Coral boards for capturing images [25]. The camera connects via the MIPI-CSI interface, and features autofocus, an 84/87-degree view angle, and various auto controls for image optimization. A VL6180X Time of Flight distance sensor is integrated for precise object distance detection. Additionally, an ADXL345 accelerometer detects gestures (tilt) to release objects.

2.2. Training and Object Detection

For object detection, we implemented the EfficientDet-Lite0 model architecture [27] as it is lightweight. For the training process, we gathered images of two objects and annotated all images using the open-source LabelImg script [28]. The images were divided into 10% for testing, 10% for validation, and 80% for training and trained for 300 epochs.

3. Result and Discussion:

The development of this novel vision-enabled rehabilitation hand exoskeleton represents a significant advancement in assistive technology for individuals with hand motor impairments. This system integrates a camera and embedded processor to enable automated hand movements, which simplifies the user interface and eliminates the need for extensive training typically required by EMG-based systems. By utilizing computer vision algorithms for object detection and an accelerometer for gesture recognition, the exoskeleton provides a more intuitive and accessible solution for users. One of this system's primary advantages is its potential to reduce costs, weight, and power consumption, making it feasible for mass production and widespread use. The focus on affordability and portability addresses some of the critical barriers to the adoption of advanced rehabilitation devices. Moreover, the use of air pressure actuators, controlled through vision-based algorithms, offers a natural and responsive user experience, mimicking the subtle movements of the human hand.

Despite these advancements, several challenges and limitations need to be addressed. The accuracy and speed of the object detection algorithms are crucial for real-time application, and any delays could impact the effectiveness of the device. Additionally, the robustness of the system in various lighting conditions and environments must be ensured to make it reliable for everyday use. Further research is needed to optimize the image processing algorithms and improve the gesture detection sensitivity of the accelerometer.

We have tested this device in the lab, from object detection to grasping and releasing. Object detection runs at six frames per second. The pilot testing phase demonstrated promising results, with users successfully performing grasp and release tasks. However, extensive clinical trials involving a larger and more diverse group of participants are necessary to validate the system's efficacy and usability comprehensively. User feedback during these trials will be invaluable for refining the design and functionality of the exoskeleton.

Compared to other existing solutions, such as the PEXO [1] and SCRIPT [2,29], EMG-based [10] systems, the vision-enabled exoskeleton offers a novel approach that simplifies user interaction while maintaining functional effectiveness. However, continuous improvements and iterations are essential to enhance its performance and user satisfaction.

4. Conclusion

The proposed vision-enabled rehabilitation hand exoskeleton presents a promising tool for assisting individuals with hand impairments, offering a blend of advanced technology and user-centered design. By addressing current limitations and incorporating user feedback, this system has the potential to significantly improve the quality of life and independence of its users. Future work will focus on refining the technology, conducting comprehensive clinical evaluations, and exploring additional applications and functionalities to broaden its impact.