A Pre-Fitting Mixed Reality System for Myosignals Evaluation and Optimal Control Training for Hand Prostheses

1. Introduction

Despite the advancements of the last decade in upper limb prosthetics field from both mechatronics and control perspectives, there are no significant improvements in the overall abandonment rate. Recently, a survey about prosthetic usage has been conducted to evaluate the user acceptance according to the current technological development [1]. In the study, for the first time, participants were recruited not only from rehabilitation centers but also from the community. A questionnaire was administered to 68 traumatic upper limb amputees (93% was a myoelectric prosthesis user). Before, only information concerning the rejection rate at all amputation levels was provided: 44% with no significant difference in acceptance in the past decade [2].

Typical powered prostheses are controlled by muscle activity of the two antagonist muscles, flexor and extensor of the wrist, generally recorded by surface electromyographic (EMG) sensors, as described in [3]: myoelectric control. The muscle contraction level modulates the speed or strength of the grasping, implementing a proportional control [4].

Although different reasons can be identified as main sources for the still high abandonment rate, the most relevant are the effectiveness of myoelectric prosthetic control and prosthesis control training, together with the lack of comfort and weight of the device [1,5].

Therefore, targeted and personalized prosthetic training may have a positive impact on device acceptance. In particular, the quality of the training seems to be one of the most the relevant factor, together with an early prosthetic fittings: prosthetic training within the first six months after amputation (before amputees get used to perform most of the everyday activities with one hand) leads to a much higher acceptance [1,6].

As consequence, highlighting the importance of muscle training throughout all phases of rehabilitation, from pre-prosthesis to post-prosthesis receiving, is crucial for enhancing usability and user acceptance of the final device [7]. Irrespective of the prosthesis control method, users must possess the ability to selectively activate specific muscles and modulate their activation for proportional control over degrees of freedom. Typically, this involves executing movements distinct from those used before amputation to achieve desired actions [8]. Consequently, evaluating myo-signals becomes a fundamental step for an early myoelectric prosthetic fitting. This evaluation includes identifying residual antagonist muscles, determining optimal electrode positions, and setting amplifier gains for precise socket manufacture and easy control [9].

Researchers have explored integrating technology to aid pre-prosthetic training for individuals with amputations or congenital limb deficiencies. Various solutions, including myoelectric signal visualization and virtual prosthesis control on screens, have been presented over the years [8,9,10]. Notably, modern technologies such as advanced VR headsets [11,12] have enabled the development of immersive applications, allowing upper-limb difference individuals to simulate virtual prosthesis control in engaging environments [13]. Studies indicate that immersive VR applications enhance the sense of ownership over a virtual limb, positively influencing prosthesis acceptance and satisfaction [14].

Literature delves into innovative approaches in prosthetic training, particularly leveraging Augmented Reality (AR). A noteworthy solution is the game-based training tool [15], offering an intuitive method for myoelectric prosthetic muscle training. Users experience a real-time mirrored view with a virtual arm superimposed on their residual limb, controlled by muscle activity and aided by fiducial markers.

With the advent of head-mounted display devices [16], the immersive environment shifted to a first-person perspective, enhancing embodiment and precise interactions with virtual objects [17]. In those works, the authors described training systems that project an augmented virtual prosthesis controlled through muscle activation and enable users to interact with holograms using gestures or voice commands.

Recent strides include Mixed Reality (MR) applications, evolving into holographic manipulation exercises [18]. In one application treating phantom limb pain, patients explore the MR environment, manipulating holographic objects with their healthy hand and a holographic arm on the stump. While focused on pain treatment rather than device control, it highlights the immersive potential of MR.

The latest MR prosthetic training system [19,20] aims to enhance advanced prosthetic controllability. Using a head-mounted display with a stereoscopic camera, it projects a virtual arm model onto the user’s arm, controlled by a Myo Armband EMG sensor on the residual limb. The system, featuring a virtual Clothespin Relocation Test, demonstrates skill transfer from simulation to real scenarios. These advancements underscore AR’s potential in providing engaging prosthetic training experiences, contributing to skill transfer from simulated to real-world applications.

Despite promising advances in AR-based prosthetic training, some limitations remain. In particular, the system’s design limits training exclusively to distal transradial amputees with thick stumps system [14,21]. This constraint arises from the necessity for the AR device to have a clear view of the marker on the stump, and the residual limb must fit the specific dimensions of the EMG/IMU armband. Moreover, the hardware employed for hologram visualization relies on a head-mounted display integrating augmented objects into the real scene captured by an external stereo camera. This setup introduces latency in visualization, with an average control delay of 250ms [19,20]. Such delays can impart an unnatural sense of control over the artificial limb, potentially affecting the user’s experience.

Importantly, the control delay surpasses the recommended “optimal control delay” threshold identified in [22], where delays exceeding 100ms can result in a consistent decrease in prosthesis performance. These limitations underscore the need for further refinement in AR-based prosthetic training systems to address issues related to stump detection, hardware constraints, and control delay for a more inclusive and effective training experience. Additionally, AR systems were mainly tested on able bodied participants and preliminarily evaluated with a single transradial limb difference individual. These limitations affect the clinical application of research findings to real prosthesis users, thus impacting consideration about the previous experience of the limb different subject in EMG control signals.

The present work introduces a Mixed Reality (MR) training system designed to address critical limitations identified in previous literature. Our architecture facilitates user control of a holographic representation of the Hannes prosthetic hand with a latency that surpasses the optimal control delay for prosthetic performance. The primary objective is to introduce a novel training solution aimed at enhancing engagement and extracting several objective information from the user during the learning process. The latter aims to explore the learning curve for new users of the Hannes hand focusing on mastering myoelectric signal control within a MR platform. In contrast to previous work, the present preliminary study involved transradial limb difference people who completed a full clinical protocol with the Hannes prosthesis, providing a more realistic preliminary assessment of MR training efficacy while taking into consideration previous myoelectric experience of the involved participants.

One notable aspect of our system is the low latency performance that maintains consistency between the virtual and real prosthesis architectures. Unlike existing solutions, our architecture allows individuals with any residual limb shape to control an augmented prosthesis with latency levels below the optimal control threshold. This not only provides users the ability to customize, evaluate, and train control parameters in a MR environment but also ensures that the hardware architecture of the proposed system completely aligns with what will be embedded in the actual prosthetic device, mimicking the real behavior of the Hannes prosthesis [23].

To preliminary validate the proposed MR system, our work introduces an innovative Target Achievement Control (TAC) [24] test as part of the Hannes pre-fitting process. This test is seamlessly integrated into the MR environment, employing a custom-made setup and a novel MR framework tailored for Microsoft Hololens2 [11]. While the sample size of the involved population limits statistical conclusions, this study aims to demonstrate the technical feasibility, usability, and potential benefits of the MR platform as a portable preparatory training before actual prosthesis fitting.

2. Materials and Methods

In this section, the overall Hannes system and HoloApp software architecture as well as an user case validation method is described. Firstly, in Section 2.1, the overall architecture is presented is detailed, focusing on the integration between the MR platform and the Hannes prosthetic device. Subsequently, in Section 2.2, the different communication methods between the hardware components of the overall system are analyzed to evaluate latency. Additionally, Section 2.3 highlights the HoloApp developed for the immersive prosthetic pre-fitting process (muscular evaluation and control parameters training). Finally, in Section 2.4, our novel bimanual TAC test is presented as a user case study for the preliminary validation of the system.

2.1. Overall Architecture

A Mixed Reality system has been developed employing an Hololens2 device. The device can be worn by the user overlapping holograms to the real environment. Further, the system can implement a prosthesis hologram that reproduce the real behavior of the Hannes prosthetic system [23]. According to that, the user can control the holographic prosthesis using muscle contraction in the same manner as commercial prosthesis. This proposed solution allows to fully customize, train and validate the Hannes system and its controller, with a realistic relationship between the real environment and the virtual ones, with the aim to realize advanced assessment protocols capable of promoting the learning process of new users and the integration between user and prosthesis. The overall system is described in Figure 1 and consists of three main components:

• Hannes prosthesis system;
• The Virtual environment;
• Hololens App.

The Hannes prosthesis system is composed of a custom-made socket that represents the physical connection interface between user and the prosthesis. The socket contains a battery pack, a set of two electromyographic sensors (EMG, 13E200 MyoBock Electrodes), and an EMG processing board (EMG-Master) [23]. The two EMG sensors (respectively placed in the forearm flexor and extensor muscles) are used to detect the residual muscular activity at the stump level, i.e., the muscular contractions of the forearm flexor and extensor muscles. The muscular activity was processed by the EMG-Master to generate control signals to actuate the Hannes prosthesis. Such a board can record up to 6 EMG sensors at 300Hz and generate control signals to move up to 3 degrees of freedom (DoFs) of the Hannes hand [25]: hand aperture, wrist rotation, and wrist flexion/extension. It is worth noticing that, for the architecture proposed by this work, the EMG-Master can communicate with a Host Pc via Bluetooth to send the control commands for controlling the virtual hand in the virtual environment according to the muscular activity.

A VR framework was developed using the Unity development suite in C# language. This application runs in a Host Pc (Dell precision 3571, Intel i7, 16Gb Ram, Windows 10), where the virtual representation of Hannes is controlled with the EMG activity of the patient’s residual limb, replicating exactly the proportional control of the real prosthesis. At the same time, the host pc sends the velocity references received from the EMG Master to Microsoft Hololens2.

The VR framework allows the clinician to perform the EMG signals evaluation and to tune the control parameters for the amputee to achieve fine motion and proportional control of the prosthesis. The modified control parameters are instantly sent to the EMG Master and saved in the EEPROM memory so that the patient could immediately test the comfort of the control of real Hannes hand and the virtual hand.

Such virtual solution can evaluate the level of controllability of the device, reached by the user, performing a Target Achievement Control test. This test was already employed in different works to evaluate on screen the performance of users and control algorithms to manage multiple DoFs. Therefore, the test on screen does not allow to replicate a real scenario where the user does not perceive the size of the device and the third dimension of the hand configuration. So, to overcome this limitation the Hololens App was developed in a Mixed Reality environment.

The Hololens App (HoloApp) is an application realized using the Unity development suite in C# language. This application runs in a Hololens2 device reproducing the augmented representation of Hannes prosthesis on the user view. The prosthesis is projected as a 3D hologram and it moves coherently with the virtual hand and the real hand prosthesis. Such a solution allows the user to perform any kind of test in a realistic condition.

In this novel scenario, a new TAC test was developed to train the user in myoelectric prosthesis control while extracting significant information about the algorithm’s performance and usability.

2.2. Device Communication

The EMG-Master board performs A/D conversion of EMG signals at 300Hz and then transmit data to the host pc via Bluetooth communication [26] using two BGX13P modules (one embedded on EMG-Master board and one on USB dongle connected to PC) making a bridge between the devices. The Bluetooth communication between the EMG-Master and the host pc is bilateral: i) the EMG data, the prosthesis Measurement and the References are sent from EMG-Master to host PC; ii) the parameter customized to the user are sent from Host PC to EMG-Master to be stored on board EEPROM during the tuning phase. At the same time, the host PC runs an Unity based application that receives and parses the data coming from Bluetooth. In that application the virtual hand moves on the screen replicating the same movements of the real hand at 200Hz. Then the application on PC communicates with the Hololens2 device wireless using a router as bridge. In that case, the Hololens2 runs the Unity based HoloApp, and the host PC sends control commands through a UDP communication protocol in order to move the holographic hand.

The most important requirement for this application is the latency: it needs to be below the perceived delay of real prosthesis control [22].

The total latency of the proposed system is the sum of: EMG-Master acquisition time, Bluetooth data packet transmission time from EMG-Master to host PC, Unity PC application frame interval, UDP data packet transmission time from host PC to Hololens2 device and Unity Hololens application frame interval (Table 1).

Several tests on BGX13P module used as dongle USB on the host pc were conducted to find the streaming data transmission time from the EMG-Master to the host pc. It was found that transmissions of continuous flows of characters with a baud rate of 115200bps (board default value), the characteristic flight time of the data packet is 5ms.

Regarding the communication between PC and Hololens, real time EMG signals data are transmitted from the host pc (server) to the AR device (client) using UDP sockets. To estimate the data packets transmission time, an echo response of each data received from the host pc was implemented on Hololens: the holographic glasses send the data back to the server immediately after having received them. The UDP transmission time from the host pc to Hololens is logically lower than the time interval between the server transmission and reception of the same data. The UDP transmission-reception times were collected during 3 days performing 3 TAC tests lasting one minute for each day. The mean of the times collected was calculated. It was found that the mean UDP transmission and reception time during real time EMG signals data streaming to Hololens is equal to 15ms. The summarization of the different delay components obtains an overall System Latency < 56.5ms.

Research in upper limb prosthetics demonstrated that the “optimal control delay” (i.e., the maximum amount of time that can be used by the controller for data collection and analysis to maximize classification accuracy without affecting prosthesis performance) must be kept under 100ms to avoid consistent decrease in prosthesis performance [22]. The system latency of the proposed system guarantee the optimal value respect to the most recent AR prosthetic training works [20].

2.3. HoloApp

The amputee can perform muscular evaluation and train the control parameters of the future Hannes device thanks to the HoloApp. The application consists in three main phases: i) prosthesis projection; ii) parameters tuning; and iii) Assessment. A specific case of assessment will be described in paragraph 2.4.

2.3.1. Prosthesis Projection

Since AR is the superposition of virtual objects in the real environment, it can provide individuals with real time interactions removing the need to use mental imagery. Similar to AR applications that enhance the realistic experience of “digitally fit” their products to aid consumers during decision making [27], it’s possible to create AR solutions in which digitally wear Hannes prosthesis during the pre-fitting process.

First, the firmware projects on the flat surface in front of the user the holograms of two boxes. Each of them contains the components of the future real prosthesis: the socket and the Hannes hand. One of the boxes contains the components of the right Hannes prosthesis and the other the components of a left Hannes prosthesis. The user needs to touch the box corresponding to his/her amputation to open its content. The holograms of the socket and the Hannes hand can be grabbed and manipulated (translated and rotated) with the healthy hand. Before controlling the holographic prosthesis, the patient needs to wear it, replicating and mimicking the real fitting phase of the physical device (Figure 2). The user must:

• Choose between right hand and left hand;
• Grab the socket and place it on the residual limb;
• Grab the hand and place it over the socket;
• Push the plug button on the socket to turn on the prosthesis.

The hologram of the socket is fixed to the position given by the user, who can always grab it and place it again. Unlike other solutions [18,19,21], the residual limb position is not constantly estimated with external devices. We preferred computational efficiency and the possibility to use the firmware with any kind of upper limb amputation over the hologram following the stump when moved. In the evaluation pre-fitting phase, the patient does not have to get used to the weight of the prosthesis (like the post fitting phases), so all the procedure can be performed with the residual limb comfortably leaned on the table. When the holographic prosthesis is turned on, the firmware starts to wait for the commands data from the host pc that animate coherently the hand.

2.3.1. Parameters Tuning

The clinician performs the EMG signals evaluation and starts tuning the control parameters. In particular, the tuned parameters involved the minimum signal amplitude (threshold) to generate a movement and the associate mapping of speed for proportional control (gain). These two parameters are fundamental to allow the amputee to control the holographic Hannes hand with fine motion. The clinician changes these values with a slider and sends the new parameters to Hololens through UDP so that the user can control the holographic hand with the new settings and test the resulting motion. The host pc also sends the new parameters values to the EMG-Master through Bluetooth: the values are saved on the EEPROM of the board.

2.4. A Case Study: Bimanual TAC Test

Once the user achieves accurate and fine control with the selected parameters, it is possible to train the participant to activate muscles selectively and to modulate their activation for proportional control with the proposed TAC test. The TAC test, presented in [28], consisted of a virtual solid arm, that can be controlled by the user with EMG, and a second virtual semi-transparent hand which shows the target configuration.

In the proposed TAC test both the virtual controlled hand and the virtual semi-transparent target hand are replaced by 3D holograms and placed in the real environment instead of a 2D representation. The user needs to move the controlled Holographic hand while reaching the target configuration of the semi-transparent holographic target hand with a predefined maximum tolerance using myoelectric signals.

Since bimanual augmented reality exercises have been studied to treat Phantom Limb Pain (especially for amputees during early prosthetic fitting) [18], the proposed TAC test is composed by bimanual trials. The proposed TAC test is composed of 3 sessions of 21 trials each. The single trial consists of the following steps, represented in Figure 3:

A.

The semi-transparent holographic target hand is projected over the healthy hand of the subject. The user is free to move the hand: the hologram will follow its position and orientation in the space. The target position is randomized while ensuring that the same position is not repeated consecutively.

B.

The user needs to reach the target configuration with the healthy hand with a predefined tolerance. The configuration error needs to be kept lower than 5% for 1 second consecutively. After that, the semi-transparent hand disappears.

C.

The user has a limited amount of time to reach the target configuration with a predefined tolerance with the Hannes hand hologram. The single trial must be considered succeeded if the configuration error is kept lower than 5% for 3 seconds consecutively. The single trial must be considered failed if the target configuration isn’t reached within 20 seconds or if the configuration of the healthy hand changes in the process.

A custom algorithm has been developed so that the healthy hand posture could be comparable with the one of the holographic Hannes hand, since their motion is different. The biggest difference in the motion is that the distal phalanges are not actuated in Hannes prosthesis, so they can’t be considered in the evaluation of the posture. For this reason, to get the posture of the healthy hand, we decided to detect the index middle joint and the thumb proximal joint and measured the distance between them (D). Since every person has a different maximum opening and a maximum closing position of the hand, each user must perform a fast calibration prior to the TAC test. The calibration phase consists in performing 5 opening/closing gestures with the healthy hand to find the maximum distance (Dmax) and minimum distance (Dmin) between the index middle joint and the thumb proximal joint of the user.

Regarding the semi-transparent hand and the Hannes hand hologram, the opening/closing references (AC) range from 0 (maximum opening) to 1 (maximum closing). In step 1, the reference value for AC movement of the semi-transparent hand is randomly generated while in step 3 the references are continually sent from the host pc. To find out if the healthy hand and the semi-transparent hand (step 1) or the holographic Hannes hand (step 3) have the same posture with a tolerance of 5% degrees, the following condition needs to be verified:

AC - 0.05 < 1 - (D - Dmin) / (Dmax - Dmin) < AC + 0.05,

(1)

The empirically identified tolerance threshold is 0.05.

2.4.1. Subjects

Two upper limb different adults who never used Hannes prosthesis before voluntarily participated in the experiment. The participants are a naïve subject with transradial congenital limb deficiency (female, 50 years old), who never used a myoelectric prosthesis, and a pro user in controlling myoelectric prosthesis (not Hannes device) with transradial amputation (male, 60 years old). After this pre-fitting training, a real Hannes hand was given to the two participants to perform some clinically validated functional tests. To validate the efficacy of such MR protocol in improving the controllability of Hannes, the performances of the two participants were compared with those of a reference group of 8 transradial amputees (N=8; 1 female; 7 males; 51.22 ± 6.31 years old) with previous experience in EMG control, who followed the same protocol of the tested subject but without the usage of MR training. The experimental protocol (CP-PP3AS1/1-03) was approved by the AVEC (Area Vasta Emilia Centro) Ethics Committee, and the subjects signed an informed consent form to perform all the following tests. It is important to acknowledge that the recruitment of participants with limb differences represents a significant challenge in clinical studies. In this work, this challenge was further increased by the additional requirement that participants had no prior experience in controlling the Hannes prosthesis. Consequently, the present study should be interpreted as a preliminary evaluation aimed at providing initial insights into the proposed approach. Moreover, comparisons were performed with a control group of eight transradial limb-different participants who followed the same protocol without MR-based training.

2.4.2. Experimental Protocol

The experimental protocol follows the steps described in chapter 2.3. In this scenario the two participants wear the Hololens device and start the app to perform the experiment during the pre-fitting phase. In the first step the subjects need to wear the virtual prosthesis. After this step the experimenter sets the EMG parameters (threshold and gain) to guarantee the user suitable controllability of the virtual hand. Then, the subjects have a single training session of 21 trials to learn how the bimanual TAC test works and set the proper time limit for the real experimental phase. The experiment phase consists of a total of 3 sessions of 21 bimanual TAC test trials each. We will call this part of the experimental protocol “V-1”. During the V-1 phase, the error along time (between the controlled hand and target hand) was collected to perform the offline analysis of the performance of the naïve subject and have the comparison with the pro user.

After the V-1, a real Hannes device was given for the first time to the two subjects to perform 3 clinically validated functional test: 3 sessions of Minnesota Dexterity test (MMDT) and the Southampton Hand Assessment Procedure (SHAP), during which the execution times were measured, and 3 sessions of the Box and Blocks test (BBT), in which the number of moved boxes was collected. We will call this part of the experimental protocol “T-1”. The T-1 functional tests session was performed also by all the other pro users that, as the two participants that performed the V-1 experimental phase, had never used Hannes prosthesis before.

After a month from the T-1, another bimanual TAC test was performed by the two subjects that participated in the V-1. We will call this second virtual experimental phase “V0”. Like the V-1, the V0 consists of 3 sessions of 21 trials each.

After the V0, the two subjects performed again all the functional tests. We will call this part of the experimental protocol “T0”. The T0 functional tests session was performed also by all the other pro users that did not perform the virtual experimental sessions (V-1 and V0).

During the T-1 and T0, performance data were collected to perform the offline analysis to compare progressively the ability in using Hannes prosthesis between the participants of the virtual experimental sessions and the reference group.

2.4.3. Data Analysis

From the measurements collected during each session of the V-1 and V0 bimanual TAC tests, the following data have been obtained for each subject: success rate, average path efficiency (considered only for success trials), average error and standard deviation, and average completion time (considered only for success trials).

The session average error represents the average of the final errors of each trial. Path efficiency was calculated by comparing the optimal trajectory (the distance between the holographic hand starting position and target position, known as starting error) to the trajectory generated by the subject (sum of the differential errors during the trial).

The data analysis during the sessions of the bimanual TAC test (V-1 and V0) aimed to compare the performance trends in controlling a virtual Hannes prosthesis with EMG signals between a naïve subject in myoelectric hand control and a pro user.

Following each bimanual TAC test, the two subjects completed two questionnaires to provide insights into the subjective experience of the proposed Mixed Reality platform: the System Usability Scale (SUS) for measuring the usability of the system, and the NASA Task Load Index (TLX) for assessing the subject workload.

Regarding the T-1 and T0 hand function tests, the MMDT and SHAP average execution times, along with the average BBT score of the subject and the reference group, were obtained.

The data analysis during the three functional tests (T-1 and T0) aimed to compare the performance trends in controlling a real Hannes prosthesis between the two subjects that underwent the virtual sessions (V-1 and V0) and the reference group who did not receive the virtual sessions.

3. Results

The bargraphs of Figure 4 display the success rate and average error of the naïve subject (blue) compared to the performance of the pro user (red) across the 3 sessions of the bimanual TAC test before Hannes fitting phase (V-1).

In the first session, the success rate of the test performed by the naïve subject (47%) is lower than the test performed by the pro user (57%). However, the graphs show a greater increase in the success rate and a greater decrease in the average error of the naïve subject compared to the results of the pro user. At the end of V-1, the naïve subject achieves better performance in performing the TAC test (76%) than the pro user (62%).

After the pre-fitting phase, the naïve subject also achieves better results in performing the 3 functional tests during the T-1 experimental phase compared to the pro user (except for SHAP power grasp), as shown in Table 2.

Table 2 also indicates that the results of the naïve subject in performing the BBT and the SHAP test are better compared to the ones of the other pro users who did not use the Mixed Reality setup (except for SHAP power grasp), while the MMDT performances are similar. The naïve subject achieved similar performances in the MMDT compared to the pro users. Contrary to the pro user, who used the MR setup, the naïve subject outperforms in all functional tests (except for SHAP power grasps). The bargraphs of Figure 5 show the success rate and average error of the naïve subject compared to the performance of the pro user across the 3 sessions of the virtual bimanual TAC test post Hannes fitting phase (V0). Like the pre-fitting phase (V-1), the graphs depict a sensitive increase in the success rate (from 71% to 100%) and decrease in the average error (from 4.3% to 2.1%) of the naïve subject compared to the results of the pro user, whose performance remains approximately stable. The naïve subject also achieves an overall better performance in the functional tests during the T0 experimental phase compared to the pro user.

Table 3 shows results of the naïve subject in performing the MMDT, and most of the SHAP grasp types were performed better by the naïve subject, while the BBT average score is consistent for all subjects. Moreover, the naïve subject attains an average better performance in all the functional tests compared to the pro users who did not use the MR setup (except for SHAP tripod and tip grasps). Unlike T-1, the results of all functional tests of the pro user who performed V0 are close to the results of the other pro users (except for SHAP spherical grasp), indicating a faster learning curve in using Hannes prosthesis.

The subjective experience with the proposed MR platform is shown in Figure 6. The SUS test indicates that the subjects appreciated the usability of the system: they found the application nice and quite easy to use during both V-1 and V0. The NASA TLX questionnaires show that the mental load was the predominant workload during both V-1 and V0. The virtual TAC tests were not physically demanding and did not cause much frustration. Both subjects experienced more workload during V0 compared to V-1, especially the naïve subject.

4. Discussion

In this work, we an architecture for a Mixed Reality training system. This system enables an upper limb deficient person with any kind of residual limb morphology to control an augmented Hannes prosthesis with a latency under the optimal control threshold for prostheses [22]. The objective was to investigate the learning phase of a new user of the Hannes prosthesis in controlling the myoelectric device after evaluation and training with the proposed system. The final goal of this preliminary investigation is to reduce the prosthesis rejection rate by enhancing signal evaluation and control training.

The results of the V-1 pre-fitting experimental phase reveal that an upper limb deficient subject who has never used a myoelectric prosthesis faces more difficulties in performing the initial sessions of the bimanual TAC test compared to a myoelectric prosthesis pro user. However, the naïve subject achieves better performance by the end of the test, showing a significant improvement in controlling the Hannes system compared to the pro user. This improvement may be attributed to the fact that the pro user required more time to modify his control strategy, having been trained with different myoelectric signals. In contrast, the naïve subject started training with completely new EMG control signals. These results are further confirmed after the Hannes fitting phase. During the T-1 practical session, the naïve subject demonstrates better performance in almost all the functional tests compared to the pro user, especially in tasks requiring fine myoelectric control like the MMDT and BBT. These preliminary results suggest that the proposed MR platform may be an effective early evaluation and training method, potentially leading to higher prosthesis acceptance for naïve subjects. However, its effectiveness appears to diminish for subjects already accustomed to controlling a different myoelectric prosthesis. This application highlights that combining traditional methods with our MR system potentially accelerates the learning phase for using myoelectric prostheses.

The virtual experimental phase V0 and the functional tests in the T0 phase were conducted to investigate the learning phase in controlling Hannes signals over time. The results of the V0 phase show a significant improvement of the naïve subject in controlling Hannes signals during the bimanual TAC test compared to the pro user. Once again, the difficulty of the pro user in changing his control strategy is likely the cause. These results are reiterated in the T0 practical session, where the naïve subject outperforms the pro user in almost all the functional tests, underscoring the importance of early training for naïve subjects. However, despite lower performances in functional tests, the pro user demonstrates a much faster learning curve in controlling the Hannes prosthesis compared to the reference group that did not use the MR platform, particularly in tasks requiring fine myoelectric control like the MMDT and BBT.

The proposed MR platform appears to be a feasible and useful system for signal evaluation and early training in new users of the Hannes prosthesis. Within this pilot study, the completely naïve participant can quickly reach and improve the performance of the myoelectric prosthesis users (reference group), while the pro user can expedite the learning curve in controlling the Hannes device.

5. Conclusions

Introducing a novel Mixed Reality platform tailored for training new upper limb prosthetic users, our system offers a swift, lifelike training solution suitable for individuals with diverse stump morphologies and myoelectric control experience. Tested with both a naïve user and an experienced user, preliminary results indicate faster adaptation and improved control performance when transitioning to the real Hannes prosthesis compared to traditional methods. Furthermore, participants reported greater satisfaction and engagement during MR-based training, suggesting that such an approach may enhance overall prosthesis acceptance. Although larger studies are required to validate these findings, the results of this pilot suggest that MR pre-fitting could represent a valuable addition to rehabilitation workflows, improving user experience and potentially supporting long-term prosthesis adoption.