Analysis of Hand Controls on a Backhoe Loader Simulator

1. Introduction

Efficient training of excavator operators is crucial for ensuring optimal performance and safety in the construction and mining industries. Traditional training methods often rely on subjective assessments, lacking detailed, quantitative feedback to guide trainees effectively. Recent advances in simulation technology have opened new possibilities for analyzing operator performance using objective metrics. For instance, studies have explored the use of virtual reality systems for training excavator operators, highlighting the benefits of simulated environments for skill acquisition [1]. Additionally, research comparing actual and theoretical performance of heavy machinery in earth-moving activities emphasizes the importance of accurate performance metrics in operator training [2]. Our study presents a novel backhoe simulator designed to bridge this gap by capturing and analyzing control signals during predefined exercises. These signals, including joystick and pedal states, are serialized and processed to generate graphs, heatmaps, and quantitative measures such as total effort, fuel consumption, and angular displacement of moving parts.

The simulator establishes performance benchmarks based on data from expert operators, which serve as references for evaluating trainees. Trainees perform identical exercises, and the system generates visual and numerical reports comparing their performance to expert standards. These reports include detailed graphs and percentage-based efficiency scores, enabling targeted feedback and skill improvement. This approach represents a significant advancement in operator training by providing measurable insights into performance dynamics. By leveraging simulation data, our work contributes to developing more effective training programs, fostering greater efficiency and safety in machine operation.

2. Simulator Description

2.1. Physical Setup

The backhoe simulator was designed to provide a realistic and immersive experience for operators, replicating the physical aspects of operating a real excavator. The simulator includes several key components that contribute to its realism and effectiveness as a training tool.

2.1.1. Visual Interface

The simulator is equipped with multiple high-definition screens that serve as the visual interface for the operator. These screens provide a panoramic view of the simulated environment, displaying realistic 3D graphics of the excavation site, machinery, and surrounding terrain. The operator can interact with the environment by manipulating the simulator’s controls, such as joysticks and pedals, which correspond to the movements of the machine’s parts in the virtual world. The display is updated in real-time, ensuring that the operator receives immediate visual feedback for every action taken.

2.1.2. Seat with Motion Feedback

To enhance the immersive experience, the simulator includes a custom-designed seat equipped with motion feedback technology. This seat mimics the movements of a real backhoe, such as vibrations, tilts, and shifts, in response to the operator’s actions. The motion system provides physical feedback during operation, allowing the operator to feel the machine’s responses to different tasks, such as lifting heavy loads or digging into the ground. The seat’s movements are synchronized with the actions of the virtual backhoe, further enhancing the realism of the training experience.

2.1.3. Control System

The simulator features a set of ergonomic controls designed to replicate the joysticks and pedals found in a real excavator. These controls are highly sensitive and are equipped with haptic feedback, which simulates the tactile sensations an operator would experience when interacting with the actual machinery. The joysticks control the movement of the backhoe’s arm and bucket, while the pedals control the speed and direction of the machine. The control system is designed to provide a natural and intuitive interaction, allowing trainees to become familiar with the machine’s operation in a virtual setting before transitioning to real-world operation. See Figure 1.

2.1.4. Sound System

In addition to the visual and motion feedback, the simulator is equipped with a high-fidelity sound system that plays realistic audio cues. These sounds include engine noises, the sound of the backhoe’s hydraulic system, and environmental sounds such as the impact of the bucket hitting the ground or materials being moved. The auditory feedback reinforces the operator’s sense of immersion, providing additional sensory input to enhance training realism.

2.2. System Integration

All components of the simulator are seamlessly integrated into a single system that synchronizes visual feedback, motion, and control inputs. The simulator operates on a dedicated computer system equipped with powerful graphics processing units (GPUs) and custom software. This configuration enables real-time rendering of the simulated environment and precise tracking of operator inputs. Additionally, the system captures and stores all relevant performance data, which is subsequently analyzed to provide feedback on the operator’s skills and efficiency.

Figure 2 illustrates the simulator in operation, highlighting the integrated design and its capability to synchronize motion, visuals, and controls effectively. This visualization demonstrates the real-time functionality and showcases the advanced hardware and software working in tandem to create an immersive and data-driven simulation environment.

3. Materials and Methods

3.1. Simulator Setup

A custom-designed backhoe simulator was developed for the purpose of evaluating excavator operator performance. The simulator collects data from the machine’s controls, including joysticks and pedals, which are digitalized and serialized to track the state transitions during predefined exercises. The collected data includes various performance metrics, such as fuel consumption, machine effort, angular displacement of moving parts, and task completion time. This information is processed and visualized using graphs and heatmaps, allowing for detailed analysis of operator performance.

3.2. Operator Tasks

The simulation includes a series of tasks that replicate common operations performed by excavator operators, such as digging, lifting, and moving materials. Expert operators performed these tasks to establish baseline performance data, which was then used as a reference for comparing trainee performance. The exercises and tasks are designed to assess the operator’s efficiency, safety, and overall skill level.

3.3. Data Collection and Analysis

The simulator continuously records data throughout each operator’s task performance. Key performance metrics include joystick and pedal states, fuel consumption, total machine effort, angular displacement of moving parts, and task completion time. Data from expert and trainee operators were systematically analyzed to extract key performance indicators (KPIs). These KPIs were visualized using graphs and heatmaps, effectively illustrating differences in operational efficiency and skill levels between experts and trainees.

To further analyze the joystick data, we implemented a detailed process to interpret the control dynamics. Each joystick can occupy one of nine positions: the central (idle) position and eight additional positions distributed along a circular path at 45-degree intervals. As operators perform tasks, the simulator records the sequence of joystick positions. This sequence is then translated into a graph by the simulator, which provides a visual representation of movement blocks and transitions between them. This approach enables a deeper understanding of the operator’s control strategies, offering insights into their skill level and operational efficiency.

3.4. Data Availability and Privacy

Due to the sensitive and proprietary nature of the data collected during the simulation, we regret that the data cannot be made publicly available. All collected data are treated as confidential and are protected under institutional privacy policies. These data are solely intended for research purposes within the scope of this study. Access to the raw data can be granted on a case-by-case basis through request to the corresponding author, subject to approval from the relevant authorities and in accordance with applicable privacy regulations.

3.5. Ethical Considerations

As the study involves only simulated tasks with no interaction with live animals or humans, no ethical approval was required. However, all participants in the study, including expert and trainee operators, provided informed consent prior to participation. The study adhered to ethical guidelines regarding data privacy and participant confidentiality.

3.6. Limitations

A limitation of this study is the unavailability of the raw data to the public due to its sensitive nature. Additionally, the simulator’s environment does not replicate all real-world conditions, and the results obtained may be influenced by these controlled simulation parameters. Future research will explore ways to overcome these limitations and improve the realism of the simulation while adhering to privacy requirements.

4. Results

4.1. Operational Graphs

Figure 1 and Figure 2 depict the graphs corresponding to the left joystick’s movements for an expert operator and a trainee operator, respectively. The expert operator’s graph reveals a significantly lower number of movements, evidenced by fewer nodes and a lower cardinality of the edges. This suggests a more deliberate and efficient control strategy, minimizing unnecessary transitions and focusing on essential actions to complete the task.

In contrast, the trainee operator’s graph exhibits a markedly higher edge cardinality, reflecting a greater number of transitions between joystick positions. This indicates less efficient control, likely due to exploratory or corrective movements as the trainee refines their motor skills and task comprehension. The differences in graph complexity between expert and trainee operators provide a clear visual and quantitative measure of operational efficiency and skill level, underscoring the value of graph-based analysis for performance evaluation and training optimization. See Figure 3.

Figure 3 and Figure 4 illustrate the graphs corresponding to the right joystick’s movements for an expert operator and a trainee operator, respectively. Similar to the left joystick, the expert operator’s graph displays fewer movements, characterized by a lower number of nodes and edges. This reflects a streamlined and efficient control approach, with minimal unnecessary transitions.

Conversely, the trainee operator’s graph demonstrates a higher edge cardinality, indicative of frequent transitions and a less efficient control strategy. This pattern likely arises from exploratory movements or corrections during task execution. The consistency in these findings across both joysticks highlights the reliability of graph-based analysis in distinguishing skill levels and control efficiency. See Figure 4.

4.2. Cummulative Effort

Our simulator also calculates the cumulative displacement of all moving parts of the excavator. The resulting values are then compared between the expert operator and the trainee operator to assess operational efficiency. The cumulative effort is defined as the total angular displacement of all moving parts of the excavator. It is calculated as the sum of each moving part’s individual displacements, expressed in degrees. Mathematically, this can be represented as:

$$E_{\mathrm{cumulative}}=\sum _{i=1}^n\Delta {\theta }_i$$

Where:

• $E_{\mathrm{cumulative}}$ is the total cumulative effort.
• $\Delta {\theta }_i$ represents the angular displacement (in degrees) of the i-th moving part.
• n is the total number of moving parts being considered.

The performance of a trainee operator is quantified by calculating the percentage deviation from the expert operator’s performance. The deviation is expressed as:

$$D_{\mathrm{performance}}={\displaystyle \frac{|E_{\mathrm{trainee}}-E_{\mathrm{expert}}|}{E_{\mathrm{expert}}}}\times 100$$

Where:

• $D_{\mathrm{performance}}$ is the percentage deviation in performance between the trainee and expert operator.
• $E_{\mathrm{trainee}}$ is the cumulative effort (or another relevant performance metric) of the trainee operator.
• $E_{\mathrm{expert}}$ is the cumulative effort (or the same metric) of the expert operator.

4.3. Control Effort

We also calculate the effort exerted on the simulator controls, as this effort corresponds directly to the forces experienced in real machinery. This evaluation provides valuable insights into the physical demands placed on the excavator’s joysticks and pedals during operation.

Figure 5 show the intensity of the work performed by the right and left joysticks for a trainee operator in each of their possible positions. Similarly, Figure 6 depict the same analysis for an expert operator. These visualizations highlight the differences in control utilization between the two operators and the areas where significant effort is required.

The interpretation of these images is left to the discretion of the training supervisor, who can use them to assess the trainee’s performance and provide targeted feedback for improvement.

4.4. Star Chart Comparison

For all possible magnitudes that can be evaluated in the simulator—thanks to its high configurability—a star chart is generated. This chart provides a generic and visual means to compare the trainee operator’s performance with that of the expert operator.

The star chart enables a comprehensive overview of multiple metrics simultaneously, making it easier to identify strengths and areas for improvement in the trainee’s performance. Figure 7 illustrates an example of such a chart, showcasing the comparative performance between the trainee and expert operators.

5. Discussion

The findings of this study shed light on key aspects of excavator operation efficiency and skill development, offering novel insights compared to previous research in this field. By employing a digitalized and serialized approach to capturing control signals, the backhoe simulator allows for detailed analyses of operator behaviour that align with and extend earlier studies on manual control systems in heavy machinery.

Our results confirm that expert operators demonstrate superior control efficiency, as evidenced by reduced machine effort and more precise movements. These findings support the working hypothesis that experienced operators optimize control inputs to minimize energy expenditure and wear on machinery. The detailed graphs generated from state sequences further enrich this understanding by providing a visual representation of control dynamics, which has been less explored in earlier studies.

In a broader context, these findings emphasize the critical role of structured training programs in addressing efficiency disparities between trainees and experts. This fact aligns with the growing literature advocating integrating simulation-based tools in construction training programs. Simulators like the one developed here contribute to safer and more cost-effective operator training by bridging the gap between controlled learning environments and real-world scenarios.

The implications of this study extend beyond operator training to optimizing machine design and performance monitoring systems. By quantifying metrics such as fuel consumption and angular displacement, our approach offers valuable data for improving equipment ergonomics and reducing environmental impacts. Additionally, the ability to compare performance across different operator profiles could inform personalized training interventions, further enhancing skill acquisition.

Despite its strengths, the simulator has limitations that warrant future investigation. For instance, while it provides a robust framework for analyzing operator inputs, it does not fully replicate sensory feedback from real-world excavator operations, such as vibrations and ground resistance. Future research should explore integrating advanced haptic feedback systems to address this limitation and enhance simulation fidelity. Moreover, expanding the simulator’s scope to include complex, multi-task scenarios could offer deeper insights into operator decision-making under varying conditions.

As part of our ongoing efforts, we are working on a system that interprets the numerical data from simulations and provides improvement recommendations in natural language. This approach aligns with recent advancements in natural language feedback systems, which have been shown to enhance interactive learning and decision-making. For example, Yan et al. [3] developed a method for simulating natural language feedback to improve semantic parsing, while Fernandez et al. [4] explored the use of natural language for training agents in complex tasks. Similarly, Abbas et al. [5] demonstrated the benefits of AI-enhanced decision support systems for improving operator efficiency in control room environments. Moreover, tools such as SIMPT [6] illustrate how interactive simulation frameworks can drive process improvements. These advancements aim to make feedback more accessible and actionable, enhancing the utility of simulators as tools for training and performance assessment.

In conclusion, this study highlights the potential of digitalized simulation tools to advance operator training and machine efficiency in the construction industry. By building on the current framework and addressing its limitations, future research can further refine these tools, ensuring their alignment with industry needs and environmental sustainability goals.