The Road to the Mine of the Future: Autonomous Collaborative Mining

1. Introduction

The use of automation technologies is critical in today’s mining industry, because it makes it possible to increase the safety of the operations, as well as their productivity and efficiency [1,2,3]. Moreover, the use of automated systems allows minimizing waste and reducing the environmental impact of mining activities [2].

Automated equipment can be static (e.g., conveyor belts) or mobile (e.g., trucks), and it is the latter type of equipment that is the focus of the analysis in this paper. Automated mobile equipment can have different degrees of autonomy, ranging from pure remote control to full autonomy. Mobile equipment (vehicles) with full autonomy are of great interest because they have the potential to implement the so-called zero-entry mining concept [2,4], where no human is required to enter the mine site, thus, drastically reducing the number of accidents.

It is interesting to review the state of the automation of mobile mining equipment in the mining industry. In the case of underground mining, particular attention has been given to the automation of LHD (Load-Haul-Dump) vehicles, because they operate in high-risk areas [5], but also because the mine production directly depends on the operation of these vehicles, which move the material along the production areas of the mines. The current standard is that semi-autonomous LHDs, which can haul and dump autonomously, but are tele-operated during loading, are already in commercial use [6,7]. The same technology used to automate the LHDs has been applied to automate the low-profile trucks that operate at the haulage level of underground mines.

In the case of open pit mines, the automation of the haul truck systems has received the most attention. Today, several mines around the world are operating with autonomous haul trucks [8,9], and given the reported benefits in terms of reduced operating and maintenance costs and increased productivity [2,8,9], many other operations are implementing this technology. Similarly, autonomous drills have recently attracted the interest of mining companies, and they are increasingly being used in open pit operations, also demonstrating benefits in terms of increased safety and cost reduction [2,10].

In all of the above cases of automated mobile equipment - trucks, LHDs and drills - they operate under human supervision for safety reasons, meaning that an operator remotely monitors the operation of the equipment.

It is relevant to mention that, in addition to trucks, LHDs and drills, there are dozens of other types of mobile equipment/vehicles used in mines that have not yet been automated. Mining includes not only unit operations related to production, but also to mine development (i.e., construction), where high-risk activities are carried out using a wide variety of mobile equipment. For example, in [5] it is shown that out of 59 unit operations in a typical block/panel caving mine, only six are fully automated worldwide (10%), eight are teleoperated (14%), and 45 are still manually operated (76%). This means that we are still a long way from having a zero-entry mine [2], i.e. a fully autonomous mine with no people in the mine. This implies that we still need to find ways to increase the safety of mining operations through the use of robotics and automation technologies.

In order to address this challenge, this paper analyses the current state of automation in mining and proposes a path to the mine of the future. This future mine will be a safe environment in which humans and autonomous machines will work collaboratively. Firstly, it is argued that the robotics and automation technology currently available is not yet ready for the implementation of fully autonomous operation of mines. The urgent and immediate goal must be to reduce the risks associated to mining operations by eliminating exposure to unsafe environments, particularly the production area. To achieve this, we propose the paradigm of autonomous collaborative mining, wherein humans and autonomous machines work together in a collaborative manner. As a case study, we present the case of the autonomous loading of minerals using LHDs. Finally, we draw some conclusions and outline future work in this area.

It is important to clarify that this paper presents the Autonomous Collaborative Mining paradigm at a conceptual level and that its main components still need to be further developed and implemented. Our goal is to motivate the need to develop automation solutions for the mining industry using this paradigm, which emphasizes the need for collaborative automation solutions.

2. Key Challenges in Mining Automation

2.1. Current state of Mining Automation

A common misconception is that mining automation, understood as the automation of static and mobile equipment, has been remarkably successful. This view holds that the majority of equipment utilized in underground and open mines can operate either autonomously or teleoperated, and this would allow, as a subsequent step, the drastic reduction of the humans inside the mine sites and the implementation of the zero-entry mining concept, where no human is required to enter the mine site. Nevertheless, we contend that this is not the case. A significant number of mobile equipment still require automation, and the manner in which current automated mobile equipment operates needs to be improved. Additionally, when different automated equipment will be operated together as a fleet of heterogeneous autonomous and teleoperated equipment, coordination and optimization will still need to be addressed. To illustrate the present state of mining automation, we may consider the case of block/panel caving mines, which are massive underground mining operations.

Block/panel caving is a mass mining method employed in hard rock mining, whereby “gravity is used in combination with internal rock stresses to fracture and break the rock mass into pieces that can be handled by miners” [11]. The considerable scale of block/panel caving operations entails a high degree of mechanization across a range of operational activities. This facilitates enhanced productivity and safety within the operational framework. The high level of mechanization inherent to these operations facilitates the implementation of new technologies and improvements to existing ones in a manner that is more straightforward than in other types of mining operations. The automation of these activities plays a significant role in addressing the safety and production challenges posed by evolving rock mass conditions. By reducing personnel exposure to risk situations and enhancing control and management of operations, automation can help mitigate potential hazards and improve overall safety and efficiency. In a block/panel mine, there are over 50 distinct units and auxiliary operations related to the processes of mine operation (production), mining preparation, equipment and infrastructure maintenance, development of civil works, and auxiliary support. The authors of [5] provide a comprehensive examination of the automation status for the various operations inherent to block/panel caving mines. These findings are summarized in Table 1.

As evidenced in Table 1, a total of 59 operations were identified, comprising 16 mining operations, 30 mine preparation operations, and 13 auxiliary/support activities. Of these operations, only six are fully automated worldwide, representing 10% of the total. Eight are teleoperated (14%), and 45 are still manually operated (76%). The six autonomous operations are as follows: ventilation (in horizontal development), LHD hauling (in production), LHD dumping (in production), rail hauling (in ore pass transport), conveyor belt hauling (in ore pass transport), and CAEX hauling (in ore pass transport). The eight teleoperated operations are as follows: LHD ore extraction (in horizontal developments), gallery cleaning (in horizontal developments and in ore pass), LHD loading (in production), fixed rock breaker (in production, in ore pass and in ore pass transport), and street cleaning (in support). These numbers demonstrates that there are significant opportunities for the automation of mining equipment, and that the majority of efforts have concentrated on the automation of the equipment used in production.

In order to enhance the safety of underground operations, it is essential to direct attention to the processes of development and auxiliary activities. In [5], this aspect is also analyzed. The study defines a risk index for the different operations and determines that 43 operations have a high-risk index (74%), one operation has a medium risk index (2%), and 14 operations have a low-risk index (24%). This indicates that there is a need for automation of these high-risk operations in order to reduce the level of worker exposure.

Other underground mining methods, as well as open pits mining methods, employ distinct mining processes, subprocesses and unit operations. However, in all cases, the majority of the automation efforts have been focused on the automation of a select few unit operations, predominantly utilizing autonomous haul trucks, LHDs, and drills, along with a few additional pieces of teleoperated equipment. In light of the aforementioned, it is evident that the immediate goal should be to have mines where humans are situated at a safe distance from risky areas, and in which humans and autonomous and teleoperated machines work in an efficient and collaborative manner, instead of having a mine with no humans. In the next subsections we will argue that the current robotic technology does not allow for the full automation of the multiple unit processes and tasks required in mining.

2.2. Robotics Technology for Autonomous Vehicles is Not Yet Sufficiently Robust

Mining is one of the few industries where autonomous mobile equipment, i.e., vehicles, are regularly used (e.g., autonomous haul trucks in open pit mines). As a result, there are few examples of other industries from which to draw technological developments. One potential source of inspiration could be the autonomous vehicle/automotive industry, where billions of dollars have been invested, but where satisfactory results have yet to be achieved. The development of general-purpose autonomous vehicles is worthy of analysis.

Although the first autonomous vehicles/cars were developed in Japan and in Germany in the early 1980s [12,13], the idea of having a massive adoption of autonomous vehicles started after the success of the second DARPA Grand Challenge (2005) in which 5 vehicles were able to complete the challenge [14,15]. This success inspired companies like Google to pursue the development of this technology. Subsequently, billions of dollars have been invested in these developments. Nearly every major automotive manufacturer has initiated its own autonomous vehicle project, and numerous new companies and joint ventures have been established with this objective [13]. Nevertheless, this technology remains insufficiently mature to be used in most cities [16,17]. Consequently, following two decades of development and substantial investment, the application of this technology remains largely confined to a few select cities, including San Francisco and Phoenix, and states (e.g., Arizona and California in the United States). The technology is not yet sufficiently robust. It relies on the use of extensive LIDAR-based scans of the routes (high-density sensor maps), requiring a constant human supervision of the vehicles from central station. Furthermore, it only works under normal traffic conditions. In conclusion, the technology remains in an immature state: (1) it is not plug-and-play solution, as it requires a previous scan of the routes, (2) it does not work without human supervision, and (3) it is not operational under heavy or under unstructured traffic conditions [17]. The reasons for this lack of success can be attributed to two primary factors. Firstly, the current state of robotics technology is not yet sufficiently robust to effectively deal with changing environmental conditions. Secondly, it remains challenging to accurately anticipate and respond to the myriad of potential scenarios that can arise on a populated road. Learning algorithms fail to make decisions in rare cases because there is no data to train the algorithms in these cases (out-of-distribution problem [18]).

This last point is significant because it may be assumed that the most recent AI algorithms, including convolutional neural networks, transformers, reinforcement learning, and diffusion models, will provide a solution to the autonomous driving problem. While AI can undoubtedly assist in addressing the aforementioned challenges, it is imperative to acknowledge the existence of a fundamental limitation inherent to the majority of AI algorithms: the out-of-distribution (OOD) problem. An AI algorithm is trained on a specific distribution of examples or cases, and its efficacy is contingent upon its ability to generalize to similar examples or cases. If the examples or cases differ from those used for training, an out-of-distribution (OOD) problem arises, preventing the AI algorithm from inferring or reasoning correctly. This phenomenon occurs in autonomous vehicles when they encounter situations not included in their training data. Such scenarios are not uncommon when driving in real-world settings, particularly in complex urban environments.

A further DARPA Challenge, the DARPA Subterranean (SubT) Challenge, which is more closely aligned with mining, was recently launched with the objective of developing innovative technologies that can enhance underground operations. “The SubT Challenge program explored new approaches to rapidly map, navigate, search, and exploit complex underground environments such as human-made tunnel systems, urban underground, and natural cave networks” [19]. The main scientific and technical outcomes of this four-year endeavor are summarized in [20]. Advancements were observed in the domains of perception and SLAM (Simultaneous Localization and Mapping) under underground conditions, as well as in multi-robot operation and multi-sensor integration. However, with regard to decision-making, all teams still include humans in the main decision-making loops (human supervisor), “For most teams, the role of the human supervisor included commanding high-level missions, providing manual intervention when needed, deciding when to deploy communication nodes, coordinating the next system deployments with the pit crew, monitoring incoming sensor streams, reviewing artifact detections for viability, and sending artifact reports to the DARPA command post” [20]. This reinforces the continued importance of humans in ensuring the successful operation of robotic systems. In fact, one of the primary reported insights emphasizes the pivotal role of humans in system performance: “The Human Teammate Is (Often) the Critical Performance Limiter” (Insigh #6 in [20]).

In conclusion, the current state of robotics technology is insufficient for full autonomous navigation (i.e. no human intervention) in complex, real-wold environments. To address this challenge, it is necessary to integrate human supervision and, in many cases, incorporate human decision-makers into the decision-making loops.

2.3. Continuous Operation of an Automated Task Requires Human Intervention

Autonomous systems cannot operate without human intervention for long periods of time. As an example, let us analyze the case of the most widely used household robot in the world, the robot vacuum cleaner. A robot vacuum cleaner can vacuum autonomously, but it requires a human being to place it in the room to be cleaned, making sure that it has access to the entire place where it must perform the task, for which he/she must open the doors to the rooms where the cleaning must be performed and, if the house has more than one floor, take it to the corresponding floor. In addition, the vacuum cleaner has a waste container that must be emptied by a human being when it is full, and if water is used for cleaning, it must be filled by a human being. Moreover, after days/weeks of operation, maintenance and repair of the robot vacuum cleaner must also be performed by a human being. In summary, the robot vacuum cleaner can operate autonomously for short periods of time, but its medium- and long-term operation depends on tasks such as placement, supply, maintenance and repair that only a human being can perform.

Similarly, if we think of any equipment that has to work inside a mine, in a shift regime, and perform tasks such as installing of infrastructure (meshes, bolts, etc.), drilling, blasting, transporting material, etc., it can operate autonomously for limited periods of time, but undoubtedly, humans will be required to bring the equipment to its place of operation and provide it with the necessary supplies. Moreover, depending on the task, humans will supervise the safe operation of the equipment and evaluate the results obtained (e.g., mesh installation). In the medium and long term, humans will also perform maintenance and repair tasks, most of the time, inside the mine facilities.

In conclusion, the performance of tasks by autonomous equipment inside a mine requires human involvement.

2.4. Autonomous Equipment Confinement and Operational Interference Management

In underground mining operations, as well as in some surface mining setups, the number of interactions between mobile autonomous equipment and human personnel is minimized by isolating them from each other to achieve a safe operation. This is achieved by confining mobile autonomous equipment to designated mine areas, where human operators and manual equipment are prohibited. The isolation of autonomous equipment is facilitated by a confinement system that should encompass the following functionalities [21]:

• It must restrict unauthorized entry of equipment and people to the operating area of the autonomous systems, which is referred to as the autonomous operation zone (AOZ).
• It must be able to detect any unauthorized entry of people to the AOZ or any transgression of the system security.
• It must prevent unplanned departure of autonomous equipment in operation from the AOZ.
• It must provide controlled access for autonomous equipment to the AOZ.
• It must provide controlled and secure access for operators and manual equipment to the AOZ, when they are required to address operational interferences.

The confinement of autonomous mobile equipment may have a notable impact on the system efficiency and in production. As an example, the segregation of autonomous LHDs allows for the operation to be conducted in a safe manner at the production level. However, this has a detrimental impact on production, as each instance of human access to production areas disrupts the operation of autonomous equipment. It is important to consider that human operators must enter the AOZ on a regular basis to address operational interferences, such as resolving draw point hang-ups and performing secondary reduction tasks. Currently, operations implement a static confinement policy, whereby all autonomous LHDs cease functioning while human workers are present in the production area. Consequently, operations are entirely halted at the production level during the periods that human workers are required to be inside this level.

According to [21], the replacement of static confinement policies with dynamic confinement policies may prove an effective means of reconciling the competing considerations of safety and efficiency.

3. The Autonomous Collaborative Mining Paradigm

The mine of the future must be a safe environment in which humans will work in areas that are free from hazardous conditions. In light of the current limitations of available robotics technology, it is not yet feasible to implement fully autonomous operation of mines. Consequently, some tasks will be conducted by autonomous mobile equipment, while many others will be performed by human teleoperation from nearby positions or from operating centers, inside or outside the mine, or directly by human operators. Furthermore, humans will continue to be essential for other activities, including equipment placement, supply provision, task evaluation, maintenance, and repair.

In this context, the key idea of the Autonomous Collaborative Mining paradigm is that humans and machines interact in a flexible, collaborative and synergistic manner to achieve safer and more efficient mine operations. This is achieved through the use of various complementary technologies, including robotics, automation, IT, and machine learning. This paradigm is aligned and inspired in the new trends of collaborative robotics which is a “new form of robotic automation built to work safely alongside human workers in a shared, collaborative workspace” [22]. Accordingly with the Association for Advancing Automation “the ability to work collaboratively with humans greatly expands the potential applications of robotic automation” [22].

The Autonomous Collaborative Mining paradigm includes the following components, which can be implemented individually or in combination:

- Autonomous Operation under Human Supervision

- Shared Autonomy

- Dynamic Confinement and Online Monitoring

- Augmented Reality to Enhance Human Capabilities

Furthermore, it is essential to consider the enabling factors that may facilitate or impede the implementation of the paradigm

Before describing the main components of the paradigm, it is important to remember that the paradigm and its components are presented in this paper at a conceptual level. They still need to be further developed and implemented. Our goal is to motivate and encourage the community to adopt the ideas behind this paradigm to further develop systems based on it, which may combine some or all of its components.

3.1. Autonomous Operation Under Human Supervision

In consideration of the present state of robotics technology, automated operations, defined as operations carried out using autonomous equipment, will continue to require human supervision for safety reasons, as is the case in the autonomous vehicle industry.

Furthermore, it is also important to note that in many unit and auxiliary operations related to mine development, installation of infrastructure, cleaning, support, maintenance and supply, in which the equipment is installed and used for a fixed period of time, an increase in the level of automation of an operation does not entail the complete withdrawal of the personnel from the mine. Rather, it entails a withdrawal of the personnel from the work zone, with high associated risks, to a safer zone. This is mainly because even after an operation is automated, human operators are still required to prepare, monitor, supervise, evaluate and complete the operation. This aspect is frequently overlooked in the design of automated mining operations. Some of the tasks that will still be carried out by humans when a manual mining operation is automated include [5]:

• Inspection of the work area and the operational variables present in the environment.
• Verification of the state of the equipment to be used.
• Transportation and installation of the equipment in the area of operation.
• Administration of supplies required in the operation.
• Execution of the unitary operation.
• Uninstallation and removal of the equipment from the work area.
• Quality assurance (QA) of the result of the operation.

Figure 1 illustrates the continued involvement of humans in the operations of autonomous units, including preparation, withdrawal, quality assurance, and supervision (either locally or remotely).

Consequently, within the autonomous collaborative mining paradigm, humans remain a presence, assuming a variety of roles. One such role is that of supervision of autonomous operations.

3.2. Shared Autonomy

The concept of shared autonomy provides an alternative to full autonomy, whereby a human operator can provide situation awareness, decision-making support and other assistance to an autonomous system [23]. In the context of mining, where the operation of autonomous mobile equipment requires human supervision for safety reasons, shared autonomy may be a more suitable option than full autonomy, as it could lead to more reliable and safer operations.

Let us explain how shared autonomy works, and its distinguishing features in comparison to alternative automation modes. The following formalization will be employed: as shown in Equation (1), the orders that the mobile equipment (machine) receives in timestep t, $u\left(t\right)$, are contingent upon a function $h_c\left(t\right)$, which will combine the human orders, $u_h\left(t\right)$, and the autonomous system orders, $u_a\left(t\right)$, depending on the context, c. The context in which these orders are given may be dependent upon a number of factors, including the task completeness, the environment, the human operator activity, or other human-related factors.

$$u\left(t\right)=h_c\left(u_h\left(t\right),u_a\left(t\right),t\right)$$

(1)

If we apply this definition to the levels of autonomy shown in Table 2, we will see that in the cases of manual operation, remote control and teleoperation without assistance, $u\left(t\right)=u_h\left(t\right)$. Conversely, in the case of autonomy $u\left(t\right)=u_a\left(t\right)$. In the case of semi-autonomy, $u\left(t\right)=u_h\left(t\right)$ or $u\left(t\right)=u_a\left(t\right)$ depending on the task at hand. For instance, in the context of a semi-autonomous LHD, $u\left(t\right)=u_h\left(t\right)$ for loading, and $u\left(t\right)=u_a\left(t\right)$ for hauling and dumping. In the case of teleoperation with assistance, $u\left(t\right)$ will be a combination of $u_h\left(t\right)$ and $u_a\left(t\right)$, as defined when the system is designed. In this case the assistance provided to the human operator may be static, or it may be contingent upon the operator’s internal state or the external environment. For instance, the assistance level may increase when the operator is fatigued or when the machine is in proximity to other vehicles or humans. Finally, in the case of shared autonomy, $u\left(t\right)$ will be a combination of $u_h\left(t\right)$ and $u_a\left(t\right)$, with the autonomous system determining the specific manner and timing of this combination. Thus, $u\left(t\right)$ is controlled by the autonomous systems of transferred to the user. For instance, in the case that the autonomous system request assistance from a human operator, $u\left(t\right)=u_a\left(t\right)$will transition to $u\left(t\right)=u_h\left(t\right)$ while the human is providing assistance, and then revert to $u\left(t\right)=u_a\left(t\right)$ when the autonomous system regains control. Naturally, we will also have the case in which the human will not assume complete control of the machine. In such case, the order received by the machine will be combination of $u_h\left(t\right)$ and $u_a\left(t\right)$. What is relevant in the shared autonomy case is that the autonomous system is the one who decides how and when the human orders and its own orders are combined. However, it is important to note that in cases where the human supervisor detects unsafe or incorrect behavior, he/she can take full control of the machine.

Thus, we propose that within the autonomous collaborative mining paradigm, shared autonomy may be regarded as a viable alternative to full autonomy in instances where the latter is not readily attainable. To illustrate this proposition, we present a case study in Section 4, wherein the application of shared autonomy to the task of loading from draw points using LHD is proposed.

It is worth noting that some current fleet management systems for mining equipment are already incorporating the concept of shared autonomy, although not necessarily by that name.

3.3. Dynamic Confinement and Online Monitoring

As previously stated, autonomous mobile equipment is confined to specific autonomous operation zones (AOZs) for reasons of safety. The presence of human operators or manual equipment is prohibited in these zones. This so-called static confinement results in inefficiencies due to the necessity for autonomous equipment to halt operations each time human operators must enter the production area. A better solution to this suboptimal approach to confinement is the implementation of dynamic confinement policies. In contrast to the static confinement paradigm, the AOZ is updated in real time as the operation progresses under the dynamic confinement paradigm. The dimensions of the AOZ are contingent upon the prevailing operational status and are not necessarily determinable in advance. Key elements for implementing dynamic confinement policies are: (1) the use of sensors embedded within the environment (e.g., laser barriers) that indicate the mobile equipment the limits of permitted areas and act as virtual gates which can be dynamically activated (see examples in [21]), and (2) the real-time provision of positional data from all mobile equipment and human operators within the AOZ to a central system. This central system is responsible for defining the AOZ at any given moment and generating alarms when any agent makes an unauthorized movement, such as an unauthorized entry or departure from the AOZ.

In order to illustrate this idea, Figure 2 presents a conceptual diagram of a central system that allows the determination of the poses of multiple agents on an AOZ. The color ellipses and arrows represent the estimated position of the different agents - humans, autonomous systems, teleoperated systems and manually operated systems - and their direction of movement. The position of each agent is estimated based on the reported position of the agent, with variance (represented by the size of the ellipse) added to account for errors in the estimation of the own position by the agent (self-localization error) and time delays in communication. Additionally, static sensors (S) estimate the position of some agents within their field of view and provide supplementary observations.

The implementation of dynamic confinement policies not only enhances operational efficiency but also provides a greater degree of operational flexibility. This enables the coexistence of automated equipment and human personnel within the same designated areas, with the establishment of virtual boundaries delineating their respective domains.

For example, in [21] simulations are carried out to assess the performance of different confinement policies in a panel caving mine with 16 production drives. The results show a reduction in the operational losses (interference time) from a 24% in the case of using a static confinement policy to 5%-7% in the case of using a dynamic confinement policy, converting most of this time into effective time and having a better utilization of the semi-autonomous LHDs. The increase in effective time results in a 35%-47% increase in production.

3.4. Augmented Reality to Enhance Human Capabilities

The term augmented reality (AR) is defined as the provision of visual interfaces that superimpose computer-generated digital information onto the perceived physical world. This enables the provision of real-time insights, visualizations, and instructions in a direct and immediate manner, within the user’s field of vision, using devices such as smartphones, special glasses and other wearables. There are numerous applications for this technology, including (i) the provision of information about parts and components during an inspection process of facilities or equipment, wherein the system guides the operator to the specific locations that need to be inspected, identifies components and defects, (ii) equipment maintenance or repair, wherein the operator is able to follow visual instructions superimposed on the equipment part to ensure precise execution of maintenance procedures, and (iii) remote support, whereby the onsite operator can stream their view of the equipment to a remote operator (expert) who can provide instructions or annotate data directly in the field of view of the onsite operator. The information provided to the operator may be obtained from annotations (notes, comments, and markings) made previously during a planning stage. Alternatively, the information may be provided online by an AI assistant or by an expert placed in another location.

The application of AR in a mining context has the potential to enhance the capabilities of operators, providing them with more effective tools and multi-purpose functionality. A given operator will be able to perform a variety of tasks, including repairs on different types of equipment, with the assistance of AR-based systems. One can envisage a scenario in which an operator is situated inside a mine facility and is required to either initialize or repair a piece of mine equipment, despite lacking complete knowledge of the task at hand. In such a situation, a virtual assistant can furnish the operator with the necessary information through the utilization of an AR interface, thereby enabling the operator to complete the task successfully.

AR can also be used to enhance the visualization employed by a teleoperator when remotely controlling equipment. In this case, supplementary data, such as the distance to objects and the identities of object components and parts, can be superimposed upon the visualization displayed to the operator. In addition, telepresence, defined as teleoperation using special sensing and display technologies that enable the human operator to feel present at the remote location despite the operator’s physical absence [24], also enhances the quality of remote control tasks. Virtual reality (VR) also offers a variety of applications in the mining industry, including operator training and teleoperation. In the latter case, the teleoperation task can be conducted in a virtual environment generated using data from sensors monitoring the environment.

In the context of autonomous collaborative mining, the use of AR, VR and telepresence technologies facilitates an enhanced interaction between human operators and the equipment to be controlled or inspected. But more importantly, it allows for the enhancement of human operators’ capabilities, transforming them into versatile operators capable of performing a diverse range of tasks. This is relevant to achieve the goal of reducing the number of humans in potential hazardous working areas, as multi-purpose human operators allow for a reduction in the required on-site personnel.

It should be noted that the use of AR, VR and telepresence technologies requires an appropriate design, in order to avoid mental fatigue of the operator/user. Mental workload must be considered in the design of these interfaces, and in general, human factors must always be considered in the design of any computer interface. The proposed idea is not to use these interfaces permanently, but only when needed to solve specific tasks.

3.5. Integration

The various components of the paradigm can be used in an integrated fashion. Figure 3 shows a conceptual diagram that integrates the different components and technologies. In this diagram, machines with different levels of autonomy and humans using different types of interfaces work together in an AOZ using dynamic confinement. All machines are subject to human supervision for safety and efficiency reasons; at any time, a supervisor can take full control of a particular machine if unsafe or incorrect behavior is detected. Of course, supervisors can also detect incorrect human behavior and take corrective actions.

Machines can be operated either autonomously using shared autonomy, remotely using telepresence technologies, or manually. People within the AOZ can be operating the machines or performing other tasks such as sampling, inspection or repair. In all cases, they can use AR or VR interfaces to augment their capabilities, as needed. The use of dynamic confinement allows for a safe interaction between the various machines and the people inside the AOZ.

3.6. Enabling Factors

The advancement of the autonomous collaborative mining paradigm requires the development of interoperability standards, new regulations, the adaptation of the mining designs and procedures to facilitate the use of autonomous and teleoperated equipment, the development of new sensors to be placed in the mines, high-bandwidth and low-latency data networks, and a workforce trained in the use of the new technologies. These factors, among others, must be considered to ensure the successful implementation of the autonomous collaborative mining paradigm. In order to achieve this, it is necessary to coordinate the efforts of mining companies, original equipment manufacturers (OEMs), research and development institutions, technological companies, and regulatory bodies.

To illustrate the significance of the aforementioned enabling factors, the lack of interoperability standards for autonomous mining vehicles hinders the development of new modules and products by non-OEM entities, thereby slowing down the development of new solutions. For instance, in light of the recent advancements in artificial intelligence (AI), particularly in the domains of machine and deep learning, there arises an opportunity to leverage novel methodologies wherein AI can enhance the functionality of existing automation technologies. This may include the development of enhanced object and human detection algorithms, more advanced optimization algorithms for navigation, more sophisticated sensing methodologies, more advanced learning algorithms, etc. However, the application of AI-based solutions developed at research and development centers or at technology companies to commercial mining mobile equipment is hindered by two main factors: the lack of interoperability standards and the presence of contractual restrictions that prohibit any intervention with the equipment. In order to effectively address these challenges, it is essential to foster collaborative initiatives among OEMs, mining companies, and technology developers.

4. Case Study: Autonomous Loading with LHDs

To date, there are no commercial autonomous loading systems for LHD that can be used in in block/panel caving mines and sublevel stoping mines due to the complexity of the loading process; loading fragmented rock from draw points is a complex task due to many factors including: bucket–rock interaction forces that are difficult to model, moisture that increases cohesion forces, and the possible presence of boulders.

In order to address this challenge, the authors in [25] proposed an autonomous loading system based on the autonomous collaborative mining paradigm. The system is designed to integrate all the relevant tasks required for ore loading, including rock pile identification, LHD positioning in front of the ore pile, charging and excavating into the ore pile, pull back and payload weighing. The system employs the concept of shared autonomy: whereby in cases where the loading process cannot be completed autonomously, the LHD’s controller is able to detect this situation and request assistance from a human operator.

4.1. On the Complexity and Requirements of the Loading Process

Loading of fragmented rock involves a sequence of steps: As the LHD’s operator approaches the draw point, a quick assessment of the condition of the rock pile is made. The LHD’s bucket is tilted down until the tip is pressed against the ground. The machine is then commanded to charge/attack the rock pile. Once the LHD has made contact, it buries the bucket into the rock pile and the bucket is tilted and lifted as it advances through the fragmented rock. When sufficient penetration is achieved, the bucket is fully retracted, and the vehicle is withdrawn with a reverse motion. The operator then shakes the bucket briefly and assesses whether or not enough ore has been loaded. If necessary, a new loading maneuver is performed.

The step in which the machine is being controlled to dig into the rock pile will herein be referred to as the excavation process. During excavation, bucket–rock interaction forces affect the bucket motion through the material, and ultimately define the amount of ore loaded. Inter-particle forces in granular material and bucket-rock interactions have been studied previously, but their complexity has led only to stochastic modeling [26]. As a result, an analytical approach to bucket motion control is not possible, and it is highly unlikely that these interactions can be accurately replicated to develop a system in simulation. Therefore, only a real-world environment can provide the appropriate conditions for developing and experimenting with an excavation method. Furthermore, fragmented rock is much more difficult to load than piled sand or gravel. This is exacerbated by the environmental conditions of underground mining. Depending on the mining method, the column of blasted rock can exert large compressive forces on the open face of the extraction point. Humidity can also increase cohesion forces, making the whole process more difficult [27]. In addition, large rocks may be present at the draw point. For this reason, the operator must be aware of the presence of large rocks as he approaches the draw point, and, depending on their size, may be forced to change the goal of the maneuver from filling the bucket to just loading one large rock. If the rock is too big to be loaded, the task is aborted. Furthermore, other conditions, such as narrow tunnels, uneven or unprepared terrain, and/or significant amounts of water and mud near the draw point, can make the loading process very challenging.

An autonomous system should be prepared to deal with all of these situations, in addition to performing certain processing and steps that a human operator performs with while driving the LHD, such as detecting a sufficiently accurate enough position of the rock pile before charging at it, making sure not to hit the tunnel while pulling back from the rock, and estimating whether the bucket is full enough.

4.2. Autonomous Loading Based on the Autonomous Collaborative Mining Paradigm

An autonomous loading routine can be thought of as a sequence of several steps that perform specific actions. Here, rock pile identification is the process of finding the location and status of the draw point (whether it is suitable for loading or not). Positioning refers to orienting the LHD machine to ensure that a forward thrust will end in a collision with the rock pile. Charging is the step that considers lowering the bucket and accelerating towards the pile. Excavation is the machine control that performs the digging action. Pull back comprises the backward movement of the vehicle and bucket shake. Finally, payload weighing estimates the amount of ore loaded. Most of these steps may fail due to the highly variable environment in underground mining. For example, (i) the on-board sensors may not accurately locate the rock pile, (ii) the machine positioning may not avoid colliding with a tunnel wall, (iii) the collision between the machine and the rock pile may not be detected, or (iv) the bucket of the LHD may become stuck during excavation. In any of these cases, human intervention would be required to resolve the problem. In fact, most advanced mining automation systems for mining operations require human interaction with automated mobile machines, at least to monitor/supervise a fleet of machines. Since large operations have a fleet of vehicles, it is common for one or more operators to monitor/supervise and assist multiple semi-autonomous machines, as each machine does not require constant attention. For these reasons, it is imperative that this human interaction be taken into account in the design phase of an autonomous loading system, and thus, included in the formulation of its behavior.

Taking these considerations into account, an autonomous loading process with these characteristics is presented in [25]. Figure 4 shows the proposed state machine that models the autonomous loading process. Two distinct assistance states are defined: positioning assistance, wherein the operator assumes control of the machine to position it in front of the rock pile, and loading assistance, wherein the operator assumes control of the LHD to perform the remaining steps of the loading maneuver up until payload weighing. Positioning assistance is called up either when the rock pile identification fails to characterize the rock or if positioning is unable to place the LHD to start the charging procedure. Once the LHD has been positioned correctly, the LHD’s controller regains control of the LHD in order to commence charging the pile. Loading assistance is called up if charging, excavation, or pull back cannot be completed successfully. Following the successful completion of the pullback step, the LHD’s controller regains control of the LHD in order to complete the payload weighing step.

In Figure 5 is shown a pseudocode that implements the state machine of the loading process.

4.3. Validation in Real Mining Environments

The validation of an autonomous loading system through full-scale experiments with a full-size LHD in a real mining environment is a complex process that can take several months. First, the excavation algorithm of the autonomous loading system proposed in [25] was validated for several months in a sublevel stoping copper mine in northern Chile. The rock pile identification module was validated using data obtained during the same tests. The results were satisfactory and are described in [25]. Later, the system was adapted for use in Room & Pillar mines, and subjected to a full validation, again during several months, in a potash mine located in Germany. The main results of this test are presented in [3]. The system is now being fine-tuned to achieve the production KPI defined by the customer, and there is a good chance that the system will be adapted by this mine.

The reader is referred to [25] and [3] for details of the technical implementation of the technology and how the validation process was carried out. In Room & Pillar mines, there is a wide variety of muck piles, and working under the principle of shared autonomy makes the system more flexible to deal with this diversity.

5. Conclusions

The future mine must be designed to provide a safe environment for humans, situated at a safe distance from potentially hazardous areas. The automation of mining operations plays a pivotal role in achieving this objective. In this context, we put forward the autonomous collaborative mining paradigm as a means of enhancing the automation of mining operations by incorporating human-robot collaborative mechanisms. This paradigm entails a flexible, collaborative and synergistic interaction between humans and machines, with the objective of achieving safer and more efficient mining operations. The paradigm posits that autonomous operations still require human involvement, particularly in supervisory roles. Furthermore, shared autonomy is regarded as a potential alternative to full autonomy in instances where the latter is not a viable option. Dynamic confinement policies facilitate the physical coexistence of automated equipment and humans within the same working areas. Furthermore, the utilization of augmented reality (AR) technologies allows for the enhancement of human operators’ capabilities, thereby transforming them into versatile operators capable of performing a diverse range of tasks. This is a crucial aspect in achieving the objective of minimizing the number of humans in potential hazardous working areas, as multi-purpose human operators allow for a reduction in the number of personnel required.

The autonomous collaborative mining paradigm represents not an end in itself, but rather a means of working towards the mine of the future. The implementation of the paradigm necessitates the consideration of pertinent enabling factors, including the development of interoperability standards, new regulations, the adaptation of the mining designs and procedures to facilitate the use of autonomous and teleoperated equipment, the development of new sensors to be placed in the mines, high-bandwidth and low-latency data networks, and the training of a workforce in the use of new technologies. The implementation of the paradigm cannot be addressed by a single institution; it requires the coordinated efforts of a number of stakeholders, including mining companies, original equipment manufacturers (OEMs), research and development institutions, technological companies, and regulatory bodies. This represents a significant challenge, but it is nevertheless a worthwhile endeavor to begin working towards the development and implementation of the paradigm.

It is notable that the autonomous collaborative mining paradigm has a direct application in space mining. The processes involved in mining on a planet or an asteroid can be conceptualized as analogous to those employed in the mining of distant locations, whether underground or in a remote desert. This indicates that the automation technologies currently utilized in terrestrial mining can be effectively applied in space mining. In this context, some key elements of the autonomous collaborative mining paradigm such as shared autonomy, human-machine collaboration and general-purpose operators may be considered.