Mixed Reality in the Operating Room: A Systematic Review

1. Introduction

1.1. Context

Over the years, surgical practice and the intraoperative space have evolved with technological advances, seeking safer, faster, and more efficient methods to succeed in the operating room (OR) [1,2]. Recently, the increasing development of computers and related technologies, such as smartphones, tablets, or head-mounted displays, has led to a growing interest in the OR has followed the trend to take advantage of the main benefits that these devices bring: easily accessible information [3,4].

In the field of modern medicine, the pursuit of excellence is a constant commitment. In the OR, where precision is crucial, there is always a need for continuous improvement. Surgeons, regardless of their experience, face the complexity of procedures, the demand for ongoing education, and the imperative to improve patient outcomes. Augmented Reality (AR) and Mixed Reality (MR) are emerging as powerful tools in surgery, promising to introduce a new era of enhancement. These innovative technologies have the potential to revolutionize the OR by seamlessly integrating digital information into the surgeon's field of view, transforming the way they learn, plan, and execute procedures.

For example, in the context of surgical procedures, the monitors are skillfully managed by the qualified operating room personnel who have both the expertise and the resources necessary to prioritize the surgeon's optimal performance. However, some very specific questions about surgical procedures require a deep insight that only the surgeon’s expertise can provide [5]. This would not be a problem if the surgeon had the answer at hand by accessing a tablet or smartphone. However, given the need to maintain aseptic conditions in the OR and the fact that the surgeon would have to operate these such devices with his hands, this solution is not practical, leaving an empty space for technology to meet these needs.

Another example, in the OR, is the positioning of the x-ray monitors or those used in endoscopic examinations, which may not be favorable to the surgeon’s positioning, increasing the need for constant adjustments during the procedure as well as an increased risk of injury for the patient caused by the associated muscular effort. It can also lead to loss of focus and prolongation of the surgical procedure [6] (also implying efficiency losses).

The concept of Extended Reality (XR) encompasses a spectrum of technologies, including virtual reality (VR), AR, and MR. Although these technologies may seem distinct, they share several common characteristics that unite them under the XR umbrella, namely digital interaction, digital visualization, immersion, and presence, with the goal of creating immersive and interactive digital experiences. Despite the recent prominence of XR, the term “mixed reality” was first introduced by Paul Milgram in 1994 to describe the merging of the physical and the digital worlds as part of a virtual continuum [7,8].

In the ever-evolving landscape of healthcare and medicine, constant technological improvements are redefining the boundaries of what is possible. MR is one of such innovations that has captured the attention of the medical community. While MR shares common elements with VR, its unique blend of real and virtual environments presents an even more compelling case for its integration into surgical rooms. MR promises to radically change the way surgeons operate, train, and, in turn, improve patient outcomes.

AR and MR are both forms of XR that blend digital and physical worlds. However, they differ in the level of immersion and interaction they provide.

AR enhances real-world images on flat screens without special devices using digital overlays. For instance, AR Snapchat filters use face-tracking software and smartphone cameras to add puppy ears to a person’s face. MR integrates physical and digital experiences by using headsets and other devices that provide 3D immersion. MR not only overlays but also anchors virtual objects to the real world, enabling users to seamlessly interact with both environments. For instant, MR headsets can project holograms of people or objects that users can manipulate with gestures or voice commands. In other words, AR and MR are similar in that they both enhance the real world with digital content, but they differ in how they achieve it. AR adds 2D overlays to flat screens, while MR creates complex 3D interactions with immersive devices [9].

The use of MR in surgical contexts goes beyond being a mere innovation or trend. It represents an evolution in a continuous search for precision and effectiveness in the medical practice. MR offers an immersive and interactive platform that seamlessly combines the physical and digital worlds. This allows surgeons to navigate complex anatomical structures, practice procedures or plan surgeries with unparalleled detail and realism, while in a controlled environment.

The OR can also utilize this technology as a teaching or support tool, providing the surgeon with access to information in a completely manipulative manner. This would allow the surgeon to consult annotations and information about the surgical technique, contact other specialists online, and access patient information, all within their point of view and without the need for physical contact with the device. Gestures, voice commands or even retinal movement can provide this. This would enable younger surgeons to better prepare and flatten their learning curve through an immersive experience. It fills the void of information access and contact with other specialists that exist in the modern OR, making it easier for them to gain knowledge and expertise.

Several authors have recognized the innovative and beneficial approach of using technology, particularly MR, computer-aided assessment, and 3D models, in healthcare, particularly in medical and surgical training. These tools offer significant advantages such as real-time remote mentoring, which allows trainee doctors to receive expert guidance during surgical procedures. They also provide the ability to offer immediate and personalized feedback, expanded visualization of medical data in immersive environments, and easier identification of anatomical anomalies. Detailed preoperative simulation also stands as a crucial advantage. In the context of training, immersive environments are set to increase focus on training activities, improve anatomical understanding and provide valuable practical experience. These advantages converge to transform surgical practice, promoting substantial advances in medical education, in the training of health professionals and, consequently, in the quality of care provided to patients.

Mixed reality (MR) is as a valuable tool in the operating room, offering various advantages for medical procedures and interventions. It allows the integration of relevant patient-specific data within real-time, real-world observations in a single display, which can enhance surgical training, education, and planning. Surgeons find MR to be a useful tool for learning and studying human anatomy, as well as for surgical planning, particularly in minimally invasive surgery. The utilization of MR technology demonstrated to enhance health outcomes, surgical procedure accuracy, and collaboration among surgeons, ultimately resulting in improved patient care [10,11].

Mixed reality (MR) has been shown to improve surgical education by providing a higher quality educational experience, improving skill progression, and ensuring greater consistency in learning when compared to traditional teaching methodologies for basic surgical skills [12]. Additionally, a narrative review of the literature has highlighted the potential for MR to improve intraoperative accuracy, surgical outcomes, and patient satisfaction. According to, MR can enhance the assessment of surgical risks, enable modification of surgical strategy as necessary, and increase patient satisfaction when used during surgery [10]. Additionally, a systematic review concluded that augmented and mixed reality technology improve surgical outcomes by increasing navigational speed and reducing navigational errors during surgery [13]. Overall, MR technology has the potential to enhance surgical training, education, and planning, leading to improved surgical outcomes, accuracy of procedures, and patient care.

1.2. Research Questions

To better guide the proposed systematic review, a set of research questions have been derived using the PICO framework (standing for Patient/Population/Problem, Intervention, Comparison, Outcome), a tool for formulating evidence-based research questions. With this approach, the following research questions were set:

• RQ1. Among surgeons performing surgery (P), does the incorporation of mixed reality tools (I), lead to increased precision, accuracy, and overall performance during surgery compared to conventional approaches (O)?
• RQ2. In patients undergoing surgery (P), does the use of mixed reality technology during the surgical procedure (I), result in improved surgical outcomes, such as reduced operation time, lower complication rates, or enhanced patient recovery compared to traditional surgical methods (O)?
• RQ3. In the context of surgery (P), how does the application of mixed reality technology (I), impact the learning curve and skill acquisition for surgical trainees compared to traditional training methods?
• RQ4. Among healthcare institutions implementing mixed reality in surgery (P), what are the cost implications and resource requirements (I), and how do these factors compare to traditional surgical approaches in terms of overall economic feasibility and sustainability (O)?

The questions formulated will guide the search for content, organization of concepts and the extraction of information from the articles to be reviewed. Answers to the above questions are presented in the Discussion section.

1.3. Document Structure

In this review article, the multifaceted world of MR in surgery, its diverse applications, its impact on surgeon training and education, its role in pre-operative planning, and its potential to enhance patient safety are the focus of exploration. Observing the referred problems, a systematic review of the use of MR in the OR was conducted to answer the research questions proposed.

2. Methods

To elaborate this systematic review, the guidelines of the PRISMA statement (Preferred Reporting Items for Systematic Reviews and Meta-analyses) were followed [14]. From now on, the protocol of this study will be available on the PROSPERO platform and can be accessed with the following code: CRD42023427699.

This study aims to provide an up-to-date review of the evidence regarding the effectiveness of MR in the OR. To identify the research question, the PICO strategy was used [15], Population: OR personnel; Intervention: the use of MR technology tools; Comparison: surgeries using MR tools versus conventional surgery and its traditional teaching and, Outcome: the impact of the use of the tools on professional and patient experience.

2.1. Search Strategy

The search was carried out in electronic databases, for which the authors of this paper have developed a strategy, namely. The following equation, with the defined keywords, was elaborated, in order to identify the relevant studies to answer the established questions: (“Mixed Reality” OR “Augmented Reality” OR “Holographic”) AND (“Operating Room” OR “Operating Theater”) AND (“Surgery” OR “Surgical Procedures” OR “Surgical Interventions” OR “Medical Training” OR “Medical Education” OR “Surgical Training” OR “Surgical Education” OR “Surgeons” OR “Surgical Specialists”). The review primary focuses on the use of mixed reality in the operating room. However, the search query also included the term “Augmented Reality” due to its innovative nature and debatable definition. It is possible that a Mixed Reality application could have been described using this term. The consulted databases included PubMed, IEEE, ScienceDirect, ACM, Academic Search Complete, Web of Science, and Scopus. The selection was based on the article’s title or abstract, and filtered by language (English and Portuguese only) and time (articles from 2018 beyond as of June 1st 2023).

2.2. Eligibility Criteria

The eligibility criteria for this systematic review were determined based on the population of the articles, their design, and characteristics. Articles must written in either English or Portuguese, have the complete text available, and not be a review or conference article. They must have been published between 2018 and 2023 and address the use of MR in the OR (surgery planning is not considered part of the OR activity).

2.3. Study Selection

The authors analyzed scientific articles in three stages, considering the title, abstract, and full text. They used inclusion and exclusion criteria and excluded duplicate articles. Next, two authors independently selected articles that met the review’s objective and inclusion criteria by reading their titles and abstracts. The articles included by one reviewer, but excluded by the other, were later subject to analysis by a third party, who had the final decision on their inclusion. After analyzing each article using this method, a second review of the full text of the remaining articles was conducted. In cases where the two authors disagreed, the third was consulted again, until a final list was reached. This process is illustrated in Figure 1, using the PRISMA diagram. Finally, the characteristics of each selected article was compiled in a table, which presents the conclusions on the subject.

2.4. Quality Assessment in Systematic Reviews

The assessment of each article’s bias is made according to the type of study reported. Two main groups were set: one for articles referring to case series and another for case-control studies. The distinguishing factor between these two groups was the presence of a control group. The group of articles describing case series did not include a control group and measured values with comparison. In contrast, articles with a control group compared both groups. To assess the risk of bias in case series studies, the scale used was the one developed by the Canadian Institute of Health Economics (IHE) [16] was used, while the Newcastle-Ottawa Scale (NOS) [17] was chosen to assess the risk of bias in case control studies. The initial scale assesses the potential for bias by considering factors such as the study’s purpose, design, population, interventions, outcomes, statistical analysis, results, conclusions, and competing interests. The NOS employs a star rating system to evaluate the risk of bias, taking into account the study’s selection, comparability, and exposure. Each article underwent the appropriate assessment.

3. Results

A total of 462 articles were collected from the electronic databases. Of those, 248 were duplicates, leaving 193 unique articles. The abstracts of these articles went through a review process, resulting in ten articles selected for full-text analysis. After analyzing the full text of these articles, four articles were excluded: two for describing VR instead of MR, one for utilizing AR instead of MR and one for being a qualitative article with no relevant measurements for further analysis.

3.1. Study Characteristics

The use of MR applications in the OR can be divided into two types: those that aim to facilitate the learning process and those that are used to assist the surgeon during the procedure. Examples of the former include works [18,19], which demonstrate the potential of MR technology as a learning tool. The first work focuses on trainee mentoring, while the second covers the creation of a virtual environment to prepare and train individuals for the surgical setting. Cen et al. [20] describe the use of MR for the 3D visualization of the patient’s organs in a perioperative context, providing support for the surgeon.

However, there are many challenges with MR. As a new technology, users are hesitant and need time to learn how to use it [19], which requires training and adaptation to new approaches. In addition, the preparation of MR content and programs is more difficult than for other established technologies [18], again requiring a step-up program for participants to gain expertise. The use of head-mounted displays also causes dizziness in some users [20]. This technology also has advantages, such as a lower cost of implementation when compared to other teaching tools in the OR [18], with satisfactory ratings from trainees, but also a new way to evaluate and prepare for different scenarios that occur in the OR [19]. In order to support the surgeon and help him understand and visualize some complex cases, MR is used to reveal 3D models of organs and their surrounding structures, facilitating the approach [20].

In the retrieved articles, five of the six were case series, which means that there was no control group with which to make a comparison. However, in these cases, measurements were made to evaluate the work performed, such as Likert scales regarding experience, established scales to evaluate performance, and measurements of patients whose surgical procedure used this type of technology.

The main aspects of the six reviewed articles are presented in Table 1. Still regarding the reviewed articles, in terms of the technology used to support MR, the Head-Mounted Displays (HMD) or Optical See-Through Head-Mounted Displays (OST-HMD) dominate the field, with Microsoft HoloLens^® and the Magic Leap^® being the most used. In Table 2, it is possible to compare the reviewed solutions in terms of the quality of their results, without forgetting the provided performance metrics. Finally, in Table 3, a cost-benefit analysis is presented for each case, considering the related challenges and limitations.

3.2. Keyword Identification and Frequency

Based on the abstracts of the reviewed articles, the most common words were selected, and a word cloud was created by counting the frequency of each word, as shown in Figure 2. Word clouds are powerful tools that allow observing aspects such as key themes, trends, conceptual relationships, or methodological focus, among others. Looking at Figure 2, as expected, words related to the application of the current research area are clearly visible (“operating”, “surgical”, “surgeons”), as well as those related to the associated technological concepts (“mixed”, “reality”, “platform”). It is interesting to note that words related to the benefits often associated with MR are well represented (“performance,” “assessment”, “visualization”, “study”). Words related to the supporting components operating mechanisms are also present (“computerized, “simulation,” “data”).

3.3. Quality Assessment

The risk of bias of reviewed articles, summarized in Table 4, was assessed according to their respective study organization and the information available in the document, using two different scales. In the first scale, related to the case series articles, all the selected articles were declared to have an insignificant risk of bias. However, some of the aspects assessed did not have such an insignificant risk. For example, for study design, all articles had a considerable risk of bias. The selection of studies was not as unbiased as desired, as it were often carried in the same institution and not continuously. Although this may seem alarming, the reason for this is the low availability of MR equipment, resulting in few cases of use to study. Some articles did not have a low risk of bias in the study population, mainly because of poor description of the selection methods [18,22]. The measures used to assess the outcome were not very well designed, resulting in a moderate risk of bias in the majority of the selected articles. One of the main problems was that the assessment of success was not done blindly, since the assessor was aware of the intervention used [19,20,21,22]. In some cases, the measurements made after the intervention and not before [19,20,21,22]. One of the reviewed articles also has a considerable risk of bias in the statistical analysis, as there were no appropriate relevant outcomes assessed with statistical tests. For parameters that were not reported, the articles presented a low risk of bias, which is why the risk of bias was assessed as low for all of them, as can be seen in Figure 3.

The system used for the remaining case-control study was the NOS [17]. Details of the star template are presented in Table 5. We found that the case definition included references to primary record sources in the selection criteria, which explains the missing star. Comparability and exposure had the maximum number of stars, which shows a good comparability of cases and controls based on the analysis and structured interview, both of which went blind to case/control status. Eight stars (out of a total of 9) were attributed to the case-control study.

The results indicated that most of the data in this systematic review came from studies of high quality.

4. Discussion

4.1. Highlights

The articles reviewed highlight various benefits and challenges associated with the implementation of innovative technologies in the surgical field. In the first article, MR proved to be a valuable tool for continuing education, allowing remote mentoring, enhanced visualization of medical data, and identification of anatomical anomalies. The second article discusses a comprehensive approach to computer-assisted assessment, emphasizing empirical validity, realistic contextualization, and the ability to provide immediate feedback. The third article demonstrates the use of 3D cardiac models in conjunction with virtual reality (VR) and mixed reality (MR) to provide better anatomical understanding, surgical simulation, and intraoperative decision support. The fourth emphasizes standardization and contextualization in surgical performance assessment, supported by evidence of validity in different domains. The fifth proposes an integrated platform for multiple imaging modalities, with an interactive 3D model and innovative functionalities, particularly applicable to vascular neurosurgery. The sixth highlights the utility of Mixed Reality for training and telementoring, emphasizing improved efficiency and image quality.

4.2. Answers to Research Questions

RQ1. From the reviewed articles, four studies addressed the use of MR during surgery [20,21,22,23]. In all cases, there is evidence of benefits compared to conventional surgery. Easier understanding of procedures, increased accuracy, improved safety, and reduced operative time were frequently mentioned.

RQ2. While the postoperative period from the patient’s perspective is not addressed in the literature, it is implicit that the enumerated benefits highlighted in RQ1 collectively contribute to an overall improvement in the quality of surgical outcomes. These improvements have promising implications for patient well-being, highlighting the positive consequences of integrating MR technology into surgical practice.

RQ3. Three studies [18,19,20] discuss learning. The authors of these studies agree that simulation shortens the learning curve and is more cost-effective and time-efficient than traditional methods. Simulation in a safe environment also shown to be crucial for skill development and retention. The ability to teach remotely and share perspectives is also a highly valued benefit of MR. In addition, MR provides assessment methods for immediate feedback and allows for scenario repetition to track performance improvement.

RQ4. In [18], the cost aspect of Mixed Reality (MR) solutions is examined, with special reference to a low-cost implementation. The other studies within the selected literature explore different supporting architectures, each serving different purposes and indicating a wide range of associated costs. It is noteworthy, however, that the dynamic nature of the technology evolution and the expected decrease in the cost of hardware devices suggest a trend toward increased performance. This trajectory paves the way for a prospective reduction in the barriers to access to MR solutions, highlighting the potential for increased affordability and widespread availability in the future. From an institution’s perspective, given these benefits, investing in MR solutions may prove to be the most appropriate way to improve patient outcomes while reducing operational costs.

4.3. Study Limitations

While a revision based on a modest selection of six articles may offer a limited scope, it remains a valuable endeavor, especially given the timeliness of the topic under consideration. Collectively, the six selected articles help provide an initial overview that illuminates the contemporary landscape of the topic. This focused examination allows for a nuanced exploration of the benefits, limitations, and emerging opportunities associated with the topic. The inherent novelty of the topic, coupled with the insights provided by these selected articles, enhances our understanding and lays the groundwork for further comprehensive research. As the field evolves, these initial findings can serve as a foundational framework for future research, guiding scholars toward a deeper understanding of the intricacies surrounding the topic.

Mixed reality glasses can provide several benefits during a surgical procedure. Some of these advantages include augmented visualization, which allows direct visual information to be superimposed on the surgeon’s field of view, providing an improved perspective of anatomy and surgical instruments.

In addition, these glasses can provide precise, real-time gesture guidance to help the surgeon navigate and perform procedures with greater accuracy. The ability to access critical clinical process data is another benefit, displaying important information such as medical images, diagrams or patient data without the need to divert attention to monitors or external devices. Real-time communication could be facilitated, allowing instant collaboration with other healthcare professionals during the procedure. In addition, these glasses could be useful in education and training, allowing medical trainees and students to observe procedures in real time and gain practical experience.

5. Conclusions

5.1. Mixed Reality in the Operating Room

Along this review, the aim was to answer the proposed questions and determine the advantages and disadvantages of incorporating MR tools in the OR. After analyzing the articles, the use of MR can be considered a helpful tool in many areas of the OR, ranging from the using the device to visualize intraoperatively, to communicating with the exterior for assistance. Many of the reviewed papers, describe the MR as a way to extend the field of view of the surgeon, adding a screen containing what is seen in the monitors regularly placed in the OR, from endoscopic imaging, but also other information of the patient, such as radiographies, blood tests, and other types of medical imaging [20,21,22,23]. Another common application of MR in the OR, as seen in the reviewed articles, is its use as a communication tool that can be used for mentoring, training and clarification of concepts from other specialists [18,21,24].

When using this technology, the users faced many challenges such as adapting to a new technology, which takes time [21], especially for older users [20], lack of confidence in this type of new technology [19]. Some of the researchers faced difficulties in developing software for MR, judging the tuning as slow [18], others argued the low capability of tracking from the device [21], its parallax error and the headset autonomy [19]. In the area of ergonomics, some articles described the use of MR devices as inducing headaches or dizziness [20], as well as reduced field of view and parallax error [23].

The use of this technology is considered as a cheaper alternative to the existing methods but also a tool of great quality, being, overall, well accepted by the questioned trainees [18]. This is seen as a way to prepare the residents for real scenarios, digitally, complementing what is the standard today, providing better training and improving education on surgical tactics and methods [19,25]. It is well known that simulation can be a powerful tool to support learning, especially in the healthcare area [26]. In the surgical procedure field, the visualization of 3D models related to the surgery are presented as a benefit for a better understanding of the condition and situation by the novice surgeons, offering different degrees of immersive experience [20]. MR technology also allows for a training environment appropriate for recreating realistic and reproducible scenarios without putting the patient at risk [19]. The use of MR contributed to improved imaging which can decrease the surgeon’s stress level and facilitate the performance of a safer and more precise operation [17]. Compared to the visualization options currently available, it enables the surgeon to consult, control, manipulate data for better decision making and remain sterile [18]. Another useful aspect is the possibility of combining mixed reality functions such as video recording for training and displaying image, text, video, or 3D objects as guidance [19]. The capability to load several DICOM data into the surgeon’s field of view in real time is also a particularly useful feature [19]. Reducing the problems with intraoperative visibility, MR applications tend to lower the procedure time, allowing multitasking and improving real-time guidance, within this context [21,22,23].

Will still evolving, these immersive technologies have the power to bridge the gap between aspiration and reality, turning the dream of an improved, patient-centered surgical experience into a tangible reality.

5.2. Open Challenges and Future Directions

While MR technology holds enormous potential, its current state of development presents challenges, particularly in the unique context of the operating room. Achieving regulatory approval for use in the OR is critical to widespread adoption, ensuring compliance with rigorous safety and efficacy standards. Practical considerations related to supporting hardware, such as limited field of view, battery constraints during long surgeries, and the need for extended user comfort, even during periods of hardware inactivity, are critical areas for ongoing development.

In addition, the need for fast processing hardware to ensure low latency and reliable synchronization for efficient data acquisition and communication with external devices is crucial. Overcoming these technical issues is essential to minimize adverse effects such as dizziness or motion sickness and to enhance the realistic and precise visualization and manipulation of virtual objects, despite potential limitations.

Finally, promoting software interoperability between Head-Mounted Displays (HMDs) and medical devices could significantly expand the applications of MR in the OR.

These challenges will soon be overcome as MR technology becomes a dependable and essential tool in the surgical setting.

The use of such technologies will be integrated with other evolving fields, such as Video Assisted Thoracic Surgery (VATS) and Robotic Assisted Thoracic Surgery (RATS), due to its ability to provide critical information and guidance during procedures.

The rise of artificial intelligence and its growing “understanding” of the real world may also open the possibility for mixed reality technology to play a more active role in surgery.