How is the Digital Surgical Environment Evolving? The Role of Augmented Reality in Surgery and Surgical Training

Background Augmented reality (AR) in surgery can offer an enhanced view of reality through the superimposition of computer-generated digital images on the real environment. It allows surgeons to integrate image visualisation, improving operative efficiency, surgical outcomes, surgical training and patient education. This review aims to evaluate the current status of augmented reality in surgery, surgical training and potential future applications. Methods We performed a non-systematic review of available literature from January 2005 to August 2021 by searching PubMed, EMBASE and the Cochrane library using a combination of terms “augmented reality”, “virtual reality”, “surgery”, “simulation” and “training”. Articles considered for this review were identified by relevant search criteria including title, keywords, abstract, and full-text. AR has also exhibited the potential to enhance surgical training and improve knowledge acquisition; it can foster international collaborations via telesurgery and telepresence. In the near future, AR will likely work in symbiosis with surgeons, serving as a complex computer-human coalition which can improve patient outcomes, patient education and surgical training.


Introduction
The exponential growth of imaging technology over the last century has revolutionised patient care from diagnosis to therapy. The evolution of medical imaging including development of ultrasound (US), computed tomography (CT) and magnetic resonance imaging (MRI) has enabled the use of three-dimensional (3D) image reconstructions for diagnosis and treatment. Further developments in imaging combine anatomy and physiology, such as functional MRI (fMRI) and single-photon emission computed tomography (SPECT). They provide physicians with a better anatomical and functional understanding of the target area, as well as improving diagnosis and prediction of prognosis.
Man-computer symbiosis has paved the way for the most recent developments in medical imaging technology to incorporate real-time feedback and visualisation. Augmented reality (AR) can be defined as the superimposition of a virtual image on the real environment, offering an enhanced view of reality using computer-generated digital images. It differs from virtual reality (VR), which immerses the user in a completely artificial, computer-generated environment. Whereas VR is predominantly used for training and preoperative planning [1,2], AR has shown the potential to transform the intraoperative environment.
The awareness of the applications of image-based augmentation during surgical procedures and its impact on patient outcomes remains low, with a gap existing between research and surgical applications of the technology. The review aims to evaluate the current status of AR and potential future applications in surgery and surgical training.

Materials and Methods
We performed a non-systematic review of available literature dating from 2005 to August 2021 by searching PubMed, EMBASE and the Cochrane library using a combination of terms "augmented reality", "virtual reality", "surgery", "simulation," "training". Articles considered for this review were identified by relevant search criteria including title, keywords, abstract, and full-text.

Fundamentals of AR
AR is a technology that provides a composite view of reality through the superimposition of a computer-generated image on the real environment, thereby enhancing the user's perception of the real world. By taking CG images such as CT and MRI scans preoperatively, and superimposing them on surgical targets intraoperatively, surgeons can expect improved visualisation of anatomical structures [3] thereby providing the potential for improved surgical outcomes. This has been increasingly tested in image-guided surgery systems [4].
The digital element of AR can take the form of either a head-mounted display (HMD), a handheld device, or a CG overlay [5]. To align preoperative digital images with the patient's anatomical landmarks, a process called registration is required. This is made difficult for many reasons, such as the need to continuously track subtle movements in the patient, so both coordinate spaces (virtual and real) can be in synchrony [6]. Further investigation is therefore vital to ergonomically refine stereotactic registration technology and new methods for this are being proposed in the literature. For example, Wang et al. proposed a novel method of registration which does not rely on external fiducial markers, overcoming the major limitation in AR regarding misalignment caused by organ deformation and patient movement [7].

Different AR systems
A range of AR technologies have been trialled and implemented for use in the surgical field. The AR technology chosen is dependent on its intended use, from patient education and surgical training to preoperative planning and intraoperative guidance. Popular examples of current AR systems include Google Glass, a HMD which accumulates and inputs surrounding sensory information to generate an augmented reality image superimposed on a real-time image [8]; AccuVein, a device that emits infrared light on a patient's skin allowing visualisation of the underlying vascular structures [9]; Microsoft HoloLens, a HMD that overlays three-dimensional holographic images of CT scans onto the surgical site to aid the surgeon's patient-specific anatomical understanding intraoperatively [10].

AR in surgical training
The gold standard of traditional surgical teaching has typically involved the use of cadavers, however recent technological advances have culminated in the rise of AR. Although still in its infancy, AR has been trialled in several teaching scenarios with a view to improve learning experience and knowledge acquisition. The ProMIS system was employed for the simulation of laparoscopic tasks including basic navigation, suturing, knot tying and sharp dissection [11]. The system is designed to provide realistic haptic feedback to enhance user ergonomics and prepare trainees for laparoscopic operations. Moreover, the system's ability to monitor and record numerous performance parameters, including time taken, path length and smoothness of motion, provides trainees with objective feedback for tracking progress. Procedures such as lumbar puncture and facet joint injections can be trained using the Perk Tutor, a system providing an AR overlay on a phantom model [12]. The Perk Tutor's intrinsic ability to measure factors such as procedure time and path length can indicate to trainees where there is scope for improvement. Randomised controlled trials illustrate the efficacy of the Perk Tutor in the training of percutaneous facet joint injections. Performance metrics such as "potential tissue damage"; "total procedure time" and "needle insertion time" were measured. Trainees trained by Perk Tutor significantly exceeded the performance of those in the control group [13].
Alongside operative simulation, AR has seen application in anatomical teaching. A study evaluating the efficacy of AR for lower limb anatomy teaching revealed that in a cohort of 211 medical students, the group receiving AR-based teaching achieved statistically significantly higher examination scores when compared to the control group, who were taught via diagrams and 2D images [14].
Evidence from literature illustrates the success of local AR-based teaching for trainee performance. A systematic review of 18 publications comparing AR-based surgical teaching with traditional techniques showed that [15]. Competency, user opinion and post-operative complication rate were found to be favourable in training when using AR. Operative duration was the only outcome measure that was improved through traditional training.
Aside from AR-based surgical teaching on a local level, the development of the VIPAR (Virtual Interactive Presence and Augmented Reality) system has enabled surgeons from across the globe to collaborate on surgical cases, thus revolutionising telesurgery and telepresence [16]. VIPAR consists of a local and remote station, linked via a robust 3G connection, ensuring reliable realtime communication. This technology provides remote surgeons the ability to highlight anatomical structures and demonstrate operative techniques on another station situated at long-distance. The sub-specialty of paediatric neurosurgery presents a distinct example of this application, with surgeons at the Children's Hospital of Alabama, USA training and assisting surgeons based at Children's Hospital #2 in Vietnam, in a range of neuroendoscopic procedures. 15 cases were trialled using VIPAR for transnational collaboration between these hospitals, with all completed successfully without complication or hardware failure [16]. This evidently demonstrates the success of AR in surgical training on an international scale.

Patient education using AR
The supplementation of traditional patient education techniques with AR can assist with the complex process of a patient's health literacy. AR can aid a patient's understanding of their diagnosis and management plan, thereby enabling informed decisions about their health and enhancing treatment compliance [17]. F.Bork presents the Magic Mirror concept as a means by which AR can be applied for patient education purposes. Here, an AR system is employed to allow the patient to see a virtually enhanced reflection of themselves containing CG overlay. This typically involves playing short explanatory animations to inform the patient on the details of their operation. However, limitations regarding its use in smaller, more complex oncological interventions exist due to the requirement of a higher sensitivity [18].
AR has also shown promise in rehabilitation [19]. The patient's movements can be transferred to a virtual anatomy model allowing the patient to view their mechanism of motion and muscle deformations in real time. This can improve motivation and patient compliance. Ingeson et al. proposed the use of AR in app-based medication management plans, which have potential to increase a patient's adherence to their drug regime [19]. Through the various applications of VR and patient education, AR systems can be exploited to support patients in comprehending abstract concepts and to promote their conceptual understanding o3f their conditions [13].

AR-aided pre-operative planning
Pre-operative planning requires a thorough understanding of the patient's anatomy. Conventional imaging modalities, such CT and MRI, traditionally only provide a limited dimensional view of anatomical structures, presenting challenges and potential misinterpretation of information as surgeons are required to mentally integrate 2D images into the 3D operative field. Devoto et al. describe how AR technology can be used to reconstruct imaging data into a computer-generated 3D format that can then be superimposed onto the surgical field with a viewing device, such as a HMD, which does not interrupt the wearer's vision [20]. The OpenSight AR system was the first FDA approved surgical navigation system for HoloLens, Microsoft for use in pre-operative planning [21]. Through creating a semi-immersive environment for the user, AR holograms in the pre-operative setting allows users to interact, analyse and edit the image, thereby providing a greater appreciation of depth perception, depth focus and field of view of anatomical structures. Greater interactivity with patient-specific imaging and 3D models can also facilitate and optimise preoperative surgical planning discussions amongst the multidisciplinary team.
Below, we will briefly highlight a selection of studies that have utilised AR for pre-operative surgical planning in neurosurgery and orthopaedic surgery.
Neurosurgery is one of the leading fields in surgery implementing AR in preoperative planning and surgical navigation. Neuro-oncology, neurovascular surgery and spinal surgery consist of intricate procedures often requiring a minimally invasive approach with a low threshold for errors. In this way, the specialty relies heavily on image-guided surgery applications, of which AR can incorporate preoperative images and patient-specific models to project a virtual image onto the patient's head. Kersten-Oertel et al. describe the use of AR for facilitating craniotomy planning in tumour resection surgery [22]. In this preliminary study, AR visualisations that were generated by a custom-built system were used in 8 cases prior to tumour resection to visualise the location and margins of the tumour and the surrounding vessels. The surgeons used the projected AR view to trace around the contour of the tumour which allowed appropriate tailoring of the skin incision and determined the size and shape of the bone flap that was to be removed. Overall, surgeons reported AR visualisation was beneficial in craniotomy planning, especially for smaller lesions, but this is largely dependent on good alignment between the virtual and real environment and minimal calibration error. Since then, multiple AR neuronavigation systems have been used for craniotomy planning including tablet, smartphone and HMD [4,23,24].
In the orthopaedic setting, AR has been utilised to preoperatively plan mechanical steps such as screw or implant insertions. Wang et al. used AR navigation for percutaneous sacroiliac screw insertion in six cadaveric specimens [25]. Preoperatively, AR was utilised to form a 3D reconstruction of the pelvis and adjacent blood vessels based on CT imaging. This was used to calculate ideal entry points and trajectories for screw placement which was projected as a cylinder onto a HMD display during the procedure. Post-operative CT scans illustrated all screws were implanted successfully with minimal deviation between planned trajectories and actual screw placement, and no perforation was documented.

Intraoperative AR applications
In every surgical procedure, sound application of anatomical knowledge to achieve the aims of the operation, whilst minimising complications, is the paramount goal. Frequently, a variety of barriers hinder a surgeon from achieving this goal and the intra-operative use of Augmented Reality (AR) aims to alleviate these barriers.
Intra-operative AR uses a display system to provide a real-time 3D virtual model of a patient's anatomy, overlaid on the real surgical field 14 . Coupling a basic intraoperative AR overlay with effective real-time tracking and registration can facilitate the creation of a more intuitive and userfriendly environment, thus offering numerous benefits to surgical outcomes. These include comprehensive knowledge acquisition of the anatomical field, ensuring enhanced procedure precision and reduction in complications of surrounding anatomical structures [26].
Below, we outline how a range of AR systems have been applied to optimise operations in a variety of surgical specialties: Neurosurgery is a highly researched specialty for the implementation of AR intra-operatively, with AR proving beneficial in 16.7% of major neurosurgical procedures [27]. In neuro-oncological surgery, AR has proved efficacious for planning the craniotomy and skin incision prior to tumour resection, with a study demonstrating clinical feasibility in five patients with malignant brain tumours 12 . Furthermore, AR was determined to clearly superimpose surrounding blood vessels, the corticospinal tract and the tumour's localisation points, thereby avoiding iatrogenic injury and enabling complete tumour resection in all cases investigated. Neurovascular surgery has also utilised AR convincingly, with a study revealing that cortical arteries and drainage veins were comprehensively identified in all cases of cerebral artery bypass surgery evaluated.
Though previously restricted to cadaveric, phantom and animal models, recent acceleration in research has resulted in a multitude of orthopaedic procedures employing AR systems to ameliorate outcomes. Spinal surgery has seen the use of AR to meticulously guide needles into position for vertebroplasty, allowing for clear visualisation of bony endpoints without the need for fluoroscopy . This culminated in a 70% reduction in time taken for needle guidance in three patients. Furthermore, the use of AR optical cameras assisting in the placement of 253 lumbosacral screws across 20 patients achieved 94.1% accuracy and no significant misplacements. The application of AR in trauma cases has also gained traction, with a systematic review demonstrating the advantages of using a HMD enabling direct image presentation in front of a surgeon's visual field. In 50 trauma cases, the number of times a surgeon's eyes left the surgical field of view decreased from 207 using conventional methods to 5 whilst using intra-operative AR HMD, thereby indicating an increase in the efficiency of operation [28].
Intra-operative AR has been demonstrated to be of particular use in ENT procedures. This is due to the critically important neurovasculature and complex anatomical structures compacted into a tight space 13 . The operative performance of 15 otolaryngologists was investigated when using both realtime intra-operative AR guidance and conventional navigation systems. Maxillary sinus expansion, sphenoidotomy, intracavernous internal carotid artery dissection and ethmoidectomy were the procedures appraised. Application of the AR system evidently indicated improved outcomes in reference to mental demand, physical demand and frustration felt by the surgeons. This suggests effective application of intra-operative AR across a broad range of ENT procedures.
Laparoscopic surgery presents the additional challenges of a limited surgical field of view, compromised instrument haptic feedback and drastically decreased depth perception [29]. Therefore, intra-operative AR is crucial in playing a role to compensate for these disadvantages by providing clearer vision of anatomical components. The application of intra-operative AR to laparoscopic adrenal tumour resections has shown promise. A study illustrated the accurate superimposition of adrenal veins in all 12 right adrenal tumour cases, with a maximal error of 2mm [30], a significant increase in reliability when compared to a surgeon's estimations of adrenal vein location. Similarly, parathyroid tumour localisation has benefitted from external AR technology due to the abundance of surrounding anatomical landmarks ensuring smooth registration. This has been denoted to optimise operation time due to faster and more seamless access to the tumour.
As the impact of intra-operative AR grows in the surgical world, other surgical specialties are beginning to incorporate this technology into procedures, namely urology, cardiothoracic surgery and general surgery. Though yet to be trialled on human kidneys, AR has exhibited efficacy in renal biopsies on the phantom kidney -a kidney-like apparition [31]. Superimposition of a lesion onto the kidney phantom via a hologram using an AR headset assisted surgeons in guiding the biopsy needle towards the target, resulting in a maximum error of 3mm [31]. Furthermore, use of AR in this regard could decrease the need for intraoperative CT imaging, thereby reducing radiation exposure and limiting operation time. In thoracic oncology, employment of AR has been useful in identifying intrathoracic nodules and lymph nodes. Further progress in this direction could lead to intraoperative AR for the detection and dissection of thoracic lymph nodes, thus potentially eliminating the need for fluorescence imaging [32]. The Vicra and CAS-one instrument guidance systems have proven effective in the treatment of hepatocellular carcinoma, particularly with regards to tumour localisation and delineation of resection margins [32]. These AR systems facilitated the successful resection of tumours with no complications in 2 hepatocellular carcinoma cases.

Limitations of AR
Although AR systems have demonstrated useful applications pre-operatively, intra-operatively and in surgical teaching, there is a plethora of speciality-specific limitations that currently inhibit AR from being widely utilised in clinical practice. Surgeons using AR in orthopaedic operations often complained of an overcrowded field of view which restricted vision and hampered operation progress [28]. Furthermore, AR systems were shown to increase focus on the task at hand, albeit negatively, as surgeons occasionally became less aware of complications manifesting in the surrounding surgical field. This is a concept known as inattentional blindness [6]. In neurosurgery, a lack of accountability for brain shift during procedures led to misalignment of the AR overlay, causing substantial inaccuracy [33]. ENT operations require exceptionally fine control, hence significant registration errors have proved particularly problematic when operating in this extremely compact surgical field of view [34]. In laparoscopy where depth perception and field of view is inherently compromised, the display of redundant information due to the absence of robust real-time overlay has frequently proved detrimental to surgical performance -negatively impacting laparoscopic performance metrics such as bimanual dexterity and tissue handling [35].
Aside from the speciality-specific limitations of AR, accessibility and set-up limitations have also arisen. One study quoted an image registration and set-up time of over 2 hours for certain procedures [36]. This certainly represents a significant hindrance in time-critical operations, or during busy elective operative lists. Furthermore, surgeons' accessibility and ease of use of current AR systems is often compromised due to restricted battery life and inconvenient cables, potentially having a detrimental impact on surgical performance [37]. Issues such as simulation sickness and discomfort during operations further inhibit widespread use of the technology by surgeons [13].
Despite the promising applications of AR across the surgical realm, significant limitations currently exist which restrict its widespread uptake into the surgical world. These limitations will need to be addressed to allow for large-scale feasible implementation of the technology.

Future directions of AR:
The ascent of AR in surgery is transforming several aspects of the field. Hence, strengthening the basics of augmentation by developing effortless overlay registration and flawless tracking, will play a notable role in maximising applications across the field. Future directions will heavily revolve around addressing the current limitations, facilitating widespread implementation of AR systems with utmost enhancement of performance. Specific solutions to address limitations have been proposed. Functional US imaging associated with the AR system can provide updated patient models and tackle the issue of brain shift during neurosurgical procedures [8]. Furthermore, the advent of an automated real-time overlay would prevent the display of obsolete information, thereby enhancing the surgeon's field of view and limiting cognitive burden during complex procedures [32]. Driving forward context-aware AR, wherein only the most relevant anatomical components are displayed, could have considerable benefits in reducing inattentional blindness and an overcrowded field of view [35]. Aside from addressing limitations of current AR systems, a focus on elaborating the ergonomics and ease-of-use could pay dividends in combating factors that currently deter surgeons from using AR systems, such as simulation sickness and discomfort during operations [27].

Conclusions
Although AR remains in its infancy, it can provide surgeons with intraoperative stereoscopic vision by continuously integrating reality with the virtual world. The creation of external superimposed projections of surgical targets and anatomical structures allows for more accurate, precise, and realistic surgical interventions, without the need for repeated exposure to ionising radiation [22,29]. The potential benefits not only relate to patient outcomes, but also surgical training; AR technology can provide novel methods of remote training and live feedback [16,39]. Despite intraoperative applications of AR being the primary focus in this field at present, AR also has applications in preoperative planning, surgical training and patient education.
AR systems have shown great promise in numerous specialties, particularly those which involve little movement of organs during surgical procedures. Organs and systems with low mobility and deformation require low tracking and tracing power from the AR devices, whereas highly mobile organs, such as bowel, require a more complicated track and display process. Taking these factors into consideration, the specialties that have shown great favour towards adopting AR technology include Neurosurgery [16,33,36,40], Ear, Nose and Throat (ENT) surgery [34], Maxillofacial surgery [7,41], and Orthopaedic surgery [25,28,29]. The use of AR in other operative modalities, including laparoscopic surgery and robotic surgery, has been expanding rapidly. The AR devices have not only shown the ability to be incorporated into the robotic surgeon's console but have also exhibited the potential to compensate for the lack of tactile feedback typically experienced during laparoscopic surgeries [42].
AR has proven to be a powerful tool which has shown the potential to revolutionise the field of surgery. With some specialties currently utilising AR more effectively than others, further research will play a pivotal role in levelling the playing field and ensuring representation of AR systems across the realm of surgery. The gap between research and widespread application is primarily due to high costs of reality technology devices. However, this will likely be averted as more affordable devices enter the market. In the near future, AR will likely work in symbiosis with surgeons, serving as a complex computer-human coalition which can improve patient outcomes, patient education and surgical training.