ARMAGNI: Augmented Reality Enhanced Surgical Magnifying Glasses Situational Awareness during surgery with AR in the Loupe

During cardiac surgery, in addition to their manual work, surgeons need to perceive a large amount of procedural intraoperative data, including information which is not directly visible to them. Therefore, we developed an augmented reality demonstrator that displays alphanumerical data into the loupe of surgical magnifying glasses. Eight cardiac surgeons tested the demonstrator in a skill task that simulates the critical part of a typical surgical procedure while being confronted to vital intraoperative parameters getting critical. The results showed a decrease of missed critical phases and improved response times when using the demonstrator instead of a customary monitor for tracking intraoperative parameters


INTRODUCTION
Situational awareness, "the perception and understanding of the surrounding environment" [13], is important in humans' performance fulfilling a complex task [6].This also applies to cardiac surgeries where safety and outcome are dependent on information flow, concerning not only preoperative data but also intraoperative procedural data such as vital parameters.Although operation rooms are equipped with multiple monitors displaying standard parameters like vital signs, other information such as heartlung machine procedural data and respiratory parameters are not readily visible for the surgeon.When needed, it requires intense communication efforts which can lead to distraction and interruption of the procedure [4].Augmented reality (AR) can be a solution to bridge the gap between information needs, communication, and display possibilities.Head-mounted displays (HMD), including smart glasses such as Google Glass, have been playing an increasing role in the health care industry [12].In a systematic review from 2019, Rahman et al. [12] determined 120 HMD applications in surgery, of which most were used for image guidance and AR.For instance, Liebert et al. [7] compared traditional vital signs monitor to an HMD for monitoring patient's vitals, showing potential for increased situational awareness and improved patient safety.The overall feedback from users was positive.In a qualitative descriptive study, Enlöf et al. [5] also showed that health care professionals have a generally positive view of using smart glasses in the medical field.In video assisted surgery AR glasses have potential to reduce bad body posture during procedure [8].Arpaia et al. [1] presented a system for retrieving and displaying patient's vitals and evaluated its effectiveness, transmission error rate, refresh rate and latency with confirming technical feasibility of such a system.The target group consisted of assistant surgeons and anesthetists.The related work mentioned in the previous lines indicate benefits from using AR devices in surgery and high user acceptance, but these applications did not consider cases where users require surgery magnifying glasses.For cardiac surgeons, wearing magnifying glasses and HMDs simultaneously can be cumbersome due to interference.As one solution, smart glasses can be mounted to surgical loupes.Yoon et al. [14] successfully used such a setup for image guidance during shunt placement.The AR images were projected above the loupe and required quick eye movement to see.Qian et al. [10], on the other hand, combined a Magic Leap One with binocular magnifying loupes.They evaluated a calibration method to align virtual content to the real world in the magnified or minified fieldof-view of the user.But since we are projecting simple alphanumerical procedural data that does not have to be aligned to real objects, there is no need for such a calibration method.Hence, we can use a simpler setup with lightweight hardware.Our approach for displaying procedural data in the field of view of surgeons is different to the ones mentioned above.Instead of extending existing HMDs which have limitations due to weight, size, or short battery life [12], we develop typical surgical magnifying glasses with an integrated AR display into the loupe.For the proof of concept, we built an AR demonstrator that can be adjusted to the user.The prototype is shown in Figure 1 and the view of the loupe with AR visualizations in Figure 2. In a feasibility study, experienced cardiac surgeons tested the usefulness of AR in the loupe with our demonstrator.We propose that AR magnifying glasses can increase situation awareness of surgeons.Beyond that, they will have potential to reduce surgeons' workload during cardiac surgeries and encounter high user acceptance, as they mimic the typical surgical magnifying glasses used during cardiac surgery.

METHODS
For this study, we asked cardiac surgeons to test our prototype referred as AR demonstrator, which we will describe in detail below along with the corresponding software, we will further outline the experimental setup.

AR Demonstrator
The prototype consists of two customary Galilean loupes with 2.5x magnification and 300 mm working distance.They were attached to a custom-built metallic mechanism, which allows different users to adjust pupil distance, intereye asymmetry or height of the loupes.The metallic mechanism is mounted on a plastic glasses frame made with a laser cutter.The 2.5 cm x 2 cm x 1.5 cm sized AR module (Figure 3) was built with a combination of a micro-OLED display, a lens, and a beam splitter.For displaying AR visualizations, we used a Sony ECX336AF-6 OLED micro display.The 0.23-inch diagonal sized RGB display has a 640 x 400 pixels resolution, 800 cd/m² maximal luminance and 10,000:1 contrast ratio.We installed the micro display on the side of a 3D printed component.There, the displayed images are projected through a lens magnifying them to fully cover the view through the loupes.A beam splitter centered between the display, the loupe, and the eye, reflects the images into the eye of the user.The AR module can be attached to one of the Galilean loupes.The micro display is connected to a Raspberry Pi Zero installed in the glasses frame.To be able to power the micro display with the required 1.8 volts and send images to it, the Raspberry Pi Zero was extended by a custom circuit board.A 3.6 volts rechargeable battery is powering the Raspberry Pi Zero through a 200 cm long cable.This battery is bundled in a 3D printed case with integrated electronics and a button, which allows us to turn the Raspberry Pi on and off.

Base station
To keep the AR magnifying glasses lightweight and conformable, we follow a client-server approach.One endpoint is the mentioned Raspberry Pi Zero built in the glasses frame.As the other endpoint, we are using a base station on Raspberry Pi 4B basis and the official Raspberry Pi 7-inch touchscreen (see Figure 4).The base station serves as a mobile control center for fetching patient data from simulated medical devices as well as generating visualizations and streaming them to the AR display.Furthermore, the base station is running a graphical user interface frontend application for configuring the AR visualizations using the touchscreen.
AR visualizations are generated in the frontend application and sent as PNG images to a REST server on the Raspberry Pi Zero.Using a Raspberry Pi 4B with the 64-Bit version of the Raspberry Pi OS as our base station, we can achieve approximately 5 frames per second with a delay of 500 milliseconds.Our frontend application developed with Vue.js enables users to customize the AR visualizations.They can pick different parameters from connected medical device simulators and choose the desired position in the field of view.We decided the visualized medical parameters to be automatically arranged in a ring formation, so they do not overlay the working surface of the user.Though, the user is free to choose the size of the ring and by that the size of parameters as well as the number of displayed parameters.Another feature of the application allows users to finetune the whole AR visualization by moving, resizing, or rotating it with touch gestures while wearing the glasses and seeing the changes near real-time.Each user can have multiple settings profiles, which are saved in a PostgreSQL database.Communication with the database and medical devices is performed by an Apollo GraphQL server, which exposes a GraphQL API for the frontend application.Backend was built in accordance with IEEE 11073 Service-oriented Device Connectivity (SDC) for being capable of communicating with real medical devices that also support SDC.Using SDCLib, an open-source SDC implementation by SurgiTAIX, we developed a SDC consumer micro service, which serves as a bridge between medical devices and our GraphQL server.For the experiment, we developed a medical device simulator, that generates random but reasonable values for heart rate (labeled as HF in Figure 2), mean arterial pressure (MAD), oxygen saturation (SPO2) and central venous pressure (ZVD).The simulated values can continuously increase and decrease every 3 seconds, and every 60 -90 seconds one of the parameters becomes critical for 15 seconds.Parameters in a critical state are displayed with a small warning sign and blink with a frequency approximately 600 ms.By means of this simulation, we evaluated the effect of the AR module on the user during a skill exercise, the setup of which we will describe next.

Experimental setup
The experiment took place in the clinic for cardiac surgery at the University Hospital Düsseldorf.We invited 8 cardiac surgeons (2 female, 6 males, between 25 and 54 years old) to perform a skill exercise while wearing our AR magnifying glasses and reacting to critical parameters.Directly afterwards, we asked them to take a survey, which included the System Usability Scale (SUS), the NASA Task Load Index (NASA-TLX) and further questions regarding the usefulness and usability of the AR.In the beginning of the experiment, the investigator taught participants how they can adjust the AR magnifying glasses via the metallic mechanism and finetuned the AR display using the frontend application.During the experiment, frontend and backend applications were running on a laptop computer and used by investigator rather than the participant.We followed a within-subject design with randomized order of conditions.In both cases, participants performed an anastomosis on the Arroyo's Anastomosis Simulator [11] while wearing our prototype.In the test condition, the AR module was turned on and the participants had to keep track of the displayed parameters in the loupe.In the control condition on the other hand, parameters were displayed on a customary 27-inch display only.We positioned the display on the right side of and 2 meters away from the participants, so they had to turn their head to the right by approximately 20-degree angle to see the values.During performing the skill exercise, participants had to react to critical values as soon as possible by telling the investigator which parameter is critical and whether it is too high or too low.We recorded their response time using a timer implemented in the simulation software.Participants had a time limit of 15 seconds to notice and react to a critical phase and the investigator registered the answer as "right" or "wrong" by pushing the corresponding button on the keyboard.If the pre-set 15-second timeout passed before the participant reacted, the software automatically registered the answer as "missed" with a response time of 15 seconds.After each trial, our simulation software generated a log file containing participant's response times to critical phases and whether the response was right, wrong, or missed.

RESULTS
We performed t-tests to compare our metrics between test condition (using AR) and control condition (using monitor).All metrics were tested for homogeneity of variance and normality before comparing them with a twotailed paired t-test for dependent means.If requirements for homogeneity or normality were not met, Wilcoxon Singed-Rank test was used.

Response performance
We used a Wilcoxon Signed-Rank test to compare response performances between test and control conditions.Response performance is the count of correct responses divided by count of critical phases for each participant.The calculated data for each participant can be seen in Figure 5.It should be noted that no participant gave wrong responses.

Response time
Data for response time is visualized in Figure 6.Since missed critical phases mean that there was no reaction for 15 seconds, data is strictly speaking not normally distributed.One participant even missed all critical phases in control condition; hence median was 15 seconds in this case.The Levene's test for response times was not significant (F(1, 7) = 1.313, p = .340)so variance homogeneity was assumed.Despite data being limited between 0 and 15 seconds, Shapiro-Wilk test for the differences between the pairs did not show a significant departure from the normality (W(8) = .915,p = .433).Participants responded faster to critical values in the AR condition (M = 6.7,SD = 1.9) compared to monitor condition (M = 10.1,SD = 3), (t(7) = 2.9, p = .025).

Total time for trials
Participants required different amount of time to complete the anastomosis on the Arroyo's simulator.We used a t-test to compare the required time for test and control conditions.The Levene's test for required times was not significant (F(1, 7) = .079,p = .783)so variance homogeneity was assumed.Shapiro-Wilk test for the differences between the pairs did not show a significant departure from the normality (W(8) = .984,p = .999).The t-test did not show any significant difference in required time for trials between AR (M = 12.4,SD = 5.2) and monitor conditions (M = 11.4,SD = 5.5), (t(7) = 1.4,p = .203).

Figure 7.
Mean scores for mental demand (MD), physical demand (PD), temporal demand, (TD), performance (P), effort (E) and frustration (F).ns means that a paired t-test result was not significant.

SUS Score
Mean SUS score was 66.875 (SD 12.02).Reponses for individual statements are visualized in Figure 8.

Further questions
Responses to further questions about AR in the demonstrator and about AR magnifying glasses in general are summarized in Figure 9. Participants were asked to rate whether they agree or disagree with the statements on a scale of 0 -100.

Objective evaluation
The experiment's results showed an improvement in participant's reactions to critical phases when using the AR display compared to a common diagnostic monitor.In the test condition, not only reaction times were 51 % faster, but also the performance for detecting critical parameters was increased by 60 %.On the other hand, given our sample size of 8 participants being small, statistical conclusions should be treated with care.Nevertheless, our initial proposition that augmented reality can improve surgeons' situational awareness could be confirmed, at least, in our experimental setup.Circumstances during real operation are different.Firstly, OP light is much brighter than in our setup, which negatively affects the visibility of the AR visualizations.Technical suitability should be evaluated in further studies.Secondly, the question arises whether an AR display in magnifying glasses is necessary in the presence of auditory alerts and assistant doctors or anesthetists.Our experimental setup demonstrates that an AR display can help surgeons keep track of selected parameters omitting some communication efforts that can possibly interrupt the operation procedure.

Subjective assessments
There was a significant difference in global NASA-TLX between test and control condition, indicating that AR magnifying glasses can also help to improve workload in surgeons.Especially, frustration was reduced by 45 % and physical demand by 36 % (borderline not significant though) when using the AR display.Higher physical demand in monitor condition could result from the fact that participants had to turn their head to see the parameters.Also, in the control condition, participants missed more critical phases (response performance dropped by 38 %), which explains higher frustration.It might be interesting to further investigate these subscales under real circumstances, where cardiac surgeons rarely look to monitors but communicate with attendants about them.

Usability and opinions
Participant's feedback to the AR demonstrator presented were mixed and an average SUS score of 66.875 can be considered as "marginally acceptable" according to determined ranges by Bangor et al [2].On the one hand, participants liked the AR functionality and showed interest in using such a device in the future.On the other hand, everyone had difficulties with the prototype itself, mostly independent of the AR module.Adjustment of the magnifying glasses could take over 10 minutes but even then, the alignment of the loupes was not ideal.Sometimes participants had to re-adjust the loupes during the skill task.Another common problem was that, due to their weight, the glasses slightly slid down during skill task so that the top two parameters disappeared.These problems primarily occurred because of the mechanics for adapting the glasses to different users.Normally, surgical loupes are custom-made for each user, to fit pupil distance, height, and preferred working distances.However, we wanted one single device that could be tried out by multiple people, which would not be possible with a tailormade device.
In the future, we are building another prototype with a different micro display that uses Bluetooth Low Energy for data transmission and a lighter battery.This omits a Raspberry Pi Zero in the glass frame reducing the weight The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.
of the device, which should resolve most issues of the current AR demonstrator as discussed above.

CONCLUSION
This first feasibility study showed that ARMAGNI has the potential to increase surgeons' situation awareness during operations requiring magnification glasses while decreasing the workload for surgeons.Participants liked the concept of the product but had difficulties with the current prototype.In an upcoming improved prototype, participants feedback will be taken into account to enhance usability.In addition, a different micro display is intended to improve overall ergonomics of the device.

Figure 1 .
Figure 1.AR demonstrator consisting of an AR module and Galilean loupes mounted on a custom-made glasses frame mimicking surgical magnifying glasses.

Figure 2 .
Figure 2. Image of the field of view of the AR loupe.

Figure 4 .
Figure 4. Base station running the software.

Figure 5 .
Figure 5. Rate for correct responses of each participant.

Figure 6 .
Figure 6.Median response times in seconds.

Figure 9 .
Figure 9. Please tell us whether you agree or disagree with the following statements.Q1.I preferred using the AR instead of looking at the monitor to detect critical parameters.Q2.The medical parameters in the AR were clearly visible.Q3.I found that the AR visualization obscured the view over the working surface.Q4.I can imagine myself using surgical magnifying glasses with AR in daily OR work.Q5.I think magnifying glasses with AR can improve the outcome of surgeries.