Introduction
The coronavirus disease 2019 pandemic rapidly accelerated the development of telemedicine, among which mixed reality is the most recent iteration, with potential to revolutionise the healthcare system. In a pilot study, mixed-reality technology was used for medical ward rounds during the pandemic, where it was demonstrated that the technology played an important role in helping to limit exposure to nosocomial infection for clinical staff and optimise personal protective equipment use while maintaining a high quality of patient care.Reference Martin, Koizia, Kooner, Cafferkey, Ross and Purkayastha1 The utility of mixed reality has been demonstrated in other healthcare settings, from emergency departments and operating theatres to medical training and education.Reference Mitsuno, Ueda, Hirota and Ogino2–Reference Scherl, Stratemeier, Karle, Rotter, Hesser and Huber5
Microsoft HoloLens 2® is an untethered, self-contained holographic headset. It is one of the most advanced mixed-reality devices currently available on the market.Reference Park, Bokijonov and Choi6 The device enables real-time interactive communication between a user and colleagues remotely. The inbuilt camera and microphone allow for bidirectional audio and visual communication between the wearer and (multiple) remote users. The HoloLens 2 can be worn by the on-call clinician as they interact with patients, and the senior consultant sees a first-person view from the HoloLens 2 wearer on a computer, allowing two-way communication with the clinician from a remote location.
As the hub and spoke model for out-of-hours healthcare is becoming increasingly common in the UK, overnight ENT care is often delivered by junior doctors with little experience in the specialty. These same clinicians are often also responsible for the care of patients in several other surgical specialties. In a national survey of first-on-call doctors for ENT, many reported not feeling comfortable in managing common ENT emergencies.Reference Biswas, Rafferty and Jassar7
The ongoing improvement in immersive technology has led to an increasing drive to find applications that might benefit clinicians in delivering patient careReference Abbas, Kenth and Bruce8 and clinical education.Reference Malik, Abbas, Jayarajah, Bruce and Tolley9 In this randomised pilot study, we used objective scoring to assess the potential benefit of mixed-reality technology in clinical communication between junior and senior colleagues in a simulated clinical scenario. If the use of HoloLens 2 can increase the level of clinical support for on-site junior doctors as well as enhance the junior doctors’ confidence in diagnosing and managing ENT conditions, not only does this empower junior doctors but it also has the potential to improve the quality of care as a result of rapid diagnosis and reduced time to treatment.
This is the first pilot study aimed at assessing the effectiveness of the HoloLens 2 in improving communication between healthcare professionals compared with conventional telemedicine in a simulated out of hours on-call setting.
Materials and methods
HoloLens 2
Developed and sold by the Microsoft corporation, the HoloLens 2 is a multifunctional platform targeted for use in multiple industries, such as engineering, automotives and construction.10 Its key feature is the ability to overlay three-dimensional holograms over the user's vision; this is what is described as mixed reality. Augmented reality is somewhat simple in comparison because the digital information is merely superimposed on the real view and does not have the same physicality that you experience with mixed reality technology.Reference Abbas, Kenth and Bruce8 It is also distinct from full virtual reality as it allows the user to interact with the real world at the same time. The HoloLens 2 also enables interaction with the holograms via motion tracking of the hands or voice control.10 Video calling facilitated via the Microsoft 365 Remote Assist® application, which enables a video call to be made to a user on another device (such as a telephone or laptop) via a Microsoft Teams® account. High-resolution, front-facing cameras on the HoloLens 2 enable the remote user to view the same perspective as the HoloLens 2 user in real time. As you can see from Fig. 1, the wearer of the HoloLens 2 and the consultant call one another, allowing the consultant to communicate with the wearer as well as seeing the perspective of the wearer.
Photographs of experimental set up. (a) Participants using wired telephone (left) and HoloLens 2 (right) as means of communication with senior consultant. Clinical cases were displayed on laminated sheets on the tables. (b) Participant communicating with senior consultant using the HoloLens 2. (c) Consultant in separate room on Microsoft Teams® call with participant using HoloLens 2.
Experimental protocol
A protocol was devised to objectively compare the quality of clinical communication using the HoloLens 2 with a traditional telephone call. This was structured as a randomised, within-patient, controlled study.
On the HoloLens 2, the remote assist application was used to conduct a video call via Microsoft Teams. For the telephone call, a landline phone was available, along with the mobile phone number for the consultant.
A diagram of the protocol is displayed in Fig. 2. A total of 30 junior specialist trainees in ENT (specialty trainee 3 level) were enrolled in the study, and they all participated on both the HoloLens 2 side and the telephone station side. Before the stations began, each trainee was given a five-minute structured brief on the basic functionality of each communication device. The trainees were then presented with simulated clinical scenarios. They were instructed to familiarise themselves with the clinical information, and to call a senior using the device (HoloLens 2 or telephone) assigned to them at that station. A time limit of 15 minutes was set. The call was made to a senior ENT consultant who was situated in another room to simulate a call made to a consultant who may be off-site.
Schematic diagram displaying the experimental protocol. Each trainee was given a short introduction before being randomly assigned to a possible sequence of stations. Each sequence involved both clinical cases and both communication devices. min = minutes. A = clinical case A; B = clinical case B.
Two different clinical scenarios were used. The clinical scenarios were designed in such a way as to require discussion with the senior consultant to formulate a safe management plan. They were presented in the form of a short clinical vignette, blood test results and photographs of clinical signs and radiological imaging. Each clinical scenario was used to test both communication modalities in an equal proportion (Fig. 2). Both the clinical scenarios and the communication modality were selected at random for each group of trainees.
The quality of the clinical communication was measured using an objective scoring matrix with 15 key clinical points for each scenario. This was scored by the senior consultants who were receiving the call. The consultants were also asked if they felt the quality of communication was good enough for them to form a safe management plan without having to see the patient. The consultants were asked to answer the following question: ‘Based on your communication experience overall, do you feel you would need assess this patient face-to-face?’. They could answer: ‘Yes – I have not gained enough information to be happy with a management plan’ or ‘No – I am happy with the information conveyed to form a plan’ (Table 1). In addition, the total time that the communication episode took place was recorded. A survey of subjective user experiences was also taken from the trainees and the consultants (Table 2).
Results of the subjective assessment made by the consultants at the end of every communication episode
Yes and no answers were given in response to the question: ‘Based on your communication experience overall, do you feel you would need assess this patient face-to-face?’
Subjective questions answered by trainees
Table data show the total number of responses out of 30
Microsoft Excel® spreadsheet software (version 16.54) was used to collate the data and perform statistical analysis. Student's t-test was used to generate p-values between the mean objective scores of the different groups. Fisher's exact test was used to generate p-values for the discontinuous variables.
Results
Objective scoring
Overall, using the objective 15-point clinical scoring matrix, communication using the HoloLens 2 scored statistically higher than the telephone call (n = 30) (11.9 of 15 vs 10.2 of 15; p = 0.001) (Fig. 3). The mean time for the communication episode was lower for the HoloLens 2 compared with the telephone (286.0 seconds vs 296.9 seconds; p = 0.655). However, there was no statistical difference between these times.
Box and whisker plot displaying the overall difference in objective scoring between the telephone and HoloLens 2. *Denotes a significant difference defined as p < 0.05.
When comparing the communication modalities directly with the same clinical case, there was a difference between the size of result between the different clinical cases. For case A, the mean score was higher for the HoloLens 2 (n = 15) compared with the telephone (n = 15), but this was not significant (11.9 of 15 vs 11.7 of 15; p = 0.67). However, in case B, the HoloLens 2 (n = 15) produced a statistically higher mean score compared with the telephone (n = 15) (12.0 of 15 vs 8.7 of 15; p = 0.0001) (Fig. 4).
Box and whisker plots displaying the difference in objective scoring between the telephone and HoloLens 2. (a) Displays the data from clinical case A only (n = 15). (b) Displays the data from clinical case B only (n = 15). *Denotes a significant difference defined as p < 0.05.
For both stations, the mean time for the communication episode showed no statistical difference between the HoloLens 2 (n = 15) and the telephone (n = 15). This was consistent between the two clinical cases (case A (279.9 seconds vs 237.6 seconds; p = 0.08) and case B (292.0 seconds vs 356.2 seconds; p = 0.1).
Subjective scoring
For each communication episode, the consultants were asked whether they felt they needed to personally review the patient because the level of information conveyed was not sufficient (question detailed above). Overall (n = 30), for every communication episode using the HoloLens 2, the consultant was satisfied with the information conveyed and did not require a personal review. For the telephone option, in seven episodes the consultant answered that they would require further in-person review (Table 1). This was a statistically significant result (0 of 30 vs 7 of 30; p = 0.01). When this is separated into the individual stations, all the unsatisfactory communication episodes were from clinical case B (0 of 15 vs 7 of 15; p = 0.006). Each trainee was asked to fill out a subjective user survey about their experience with the HoloLens 2 as a communication aid.
Discussion
Key findings
This pilot study has demonstrated the potential benefits of using mixed-reality technology to aid clinical communication. When communicating using the HoloLens 2, communication of difficult clinical cases was significantly more accurate when compared with traditional telephony communication. In addition, this result was reflected in the subjective experience of the consultant. Using the HoloLens 2 allowed the consultant to be significantly more comfortable with the clinical assessment and agreed plan.
The subjective user experience from the junior trainees demonstrated that the majority of trainees were comfortable using the HoloLens 2 and found it simple to operate. In addition, the majority of trainees felt more connected and supported by the advising consultant when using the HoloLens 2 and felt that it would add value when seeking senior support. The trainees were more mixed in their responses as to whether the use of the HoloLens 2 would prevent an unnecessary face-to-face assessment by the consultant.
Two clinical cases were used to enable each trainee to use both the communication devices without repeating the clinical case. However, we saw a relatively large difference in results between the two clinical cases (A and B). Case B demonstrated a larger improvement in objective scoring between using the telephone and the HoloLens 2 compared with case A. This may be a result of several factors. First, it is likely that some clinical presentations would benefit more from a communication technology utilising mixed reality options such as HoloLens 2. For example, clinical case B was a case of periorbital cellulitis where the trainees were presented with a photograph of the pathological signs combined with a photograph of the computed tomography scan, which showed that there was no abscess. This would have been a challenging diagnosis for the trainees and one where the consultant's ability to see the clinical signs and scan results would have a clear benefit. This is in comparison with clinical case A, which was a simulated case of mastoiditis. For this station, there was no significant difference in the quality of communication between the HoloLens 2 and telephone. This was likely because of the nature of the clinical case, with most of the important information, such as observations and blood results, being clearly communicated with both devices. Therefore, we may be able to infer that the value of using a mixed-reality headset for clinical communication will vary from case to case. This is expected given the natural variability in clinical presentations, examination findings and imaging results throughout medicine.Reference Arrow11 Further work would need to be undertaken to determine the most valuable situations for using mixed-reality technology.
• It can be challenging for team members to communicate the complexities of a clinical case using traditional telephone communication
• This pilot study shows the potential benefit of mixed-reality devices, such as the HoloLens 2
• Using a simulated on-call environment, this study showed the HoloLens 2 device improves both objective and subjective communication between team members
• Using an objective clinical scoring matrix, communication using the HoloLens 2 scored statistically higher than the telephone
• Subjectively, the consultants judged more communication episodes to be inadequate to form a plan remotely when using the telephone compared with using the HoloLens 2
• Further studies should be undertaken in a real clinical setting to allow assessment of the practicalities of using such a device
Limitations
There are several important limitations to this study that should be recognised. Because of the nature of the experimental environment, it was impossible to have either real or simulated patients. Instead, printouts were used to describe clinical history, display clinical signs and show the results from blood tests and imaging. The benefit of this was that every trainee experienced the exact same clinical case, ensuring internal consistency between stations.Reference Kuo, Delvecchio and Preminger12,Reference Cheng, Auerbach, Hunt, Chang, Pusic and Nadkarni13 However, what was lost was the ability of the trainee to interact with the patient by performing different examinations. This would have potentially been more apparent when using the HoloLens 2 as the consultant would have been able to see the results of clinical examination live, from the trainee's perspective.
This was a randomised, within-participant, controlled study, with each participant assessed on both forms of communication device. The advantages of this design are the increased number of trials on each device and a reduction in random noise. However, there is a possibility that the participants may produce different results depending on the order they used the devices because of a possible learning effect. We attempted to minimise this by using different clinical scenarios as well as blinding the consultants to the process.
There are some potential practical limitations of the HoloLens 2 device, which, although not experienced in this study, may become evident in a real clinical setting. This study was performed in a controlled environment using three HoloLens 2 devices. This allowed at least one device to be fully charged at all times so it could be swapped between trials; therefore, the battery life was not a factor that was assessed. Furthermore, users require an account to access the device, which would be linked to their Microsoft Teams account allowing the videocall feature of Remote Assist. For the study, several accounts were made for the researchers; then, once logged-in, the device could be used by anyone. This is, of course, not reflective of a real-life situation where each user would require their own account, which may have both practical and financial implications. In this study, WiFi® connectivity with the HoloLens 2 was good. We experienced only one loss of connection over 30 trials. However, this may not be the case in a real-world clinical situation, where different areas of the hospital may have varying connectivity.
Information governance is an important factor to consider when exploring new forms of clinical communication that involve transmitting patient-specific information. It is therefore a requirement that both the hardware and software adhere to local and national information governance and data protection rules. HoloLens 2 is a product of the Microsoft Corporation: the Microsoft 365 Remote Assist application uses Microsoft Teams to allow secure video calling. Within the National Health Service (NHS), Microsoft Teams is used for clinical communication and is covered by NHS digital information governance.14 Individual users will need accounts that use password-protected secure logins.
Conclusion
Complex and subtle clinical cases will often require communication with a senior clinician. However, it can often be hard to fully convey the intricacies of some presentations over the phone. Often in these cases, clinicians will resort to sending photographs or videos over messaging applications. This is a somewhat limited method as it may miss key information. In addition, many of the applications used do not meet the standards required for full information governance within the NHS. Effective communication between levels of seniority is key to ensuring high standards and good morale across the hierarchy.
For many sub-specialties, namely ENT, the resident clinician is often one of the most junior and inexperienced in the team. Indeed, they are often cross covered by clinicians who do not normally work in that specialty. This will impact the quality of clinical judgement and assessment by the resident clinician as well as how confident the senior clinician is in making management plans remotely. This study has demonstrated that mixed-reality technology, such as the HoloLens 2, may be able to help improve the quality of clinical communication and increase the confidence of both junior and senior clinicians when agreeing on a management plan.
Competing interests
None declared.
