Bringing Virtual and Augmented Reality Into Healthcare
I see three connected opportunities for virtual and augmented reality in healthcare: learning anatomy through procedures, practicing patient interaction earlier, and using patient-specific models to prepare for surgery and robotics. Yeung and colleagues analyzed 8,399 publications through September 2020, linking education, simulation, and surgery, but evidence from one application cannot establish effectiveness for another. My focus is supervised education and supplementary surgical guidance in a hospital and medical school, retaining conventional imaging and local clinical responsibility; remote instrument control needs a separate concept of operations.
Learning anatomy through procedures
Recognizing a structure in an illustration differs from understanding its position relative to an instrument and how an intervention affects function. VR could let students examine those relationships from multiple perspectives, while AR connects anatomy to a physical teaching model; both offer repeated practice and feedback before clinical work.
In McGill’s randomized chest-tube pilot, 30 first- and second-year students, mostly first-years, completed an online module; half also used immersive VR. Blinded surgeons assessed mannequin work; the VR group scored a median 46 versus 39 out of 55.
I would add a physiological model so students could examine the chest wall, pleural space, and lung, then trace the modeled effect of an intervention. They could explain their reasoning and apply it to another case; I would investigate whether repeated visualization develops spatial skills that transfer to clinical work. Computable systems engineering could connect geometry and instrument interactions to that response, learning objectives, and assessment, making assumptions and supported responses explicit.
Practicing the conversation
Technical knowledge does not teach a student to recognize distress, explain uncertainty, or discover what a patient is worried about. Virtual encounters could provide repeated opportunities to take a history, explain a next step, and check understanding before reviewing their reasoning and the patient’s response with an instructor. At the University of Pécs, 11 preclinical and clinical students practiced with virtual patients during communication exercises, followed by debriefing. Their reported communication confidence rose from 3.0 to 4.5 on a five-point scale, which does not establish patient-care competence.
I would introduce practice earlier, varying background noise, restricting camera framing, and adding interruptions to telehealth encounters; immersive encounters could also vary proximity, gaze, and posture. AI could support responsive dialogue while educators control patient facts and objectives, but Cools and colleagues found interface demands and limited avatar cues complicated communication for 22 students, whose team and task talk sometimes displaced attention to the patient.
The exercise could test whether students notice missing cues, clarify what they heard, and restore the conversation. After familiarization, I would compare training approaches with equal practice time on unfamiliar physical tasks and unfamiliar standardized-patient encounters. A patient actor and an independent observer could assess listening, explanation, and checking understanding, then give feedback; debriefing and delayed retention tests would follow.
Keeping the patient in view
Patient-specific models could show surgeons how a lesion relates to vessels, airways, and tissue before choosing an approach. In Sadeghi’s feasibility study, CT-derived models supplemented imaging for ten planned lung segmentectomies; strategy changed after VR review in four. A 150-patient, three-center trial found improved pedicle-screw placement accuracy with a particular AR system in spine surgery. I keep returning to whether the representation matches the patient; these illustrations show deliberate misalignment.
Surgical field 
Overlay Displaced overlay In Chang and colleagues’ 20-patient study, one localization deviated 26 mm after tube displacement and lung collapse, prompting conversion to conventional cone-beam CT guidance; that is a documented failure and recovery, not a transferable error tolerance. I would link source images, identify segmentation revisions, and flag uncertain structures for clinical review. Registration and tracking affect overlay reliability; so do tissue movement and the age of the source information. An operative display might reduce attention switching.
Displays should flag missing or invalid guidance; clinicians need source-image review and independent checks of current anatomy, informed by direct observation. They should remove unreliable overlays and rehearse a return to conventional guidance. Simulation could test workload and missed cues.
Connecting preparation with performance
Source images and segmented anatomy could link plans to observed actions and reasons for changing course. The links could support anatomical study, team rehearsal, and research on robotic instrument movement. Computable systems engineering could associate each structure’s identity with its coordinate frame and measurement units. Timestamps and model revisions would help investigate discrepancies when a vessel appears in an image, a teaching model, and a navigation display.
The JIGSAWS dataset, developed by Johns Hopkins and Intuitive Surgical, combines robot kinematics, stereo video, and gesture and skill annotations from eight surgeons performing bench-top suturing, knot tying, and needle passing. I would extend this to patient-specific procedures, retaining decision context and both successful and unsuccessful attempts.
With permission and provenance, records could support teaching, simulation, and robotics research. Long and colleagues’ AR mentoring prototype streamed a three-dimensional scene to a remote mentor, who returned spatial guidance to a robotic console; it was evaluated with recorded surgical video and ex vivo tasks. Remote patient procedures would add requirements for latency and information loss, clear local responsibility, and recovery from an unreliable connection; rehearsal and mentoring could help investigate those dependencies.
Making the work sustainable
Simulation can reduce immediate patient exposure yet teach incorrect expectations or displace effective practice: a chest-tube exercise might reward the right sequence while misrepresenting physiology, so educators need separate objectives for procedural reasoning and communication. I would use Engineering Better Care to connect intended benefits to people and systems while accounting for design and risk; SEIPS 3.0 helps examine how people perform tasks with tools in a given setting. Small Plan–Do–Study–Act cycles could refine the service before expansion, monitoring staff time, cancellations, and cost; symptoms and exclusion should be weighed alongside benefit.
A multidisciplinary healthcare failure mode and effect analysis could follow an image through segmentation and registration to the display and clinical decision. The team could assign controls and owners, testing for hazards elsewhere; FDA human-factors methods would guide critical-task testing with representative users. I would test scenarios involving an incorrect revision, lost tracking, or anatomical movement, then measure task error, recognition of invalid guidance, and recovery. Clinical owners would approve procedure-specific limits.
A headset purchase does not assign responsibility for software changes or support. Recording reuse also needs an owner; an update or staff absence could interrupt sessions or prompt recording transfers to unapproved accounts. Clinical and education teams need defined responsibilities. IT and privacy roles need definition, and service support needs funding. The plan needs tested updates and rollback, minimized recordings with encrypted transfers, and role-based access with retention rules. Cleaning, setup, and an accessible nonimmersive route also matter. Each use needs evidence appropriate to its purpose.