Augmented reality and its usefulness in Computer Assisted Interventions

by Simon Drouin, Matthias Pfeifle & Marta Kersten-Oertel

Over 20 years ago, Gleason et al (1994) proposed an AR system that combined 3D segmented virtual objects from preoperative patient scans (e.g., tumors and ventricles) with live video images of the patient to guide neurosurgeons. Since then many groups have explored the use of AR in computer assisted interventions — resulting in hundreds of publications in the area. Although there’s been much progress, AR hasn’t made it into daily clinical use. In this segment, we asked two senior doctoral students and software engineers who have been working in the area for over the last 7+ years, whether AR is just a neat gimmick, or whether it can offer real solutions that aid clinicians.

Point: Augmented reality is a neat technology but current AR systems suffer from major drawbacks and obstacles remain for surgeons to accept it.

MP: Current AR systems suffer from major drawbacks including unsuitable display devices, lack of validated precision and lack of integration into the surgical workflow.

In terms of displays, for non-neurosurgical procedures (where typically the microscope is used for AR), suitable devices are still sparse and current solutions tend to interfere with the mobility of the surgeon, especially in regards to sterility requirements.

Precision and reliability are mandatory requirements for devices. AR image-guided surgery systems rely heavily on the accuracy of the image-guided surgery systems, which in turn rely on accurate tracking, and registration each of these have known inaccuracies. Furthermore, for AR accurate camera calibration is also necessary to ensure an accurate and meaningful overlay. Misalignments and discrepancies between real and virtual worlds could cause significant errors rendering a system unusable. Evaluation and validation of the precision required for AR visualization to be trustworthy for particular surgical tasks is required.

Current AR systems don’t integrate with the surgical workflow and don’t work ‘out of the box’ — rather they require significant interaction. This includes tasks associated with the preparation of data sources, as well as, calibration and registration. In addition, depending on the complexity of a system, additional technical personnel may be needed in the operating room further interrupting the surgical workflow and increasing costs. As time is always an issue in the clinical domain, the more steps that have to be performed manually, the less likely a novel technology is to be adapted – unless of course the novelty provides a significant benefit to either the clinician or the patient.

And this is the key point – in the field of image-guided surgery – a major motivation or tenable use case where AR can provide an essential benefit over every other technique that is available is missing. Without that, medical device manufacturers will lack the incentive for product development and clinical validation and certification. Until this first use case, where AR can really prove itself to be beneficial beyond other current technologies and techniques, is shown, AR will be limited to research and training setups.

Counterpoint: Augmented reality offers new solutions that can aid clinicians in the OR

SD: In the 20 years that have past since Gleason et al’s demonstration, most technical obstacles put forward to discard AR for clinical use have been resolved.

The availability of proper display devices for AR has been a false problem from the start. The simple combination of a camera and computer screen positioned close to the surgical field and showing an AR view is an improvement over traditional navigation systems because it shows reality and preoperative plans in the same space. This is not only more intuitive; it is also safer as it allows the surgeon to navigate while looking at the operating field.

Most neurosurgical procedures are performed entirely through a microscope, which has already been shown to be a valid display device for AR [1]. Moreover, with the renewed interest of the entertainment industry for head-mounted displays, the market should produce a series of even lighter wearable AR devices in the next few years, which will open a new range of possibilities.

Another technical obstacle often brought up in the case against AR is the lack of validated accuracy. In the past couple of years, most AR systems literature yielded sub-millimetre accuracy. In contrast, traditional navigation systems already used routinely in the OR report accuracies between 2.7 and 6.2 mm for patient registration [2], suggesting AR itself is not the weak link in the chain. Furthermore, Cabrillo et al recently commented that AR could be useful even without a perfectly accurate overlay of real and virtual images [3]. The challenge for the future may be to learn how to properly represent uncertainty.

Finally, it is often argued that AR can be complicated to set up in the OR. For most systems presented in the literature, the setup process is essentially the same as with traditional navigation systems. The only additional task consists in calibrating the optical device (camera, microscope, endoscope, etc.) used to capture images of the operating field, a task that is increasingly automated using various computer vision techniques such as camera self-calibration.

With the growing availability of technically verified AR solutions for various procedures, AR systems can now be brought into the OR routinely, at least for research purposes. The field of surgical AR research must now go through a paradigm shift to focus on improving the quality of visual perception and interaction as well as identify the particular surgical tasks where it can be useful (e.g. [4]).

The bottom line: Significant progress has been made in the research community on developing new AR technologies and techniques however, our experts agree that there’s a big need for research on the specific use cases of AR in computer-assisted interventions. Developed methods must be tested to show their impact on clinicians, surgeons and ideally on patients. This would ensure AR moving from research to the clinical domain.

References

1. Edwards PJ, King a P, Maurer CR, de Cunha D a, Hawkes DJ, Hill DL, et al. Design and evaluation of a system for microscope-assisted guided interventions (MAGI). IEEE Trans Med Imaging 2000;19:1082–93.
2. Stieglitz LH, Fichtner J, Andres R, Schucht P, Krähenbühl A-K, Raabe A, et al. The silent loss of neuronavigation accuracy: a systematic retrospective analysis of factors influencing the mismatch of frameless stereotactic systems in cranial neurosurgery. Neurosurgery 2013;72:796–807.
3. Kersten-Oertel, Marta, et al. “Augmented Reality for Specific Neurovascular Surgical Tasks. Augmented Environments for Computer-Assisted Interventions. Springer International Publishing, 2015; 92-103.
4. Cabrilo, Ivan, Philippe Bijlenga, and Karl Schaller. “Augmented reality in the surgery of cerebral aneurysms: a technical report.” Operative Neurosurgery 2014;10: 252-261.

Matthias Pfeifle is a doctorate student at the University of Tübingen working in the field of medical visualization and video processing.

Simon Drouin is a doctorate student at McGill and the Montreal Neuro working on intuitive visualization and interaction methods for augmented reality neuronavigation systems.

Marta Kersten-Oertel is a postdoctoral fellow at the McConnell Brain Imaging Centre, (Montreal Neuro) working on medical image visualization in the context of image-guided surgery.