Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation

Taylor Frantz; Bart Jansen; Johnny Duerinck; Jef Vandemeulebroucke

Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation

View Fulltext

Author(s): Taylor Frantz^{1, 2} ; Bart Jansen^{1, 2} ; Johnny Duerinck³ ; Jef Vandemeulebroucke^{1, 2}
- Affiliations: 1: Vrije Universiteit Brussel (VUB), Department of Electronics and Informatics (ETRO) , Pleinlaan 2, B-1050 Brussels , Belgium ;
  2: imec , Kapeldreef 75, B-3001 Leuven , Belgium ;
  3: Vrije Universiteit Brussel (VUB), Department of Neurosurgery , Laarbeeklaan 101, 1090 Brussels , Belgium
Source: Volume 5, Issue 5, October 2018, p. 221 – 225
DOI: 10.1049/htl.2018.5079 , Online ISSN 2053-3713

This is an open access article published by the IET under the Creative Commons Attribution-NonCommercial-NoDerivs License (http://creativecommons.org/licenses/by-nc-nd/3.0/)

Received 16/08/2018, Accepted 03/09/2018, Published 04/10/2018

Major hurdles for Microsoft's HoloLens as a tool in medicine have been accessing tracking data, as well as a relatively high-localisation error of the displayed information; cumulatively resulting in its limited use and minimal quantification. The following work investigates the augmentation of HoloLens with the proprietary image processing SDK Vuforia, allowing integration of data from its front-facing RGB camera to provide more spatially stable holograms for neuronavigational use. Continuous camera tracking was able to maintain hologram registration with a mean perceived drift of 1.41 mm, as well as a mean sub 2-mm surface point localisation accuracy of 53%, all while allowing the researcher to walk about a test area. This represents a 68% improvement for the later and a 34% improvement for the former compared with a typical HoloLens deployment used as a control. Both represent a significant improvement on hologram stability given the current state-of-the-art, and to the best of the authors knowledge are the first example of quantified measurements when augmenting hologram stability using data from the RGB sensor.

References

1. 1)
  - 7. Chauvet, P., Collins, T., Debize, C., et al: ‘Augmented reality in a tumor resection model’, Surg. Endosc. Other Interv. Tech., 2018, 32, (3), pp. 1192–1201. (doi: 10.1007/s00464-017-5791-7).
2. 2)
  - 11. Auvinet, E., Galna, B., Aframian, A., et al: ‘O100: validation of the precision of the Microsoft HoloLens augmented reality headset head and hand motion measurement’, Gait Posture, 2017, 57, pp. 175–176. (doi: 10.1016/j.gaitpost.2017.06.353).
3. 3)
  - 6. Tepper, O.M., Rudy, H.L., Lefkowitz, A., et al: ‘Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room’, Plast. Reconstr. Surg., 2017, 140, (5), pp. 1066–1070. (doi: 10.1097/PRS.0000000000003802).
4. 4)
  - 5. Mascitelli, J.R., Schlachter, L., Chartrain, A.G., et al: ‘Navigation-linked heads-up display in intracranial surgery: early experience’, Oper. Neurosurg., 2017, 15, (2), pp. 184–193. (doi: 10.1093/ons/opx205).
5. 5)
  - 15. Garon, M., Boulet, P.-O., Doiron, J.-P., et al: ‘Real-time high resolution 3D data on the HoloLens’. doi: 10.1109/ISMAR-Adjunct.2016.0073.
6. 6)
  - 17. Graham, P.: ‘Scopis announces holographic navigation platform using HoloLens for open and minimally-invasive spine surgery’, 2017. Available at https://www.vrfocus.com/2017/05/scopis-announces-holographic-navigationplatform-using-hololens-for-open-and-minimally-invasive-spine-surgery/.
7. 7)
  - 20. Eckert, M., Blex, M., Friedrich, C.M., et al: ‘Object detection featuring 3D audio localization for Microsoft HoloLens’. Proc. 11th Int. Joint Conf. on Biomedical Engineering Systems and Technologies, 2018, 5, pp. 555–561. doi: 10.5220/0006655605550561.
8. 8)
  - 9. Rae, E., Lasso, A., Holden, M.S., et al: ‘Neurosurgical burr hole placement using the Microsoft HoloLens’, March 2018. doi: 10.1117/12.2293680.
9. 9)
  - 8. Pratt, P., Ives, M., Lawton, G., et al: ‘Through the HoloLens looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels’, Eur. Radiol. Exp., 2018, 2, pp. 1–7 (doi: 10.1186/s41747-017-0033-2).
10. 10)
  - 12. Vassallo, R., Rankin, A., Chen, E.C.S., et al: ‘Hologram stability evaluation for Microsoft HoloLens’, Proc. SPIE, 2017, 10136, p. 1013614. (doi: 10.1117/12.2255831).
11. 11)
  - 13. Liu, Y., Dong, H., Zhang, L., et al: ‘Technical evaluation of HoloLens for multimedia: a first look’. IEEE Multimedia, 2018, pp. 1–7.
12. 12)
  - 14. Guha, D., Alotaibi, N.M., Nguyen, N., et al: ‘Augmentedreality in neurosurgery: a review of current concepts and emerging applications’, Can. J. Neurol. Sci., 2017, 44, (33), pp. 235–245 (doi: 10.1017/cjn.2016.443).
13. 13)
  - 2. Meola, A., Cutolo, F., Carbone, M., et al: ‘Augmented reality in neurosurgery: a systematic review’, Neurosurg. Rev., 2017, 40, (4), pp. 537–548. (doi: 10.1007/s10143-016-0732-9).
14. 14)
  - 8. Inoue, D., et al: ‘Preliminary study on the clinical application of augmented reality neuronavigation’, J Neurol Surg A Cent Eur Neurosurg, 2013, 74, (2), pp. 71–76 (doi: 10.1055/s-0032-1333415).
15. 15)
  - 16. Kuzhagaliyev, T., Clancy, N.T., Janatka, M., et al: ‘Augmented reality needle ablation guidance tool for irreversible electroporation in the pancreas’, 2018. doi: 10.1117/12.2293671. Available at http://arxiv.org/abs/1802.03274.
16. 16)
  - 8. Orringer, D.A., Golby, A., Jolesz, F.: ‘Neuronavigation in the surgical management of brain tumors: current and future trends’, Expert. Rev. Med. Devices, 2012, 9, (5), pp. 491–500. (doi: 10.1586/erd.12.42).
17. 17)
  - 14. Xie, T., Islam, M.M., Lumsden, A.B., et al: ‘Holographic iRay: exploring augmentation for medical applications’. Adjunct Proc. 2017 IEEE Int. Symp. on Mixed and Augmented Reality, ISMAR-Adjunct 2017, Nantes, France, 2017, pp. 220–222. doi: 10.1109/ISMARAdjunct.2017.73.
18. 18)
  - 19. McDuff, D., Hurter, C., Gonzalez-Franco, M.: ‘Pulse and vital sign measurement in mixed reality using a HoloLens’. Proc. 23rd ACM Symp. on Virtual Reality Software and Technology – VRST ‘17, Gothenburg, Sweden, 2017, pp. 1–9. Available at http://dl.acm.org/citation.cfm?doid=3139131.3139134.
19. 19)
  - 4. Cutolo, F., Meola, A., Carbone, M., et al: ‘A new head-mounted display-based augmented reality system in neurosurgical oncology: a study on phantom’, Comput. Assist.Surg., 2017, 40, (4), pp. 537–548. doi: 10.1080/24699322.2017.1358400.
20. 20)
  - 18. Manke, K.: ‘Brain surgery may get a bit easier, with augmented reality’, 2016. Available at https://today.duke.edu/2016/10/brain-surgery-may-get-bit-easieraugmented-reality.

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation

References

Related content