Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery

Sing Chun Lee; Bernhard Fuerst; Keisuke Tateno; Alex Johnson; Javad Fotouhi; Greg Osgood; Federico Tombari; Nassir Navab

Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery

View Fulltext

Author(s): Sing Chun Lee¹ ; Bernhard Fuerst² ; Keisuke Tateno^{3, 4} ; Alex Johnson⁵ ; Javad Fotouhi¹ ; Greg Osgood⁵ ; Federico Tombari³ ; Nassir Navab^{1, 3}
- Affiliations: 1: Computer Aided Medical Procedures, Laboratory for Computational Sensing & Robotics , Johns Hopkins University , Baltimore, MD , USA ;
  2: Verb Surgical , Mountain View, CA , USA ;
  3: Fakultät für Informatik , Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality, Technische Universität München , Garching, Bayern , Germany ;
  4: Canon Inc. , Shimomaruko, Tokyo , Japan ;
  5: Orthopaedic Trauma, Department of Orthopaedic Surgery , Johns Hopkins Hospital , Baltimore, MD , USA
Source: Volume 4, Issue 5, October 2017, p. 168 – 173
DOI: 10.1049/htl.2017.0066 , Online ISSN 2053-3713

This is an open access article published by the IET under the Creative Commons Attribution -NonCommercial License (http://creativecommons.org/licenses/by-nc/3.0/)

Received 26/07/2017, Accepted 02/08/2017, Published 14/09/2017

Orthopaedic surgeons are still following the decades old workflow of using dozens of two-dimensional fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This Letter presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. A red–green–blue–depth camera is rigidly attached to a mobile C-arm and is calibrated to the cone-beam computed tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model-based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localising the entry point and orienting the surgical tool during screw placement. The authors validate the augmentation by measuring target registration error and also evaluate the tracking accuracy in the presence of partial occlusion.

References

1. 1)
  - 15. Fotouhi, J., Fuerst, B., Johnson, A.,, et al: ‘Pose-aware C-arm for automatic re-initialization of interventional 2d/3d image registration’, Int. J. Comput. Assist. Radiol. Surg., 2017, 12, (7), pp. 1221–1230 (doi: 10.1007/s11548-017-1611-8).
2. 2)
  - 8. Navab, N., Blum, T., Wang, L.,, et al: ‘First deployments of augmented reality in operating rooms’, Computer, 2012, 45, pp. 48–55 (doi: 10.1109/MC.2012.75).
3. 3)
  - 24. Fotouhi, J., Fuerst, B., Lee, S.,, et al: ‘Interventional 3d augmented reality for orthopedic and trauma surgery’. 16th Annual Meeting of the International Society for Computer Assisted Orthopedic Surgery (CAOS), 2016.
4. 4)
  - 11. Reaungamornrat, S., Otake, Y., Uneri, A.,, et al: ‘An on-board surgical tracking and video augmentation system for C-arm image guidance’, Int. J. Comput. Assist. Radiol. Surg., 2012, 7, (5), pp. 647–665 (doi: 10.1007/s11548-012-0682-9).
5. 5)
  - 20. Tateno, K., Tombari, F., Navab, N.: ‘Large scale and long standing simultaneous reconstruction and segmentation’, Comput. Vis. Image Underst., 2017, 157, pp. 138–150 (doi: 10.1016/j.cviu.2016.05.013).
6. 6)
  - 7. Navab, N., Heining, S.M., Traub, J.: ‘Camera augmented mobile C-arm (CAMC): calibration, accuracy study, and clinical applications’, IEEE Trans. Med. Imaging, 2010, 29, pp. 1412–1423 (doi: 10.1109/TMI.2009.2021947).
7. 7)
  - 22. Uckermann, A., Elbrechter, C., Haschke, R.,, et al: ‘3D scene segmentation for autonomous robot grasping’. 2012 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, October 2012, pp. 1734–1740.
8. 8)
  - 16. Fotouhi, J., Fuerst, B., Wein, W.,, et al: ‘Can real-time RGBD enhance intraoperative cone-beam CT?’, Int. J. Comput. Assist. Radiol. Surg., 2017, 12, (7), pp. 1211–1219 (doi: 10.1007/s11548-017-1572-y).
9. 9)
  - 23. Rusinkiewicz, S., Levoy, M.: ‘Efficient variants of the ICP algorithm. 3D digital imaging and modeling’. Int. Conf. 3D Digital Imaging and Modeling, 2001.
10. 10)
  - 18. Fischer, M., Fuerst, B., Lee, S.C.,, et al: ‘Preclinical usability study of multiple augmented reality concepts for k-wire placement’, Int. J. Comput. Assist. Radiol. Surg., 2016, 11, (6), pp. 1007–1014 (doi: 10.1007/s11548-016-1363-x).
11. 11)
  - 12. Grützner, P.A., Zheng, G., Langlotz, U.,, et al: ‘C-arm based navigation in total hip arthroplasty background and clinical experience’, Injury, 2004, 35, (1), pp. 90–95, supplement (doi: 10.1016/j.injury.2004.05.016).
12. 12)
  - 21. Rusu, R.B., Blodow, N., Beetz, M.: ‘Fast point feature histograms (FPFH) for 3d registration’. Proc. 2009 IEEE Int. Conf. Robotics and Automation, ICRA'09, Piscataway, NJ, USA, 2009, pp. 1848–1853.
13. 13)
  - 3. Matthews, F., Hoigne, D., Weiser, M.,, et al: ‘Navigating the fluoroscope's C-arm back into position: an accurate and practicable solution to cut radiation and optimize intraoperative workflow’, J. Orthop. Trauma, 2007, 21, pp. 687–692 (doi: 10.1097/BOT.0b013e318158fd42).
14. 14)
  - 14. Held, R., Gupta, A., Curless, B.,, et al: ‘3d puppetry: a kinect-based interface for 3d animation’. Proc. 25th Annual ACM Symp. User Interface Software and Technology, UIST ‘12, New York, NY, USA, 2012, pp. 423–434.
15. 15)
  - 1. Parker, P.J., Copeland, C.: ‘Percutaneous fluoroscopic screw fixation of acetabular fractures’, Injury, 1997, 28, (9), pp. 597–600 (doi: 10.1016/S0020-1383(97)00097-1).
16. 16)
  - 2. Starr, A., Jones, A., Reinert, C.,, et al: ‘Preliminary results and complications following limited open reduction and percutaneous screw fixation of displaced fractures of the actabulum’, Injury, 2001, 32, pp. 45–50 (doi: 10.1016/S0020-1383(01)00060-2).
17. 17)
  - 9. Diotte, B., Fallavollita, P., Wang, L.,, et al: ‘Radiation-free drill guidance in interlocking of intramedullary nails’ (Springer Berlin Heidelberg, Berlin, Heidelberg, 2012), pp. 18–25.
18. 18)
  - 4. Liu, L., Ecker, T., Schumann, S.,, et al: ‘Computer assisted planning and navigation of periacetabular osteotomy with range of motion optimization’ (Springer International Publishing, Cham, 2014), pp. 643–650.
19. 19)
  - 17. Lee, S.C., Fuerst, B., Fotouhi, J.,, et al: ‘Calibration of RGBD camera and cone-beam CT for 3d intra-operative mixed reality visualization’, Int. J. Comput. Assist. Radiol. Surg., 2016, 11, (6), pp. 967–975 (doi: 10.1007/s11548-016-1396-1).
20. 20)
  - 10. Habert, S., Gardiazabal, J., Fallavollita, P.,, et al: ‘Rgbdx: first design and experimental validation of a mirror-based RGBD X-ray imaging system’. 2015 IEEE Int. Symp. Mixed and Augmented Reality (ISMAR), September 2015, pp. 13–18.
21. 21)
  - 5. Boszczyk, B.M., Bierschneider, M., Panzer, S.,, et al: ‘Fluoroscopic radiation exposure of the kyphoplasty patient’, Eur. Spine J., 2006, 15, (3), pp. 347–355 (doi: 10.1007/s00586-005-0952-0).
22. 22)
  - 6. Synowitz, M., Kiwit, J.: ‘Surgeon's radiation exposure during percutaneous vertebroplasty’, J. Neurosurg., Spine, 2006, 4, (2), pp. 106–109 (doi: 10.3171/spi.2006.4.2.106).
23. 23)
  - 13. Siewerdsen, J.H., Moseley, D.J., Burch, S.,, et al: ‘Volume CT with a flat-panel detector on a mobile, isocentric C-arm: pre-clinical investigation in guidance of minimally invasive surgery’, Med. Phys., 2005, 32, (1), pp. 241–254 (doi: 10.1118/1.1836331).
24. 24)
  - 19. Tateno, K., Tombari, F., Navab, N.: ‘Real-time and scalable incremental segmentation on dense slam’. 2015 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), September 2015, pp. 4465–4472.

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery

References

Related content