RT Journal Article
A1 Sing Chun Lee
A1 Bernhard Fuerst
A1 Keisuke Tateno
A1 Alex Johnson
A1 Javad Fotouhi
A1 Greg Osgood
A1 Federico Tombari
A1 Nassir Navab

PB iet
T1 Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery
JN Healthcare Technology Letters
VO 4
IS 5
SP 168
OP 173
AB Orthopaedic surgeons are still following the decades old workflow of using dozens of two-dimensional fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This Letter presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. A red–green–blue–depth camera is rigidly attached to a mobile C-arm and is calibrated to the cone-beam computed tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model-based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localising the entry point and orienting the surgical tool during screw placement. The authors validate the augmentation by measuring target registration error and also evaluate the tracking accuracy in the presence of partial occlusion.
K1 real-time automatic fusion
K1 iterative closest point algorithm
K1 workflow
K1 CBCT imaging
K1 automatic tool segmentation
K1 complex 3D structures
K1 3D point clouds
K1 mobile C-arm
K1 cone-beam computed tomography imaging space
K1 orthopaedic surgery
K1 pelvis
K1 mixed reality environment
K1 model-based surgical tool tracking
K1 multimodal imaging
K1 two-dimensional fluoroscopic images
K1 adapted 3D model-based tracking algorithm
K1 synthetic fluoroscopic images
K1 target registration error
K1 red-green-blue-depth camera
K1 entry point
K1 tracking accuracy
K1 surgical tools
K1 partial occlusion
K1 reconstructed surface
K1 multimodal data fusion
K1 screw placement
K1 mixed reality visualisation
K1 interactive 3D mixed reality environment
DO https://doi.org/10.1049/htl.2017.0066
UL https://digital-library.theiet.org/;jsessionid=6ung39qc41vnn.x-iet-live-01content/journals/10.1049/htl.2017.0066
LA English
SN
YR 2017
OL EN