Augmented Reality Navigation in Robot-Assisted Surgery with a Teleoperated Robotic Endoscope

Augmented reality (AR) is considered one of the most promising solutions for safer procedures in several surgical specialities. Fusing patient-specific pre-operative information, typically 3D models extracted from CT scans or MRI, with real-time surgical images allows the surgeon to have detailed information on the anatomical structure of the surgical target intra-operatively. The coupling of AR and Robotics represents the next step towards introducing awareness into the surgical room, thus enhancing the surgeon's perceptual, cognitive and manipulative capabilities. This paper presents a novel integrated system for real-time AR navigation in robotic minimally invasive surgery (RMIS), composed of a robotic endoscopic camera, a robotic teleoperation implementing a software-based Remote Center of Motion (RCM), and an AR navigation software based on an initial manual registration of virtual 3D models with the real anatomy. The integrated system, as well as the individual modules, were evaluated in simulated surgical-like setups for accuracy and repeatability. The proposed system can perform high-precision tasks (position accuracy around $1 mm$ and AR error lower than 7%), showing potential for application in different surgical procedures and setting the basis for autonomous robotic surgery operations.