Amir H. Sadeghi, MD, PhD; Quinten Mank, MSc; Alper S. Tuzcu, BSc; Jasper Hofman, MSc; Sabrina Siregar, MD, PhD; Alexander Maat, MD, PhD; Alexandre Mottrie, MD, PhD; Jolanda Kluin, MD, PhD; Pieter De Backer, MD, PhD

Abstract

In recent years, thoracic surgery has undergone a transformative shift, propelled by technological advancements such as robotic-assisted thoracic surgery (RATS), artificial intelligence (AI), and extended reality.1 RATS has gained global popularity, offering innovative tools like robotic stapling, fluorescence imaging, and the integration of 3-dimensional (3D) reconstructions in the console. However, challenges persist, including a lack of tactile feedback, constrained visual access because of the proximity of robotic instruments to target tissues, and a significant learning curve.2 In addition, preoperative (3D) computed tomography scans do not accurately represent the anatomical deformations in the lung's shape during RATS.3 The integration of AI/extended reality has emerged as a potential solution, providing real-time anatomical updates and ensuring that the surgeon has an accurate and unobscured representation of the patient's 3D anatomy.4,5 Integrating these technologies in RATS anatomical lung resections will hopefully enhance precision, safety, surgeon satisfaction, and surgical outcomes of lung cancer patients. This article explores the feasibility of a technological milestone by combining AI, augmented reality (AR), and pulmonary robotic surgery, showcasing the first-in-human AR robotic lobectomy.

Technology

In collaboration with MedicalVR, we have developed PulmoVR's (MedicalVR) deep learning (DL)-based segmentation method, which automatically generates 3D reconstructions from the computed tomography scan, focusing on pulmonary artery, vein, lobes, and airways. In order to create these models, 126 computed tomography scans from patients who underwent lung resection at Erasmus MC were collected over years. To train a DL model, the pulmonary arteries and veins were manually annotated and subsequently used for the development of the AI-based algorithm.

These segmented images are then transformed into interactive and dynamic simulated realities (iSRs; interactive 3D models that have the ability to closely simulate the reality in terms of lung deformation) through finite element method simulations (virtual computer-based simulations, enabled by applying physics-based principles) creating deformable virtual 3D models (Figure E1). Subsequently, users (eg, the surgeon) choose the segmented images created with the aforementioned DL model. The software automatically selects a physics template, defining biomechanical properties. iSR allows the user to deform (with a mouse) the models in real time with simple motion-based input, which then adapt boundary conditions of the finite element method simulation. This means that the user (bedside technician) can deform the model manually according to his/her interpretation on the extent of compression/deformation that is carried out by the console surgeon. However, it is not possible yet to deform the virtual models automatically.

For real-time instrument deocclusion, a DL algorithm segments nonorganic items.5 The technology consists of a robust real-time binary segmentation pipeline for nonorganic items, where the segmented instruments are rendered on top of the patient-specific 3D model to mitigate instrument deocclusion and add a sense of depth to the augmented reality. The dataset for binary segmentation comprises 31,812 images in which nonorganic items were outlined using the SuperAnnotate annotation platform. These nonorganic items cover a variety of categories such as robotic and laparoscopic instruments, needles, wires, clips, vessel loops, bulldogs, gauzes, and more. The selection of images was uniformly drawn from 100 full-length robotic-assisted partial nephrectomy procedures. The segmented instruments overlay the iSR 3D-model, enhancing AR with depth perception. Developed with NVIDIA Holoscan SDK (leveraging the Clara AGX developer kit), the intraoperative AI and AR application captures endoscopic video processed by a Deltacast capture card. The output feeds into the Tilepro input for the surgical console and a bedside monitor simultaneously, allowing real-time AR views for the surgeon and real-time model alignment by a bedside technician (Figure 1). This will enable a real-time colored overlay of a dynamic 3D model, mirroring the lung as it is manipulated by the surgeon.