Surgical phase recognition

Cholec80 is an endoscopic video dataset containing 80 videos of cholecystectomy surgeries performed by 13 surgeons. The videos are captured at 25 fps and downsampled to 1 fps for processing. The whole dataset is labeled with the phase and tool presence annotations. The phases have been defined by a senior surgeon in Strasbourg hospital, France. Since the tools are sometimes hardly visible in the images and thus difficult to be recognized visually, a tool is defined as present in an image if at least half of the tool tip is visible. Source: EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos[https://arxiv.org/pdf/1602.03012.pdf]

The MISAW data set is composed of 27 sequences of micro-surgical anastomosis on artificial blood vessels performed by 3 surgeons and 3 engineering students. The dataset contained video, kinematic, and procedural descriptions synchronized at 30Hz. The procedural descriptions contained phases, steps, and activities performed by the participants. Source: MIcro-Surgical Anastomose Workflow recognition challenge report

Analyzing the surgical workflow is a prerequisite for many applications in computer assisted surgery (CAS), such as context-aware visualization of navigation information, specifying the most probable tool required next by the surgeon or determining the remaining duration of surgery. Since laparoscopic surgeries are performed using an endoscopic camera, a video stream is always available during surgery, making it the obvious choice as input sensor data for workflow analysis. Moreover, this offers the opportunity for structured assessment of surgical skill for safety, teaching and quality management. The sub-challenge “Surgical Workflow and Skill Analysis” focuses on the online workflow analysis of laparoscopic surgeries. Participants are challenged to segment laparoscopic surgeries for gallbladder removal (cholecystectomy) into surgical phases, to recognize instrument presence and surgical actions as well as to classify surgical skill based on video data. Participants are encouraged (but not required!) to submit different results for phase segmentation, action recognition, instrument presence and skill classification . This novel kind of challenge investigates the current state-of-the-art results on surgical workflow analysis and skill assessment on one comprehensive dataset.

Holistic and Multi-Granular Surgical Scene Understanding of Prostatectomies (GraSP) dataset, a curated benchmark that models surgical scene understanding as a hierarchy of complementary tasks with varying levels of granularity. Our approach enables a multi-level comprehension of surgical activities, encompassing long-term tasks such as surgical phases and steps recognition and short-term tasks including surgical instrument segmentation and atomic visual actions detection. To exploit our proposed benchmark, we introduce the Transformers for Actions, Phases, Steps, and Instrument Segmentation (TAPIS) model, a general architecture that combines a global video feature extractor with localized region proposals from an instrument segmentation model to tackle the multi-granularity of our benchmark. Through extensive experimentation, we demonstrate the impact of including segmentation annotations in short-term recognition tasks, highlight the varying granularity requirements of each task, and establish TAPIS's superiority over previously proposed baselines and conventional CNN-based models. Additionally, we validate the robustness of our method across multiple public benchmarks, confirming the reliability and applicability of our dataset. This work represents a significant step forward in Endoscopic Vision, offering a novel and comprehensive framework for future research towards a holistic understanding of surgical procedures.

Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assistance, situational awareness, and patient safety. Current datasets fall short in scale, realism and do not capture the multimodal nature of OR scenes, limiting progress in OR modeling. To this end, we introduce MM-OR, a realistic and large-scale multimodal spatiotemporal OR dataset, and the first dataset to enable multimodal scene graph generation. MM-OR captures comprehensive OR scenes containing RGB-D data, detail views, audio, speech transcripts, robotic logs, and tracking data and is annotated with panoptic segmentations, semantic scene graphs, and downstream task labels. Further, we propose MM2SG, the first multimodal large vision-language model for scene graph generation, and through extensive experiments, demonstrate its ability to effectively leverage multimodal inputs. Together, MM-OR and MM2SG establish a new benchmark for holistic OR understanding, and open the path towards multimodal scene analysis in complex, high-stakes environments. Paper: https://arxiv.org/abs/2503.02579

49 videos of gastrojejunostomy procedure

BernBypass70 is a dataset consisting of 70 surgical videos of LRYGB at Inselspital, Bern University Hospital, Switzerland. The videos were recorded at a resolution of 720 × 576 at 25 fps. StrasBypass70, extending the Bypass40 dataset, is a collection of 70 videos of LRYGB surgeries performed by surgeons at the University Hospital of Strasbourg, France. The videos were recorded at a resolution of 854 × 480 or 1920 × 1080 resolution at 25 fps and were uniformly edited to a resolution of 854 × 480. MultiBypass140 is the combined dataset of 140 videos from Bern and Strasbourg medical centers. All videos have been anonymized by blacking out the potentially identifying frames outside the patient’s body.

Cataract is the leading cause of blindness worldwide, most affecting life in low- and middle-income countries (LMICs). The mainly used, most appropriate, and most cost-effective cataract surgical technique for LMICs is small incision cataract surgery (SICS). While algorithms have been developed for automated video analysis of surgical performance parameters for the cataract surgical technique predominantly used in high-income settings, so far there were no datasets nor algorithms for SICS available. This MICCAI challenge introduces the first SICS video dataset and offers teams the opportunity to evaluate the effectiveness of their phase recognition algorithms. The dataset of 155 patients was recruited at Sankara Eye Hospital in India. Analysis of surgical phases is important because it allows for quantitative comparison between different surgeons, feedback on identified critical steps, and detection of discrepancies from surgical protocols and because it is the first step for automatic assessment of surgical quality (Sim-OSSCAR). Our contribution is the first public dataset for SICS holding surgical videos and phase annotations of 155 surgeries with 18 distinct phases. Currently, there are other public cataract surgery phase datasets like Cataract-101 (n=101 videos) or the IEEE Cataracts (n=50 videos) but they only show phacoemulsification surgery which is distinct from SICS. Despite SICS widespread adoption in countries of the global south, no publicly available dataset exists for for this surgery, leaving a critical gap in cataract surgery research. Competitors are expected to submit an algorithm for predicting surgical phases based on the video data we supply and a short paper describing their approach.