Skip to content
SCILABEL INSIGHTS

Annotating Surgical and Endoscopic Video: Phases, Instruments and Critical Events

Surgical and endoscopic videos are becoming an important source of training data for artificial intelligence (AI), computer vision, robotic surgery, surgical workflow analysis and medical-device technologies. However, raw video alone is not enough to train reliable AI systems. The video must be transformed into structured, consistent and clinically meaningful data through carefully designed annotation.

Surgical video annotation involves identifying what is happening, when it is happening, which instruments are being used, what anatomical structures are involved and which clinically important events occur. Modern surgical datasets increasingly use hierarchical annotation approaches that represent procedures through phases, tasks and fine-grained actions.

Understanding Surgical Video Annotation

A surgical procedure may contain thousands of frames, but not every frame represents the same activity. A procedure moves through different stages, and instruments enter and leave the field while the surgeon performs different actions.

For this reason, an effective annotation framework should capture both temporal information and spatial information. Temporal labels describe when a phase or event begins and ends, while spatial annotations identify instruments, anatomical structures or other objects within individual frames.

A useful hierarchical structure is:

Surgical Phase → Task → Action → Instrument → Anatomical Target → Critical Event

For example, during a dissection phase, an annotator may identify the specific dissection task, the instrument being used, the tissue being manipulated and any important event occurring during that activity.

1. Surgical Phase Annotation

Surgical phase annotation divides a procedure into clinically meaningful stages. Depending on the operation, phases may include preparation, access, exploration, dissection, clipping, cutting, coagulation, suturing, specimen removal, closure and instrument removal.

The exact phases should always be defined according to the procedure and the annotation protocol. Recent surgical datasets demonstrate the use of clearly defined start and end frames for individual surgical phases, with intermediate frames assigned according to predefined rules.

For each phase, the annotation can follow a simple structure:

Phase name → Start frame → End frame → Attributes

Clear definitions of phase boundaries are particularly important because different annotators may otherwise interpret the beginning or end of an activity differently.

2. Surgical Task and Action Annotation

Phase-level information can be expanded into more detailed task and action labels. This allows an AI system to understand not only the broad stage of a procedure but also the specific activities taking place within that stage.

For example, a dissection phase could contain tasks such as identifying the dissection plane or exposing a structure. Individual actions could then include grasping, cutting, retracting, dissecting, suctioning or coagulating.

The TEMSET-24K endoscopic dataset provides an example of this hierarchical approach, using phase, task and action labels to represent surgical workflows at different levels of detail[1].

This type of structure can be particularly useful for AI systems designed for surgical workflow recognition, procedure indexing, education and automated video analysis.

3. Surgical Instrument Annotation

Identifying surgical instruments is another important component of video annotation. AI developers may need to know not only whether an instrument is present but also its location, identity, movement and interaction with tissue.

Common instruments that may be annotated include:

  • Graspers and forceps
  • Surgical scissors
  • Needle holders
  • Dissection instruments
  • Electrocautery devices
  • Clip appliers
  • Staplers
  • Suction and irrigation instruments
  • Endoscopic instruments
  • Robotic surgical instruments

Different annotation methods can be selected according to the intended AI application. Bounding boxes can identify the approximate location of an instrument, while polygons or segmentation masks can provide more precise boundaries. Keypoints can identify important parts of an instrument, and tracking can maintain its identity across consecutive frames.

Research datasets demonstrate the use of instrument detection, segmentation, tracking and keypoint annotation for surgical video analysis[2].

4. Endoscopic Video Annotation

Endoscopic video presents additional annotation challenges because the visual field can change continuously. Camera movement, zooming, irrigation, bleeding, smoke, blur, instrument occlusion and temporary loss of visualization can all affect the appearance of the scene.

For this reason, endoscopic annotation may combine procedure phases with instrument, action and visual-condition labels.

For example, an annotation scheme might record:

Procedure phase → Instrument → Action → Anatomical target → Visual condition

Endoscopic datasets such as TEMSET-24K and the Renji endoscopic datasets demonstrate how detailed temporal phase annotations can be used to represent endoscopic procedures and support AI-based surgical workflow analysis[1].

5. Critical Surgical Events

Some AI projects require more than phase and instrument labels. They may also need annotations for important events that occur during a procedure.

Examples include bleeding, haemostasis, tissue manipulation, instrument exchange, loss of visualization, specimen removal, suturing, knot tying, instrument collision and other procedure-specific events.

Critical-event annotation should be based on clearly defined project criteria. An annotator should not label an event simply because a frame appears unusual. The annotation protocol should define what constitutes the event and specify its temporal boundaries.

This is especially important for developing systems intended to recognize surgical complications, workflow interruptions or clinically significant events.

6. Frame-Level and Temporal Annotation

Not every project requires every frame to receive the same level of annotation.

A project may use frame-level annotation when the AI model needs detailed information about each image. Temporal annotation is useful for defining the duration of a phase or event. Keyframe annotation can be used when detailed spatial labeling is required only at selected frames, while object tracking can maintain instrument identity as an object moves through the video.

Some surgical datasets deliberately sample frames rather than annotate every frame, while others provide dense temporal annotations. The appropriate strategy depends on the AI application, dataset size, annotation budget and required level of precision.

7. Building Clear Annotation Guidelines

High-quality surgical video annotation begins before the first production frame is labeled.

Annotation guidelines should define the terminology, label hierarchy, start and end conditions, object categories and procedures for handling difficult cases.

For example, the guidelines should explain:

  • When does a surgical phase begin?
  • When does it end?
  • Which instruments should be labeled?
  • How should partially visible instruments be handled?
  • How should occluded instruments be labeled?
  • What qualifies as a critical event?
  • How should ambiguous frames be treated?
  • How should camera movement or poor visibility be handled?
  • Can labels overlap?
  • Which attributes must be recorded?

Recent work on standardized temporal annotation emphasizes the importance of procedure-specific definitions and consistent start/stop criteria for surgical video annotation.

8. Clinical Expertise and Annotation Quality

Surgical video contains clinical information that may not be reliably interpreted using purely visual rules. Clinical expertise can therefore be important when defining annotation taxonomies, reviewing difficult cases and resolving disagreements.

A strong workflow may begin with expert-designed guidelines, followed by annotator training and calibration. A sample of videos can then be independently annotated and reviewed before large-scale production begins.

Recent multicentre work in surgical video annotation used trained clinical annotators, predefined phase definitions and structured quality-control procedures.

9. Quality Assurance and Validation

Quality assurance is essential because annotation errors can become part of the training signal used by an AI model.

A robust workflow can include:

Guideline development → Annotator training → Calibration → Production annotation → Quality review → Clinical validation → Adjudication → Final dataset validation

Quality checks may examine whether phase boundaries are correct, instruments have been classified consistently, segmentation boundaries are accurate and critical events have been captured.

Where multiple annotators are involved, inter-annotator agreement can also be assessed to identify areas where the annotation protocol needs clarification. Some recent surgical-video studies have used independent annotation and cross-validation or consensus processes to reduce subjectivity.

10. From Surgical Video to AI-Ready Data

The ultimate objective is not simply to label video. It is to convert complex surgical recordings into structured, consistent and clinically meaningful datasets that can support AI development and evaluation.

A typical workflow can therefore look like:

Raw Surgical/Endoscopic Video
↓
Video Preparation & Quality Control
↓
Surgical Phase Annotation
↓
Task & Action Annotation
↓
Instrument Detection/Segmentation
↓
Critical Event Annotation
↓
Clinical Review
↓
Quality Assurance
↓
AI-Ready Dataset

These datasets can support applications such as surgical workflow recognition, robotic surgery, computer vision, procedure indexing, surgical education, medical-device AI, clinical research and automated surgical video analysis.

How SCILabel Supports Surgical Video AI

SCILabel provides healthcare AI data services covering data sourcing, clinical annotation, AI evaluation and quality assurance. For surgical and endoscopic video projects, these capabilities can be applied to the development of structured datasets for medical AI and computer-vision applications.

Our capabilities can include surgical phase annotation, procedure and workflow labeling, instrument detection and segmentation, object tracking, temporal event annotation, clinical review, annotation QA and dataset validation.

Whether you are developing surgical AI, robotic surgery systems, endoscopic computer vision, medical-device technologies or clinical workflow analytics, the quality of your training data is fundamental to the performance and reliability of the resulting AI system.

Build Better Surgical AI with Better Data

Surgical and endoscopic video contains valuable information about procedures, instruments, actions, anatomy and clinical events. With the right annotation framework, that information can be transformed into structured datasets capable of supporting the next generation of healthcare AI.

Have a surgical, endoscopic or medical-device video annotation project?

📧 Contact SCILabel: info@shevslabel.com

SCILabel — Powering Safer, Smarter Medical AI.


References

  1. TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation. Scientific Data, 2025.
  2. Multicenter Fine Annotated Surgical Video Dataset for Minimally Invasive Glaucoma Surgery. Scientific Data, 2026.
  3. Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room. Scientific Data/PMC, 2021.
  4. Development and Validation of a Model for Laparoscopic Colorectal Surgical Instrument Recognition Using Convolutional Neural Network-Based Instance Segmentation and Videos of Laparoscopic Procedures. JAMA Network Open/PMC, 2022.
  5. SegMatch: Semi-Supervised Surgical Instrument Segmentation. PMC, 2025.
  6. ROBUST-MIPS: A Combined Skeletal Pose and Instance Segmentation Dataset for Laparoscopic Surgical Instruments. PMC, 2026.
  7. Spine Endoscopic Atlas: An Open-Source Dataset for Surgical Instrument Segmentation. PMC, 2025.
  8. Renji Endoscopic Submucosal Dissection Video Dataset for Early Gastric Cancer. Scientific Data, 2025.
  9. Renji Endoscopic Submucosal Dissection Video Dataset for Colorectal Neoplastic Lesions. Scientific Data, 2025.
  10. Surgical Data Science – From Concepts Toward Clinical Translation. PMC, 2022.
  11. A Standardized Temporal Segmentation Framework and Annotation Resource Library in Robotic Surgery. PMC, 2025.
  12. Surgical Video Workflow Analysis via Visual-Language Learning. npj Health Systems, 2025.
Portal Login