Skip to content

About

[MICCAI'26] From Perception to Anticipation: Forecasting Vessel–Instrument Interactions in Endoscopic Surgery under Unreliable Observations

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

8 Commits

Folders and files

Repository files navigation

EndoInteractionAnticipation

[MICCAI'26] From Perception to Anticipation: Forecasting Vessel–Instrument Interactions in Endoscopic Surgery under Unreliable Observations

Introduction

This repository provides the main-model PyTorch implementation accompanying our MICCAI 2026 publication:

From Perception to Anticipation: Forecasting Vessel–Instrument Interactions in Endoscopic Surgery under Unreliable Observations

Overview of the vessel–instrument interaction anticipation framework

Timely anticipation of vessel–instrument interactions is critical for improving surgical safety in Endoscopic Submucosal Dissection (ESD). Existing computer-assisted systems mainly rely on vessel segmentation, which provides spatial awareness but cannot predict hazardous interactions before they occur. Furthermore, segmentation errors caused by bleeding, specular reflections, and tissue deformation introduce unreliable observations, making anticipation particularly challenging.

In this work, we introduce vessel–instrument interaction anticipation, a task that predicts future interaction states from a short observation window of surgical video and segmentation masks. The model uses 16 frames from a 3-second observation window to predict the interaction state at second 4: no_interaction, approaching, or touching.

To address unreliable observations, our method consists of two key components:

  • Cross-Modal Latent Recovery (CLR): learns a shared latent representation by enforcing consistency between visual appearance and geometric observations, reducing the impact of noisy segmentation.
  • Dual Asynchronous Memory (DAM): separates transient interaction dynamics from reliability-filtered interaction prototypes through distinct admission and retrieval rules.

The model uses a VideoMAE ViT-B backbone, with CLR between transformer blocks 8 and 9, a 10-dimensional geometric descriptor, reliability-aware memory fusion, and a three-class prediction head.

Reported results

  • Macro F1: 70.80%
  • Precision: 71.34%
  • Recall: 71.52%
  • Accuracy: 71.11%
  • Inference speed: 97.82 FPS

These values are reported in the publication; numerical reproduction with this source release has not yet been verified. The smoke test below checks execution, not experimental performance.

Installation

Use Python 3.10 or newer and a PyTorch build appropriate for your machine. Run the following commands from the repository root:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python tools/smoke_test.py

The smoke test uses synthetic inputs and a reduced-size model on CPU, without input files, pretrained weights or a GPU. It checks forward and backward execution, output shapes, finite values and bounded reliability scores. It retains the default transformer depth, CLR insertion point and number of input frames, but reduces image resolution and feature dimensions to keep the check lightweight. It does not test preprocessing or full-size training.

Training and evaluation

Use configs/default.json as a template for your local configuration. Set the input locations, annotation labels, case split files and pretrained backbone checkpoint as described in USAGE.md. Relative paths inside a configuration resolve from the repository root; absolute paths are used unchanged.

# Validate the configuration, case split, observation sampling and input files.
CONFIG=/path/to/local.json MODE=validate bash run.sh

# Train and evaluate the final checkpoint.
CONFIG=/path/to/local.json MODE=train bash run.sh \
  --output-dir outputs/run_01

# Evaluate an existing compatible checkpoint in a separate output directory.
CONFIG=/path/to/local.json MODE=eval bash run.sh \
  --checkpoint outputs/run_01/checkpoint_last.pth \
  --output-dir outputs/eval_01

The default recipe uses AdamW with a learning rate of 2.5e-4, batch size 16 and 60 epochs. The backbone is frozen during the first 10 epochs; blocks 9–12 and the final normalization are then unfrozen while blocks 1–8 remain frozen. The objective is CE + 0.7 * CLR.

Choose a new output directory for every run. Outputs include the final checkpoint, training history, classification metrics, per-clip predictions, runtime environment and selected observation timestamps.

Both constant-FPS frames and explicit frame timestamps are supported. prepared_fps must describe the actual stored frame rate: use frames_in=30 at 10 fps or frames_in=150 at 50 fps for the 3-second window. See USAGE.md for timestamp manifests and offset windows.

Weights and inference

To export a checkpoint trained with this implementation:

python tools/export_weights.py \
  --checkpoint outputs/run_01/checkpoint_last.pth \
  --output checkpoints/release_01

The exported bundle contains model.pth (model tensors) and model.json (model settings, class order, implementation fingerprint and checksum). No verified pretrained download is provided by this source release.

Prepare a clip using the same sampling, normalization and geometry extraction as training, then run prediction:

python tools/prepare_observation.py --config /path/to/local.json \
  --clip-id example_clip --output outputs/observation.npz

python -m src.predict --weights checkpoints/release_01 \
  --input outputs/observation.npz --device cpu

Replace example_clip with a clip ID in your metadata. The NPZ contains only preprocessed RGB and geometry tensors. Prediction returns class probabilities, reliability and the final memory fusion weight. The tensor contract is documented in USAGE.md.

Checkpoint loading is restricted by default. Only for your own trusted legacy checkpoints, use --trusted-checkpoint with the relevant command; see USAGE.md.

Throughput measurement

python tools/benchmark.py --config /path/to/local.json \
  --checkpoint checkpoints/release_01/model.pth \
  --batch-size 16 --warmup 50 --iterations 200

Use the model configuration that produced the checkpoint. This measures model-only clips per second and milliseconds per batch, excluding preprocessing and file reads. These units are not interchangeable with raw-video FPS or end-to-end latency.

Runtime scope

This release includes the clip-level model, preprocessing, training, evaluation, standalone inference, weight export and a lightweight smoke test.

Memories reset per clip. With tubelet size 2, 16 input frames produce eight internal temporal states; memory capacities default to 8 and 20. Increasing capacity beyond the available states does not add historical observations. The transformer uses the entire observation window. Continuous-video use requires a separate sliding-window and event-alignment protocol. See USAGE.md for input conventions and runtime details. Research use does not establish clinical suitability.

License

No software license has been selected for this source snapshot.

Contact

For any questions, please contact:

Yueyao Chen — yueyaochen0823@gmail.com

Citation

If you find this work useful, please cite:

@inproceedings{chen2026anticipation,
  title={From Perception to Anticipation: Forecasting Vessel--Instrument Interactions in Endoscopic Surgery under Unreliable Observations},
  author={Chen, Yueyao and Wang, Kai-Ni and Yip, Hon-Chi and Dou, Qi},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)},
  year={2026}
}

About

[MICCAI'26] From Perception to Anticipation: Forecasting Vessel–Instrument Interactions in Endoscopic Surgery under Unreliable Observations

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages