Sensorium: An Open-Source Multimodal Data Collection and Curation Library for Embodied Surgical AI
cs.AI, cs.RO
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: GenAI for Health Workshop @ NeurIPS 2026, Sydney
Code: https://github.com/nct-tso-robotics/tso_sensorium
License: http://creativecommons.org/licenses/by/4.0/
The gist: Embodied surgical AI requires reliable multimodal data, yet surgical data collection remains costly, difficult to repeat, and strongly dependent on the hardware setup.
Terminology
Abstract
Embodied surgical AI requires reliable multimodal data, yet surgical data collection remains costly, difficult to repeat, and strongly dependent on the hardware setup. We present Sensorium, an open-source Python library for recording, processing, annotating, and curating multimodal surgical data from configurable robot and sensor streams using the Robot Operating System (ROS) and ROS2 middleware. The library combines operator-facing data acquisition, configurable processing, semantic annotation, and episodic dataset generation. We evaluate Sensorium across 1516 recordings of phantom, ex vivo, and in vivo porcine experiments involving multiple surgical tasks and sensing configurations, and report preliminary usability feedback.
Sources
- LeRobot: An Open-Source Library for End-to-End Robot Learning
- SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos
- SutureBot: A Precision Framework & Benchmark For Autonomous End-to-End Suturing
- Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling
- Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation
- Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models
- LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning
- Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
- LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning
- An Open-Source Robotics Research Platform for Autonomous Laparoscopic Surgery
- GP-VLS: A general-purpose vision language model for surgery
- MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery
- SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
- SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents