VisionSIM: Beyond Idealized Sensors - A Multi-Modal Simulator for Robot Perception
cs.RO, cs.AI, cs.CV
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/Genesis-Embodied-AI/Genesis
License: http://creativecommons.org/licenses/by/4.0/
The gist: Robotic perception increasingly relies on heterogeneous sensor suites combining cameras with emerging modalities (event, SPAD, iToF/dToF, and thermal).
Terminology
Abstract
Robotic perception increasingly relies on heterogeneous sensor suites combining cameras with emerging modalities (event, SPAD, iToF/dToF, and thermal). Yet existing simulators idealize these sensors, hiding critical modality-specific failure modes that limit real-world deployment. We present VisionSIM, a unified, physically grounded simulation platform that elevates sensor realism to a first-class concern by modeling the raw physical measurement process for diverse conventional and emerging modalities. We expose fine-grained control over physical sensor parameters, reproducing failure modes overlooked by general-purpose simulators: pileup and glare for LiDAR; phase wrapping, saturation, and motion artifacts for iToF; refractory drops for event cameras; and history-dependent emission for thermal cameras. This enables principled cross-modality comparison on identical scenes and trajectories. We release the platform and a large indoor dataset with synchronized raw measurements and rich ground-truth annotations.
Sources
- Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
- v2e: From Video Frames to Realistic DVS Events
- A Comprehensive Survey on Synthetic Infrared Image synthesis
- Habitat: A Platform for Embodied AI Research
- nuScenes: A multimodal dataset for autonomous driving
- A2D2: Audi Autonomous Driving Dataset
- A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
- Kubric: A scalable dataset generator
- BlenderProc
- CARLA-Loc: Synthetic SLAM Dataset with Full-stack Sensor Setup in Challenging Weather and Dynamic Environments
- Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation
- A Study on Quantifying Sim2Real Image Gap in Autonomous Driving Simulations Using Lane Segmentation Attention Map Similarity
- Radiance Fields from Photons
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation