Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments
cs.MM
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: 21 pages, 9 figures
Project page: https://black1948.github.io/lab2sim
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners.
Terminology
Abstract
Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.
Sources
- WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader
- Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning
- WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics
- Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation
- VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction
- FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videos
- Kimi K3: Open Frontier Intelligence
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisit
- FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation
Related papers
- FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries
- When does the head leave? Shared timing and a movement clock in 360-degree video viewing
- Augmented Hybrid TV Services. Habits, Preferences, Opinions and Expectations of Spanish Consumers
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR