Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish
summary
The gist
Swim2Real presents an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone This approach closes the sim-to-real gap for…
In short
Swim2Real creates an automated system to calibrate a fish simulator using only video input to set all 16 parameters at once, closing the sim-to-real gap for aquatic robots. It uses a Vision Language Model (VLM) to reason about physical differences in videos and employs a backtracking line search to find precise parameter adjustments without manual identification.
Key concepts
- System Identification Methodology
- This core method uses a Vision Language Model (VLM) instead of traditional optimization methods. The VLM analyzes video overlays, identifies physical problems like 'excess tail amplitude,' and maps these visual observations directly to specific simulator parameters, providing physically grounded suggestions rather than just scalar error scores.
- Backtracking Line Search
- This strategy validates the parameter updates proposed by the VLM. Because VLMs can sometimes propose updates that are too large, this search tests different step sizes (like 1.0 or 0.5) to find the smallest change that actually improves performance, significantly increasing the chance of accepting a good update.
- VLM-Guided Calibration
- The VLM acts as a physical diagnostician by reasoning about discrepancies in video, such as identifying 'excess damping forces.' This provides physically informed corrections that traditional objective functions cannot offer. The iterative loop ensures the system refines its understanding of the physical simulation parameters based on visual evidence.
- Zero-Shot RL Transfer
- The successful calibration allows Reinforcement Learning (RL) policies trained in simulation to be transferred directly to a physical fish without needing manual identification steps. This means robots can swim effectively in the real world immediately after calibration, bypassing tedious and error-prone manual system identification.
Terminology used across episodes
This episode discusses
- Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish · Paper Radio
- The CMA Evolution Strategy: A Tutorial
- Vid2Sid: Videos Can Help Close the Sim2Real Gap
- Gemini: A Family of Highly Capable Multimodal Models
- Simple Models, Real Swimming: Digital Twins for Tendon-Driven Underwater Robots
- RoboMorph: Evolving Robot Morphology using Large Language Models
- Debate2Create: Robot Co-design via Multi-Agent LLM Debate
The paper
Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish · Read on arXiv
University of Warsaw · IDEAS NCBR EPFL ETH Zurich
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish".
Dev: Here's the gist of the paper. Swim2Real presents an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone This approach closes the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification System Identification Methodology The core of the method involves using a Vision Language Model (VLM) to guide system identification, which replaces traditional black-box optimization approaches The VLM is provided with both video and numerical data at each iteration to reason about physical discrepancies and propose parameter updates Specifically, the prompt instructs the VLM to identify physical discrepancies in overlay videos, such as "excess tail amplitude" or "incorrect body curvature," and then map these observations to specific simulator parameters using parameter semantics This contrasts with black-box optimizers that only return scalar improvements Calibration Refinement Strategy To handle the imprecision of VLM proposals, a backtracking line search is employed to validate each VLM proposal at geometrically decreasing step sizes This mechanism addresses the issue where the VLM proposes an update direction but overestimates its magnitude, as seen when "the full step (β 0 = 1.0) overshoots, but halving the step size (β 1 = 0.5) yields an improvement" This process has been shown to triple the accept rate from 14% to 42% compared to evaluating only the full step Performance Metrics and Validation The calibration accuracy is validated across several metrics, including marker error, swimming velocity, and downstream RL performance Swim2Real achieves a mean L2 marker error of 51.3 ± 1.2 mm with zero outliers across five runs Furthermore, the calibrated simulator produces forward swimming velocities that match real fish dynamics with a Mean Absolute Error (MAE) of 7.4 mm/s, which is half that of BayesOpt This fidelity translates to downstream RL policies that swim 12% farther than those from BayesOpt-calibrated simulators Key Findings and Interpretability The VLM's reasoning provides a physically grounded correction, as demonstrated by its diagnosis of "excess damping forces that become dominant at high angular velocities" leading to a proposal to "decrease fluid drag and increase hinge stiffness" This contrasts with scalar objective functions which cannot provide such physically grounded corrections The iterative loop is essential because the VLM often conflates fluid coefficient effects with joint stiffness effects, and the line search recovers directionally correct but oversized proposals The results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification Swim2Real achieves this reliability with no hand-designed search stages, operating directly from video comparisons The pipeline currently relies on a proprietary VLM (Gemini), but the approach suggests portability to opensource VLMs through the prompt requirement for standard vision-language capabilities The VLM call efficiency is also optimized, as the line search makes each VLM call 2.5× more productive The pipeline successfully transfers motor commands to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment Page 1 --- Swim2Real calibrates a robotic fish simulator from video and deploys the resulting RL policy on hardware, with no hand-designed search stages.
Rosa: Stage 1: a VLM compares simulated and real swimming videos, proposes parameter adjustments, and a backtracking line search validates the step size, iterating for up to 40 evaluations.
Dev: Stage 2: the calibrated simulator trains an RL policy that swims 12% farther than BayesOpt-calibrated policies.
Rosa: Motor commands from the trained policy are deployed on the physical fish at 50 Hz.
Paper summary: Rosa: So looking at "Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish," the authors are essentially saying they’ve built an end to end pipeline that calibrates all sixteen parameters of a tendon driven fish simulator simultaneously from video input alone <ref:two thousand six hundred three point two zero eight two seven#pg1.
Dev: The main implication is that VLM guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero shot reinforcement learning transfer to physical swimmers without manual system identification <ref:two thousand six hundred three point two zero eight two seven#pg1.
Taro: They achieved this reliability with no hand designed search stages operating directly from video comparisons, which is a big deal because it removes the need for those tedious manual tuning steps <ref:two thousand six hundred three point two zero eight two seven#pg1.
Rosa: They also mentioned that while the pipeline currently relies on a proprietary VLM called Gemini, the approach suggests portability to open source VLMs through the prompt requirement for standard vision language capabilities <ref:two thousand six hundred three point two zero eight two seven#pg1.
Dev: And they optimized the VLM call efficiency, because that backtracking line search makes each VLM call two point five times more productive <ref:two thousand six hundred three point two zero eight two seven#pg1.
Taro: It’s interesting that their results show qualitative differences too, where Swim2Real reproduces the body shape and forward progression of the real fish, while CMA ES exhibits incorrect posture and reduced thrust <ref:two thousand six hundred three point two zero eight two seven#pg6.
Rosa: The overall message is that this method offers a way to do system identification using video structure rather than just scalar metrics, which is what makes it different from prior black box optimization methods <ref:two thousand six hundred three point two zero eight two seven#pg2.
Conclusion: Rosa: So we’re wrapping up on "Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish" and the authors are just showing how to take video input alone to calibrate all sixteen parameters of a tendon driven fish simulator <ref:two thousand six hundred three point two zero eight two seven#pg1.
Dev: It’s about using a vision language model Gemini to look at swimming videos, find physical mistakes, and then propose parameter adjustments based on what it sees rather than just using some black box optimization stuff <ref:two thousand six hundred three point two zero eight two seven#pg4.
Taro: The big deal there is that this bypasses the need for hand crafted cost functions, which means you don't have to spend all that time designing what the simulator should be trying to minimize <ref:two thousand six hundred three point two zero eight two seven#pg2.
Rosa: Exactly. It’s a way to get physically grounded corrections because the model is reasoning about things like "excess tail amplitude" or "incorrect body curvature" instead of just spitting out some random number <ref:two thousand six hundred three point two zero eight two seven#pg1.
Dev: And they handle the imprecision of those VLM suggestions with a backtracking line search, so they test different step sizes to make sure the parameter update actually gets better instead of just overshooting things <ref:two thousand six hundred three point two zero eight two seven#pg5.
Taro: That line search is crucial because it’s what recovers those directionally correct but oversized proposals that the VLM might suggest on its own <ref:two thousand six hundred three point two zero eight two seven#pg5.
Rosa: The validation shows they got a mean marker error of about fifty-one point three millimeters with zero outliers across five runs, which is pretty solid for getting it right outside the lab <ref:two thousand six hundred three point two zero eight two seven#pg1.
Dev: And the swimming velocity matches real fish dynamics with a mean absolute error of seven point four millimeters per second, which is half what another calibration method got <ref:two thousand six hundred three point two zero eight two seven#pg4.
Taro: It’s that fidelity that matters because it means when you take the policy trained on this simulator and send it to a real fish, it swims twelve percent farther than policies from another calibrated simulator <ref:two thousand six hundred three point two zero eight two seven#pg1.
Rosa: So the bottom line is they closed the sim-to-real gap directly from video using this VLM guidance without needing any manual system identification steps <ref:two thousand six hundred three point two zero eight two seven#pg1.
Dev: It’s a powerful way to transfer learning to physical swimmers, but it does mean you still have that reliance on the vision language model for the actual tuning part <ref:two thousand six hundred three point two zero eight two seven#pg1.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies