Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish

arXiv:2603.20827 · cs.RO · Submitted 2026-03-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish".

Dev: Here's the gist of the paper. Swim2Real presents an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone This approach closes the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification System Identification Methodology The core of the method involves using a Vision Language Model (VLM) to guide system identification, which replaces traditional black-box optimization approaches The VLM is provided with both video and numerical data at each iteration to reason about physical discrepancies and propose parameter updates Specifically, the prompt instructs the VLM to identify physical discrepancies in overlay videos, such as "excess tail amplitude" or "incorrect body curvature," and then map these observations to specific simulator parameters using parameter semantics This contrasts with black-box optimizers that only return scalar improvements Calibration Refinement Strategy To handle the imprecision of VLM proposals, a backtracking line search is employed to validate each VLM proposal at geometrically decreasing step sizes This mechanism addresses the issue where the VLM proposes an update direction but overestimates its magnitude, as seen when "the full step (β 0 = 1.0) overshoots, but halving the step size (β 1 = 0.5) yields an improvement" This process has been shown to triple the accept rate from 14% to 42% compared to evaluating only the full step Performance Metrics and Validation The calibration accuracy is validated across several metrics, including marker error, swimming velocity, and downstream RL performance Swim2Real achieves a mean L2 marker error of 51.3 ± 1.2 mm with zero outliers across five runs Furthermore, the calibrated simulator produces forward swimming velocities that match real fish dynamics with a Mean Absolute Error (MAE) of 7.4 mm/s, which is half that of BayesOpt This fidelity translates to downstream RL policies that swim 12% farther than those from BayesOpt-calibrated simulators Key Findings and Interpretability The VLM's reasoning provides a physically grounded correction, as demonstrated by its diagnosis of "excess damping forces that become dominant at high angular velocities" leading to a proposal to "decrease fluid drag and increase hinge stiffness" This contrasts with scalar objective functions which cannot provide such physically grounded corrections The iterative loop is essential because the VLM often conflates fluid coefficient effects with joint stiffness effects, and the line search recovers directionally correct but oversized proposals The results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification Swim2Real achieves this reliability with no hand-designed search stages, operating directly from video comparisons The pipeline currently relies on a proprietary VLM (Gemini), but the approach suggests portability to opensource VLMs through the prompt requirement for standard vision-language capabilities The VLM call efficiency is also optimized, as the line search makes each VLM call 2.5× more productive The pipeline successfully transfers motor commands to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment Page 1 --- Swim2Real calibrates a robotic fish simulator from video and deploys the resulting RL policy on hardware, with no hand-designed search stages.

Rosa: Stage 1: a VLM compares simulated and real swimming videos, proposes parameter adjustments, and a backtracking line search validates the step size, iterating for up to 40 evaluations.

Dev: Stage 2: the calibrated simulator trains an RL policy that swims 12% farther than BayesOpt-calibrated policies.

Rosa: Motor commands from the trained policy are deployed on the physical fish at 50 Hz.

Paper summary: Rosa: So looking at "Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish," the authors are essentially saying they’ve built an end to end pipeline that calibrates all sixteen parameters of a tendon driven fish simulator simultaneously from video input alone <ref:two thousand six hundred three point two zero eight two seven#pg1.

Dev: The main implication is that VLM guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero shot reinforcement learning transfer to physical swimmers without manual system identification <ref:two thousand six hundred three point two zero eight two seven#pg1.

Taro: They achieved this reliability with no hand designed search stages operating directly from video comparisons, which is a big deal because it removes the need for those tedious manual tuning steps <ref:two thousand six hundred three point two zero eight two seven#pg1.

Rosa: They also mentioned that while the pipeline currently relies on a proprietary VLM called Gemini, the approach suggests portability to open source VLMs through the prompt requirement for standard vision language capabilities <ref:two thousand six hundred three point two zero eight two seven#pg1.

Dev: And they optimized the VLM call efficiency, because that backtracking line search makes each VLM call two point five times more productive <ref:two thousand six hundred three point two zero eight two seven#pg1.

Taro: It’s interesting that their results show qualitative differences too, where Swim2Real reproduces the body shape and forward progression of the real fish, while CMA ES exhibits incorrect posture and reduced thrust <ref:two thousand six hundred three point two zero eight two seven#pg6.

Rosa: The overall message is that this method offers a way to do system identification using video structure rather than just scalar metrics, which is what makes it different from prior black box optimization methods <ref:two thousand six hundred three point two zero eight two seven#pg2.

Conclusion: Rosa: So we’re wrapping up on "Swim2Real: Vision-Language Models for Simulator Calibration of a Soft Robotic Fish" and the authors are just showing how to take video input alone to calibrate all sixteen parameters of a tendon driven fish simulator <ref:two thousand six hundred three point two zero eight two seven#pg1.

Dev: It’s about using a vision language model Gemini to look at swimming videos, find physical mistakes, and then propose parameter adjustments based on what it sees rather than just using some black box optimization stuff <ref:two thousand six hundred three point two zero eight two seven#pg4.

Taro: The big deal there is that this bypasses the need for hand crafted cost functions, which means you don't have to spend all that time designing what the simulator should be trying to minimize <ref:two thousand six hundred three point two zero eight two seven#pg2.

Rosa: Exactly. It’s a way to get physically grounded corrections because the model is reasoning about things like "excess tail amplitude" or "incorrect body curvature" instead of just spitting out some random number <ref:two thousand six hundred three point two zero eight two seven#pg1.

Dev: And they handle the imprecision of those VLM suggestions with a backtracking line search, so they test different step sizes to make sure the parameter update actually gets better instead of just overshooting things <ref:two thousand six hundred three point two zero eight two seven#pg5.

Taro: That line search is crucial because it’s what recovers those directionally correct but oversized proposals that the VLM might suggest on its own <ref:two thousand six hundred three point two zero eight two seven#pg5.

Rosa: The validation shows they got a mean marker error of about fifty-one point three millimeters with zero outliers across five runs, which is pretty solid for getting it right outside the lab <ref:two thousand six hundred three point two zero eight two seven#pg1.

Dev: And the swimming velocity matches real fish dynamics with a mean absolute error of seven point four millimeters per second, which is half what another calibration method got <ref:two thousand six hundred three point two zero eight two seven#pg4.

Taro: It’s that fidelity that matters because it means when you take the policy trained on this simulator and send it to a real fish, it swims twelve percent farther than policies from another calibrated simulator <ref:two thousand six hundred three point two zero eight two seven#pg1.

Rosa: So the bottom line is they closed the sim-to-real gap directly from video using this VLM guidance without needing any manual system identification steps <ref:two thousand six hundred three point two zero eight two seven#pg1.

Dev: It’s a powerful way to transfer learning to physical swimmers, but it does mean you still have that reliance on the vision language model for the actual tuning part <ref:two thousand six hundred three point two zero eight two seven#pg1.

University of Warsaw · IDEAS NCBR EPFL ETH Zurich

cs.RO

Submitted: 2026-03-21

Updated: 2026-11-09

Comments: 8 pages, 8 figures, 5 tables. Accepted for publication in IEEE Robotics and Automation Letters. v2: accepted version, substantially revised after peer review (new title, new experiments). Project page: https://swim2real.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Swim2Real presents an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone This approach closes the sim-to-real gap for…

Key concepts

System Identification Methodology
This core method uses a Vision Language Model (VLM) instead of traditional optimization methods. The VLM analyzes video overlays, identifies physical problems like 'excess tail amplitude,' and maps these visual observations directly to specific simulator parameters, providing physically grounded suggestions rather than just scalar error scores.
Backtracking Line Search
This strategy validates the parameter updates proposed by the VLM. Because VLMs can sometimes propose updates that are too large, this search tests different step sizes (like 1.0 or 0.5) to find the smallest change that actually improves performance, significantly increasing the chance of accepting a good update.
VLM-Guided Calibration
The VLM acts as a physical diagnostician by reasoning about discrepancies in video, such as identifying 'excess damping forces.' This provides physically informed corrections that traditional objective functions cannot offer. The iterative loop ensures the system refines its understanding of the physical simulation parameters based on visual evidence.
Zero-Shot RL Transfer
The successful calibration allows Reinforcement Learning (RL) policies trained in simulation to be transferred directly to a physical fish without needing manual identification steps. This means robots can swim effectively in the real world immediately after calibration, bypassing tedious and error-prone manual system identification.

Terminology

Summary

Swim2Real presents an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone This approach closes the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification

System Identification Methodology

The core of the method involves using a Vision Language Model (VLM) to guide system identification, which replaces traditional black-box optimization approaches The VLM is provided with both video and numerical data at each iteration to reason about physical discrepancies and propose parameter updates Specifically, the prompt instructs the VLM to identify physical discrepancies in overlay videos, such as excess tail amplitude or incorrect body curvature, and then map these observations to specific simulator parameters using parameter semantics This contrasts with black-box optimizers that only return scalar improvements

Calibration Refinement Strategy

To handle the imprecision of VLM proposals, a backtracking line search is employed to validate each VLM proposal at geometrically decreasing step sizes This mechanism addresses the issue where the VLM proposes an update direction but overestimates its magnitude, as seen when the full step (β 0 = 1.0) overshoots, but halving the step size (β 1 = 0. p. 2 p. 25) yields an improvement This process has been shown to triple the accept rate from 14% to 42% compared to evaluating only the full step

Performance Metrics and Validation

The calibration accuracy is validated across several metrics, including marker error, swimming velocity, and downstream RL performance Swim2Real achieves a mean L2 marker error of 51.3 ± 1.2 mm with zero outliers across five runs Furthermore, the calibrated simulator produces forward swimming velocities that match real fish dynamics with a Mean Absolute Error (MAE) of 7. p. 64 mm/s, which is half that of BayesOpt This fidelity translates to downstream RL policies that swim 12% farther than those from BayesOpt-calibrated simulators

Key Findings and Interpretability

The VLM's reasoning provides a physically grounded correction, as demonstrated by its diagnosis of excess damping forces that become dominant at high angular velocities leading to a proposal to decrease fluid drag and increase hinge stiffness This contrasts with scalar objective functions which cannot provide such physically grounded corrections The iterative loop is essential because the VLM often conflates fluid coefficient effects with joint stiffness effects, and the line search recovers directionally correct but oversized proposals

Conclusion on Efficacy

The results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification Swim2Real achieves this reliability with no hand-designed search stages, operating directly from video comparisons The pipeline currently relies on a proprietary VLM (Gemini), but the approach suggests portability to opensource VLMs through the prompt requirement for standard vision-language capabilities The VLM call efficiency is also optimized, as the line search makes each VLM call 2.5× more productive The pipeline successfully transfers motor commands to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment

--- Page 1 ---

Swim2Real calibrates a robotic fish simulator from video and deploys the resulting RL policy on hardware, with no hand-designed search stages. Stage 1: a VLM compares simulated and real swimming videos, proposes parameter adjustments, and a backtracking line search validates the step size, iterating for up to 40 evaluations. p. 0 Stage 2: the calibrated simulator trains an RL policy that swims 12% farther than BayesOpt-calibrated policies. p. 0 Motor commands from the trained policy are deployed on the physical fish at 50 Hz. p. 0

--- Page 2 ---

Recent work has demonstrated that video comparisons between simulated and real robots can drive system identification without hand-crafted cost functions, exploiting the visual structure that scalar metrics discard [9]. Visionlanguage models (VLMs) such as Gemini [10] provide a structured diagnostic signal. p. 1 p. 1 Rather than returning a scalar error, they reason about physical discrepancies from video, identifying when a simulated fish bends too sharply, moves too slowly, or exhibits incorrect tail dynamics. p. 1 p. 1 This VLMbased physical reasoning complements traditional optimizers, proposing directions informed by physical intuition, and a line search determines the correct magnitude

--- Page 3 ---

The goal is to find θ∗ that minimizes discrepancies between simulated and real marker trajectories: θ∗ = arg min θ∈Θ 1 F X f∈F 1 MTf XM m=1 XTf t=1 p s m(t, f, θ) − p r m(t, f) squared (1) where Θ ⊂ R d is the bounded parameter space, F is a set of actuation frequencies, M is the number of body markers, Tf is the number of time steps at frequency f, and p s m, p r m ∈ R 2 are simulated and real marker positions. We denote this objective L(θ). p. 2 Each evaluation of L(θ) requires running F MuJoCo simulations and constitutes the dominant computational cost. p. 2

--- Page 4 ---

Algorithm 1 Swim2Real: VLM-Guided Calibration with Line Search Require: Simulator Sθ, eval. budget B, param. p. 3 bounds Θ, decay β, max steps K 1: θbest ← RandomInit(Θ); Lbest ← Evaluate(θbest); B ← B−1 2: H ← ∅

3: while B > 0 do

4: V ← RenderOverlay(θbest) Sim vs. real video 5: θ′ ← QueryVLM(V, θbest, H) VLM proposes direction 6: ∆ ← θ′ − θbest; accepted ← false 7: Backtracking line search

8: for k = 0, 1,..., K−1 do

9: θk ← Clip(θbest+β k∆, Θ)

10: L ← Evaluate(θk); B ← B−1

11: if L < Lbest then

12: θbest ← θk; Lbest ← L; accepted ← true; break

13: end if

end for 14: end while 15: Append (∆, βk, accepted) to H 16: end while 17: return θbest, Lbest

--- Page 5 ---

Fig. 2 One iteration of the backtracking line search. p. 2 The VLM diagnoses a physical discrepancy and proposes updated parameters θ′. p. 2 The full step (β 0 = 1. p. 20) overshoots, but halving the step size (β 1 = 0.5) yields an improvement over Lbest and is accepted. p. 2 This triples the accept rate from 14% to 42% compared to evaluating only the full step

--- Page 6 ---

Swim2Real achieves this reliability with no hand-designed search stages, operating directly from video comparisons. Fig. 4 shows the qualitative difference: Swim2Real reproduces the body shape and forward progression of the real fish, while CMA-ES exhibits incorrect posture and reduced thrust

--- Page 7 ---

Swim2Real tracks the real fish most closely (MAE = 7.4 mm/s, half that of BayesOpt at 12. p. 59 mm/s), with the gap widening at higher frequencies where calibration quality matters most

--- Page 8 ---

The tail tip (M8) alone reaches 101 mm. Reducing this floor would require per-segment fluid coefficients or a higherfidelity fluid model

--- Page 9 ---

This is the paper on arXiv. This paper is 2603. p. 0 p. 020827. EVERY sentence must end with a reference token The token names the page and quotes the words it rests on, copied word for word from that page every sentence must end with a reference token

Swim2Real presents an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone The single most informative sentence about the paper's subject and finding is Swim2Real, an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone

The results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification Swim2Real achieves this reliability with no hand-designed search stages, operating directly from video comparisons The pipeline currently relies on a proprietary VLM (Gemini), but the Gemini 3.1 ablation demonstrates robustness across model versions and the prompt requires only standard vision-language capabilities, suggesting portability to opensource VLMs The VLM API costs are minor compared to real-world data collection The pipeline successfully transfers motor commands to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment

--- Page 9 ---

This is the paper on arXiv. This paper is 2603.20827. EVERY sentence must end with a reference token The single most informative sentence about the paper's subject and finding is Swim2Real, an end-to-end pipeline that calibrates all 16 parameters of a tendon-driven fish simulator simultaneously from video input alone

--- Page 2 ---

Recent work has demonstrated that video comparisons between simulated and real robots can drive system identification without hand-crafted cost functions, exploiting the visual structure that scalar metrics discard [9]. Visionlanguage models (VLMs) such as Gemini [10] provide a structured diagnostic signal. Rather than returning a scalar error, they reason about physical discrepancies from video, identifying when a simulated fish bends too sharply, moves too slowly, or exhibits incorrect tail dynamics.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. VLM-guided calibration pipeline: The system can automatically calibrate all 16 parameters of a tendon-driven fish simulator directly from swimming videos, requiring no hand-designed search stages or perparameter method selection. This replaces prior methods that required three manually tailored stages to handle complexity.

  2. Backtracking line search integration: The system can validate VLM proposals by testing them at geometrically decreasing step sizes, which triples the accept rate from 14% to 42% by recovering proposals where the direction is correct but the magnitude is too large. This significantly improves convergence speed during parameter tuning.

  3. Physics-grounded reasoning: The system can use a VLM to provide interpretable and physically grounded feedback by identifying discrepancies like excess tail amplitude, insufficient thrust, or incorrect body curvature, allowing it to propose updates based on physical intuition rather than just scalar error signals.

Sources

Related papers