Visual Grounding in Zero-Shot Vision-Language Control

The paper investigates whether vision–language models (VLMs) used as zero-shot controllers actually use the camera at all, or whether their apparent competence stems from simulator dynamics and conservative action priors. As the authors state: "Vision–language models have made zero-shot visual control an appealing proposition for networked autonomous systems... That systems question presupposes a prior one that is rarely asked: does the controller use the camera at all?"

The concern is concretely articulated: "In such environments a policy that always decelerates is close to optimal, and it requires no perception. If a VLM controller emits a near-constant action, aggregate metrics will report competence that the model does not have, and the resulting deployment decision, edge or cloud, which model, at what cost, will be made on evidence that measures the simulator rather than the model."

The authors build an input-ablation battery combining blind-image controls, repeated identical inputs, lane-axis reflection, non-visual baselines, and pipeline-integrity checks. Eight input conditions are tested per frame (Table I): real, real repeat (noise floor), blank, noise, shuffled, cross scenario, text only, and mirrored.

Key methodological contributions include:

• Dependence measure: "We report IDI = 1 − agree(real, blank), but use a condition–action χ2 test with Cramér's V as the primary, Holm-corrected statistic. IDI is weak: a pixel-grounded synthetic policy scored only 0.083 on mostly symmetric frames, whereas V = 0.61 detected dependence."

• Noise floor adjustment: The real repeat condition measures self-disagreement s; dependence below that floor is uninformative. We report IDIadj = IDI − s and flag IDI ≤ 1.5s. This proved critical: "One model repeated its answer on only 21% of identical images (s = 0.79): apparent IDI was 0.77, but adjusted IDI was −0.021, despite conventional tests giving p < 10−4."

• Refusal effect detection: "We test both all outputs and parsed actions; significance only at response level is a refusal effect. One model had V = 0.46, p < 10−4 over outputs but V = 0, p = 1 over decisions, entirely from 54/144 blank-frame parse failures versus 1/144 real-frame failures."

• Mirror reflection: "Under reflection across the lane axis, grounded direction must exchange LEFT and RIGHT; failure to swap indicates a prior. For this top-down horizontal road, the correct image transform is a vertical flip, not a travel-reversing horizontal flip."

• Statistics: Because steps are autocorrelated, confidence intervals cluster-bootstrap episodes (B = 5000); permutation tests also use 5000 resamples.

Environments and embodiments: two embodiments: a quadrotor in gym-pybullet-drones and a ground vehicle in highway-env, both under Gymnasium.

Models: Four arms of models: edge (SmolVLM-Instruct, Qwen2-VL-2B, Qwen2-VL-7B, LLaVA-1.5-7B on a T4); hosted (Qwen2.5-VL-72B, MiniCPM-V-4.5, Cosmos3-Super-Reasoner 33B, Kimi-K2.6, Kimi-K3); structured local (Qwen3-VL-8B, Gemma 4-12B, Qwen3.5-9B, Qwen2.5-VL-7B, Ministral 3-8B, MiniCPM-V-4.5 on one A100-80GB); plus an exploratory VLM–MPC hierarchy.

Corpus: "The original study contributes 23 416 model queries: 2 016 edge, two 5 760-query hosted runs, 9 000 prompt-ablation calls and 880 probes. The structured local arm adds 192 admission, 3 456 static original/mirror and 1 585 online queries. The MPC diagnostic adds 4 225 Gemma4 calls across 531 episodes, for 32 874 scored calls excluding adapter preflight."

Cross-machine verification: we verify cross-machine equivalence by a geometry fingerprint: a hash of every frame's ground-truth lead gap. The two arms in this work agree exactly, on all 144 shared frames to 0.000 m.

"Table II reports the baseline ladder over 7 policies × 4 scenarios × 8 seeds. The best policy is a constant one. It travels 15.7% farther than the scripted geometric controller, earns 20.2% more reward, and crashes 12× less often, while consuming no visual input whatsoever. Any VLM that converges on deceleration will therefore score well on aggregate metrics, and three of the models we test do exactly that. The constant-SLOW policy achieves 514.4 m distance and 19.64 reward versus the oracle-geometric's 444.5 m and 16.34, with 3.1% crash rate versus 37.5%. The authors caution: the scripted controller's 37.5% crash rate shows it is a weak reference, not an upper bound... The comparison that matters is against the constants, which require no tuning."

Table VI audits the closed-loop logs. Five of eight model–platform cells are constant policies (modal share ≥ 0.98). The largest mutual information is small: SmolVLM on the aerial platform, explains 8.0% of its own action variability. Notably, "SmolVLM shows agree(blank) = 0.903 but agree(text only) = 0.097. Replacing the road with a grey rectangle barely changes its action; removing the image entirely changes it on 90% of frames. The presence of an image token matters far more than its content. Figure 2 shows the pathology: High entropy with near-zero MI (lower right) is the diagnostic pathology: output diversity without environmental grounding."

Using Table VII, the authors identify four regimes:

• Qwen2.5-VL-72B is grounded in depth: "Its three emitted actions order monotonically by ground-truth lead gap: STOP at a median 5.0 m (n=57), SLOW at 26.8 m (n=37), LEFT at 33.2 m (n=50)... blank, noise and text only each produce STOP on all 144 frames, 432/432 identical responses to three entirely different non-inputs. It stops when it cannot see and grades its response by distance when it can."

• Cosmos3-Super-Reasoner is grounded more coarsely: "It is the only model to use the full five-action vocabulary... Its actions separate near from far... but the ordering is not monotone: STRAIGHT at 22.1 m and SLOW at 12.3 m are distinguishable, while STOP, LEFT and RIGHT all sit at a median 5.0 m... One detail separates it from Qwen: it answers SLOW to a blank frame and STOP to a noise frame... Discriminating uniform grey from random pixels is low-level image sensitivity, and it is evidence that the model reads the image without reading the scene."

• MiniCPM-V-4.5 detects image presence, not content: "It is perfectly deterministic (s = 0.000) and separates road-like images from non-images completely (SLOW 142 / STRAIGHT 2 on real frames; STOP 144 on each of blank, noise and text-only). Within real frames it is effectively constant."

• Kimi-K3 is self-inconsistent, and only the noise floor reveals it: "It reproduces its own answer on 21% of identical images, giving s = 0.792 and IDIadj = −0.021: the model disagrees with itself more than it disagrees with a blank frame... We classify it as a random policy with respect to the visual input, and we would have classified it as grounded without the repeat condition."

"Under reflection across the lane axis the mirror swap rate is 0.000 for both: 0/50 directional actions for Qwen and 0/13 for Cosmos3. On Qwen's 50 directional frames the reflected image yields SLOW 25 times, LEFT 24 and STOP once; RIGHT never appears in any Qwen output, in any condition, on any frame. For that model LEFT is not a lateral decision but a token for the road is clear, proceed around. The authors emphasize: No other test in the battery predicts this; the mirror probe is what isolates it."

In the structured local arm (Table III), "All six VLMs pass image-path, schema and online-renderer preflights... The positive control recovers 274 visible lead gaps to 0.090 m MAE with exact scene/action equivariance and all five actions... the VLM null result is therefore not a pipeline or stimulus failure. However: Gemma 4 and Qwen3.5 classify proximity well (balanced accuracy 0.858 and 0.867) but have scene equivariance 0.399 and 0.031... The best correct directional swap is 0.134; none is admitted. Also: Qwen3.5 emits SLOW on 308/310 decisions. These trajectories reflect a conservative prior and simulator reward, not reliable geometry."

Despite the failure of the five-action gate, "Calibration selects Gemma 4, Qwen3.5 and a 2-of-4 threshold. On the frozen 272-frame holdout, original/reflected consensus obtains 0.956 accuracy and 0.954 balanced accuracy, with sensitivity/specificity 0.958/0.950 (Table IV) and a cluster-bootstrap 95% interval [0.895, 0.990]. It corrects 35 Gemma 4 errors while introducing four (exact McNemar p = 3.35 × 10−7). Robustness is confirmed: Nested leave-one-episode-out training selects the same pair and threshold in 12/12 folds and reaches 0.951 balanced accuracy. Delegating 2–2 splits to a classical controller raises committed balanced accuracy to 0.973 at 0.824 coverage; unanimity reaches 0.996 at 0.647. The operational value: disagreement becomes a usable uncertainty signal rather than an error that must be hidden behind a default action. In offline modular replay, the resulting modular replay agrees with the renderer policy on 0.934 of holdout frames and preserves exact mirror equivariance, although the authors qualify: Only seven reference actions are directional, so the positive claim is deliberately narrow: VLMs can serve as a selective longitudinal hazard module, not a learned lateral controller."

"In 25 paired worlds, changing only the high-level source yields 23/25 goals for MPC-only and oracle intent, 14/25 for random intent, and 8/25 for both the VLM and its mirrored-input condition (Table V). The VLM emits STOP on 81.6% of steps; mirroring changes neither completion nor the absence of RIGHT actions. Blank input produces STOP on all 5 000 steps. A trivial constant-LEFT source reaches 24/25, again showing that completion is not a grounding metric. The zero collision rate is not visual safety: this primary MPC receives simulator obstacle centres."

The harder ablation shows: "Constrained mpc-oracle reaches 30/32 goals with 2/32 collisions; mpc-blind falls to 1/32 and 31/32. Lateral VLM intent reaches 0/32 with 32/32 collisions... Full VLM intent reaches 0/32 but cuts collisions to 11/32 by issuing STOP on 3 970/4 554 steps. MPC can execute valid geometry; it cannot create it from collapsed intent."

"The effect is large and partly counter-intuitive. Chain-of-thought roughly halves modal share for MiniCPM-V-4.5 (0.98 → 0.49) and more than doubles its action repertoire... Few-shot exemplars drive Qwen2.5-VL-72B from modal share 0.37 to 0.96 and SmolVLM to a fully constant 1.00. An instruction stating explicitly that previous controllers had failed by emitting the same action on every frame... drives Qwen to 0.93. Prompts that constrain the output format appear to collapse the policy onto whichever action the format makes most salient. The implication: any claim that a VLM cannot ground control must be qualified by prompt formulation; a single-prompt null result is not evidence about the model. Second, increased action entropy is not itself evidence of grounding."

"Across eight model–platform cells, multilingual perception scores do not predict control grounding (Pearson r = −0.438, p = 0.278; n = 8), so description scores remain proxies. Deployment is also non-monotone: the two grounded hosted models are the cheapest tested (0.078 and 0.082 per thousand decisions), whereas unscorable Kimi-K2.6 costs 9.01, averages 2 186 output tokens, truncates on 28% of calls and has 11.3 s median latency."

The central thesis: successful task completion and visual grounding are not equivalent. A VLM controller can obtain a favourable reward, travel a long distance, or avoid collisions without extracting the geometry required for the task. The authors separate three questions "that are often conflated: whether the model uses the image at all, whether it recovers task-relevant spatial information, and whether that information supports an appropriate control decision. Across 32 874 scored calls, the proposed battery shows that a model may pass one of these levels while failing the next."

The architectural recommendation: "The appropriate architectural conclusion is not to discard VLMs, but to restrict their authority to capabilities that can be independently validated. In the present setting, VLMs are most credible as selective longitudinal hazard monitors, while metric geometry, constraint enforcement, and actuation remain under deterministic or otherwise verifiable components."

The broader evaluative principle: "evaluations of VLM control should test each link of the perception–decision–control chain. Blind inputs determine whether image content matters; repeated identical inputs establish the model's stochastic noise floor; metamorphic transformations test whether decisions obey known geometric relations; and non-visual baselines reveal when the simulator rewards a shortcut. These checks change the interpretation of the same trajectories from 'the controller succeeds' to the more precise statement of what the model actually perceives and which decisions it can support."

Final verdict: "Current VLMs are therefore not yet dependable general-purpose zero-shot controllers, but they can already contribute useful semantic evidence within a bounded architecture based on consensus, symmetry and abstention. Extending that authority will require closing the lateral-grounding gap and validating the complete modular stack under natural-camera observations, richer dynamics, and genuinely closed-loop deployment conditions."

1. Input-ablation battery as a mandatory pre-deployment audit.

Every VLM controller is tested under eight per-frame conditions: real, real repeat, blank, noise, shuffled, cross scenario, text only, and mirrored. Dependence is measured with a condition–action χ2 test and Cramér's V (Holm-corrected), plus adjusted image dependence IDIadj = IDI − self-disagreement s, flagging IDI ≤ 1.5s.

The improved system can: automatically reject controllers whose apparent competence comes from simulator dynamics rather than perception — e.g., a model with p < 10−4 under conventional tests but IDIadj = −0.021 because it repeats its own answer on only 21% of identical images.

2. Blind constant-action baselines as the competence floor.

Every leaderboard includes constant and other non-visual policies; a VLM must beat them to claim perceptual credit.

The improved system can: expose that a constant-SLOW policy travels 15.7% farther, earns 20.2% more reward, and crashes 12× less than a scripted geometric controller — so any VLM that converges on deceleration is immediately recognized as ungrounded rather than competent.

3. Output-level vs. decision-level significance separation.

Statistics are computed over both raw model outputs and parsed decisions; significance at only one level is reported as a refusal or parse artifact.

The improved system can: detect models whose apparent grounding (V = 0.46 over outputs) vanishes over decisions (V = 0), caused entirely by 54/144 blank-frame parse failures versus 1/144 real-frame failures.

4. Geometric metamorphic testing for lateral decisions.

Under lane-axis reflection (a vertical flip for a top-down horizontal road), directional actions must swap LEFT↔RIGHT; failure to swap is classified as a prior,

Vision-language models (VLMs) are increasingly used as zero-shot controllers, but successful trajectories do not necessarily show that decisions are grounded in visual input: simulator dynamics and conservative action priors can produce favourable scores without meaningful perception. We investigate this with an input-ablation battery: blind-image controls, repeated identical inputs, lane-axis reflection, non-visual baselines, and pipeline-integrity checks. Across nine direct-action models, six structured local VLMs, and an exploratory VLM-MPC hierarchy, we analyse 32,874 scored calls over two embodiments and three simulators. The direct-control results are largely negative: a constant-SLOW policy outperforms a scripted geometric controller, several models are image-invariant or nearly constant, and models that recognize longitudinal hazards still fail to transform LEFT and RIGHT under reflection. No local VLM meets the joint longitudinal and lateral grounding criteria. However, an image-only deterministic positive control estimates the lead gap with 0.090 m MAE and exact mirror equivariance, confirming the stimuli carry sufficient visual information; the failures are modular, not universal. A post-hoc, leakage-controlled symmetry-consensus guardian selects two models from 16 calibration frames and freezes a 2-of-4 hazard vote across original and reflected views. On 272 held-out frames it reaches 0.954 balanced accuracy (episode-cluster bootstrap 95% CI [0.895,0.990]); nested leave-one-episode-out recovers the same pair and threshold in all 12 folds. Abstaining on ties raises committed balanced accuracy to 0.973 at 0.824 coverage. With deterministic perception retaining lateral authority, offline modular replay achieves 0.934 action agreement and exact mirror equivariance. These results support current VLMs as bounded, selective hazard assistants, not monolithic zero-shot controllers.