Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models

The gist The gist GPD, an on-policy self-distillation framework, augments GRPO with an OPSD teacher conditioned on question-routed depth, semantic, and BEV textual cues together with the reference answer to enable a student model to internalize geometric reasoning during training while retaining RGB-only inference.

How it works

1. GPD operates in two stages: first, for each training sample, GPD constructs compact textual summaries of depth, semantic, and BEV cues offline and then routes only the question-relevant subset to form a privileged context for the teacher. Second, during training, the same VLM plays two roles: as a student conditioned only on RGB and question, it samples on-policy trajectories; as a privileged teacher additionally conditioned on the routed 3D cues and reference answer, it re-scores those trajectories token by token.

Stage 1: Offline 3D Privilege Construction

The construction of the privilege involves several steps for each training example (v, q, a⋆), where v denotes the RGB observations, q the question, and a⋆ the answer. This includes Textual 3D Cue Extraction where depth, semantic segmentation masks and labels via Grounded-SAM-2 are estimated if aligned 3D annotations are unavailable. Question-Conditioned Routing is performed offline to select a relevant subset r(q) ⊆ M for each question, where the routed scene context is cscene(q) = concat(sm: m ∈ r(q)) and the complete privilege provided to the teacher is cpriv(q, a⋆) = concat(cscene(q), a⋆).

Stage 2: Geometry-Privileged Self-Distillation

During training, the student receives only the RGB observations and question, pS = concat(v, q), while the teacher additionally receives the offline-constructed privilege, pT = concat(v, q, cpriv(q, a⋆)). The optimization objective combines outcome-based GRPO with privilege-guided distillation applied selectively to incorrect trajectories.

Outcome-Guided Policy Optimization

The GRPO objective is defined as LGRPO(θ) = −E[yi]G i=1 (1/G i=1) Σ T j=1 min ρi,jAi, clip(ρi,j, 1 ± ε)Ai − β (1/G i=1) DKL[πθ∥πref] i where ρi,j = πθ(tj t<j, pS) / πθold (tj t<j, pS) is the token-level importance sampling ratio and DKL[πθ∥πref] = PT j=1 [log πθ(tj t<j, pS) − log πref(tj t<j, pS)] is the token-level KL divergence with the frozen reference policy πref.

Privilege-Guided Policy Distillation

The distillation loss is defined as LOPSD(θ) = Σ G i=1 Σ T j=1 ui,j di,j where ui,j ∈ [0, 1] indicates a valid response token and di,j = exp(∆i,j) − ∆i,j − 1 where ∆i,j = log πθ(tj t<j, pT) − log πθ(tj t<j, pS) is the token-level teacher–student log-probability gap. The Error-Gated Joint Objective is L = LGRPO + λL˜OPSD where λ controls the strength of the distillation signal, and L˜OPSD = min (Σ i∈E Σ T j=1 ui,j di,j.X i∈E Σ T j=1 ui,j, 1.5) where E = [i: Ri < τ] denotes the set of incorrect trajectories and we set τ = 0.5.

Key Findings

The results show that on the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD. Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories. Furthermore, zero-shot privilege injection demonstrates that routed text is the only variant that exceeds the baseline accuracy by reaching 52.2 (+3.1 points), motivating its use as the privilege form in GPD. The case study illustrates a failure mode of outcome-only supervision, where GRPO misidentifies the fountain’s position relative to Question: Based on these four, with each camera aligned with room walls and partially capturing the surroundings: If I am standing at the same spot and facing the same direction as shown in image 2, then I turn left and move forward, will I get closer to the stone fountain? Options: A. Yes B. No. GPD correctly localizes the camera, maps the left-forward motion to the spatial frame, and grounds the fountain via cross-view correspondence.

The final joint objective is L = LGRPO + λL˜OPSD where λ controls the strength of the distillation signal. Across both model scales and benchmarks, GPD improves overall accuracy over GRPO and outperforms answer-privileged OPSD. The smaller privilege gap suggests that GPD internalizes geometric cues during training rather than remaining dependent on them at inference.

The paper presents GPD, which augments GRPO with an OPSD teacher conditioned on question-routed depth, semantic, and BEV textual cues together with the reference answer to enable a student model to internalize geometric reasoning during training while retaining RGB-only inference. Across both model scales and benchmarks, GPD improves overall accuracy over GRPO and outperforms answer-privileged OPSD. The method is designed to turn geometric evidence into supervision that an RGB-only model can internalize. This approach addresses the trade-off between explicit 3D methods and training-based methods by using 3D signals as privileged information during training. The core mechanism relies on question-routed geometric privilege to guide token-level distillation on incorrect trajectories. This work establishes question-routed geometric privilege as effective supervision for spatial reasoning. The results across two model scales and five benchmarks demonstrate consistent improvements over GRPO and answer-privileged OPSD, with ablations validating the key design choices. The geometric cues have been internalized by the student model.

--- Page 8 ---

References Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W., et al. 2025a. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631 Bai, S.; Chen, K.; Liu, X.; Wang, J., et al. 2025b. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 Cai, Y.; Liu, J., et al. 2026. Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillation. arXiv preprint arXiv:2606.08719 Chen, B.; Xu, Z., et al. 2024. SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14455–14465 Chen, Z.; Lu, X., et al. 2026. Geometrically-constrained agent for spatial reasoning.

1. textbfGeometry-Aware Supervision via GPD Implementation + VLM Architecture Preservation + RGB-Only Inference Maintenance (GPD): The improved system can perform spatial reasoning tasks—such as judging distance, direction, and layout—while maintaining an RGB-only inference interface, avoiding the architectural modification costs of explicit 3D methods.

2. textbfQuestion-Conditioned Textual Privilege Construction: GPD's Stage 1 process allows for question-conditioned routing offline, which selects a relevant subset of geometric cues from depth, semantic segmentation, and BEV cues based on the query, ensuring that different spatial questions require different forms of geometric evidence.

3. textbfSelective Distillation via Incorrect Trajectory Gating: The system employs an Error-Gated Joint Objective by applying the distillation loss only to incorrect trajectories (gate LOPSD to incorrect trajectories only), which prevents noise from disturbing already-correct trajectories and ensures process supervision on incorrect trajectories.

Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer; a privileged KL, applied only to incorrect trajectories, augments GRPO, and the deployed model remains RGB-only. On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks. Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories.