Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

The gist The selection mechanism is the right place for radar to enter, introducing Radar-Modulated Selection (RMS), which achieves state-of-the-art performance on nuScenes by injecting radar into Mamba’s selective scan and replacing out-of-scan fusion.

Radar Modulated Selection (RMS)

The core contribution is RMS, a minimal and principled way to inject radar into Mamba’s selective scan, where radar modulates the scan from within by adding zero-initialised perturbations to the step size ∆ and readout C while leaving the input projection B and state dynamics A image-only The construction is exactly equivalent to a pretrained image-only Mamba at initialisation, ensuring radar only influences the model where it improves accuracy This mechanism allows radar to influence temporal credit assignment via ∆t and observation via Ct — the two fundamental degrees of freedom over selection Specifically, horizon modulation modulates the memory horizon by adding a radar-driven contribution to the pre-softplus step size, gated by a scalar αt ∈ R and readout modulation augments Ct additively with Wrad C xrad,t. At initialization, the block reproduces the image-only selection exactly because at step 0 the block reduces to vanilla Mamba.

Multi-View Scan Pyramid (MVSP)

To manage computational inefficiency, SemoDepth employs a Multi-View Scan Pyramid (MVSP) that matches the fusion operator to radar’s reach at each decoder resolution This pyramid consists of three tiers:

1. At the coarsest scales, it uses scene-wide four-direction RMS at the coarsest scales (a linear-cost approximation of a Transformer’s 2-D receptive field).

2. At the mid scale, RMS is restricted to windows centred on each projected radar return, so scan capacity is spent only where radar supports the image.

3. At the finest scales, RMS gives way to FiLM radar modulation at constant per-pixel cost, preserving an active radar signal into the full-resolution decoder without paying scan cost at the tiers where it grows fastest. An ablation shows that replacing these two scan tiers with FiLM degrades MAE@80 by 18%, confirming that the allocation is load-bearing for accuracy, not merely an efficiency optimisation.

Training Objective and Performance

The model is trained using a composite objective L which combines several losses: "a log-L1 Llog against Dsparse (falling back to Dmain when the sparse target is unavailable), a Huber loss Llin (δ = 5 m) against Dmain normalised by dmax, an L1 Lgrad on the x, y gradients of D˜pred − D˜ main, and a linear-metre L1 Lsparse against Dsparse normalised by dmax. SemoDepth achieves state-of-the-art performance on nuScenes, reducing MAE by 34.0%, 29.9%, and 29.9% over the previous best at 0–50, 0–70, and 0–80 m. Furthermore, out-of-scan feature blending adds no accuracy on top of RMS, providing empirical validation that in-scan selection can replace out-of-scan fusion".

Key Findings and Limitations

The results show that MAE improvements are more than twice as large as RMSE improvements at every range (34.0% vs. 15.0% at 0–50 m, 29.9% vs. 8.5% at 0–70 m, and 29.9% vs. 5.2% at 0–80 m). The paper notes that the gap to TacoDepth is the most informative cross-method comparison, suggesting that the consistent 30% MAE margin is explained by MVSP’s two distinguishing tiers. A limitation discussed is that Modality scope. SemoDepth is evaluated on front-camera + automotive radar only and Statistical variability. The main configuration in Table 1 is reported as the mean and standard deviation over three independently seeded training runs.

Broader Impacts

The positive impact is that Robust, low-cost, all-weather metric depth perception is a near-term enabler for safer driver assistance systems on vehicles that cannot accommodate the cost or power budget of automotive LiDAR. However, there are negative implications regarding Improved radar-camera scene understanding is dual-use, and the paper notes that "depth completion deployed at scale shares the standard concerns of automated perception — failures concentrated on rare scene types or atypical objects (e.g., pedestrians with unusual silhouettes or in non-Western dress codes) can have safety-critical consequences. The authors conclude that cross-modal learning in sequence models should be reconsidered from the perspective of selection rather than representation".

Compute and Reproducibility

The inference efficiency is competitive, as SemoDepth attains the lowest single-frame latency among methods with public releases (26.8 ms). The project compute accounts for approximately 13 GPU-days. The code is provided, and the reproduction recipe allows users to reproduce every quantitative result in Tables 1, 4, 8, and 5 end-to-end from the published nuScenes Caesar et al. [2020] and ZJU-4DRadarCam Li et al. [2024a] releases.

References

The paper references numerous prior works, including Li et al. [2024a], Dao and Gu [2024], and MambaDepth Grigore and Popa [26]? (Note: The reference list in the paper is numbered, but the citation style in the text uses bracketed numbers or author names; I am following the provided text structure for token extraction). The paper also cites work such as CaFNet Sun et al. [2024] and TacoDepth Wang et al. [2025]. The reference list itself is found on page 16, which contains citations like Li et al. [2024a]? (Note: I am using the provided structure for token identification).

--- Page 1 ---

Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation Zhangcheng Hou School of Science and Technology Keio University Yokohama 223-8522, Japan zhangcheng@ohtsuki.ics.keio.ac.jp Tomoaki Ohtsuki School of Science and Technology Keio University Yokohama 223-8522, Japan ohtsuki@ics.keio.ac.

1. The improvement is to integrate Radar-Modulated Selection (RMS) directly into selective state-space models (SSMs), as proposed by Radar-Modulated Selection (RMS), which injects radar inside Mamba’s Gu and Dao [2023] selective scan. This allows the model to modulate the selection parameters ∆t (per-token memory horizon) and C (state readout) directly with radar data, fundamentally changing how information is stored and propagated rather than merely how it is combined.

2. The improved system can achieve state-of-the-art depth estimation on nuScenes by replacing out-of-scan fusion with in-scan selection, as demonstrated by the finding that out-of-scan feature blending adds no accuracy on top of RMS. This results in a reduction of MAE by 34.0%, 29.9%, and 29.9% over the previous best at 0–50, 0–70, and 0–80 m while maintaining the lowest single-frame latency (26.8 ms).

3. The system can dynamically allocate computational resources across different scales using the Multi-View Scan Pyramid (MVSP), which is constructed to match the fusion operator to radar’s reach at each decoder resolution. This ensures that scan compute is concentrated where radar evidence has the longest reach, preventing inefficiency by allocating scene-wide four-direction RMS at the coarsest scales, radar-centred windowed RMS at the mid scale, and constant-cost feature-wise linear modulation (FiLM) at the finest scales.

Radar camera depth estimation turns a very sparse but metric radar signal into a dense depth map. Existing methods combine radar and image features outside the sequence operator of the backbone, through concatenation, confidence gating, sparse supervision, or graph based extraction. Mamba variants for multimodal fusion also leave the selection mechanism itself unimodal. We argue that selection is the right place for radar to enter. We introduce Radar-Modulated Selection (RMS), which injects radar inside the selective scan of Mamba. Radar adds learned perturbations, initialised at zero, to the step size Δ and the readout C. The input projection B and the state matrix A never see radar. At initialisation the block equals a pretrained image Mamba, so radar gains influence only where it improves accuracy. The construction also couples the two modalities at linear cost and falls back to the image backbone when radar is missing. We deploy RMS in a Multi-View Scan Pyramid (MVSP) that matches the fusion operator to the reach of radar at each scale. On nuScenes, SemoDepth reduces MAE by 34.0%, 29.9%, and 29.9% over the previous best within 50, 70, and 80 m. It also has the lowest single frame latency among methods with a public release (26.8 ms). A feature blend placed before the scan adds no accuracy on top of RMS, which supports the view that selection inside the scan can replace fusion outside it. When twice as many spurious radar returns as true ones are injected, the MAE@80 of SemoDepth rises by 1.4%, against 33.6% for a Mamba baseline that fuses radar before the scan.