RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection".
Jane: Here's the gist of the paper. Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry We find that current depth foundation models, despite their strong zero-shot generalization, lack the object-level precision 3D detection demands: substituting a state-of-the-art depth foundation model for a strong detector’s predicted depth degrades accuracy, even falling below the detector’s own prediction Rather than pushing detectors or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model that corrects depth without ever predicting a numerical value Our key insight is that depth error has a direct visual signature in image space: when projected onto the image, a correctly placed box tightly encloses the object, while a too-far box projects too small and a too-close box projects too large Depth refinement thus reduces to a visual alignment problem rather than a metric regression problem, which we instantiate by extending the VLM’s vocabulary with action tokens that replace numerical depth output with categorical decisions, and by supervising the model on a large-scale chain-of-thought dataset that grounds each decision in explicit visual evidence Applied as a single post-hoc step, RefineAny3D delivers consistent gains across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools, and generalizes to novel categories, scenes, and cameras without retraining.
Tom: The core problem addressed is the lack of object-level precision in current depth foundation models for monocular 3D detection This limitation means that substituting a state-of-the-art depth foundation model for a strong detector’s predicted depth degrades accuracy, even falling below the detector’s own prediction.
Jane: The authors propose RefineAny3D, which reframes depth refinement as a semantic alignment problem in image space rather than a metric regression problem.
Tom: Reframing Depth as Semantic Alignment The key insight is that once the candidate 3D box is projected onto the image plane, the camera intrinsics and metric scale are absorbed into the projection itself, and what remains is a purely 2D question: does the projected box tightly enclose the target object To realize this view, we cast depth refinement as a conditional generation problem over a VLM’s token sequence.
Paper summary: Tom: So we're looking at this paper called RefineAnythree dee: Depth Refinement as Semantic Alignment for Monocular three dee Detection <ref:2608.09147#pg1,RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection>. Basically, the big problem they're tackling is how current depth foundation models struggle with getting that fine-grained object-level precision that three dee detection really demands <ref:2608.09147#pg1>.
Jane: Right, so what does that mean for us on the ground? It means if you use a super strong detector to guess a three dee box, and then you plug in another state-of-the-art depth foundation model to give it the final depth value, that whole system gets worse <ref:2608.09147#pg1,state-of-the-art depth foundation model>.
Lu: That's the core issue they pinpoint: substituting a top depth foundation model for an already strong detector’s prediction actually degrades accuracy, sometimes falling below what the detector itself predicted >
Meng: So instead of trying to make both parts better all at once, they treat this object-level depth refinement as its own separate task and propose RefineAnythree dee, which is a vision language model that fixes the depth without ever predicting a specific number >
Tom: That’s interesting because it shifts the focus away from metric regression. What’s their main idea for fixing the problem then?
Jane: The key insight they have is that depth error has a direct visual signature in the image space. When you project a box onto an image, if it's placed correctly, it tightly encloses the object. If it's too far or too close, the projected box looks distorted in a specific way >
Tom: So they’re treating depth correction as a visual alignment problem instead of just trying to solve a math regression problem. How do they actually do that?
Lu: They turn it into a conditional generation problem over the vision language model's token sequence. At each step, the candidate box is projected onto the image and rendered as a wireframe overlay >
Jane: And this refinement process is driven by these action tokens they add to the model's vocabulary. These aren't numbers; they are categorical decisions about what to do next >
Meng: They extend the VLM vocabulary with six new special tokens. Three for direction and three for magnitude, each tuned for a specific depth adjustment, defining a closed output space that makes decoding easier >
Tom: So if the box is too big, the model can pick a direction token like <depth farther> and a magnitude token like <step large>, which tells it how much to adjust based on the object's own size >
Paper summary: Jane: And then the depth update becomes this calculation: you take your current depth and add the change calculated from that direction token multiplied by that magnitude token, scaled by some factor related to the object's characteristic size >
Lu: They train this in two stages. Stage one stabilizes those new action-token embeddings before they start joint optimization with the main language model >
Meng: Stage two is where they jointly fine-tune the language model backbone and those action tokens, while keeping the vision encoder frozen to keep those general visual priors >
Tom: And what about the training data? How do you get enough examples for this kind of specific token decision making?
Jane: They construct a training set from public three dee images, but they also synthesize about three million depth-perturbation samples <ref:2608.09147#pg1>. These samples span all the action-token outcomes, which is important because it teaches the model all those directions >
Lu: It’s interesting how they structure the training signal. The target string for each sample is basically the reasoning prose followed by a canonical action line that combines direction and magnitude tokens >
Tom: So when you run this in practice, what does an end-to-end refinement look like? Is it instantaneous?
Jane: It’s not always instant, but the iterative refinement loop shows that most cases get corrected in just one step. For some harder cases, the model applies multiple action steps until the projected box aligns with the target >
Meng: They report three metrics to show where things are failing: Direction accuracy, Full action accuracy, and Depth error. This helps them diagnose whether the model is struggling with deciding which direction to go or just how much to move along that direction >
Tom: The results they show across the different tasks are pretty encouraging. They claim consistent gains across closed-set detectors, open-vocabulary detectors, and even three dee auto-labeling tools without needing any retraining > <ref:2608.09147#pg1>
Jane: Yes, they report lifting APthree dee by four point three five overall just through this depth refinement alone when applied on top of DetAnythree dee >
Lu: And when refining the pseudo-labels for auto-labeling, they see improvements of two point one nine on Easy, two point three seven on Moderate, and zero point seven two on Hard >
Meng: It’s good that it generalizes to novel categories and scenes without retraining because that means we don't have to rebuild the whole system when we find a new object or drive in a new area >
Tom: So, what’s the actual limitation they admit? Because I know these kinds of visual alignment tricks usually break when things get really weird >
Paper summary: Jane: They flag that this method relies on the visual signature of depth misalignment being readable from the image. If an object is heavily occluded or truncated, that cue weakens and the refinement becomes less reliable, especially when you have tight three dimensional intersection measurements >
Lu: And they also note a scope condition: it depends on the upstream detector providing a reasonable initial three dee box to start with > <ref:2608.09147#pg1>
Meng: So for deployment right now, it means you still need a decent starting point from whatever your initial detector is, and if the object is totally hidden, this method might just not work as well >
Tom: It sounds like they’ve successfully made depth refinement a practical capability that plugs right into existing monocular three dee systems without requiring massive retraining on the entire perception stack > <ref:2608.09147#pg1>
Jane: Exactly. It establishes three dee box refinement as something orthogonal to existing detectors, which is a big shift in how we think about improving those systems > <ref:2608.09147#pg1>
Lu: The idea of recasting depth refinement from metric regression into semantic alignment in image space is what really stands out; it’s moving the problem from continuous numbers to visual perception >
Meng: It’s a lot of work, though. They took about ninety hours for Stage two training on just one configuration, which tells you that getting those action tokens right takes serious tuning >
Tom: Yeah, it does. But the idea is that once you have those tokens tuned, the iterative refinement loop is pretty efficient in practice >
Jane: So, when we look at the title again, RefineAnythree dee: Depth Refinement as Semantic Alignment for Monocular three dee Detection—it really frames the solution as a specific way to solve a problem that has been around for a long time > <ref:2608.09147#pg1,RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection>
Lu: It positions refinement not as fixing a broken metric regression, but as adding a new capability that complements existing Monothree dee systems >
Meng: It’s plug-and-play, which is what matters for the folks shipping models to production because it lowers the barrier to adopting this kind of improvement >
Tom: So for anyone working on autonomous driving or robotics, this suggests you can take an existing detector and add RefineAnythree dee on top without having to retrain everything from scratch >
Jane: It definitely suggests that improving object-level depth precision can make those systems safer and more reliable, especially in those open-vocabulary scenarios where current detectors are weakest >
Conclusion: Tom: So we’re wrapping up our look at RefineAnythree dee, which basically frames depth refinement not as solving a math regression problem, but as a visual alignment problem in image space.
Jane: Right, so what does that title really mean for us on the ground when you're just looking at a picture?
Tom: It means they’re treating the error visually. If you project your three dee box onto the camera lens, if it looks too big or too small in that 2D projection compared to what’s actually there, that tells you exactly how to fix it > <ref:2608.09147#pg1>
Lu: And instead of just outputting a number like "depth is ten meters," this model outputs categorical decisions about direction and magnitude based on that visual look >
Meng: From an engineering standpoint, it’s neat because they don't need to predict the exact physical measurement; they just need to align the shape correctly >
Jane: But how does this actually change things for someone who doesn't read papers? Does it mean a better self-driving car or something?
Tom: It means that even if you use a strong detector, adding this step on top can lift the overall accuracy by four point three five percentage points just through that depth refinement alone >
Lu: The implication is that you don't have to retrain your entire perception stack from scratch to get better object-level precision >
Jane: So, if I’m driving and my car sees a truck, this method could make the localization of that truck much tighter, which means safer trajectory predictions down the road >
Tom: Exactly. And it works across different types of detectors—closed-set ones like DetAnythree dee, and open-vocabulary systems too—without needing retraining >
Meng: The caveat they give is that it relies on that initial box being reasonable; if your starting guess is totally wrong, the refinement won't help much >
Jane: That makes sense. So we’ve seen how this works and the big gains it brings, but what are the real limits of this approach?
Zhihao Zhang, Gengwei Zhang, Tianlong Chen, Xiaoming Liu
Michigan State University · University of North Carolina at Chapel Hill
cs.CV
Submitted: 2026-08-10
Updated: 2026-11-08
Comments: Accepted to NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary…
Key concepts
- Object-Level Precision
- This refers to the need for 3D detectors to accurately place objects in space, not just recognize them. Current models often fail at this fine-grained placement. RefineAny3D aims to fix this by ensuring that the predicted 3D bounding box tightly encloses the actual object in image space, addressing the core limitation of existing depth foundation models.
- Semantic Alignment Problem
- The paper reframes depth correction as a visual alignment task. Instead of calculating exact numerical depths, it asks a vision-language model if the projected 2D box tightly encloses the target object. This shifts the focus from complex mathematical regression to a simpler visual check based on whether the box is too far or too close.
- Action Tokens
- The VLM's vocabulary is extended with special tokens that replace numerical depth outputs with categorical decisions. These tokens encode qualitative outcomes like 'depth_closer' or 'depth_farther,' allowing the model to make discrete adjustments to the 3D box position based on visual evidence, simplifying the refinement process.
Terminology
Summary
Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry We find that current depth foundation models, despite their strong zero-shot generalization, lack the object-level precision 3D detection demands: substituting a state-of-the-art depth foundation model for a strong detector’s predicted depth degrades accuracy, even falling below the detector’s own prediction Rather than pushing detectors or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model that corrects depth without ever predicting a numerical value Our key insight is that depth error has a direct visual signature in image space: when projected onto the image, a correctly placed box tightly encloses the object, while a too-far box projects too small and a too-close box projects too large Depth refinement thus reduces to a visual alignment problem rather than a metric regression problem, which we instantiate by extending the VLM’s vocabulary with action tokens that replace numerical depth output with categorical decisions, and by supervising the model on a large-scale chain-of-thought dataset that grounds each decision in explicit visual evidence Applied as a single post-hoc step, RefineAny3D delivers consistent gains across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools, and generalizes to novel categories, scenes, and cameras without retraining. p. 0 RefineAny3D: Depth Refinement as Semantic
Alignment for Monocular 3D Detection
Zhihao Zhang1 Gengwei Zhang2 Tianlong Chen2 Xiaoming Liu1,2
1Michigan State University 2University of North Carolina at Chapel Hill
Abstract
Monocular 3D object detection spans two regimes: closed-set detectors operating
within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry. We find
that current depth foundation models, despite their strong zero-shot generalization,
lack the object-level precision 3D detection demands: substituting a state-of-the-art
depth foundation model for a strong detector’s predicted depth degrades accuracy,
even falling below the detector’s own prediction.Where the paper says it
The core problem addressed is the lack of object-level precision in current depth foundation models for monocular 3D detection This limitation means that substituting a state-of-the-art depth foundation model for a strong detector’s predicted depth degrades accuracy, even falling below the detector’s own prediction. p. 0 RefineAny3D: Depth Refinement as Semantic
Alignment for Monocular 3D Detection
Zhihao Zhang1 Gengwei Zhang2 Tianlong Chen2 Xiaoming Liu1,2
1Michigan State University 2University of North Carolina at Chapel Hill
Abstract
Monocular 3D object detection spans two regimes: closed-set detectors operating
within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry. We find
that current depth foundation models, despite their strong zero-shot generalization,
lack the object-level precision 3D detection demands: substituting a state-of-the-art
depth foundation model for a strong detector’s predicted depth degrades accuracy,
even falling below the detector’s own prediction. Our key insight is that depth error has a direct visual signature in image space, which lets us
recast refinement from metric regression into a semantic alignment problem that a vision-language
model can solve directly. We instantiate this view as RefineAny3D, which refines object depth through
categorical action tokens grounded in chain-of-thought visual reasoning, without ever predicting a
numerical depth value.Where the paper says it
Where the paper says it
Reframing Depth as Semantic Alignment
The key insight is that once the candidate 3D box is projected onto the image plane, the camera intrinsics and metric scale are absorbed into the projection itself, and what remains is a purely 2D question: does the projected box tightly enclose the target object To realize this view, we cast depth refinement as a conditional generation problem over a VLM’s token sequence. At step t, the candidate box b(t) is projected onto the image and rendered as a wireframe overlay. p. 3 To realize this view, we cast depth refinement as a conditional generation problem over a VLM’s
token sequence. At step t, the candidate box b
(t)
is projected onto the image and rendered as a
wireframe overlay:
I
(t) = render
I, π(b
(t)
; K)
, (1)
where π(b; K) projects the eight corners of b to 2D image coordinates using K.Where the paper says it
Instantiating RefineAny3D with Action Tokens
The authors extend the VLM’s vocabulary with six new special tokens (three for direction, three for magnitude), each carrying its own learnable embedding tuned for depth adjustment, defining a closed output space that simplifies decoding and supervision. Direction tokens encode the qualitative outcome of the alignment check: ⟨depth closer⟩ when the projected box is too small (the box should be moved closer to the camera, decreasing depth), ⟨depth farther⟩ when it is too large (the box should be moved farther away, increasing depth), and ⟨depth ok⟩ when the projection already aligns with the target. p. 4 Direction tokens. The direction token ad ∈ {⟨depth_closer⟩, ⟨depth_ok⟩, ⟨depth_farther⟩}
encodes the qualitative outcome of the alignment check: ⟨depth_closer⟩ when the projected box
is too small (the box should be moved closer to the camera, decreasing depth), ⟨depth_farther⟩
when it is too large (the box should be moved farther away, increasing depth), and ⟨depth_ok⟩ when
the projection already aligns with the target. 5 m shift is negligible for a distant truck but catastrophic for a nearby cup. We
therefore tie magnitudes to the object’s own characteristic size sobj = (w + h + l)/3, the arithmetic
mean of the box’s 3D dimensions, so that step sizes are relative to the object scale:
mag(am, sobj) = αam sobj, α⟨step_small⟩ < α⟨step_medium⟩ < α⟨step_large⟩
. (5)
This scale-relative design keeps the same action semantically consistent across objects of different
sizes, which is important for generalization across categories and scenes. The full action is then
a
(t) = (a
(t)
d
, a
(t)
m ), and the depth update becomes
∆d
(t) = dir(a
(t)
d
) · mag(a
(t)
m , sobj).Where the paper says it
Where the paper says it
Where the paper says it
Training and Inference Protocol
RefineAny3D is trained in two stages, both using single-step supervision. Stage 1 initializes each new action-token embedding to the mean embedding of a short phrase describing its alignment semantics, freeze the VLM, and train only these six embeddings to stabilize them before joint optimization. p. 4 3 Training and Inference
RefineAny3D is trained in two stages, both using single-step supervision: given a noisy box paired
with its ground-truth chain-of-thought and target action tokens, the model predicts (ad, am) in one
pass. In Stage 1, we initialize each new action-token embedding to the mean embedding of a short
phrase describing its alignment semantics, freeze the VLM, and train only these six embeddings to stabilize them before joint optimization. In Stage 2, we jointly fine-tune the LLM backbone and
the action-token embeddings while freezing the vision encoder, preserving the general visual priors
from VLM pretraining that we find essential for generalization across novel categories, scenes, and
cameras. Given a noisy box, RefineAny3D projects it onto the
image, queries a vision-language model to judge how the projection misaligns with the target object,
and updates the depth based on a discrete action emitted by the model. The same step is applied
iteratively, so that fine-grained corrections emerge from chaining categorical decisions rather than
regressing a continuous metric quantity.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The performance improvements are driven by the visual signature of depth error, where a too-close box too large (left), while a correctly placed box tightly encloses the target object (right). The iterative refinement loop shows that most cases are corrected in a single step; for some cases the model applies multiple action steps until the projected box aligns with the target. p. 7 RefineAny3D refines DetAny3D [72]’s
initial prediction (red) toward the refined result (green). Most cases are corrected in a single step; for
some cases the model applies multiple action steps until the projected box aligns with the target. 5 compares our default against natural alternatives to support the design choices made
in Secs. We report DirAcc (direction prediction accuracy) and FullAcc (both direction and
magnitude correct).Where the paper says it
Where the paper says it
Impact Across Downstream Tasks
RefineAny3D consistently improves closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools across novel categories, scenes, and cameras without retraining. Specifically, applying RefineAny3D on top of DetAny3D conditioned on ground-truth 2D boxes lifts AP3D by +4. p. 6 RefineAny3D refines
DetAny3D [72] conditioned on ground-truth 2D boxes, the strongest variant of DetAny3D, evaluated
across the six Omni3D sub-datasets. RefineAny3D lifts AP3D by +4. RefineAny3D lifts AP3D by +4. 35 overall through depth
refinement alone, demonstrating that the gain holds even on top of an oracle-grade detector. 4 reports KITTI validation AP3D for detectors trained on
LabelAny3D’s pseudo-labels with and without RefineAny3D refinement. Refining the pseudo-labels
lifts the downstream detector by +2. Refining the pseudo-labels
lifts the downstream detector by +2. 72 on Easy / Moderate / Hard, narrowing
the gap to GT-trained accuracy.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Directions
RefineAny3D assumes the visual signature of depth misalignment is readable from the image; for heavily occluded or truncated objects this cue weakens and refinement becomes less reliable, particularly under tight 3D IoU thresholds. Future work could combine visual alignment with complementary cues such as scene context or temporal consistency. p. 2 Visual Prompting and Semantic Alignment. Recent work shows that multimodal models can
benefit from explicit visual cues rendered directly on top of the input image. We
formalize this as the task of 3D box refinement: given an image and a candidate 3D box from any
upstream source (a closed-set detector, an open-vocabulary detector, or an auto-labeling tool), output
a refined 3D box whose object depth is closer to the ground truth, leaving other attributes untouched. A successful refinement module is plug-and-play, depends on no specific upstream architecture, and
generalizes across diverse object categories, scene types, and camera setups.Where the paper says it
Where the paper says it
Experimental Rigor and Resources
All experiments use 8 NVIDIA H100 80GB GPUs with bfloat16 precision and FlashAttention-2. Stage 1 takes approximately 1 hour and Stage 2 takes approximately 90 hours per configuration. p. 5 2, and stratified
so that all action-token outcomes are uniformly represented. This yields approximately 3. We find
that current depth foundation models, despite their strong zero-shot generalization,
lack the object-level precision 3D detection demands: substituting a state-of-the-art
depth foundation model for a strong detector’s predicted depth degrades accuracy,
even falling below the detector’s own prediction. Rather than pushing detectors
or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model
that corrects depth without ever predicting a numerical value. Rather than pushing detectors
or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model
that corrects depth without ever predicting a numerical value. Our key insight is
that depth error has a direct visual signature in image space: when projected onto
the image, a correctly placed box tightly encloses the object, while a too-far box
projects too small and a too-close box projects too large.Where the paper says it
Where the paper says it
Where the paper says it
Data Curation Details
The training set is constructed from public 3D-annotated images in three stages: source dataset selection (Sec. D.1), per-annotation filtering (Sec. p. 5 This diversity is essential for
RefineAny3D’s generalization across categories, scenes, and cameras. Annotation filtering. For each filtered annotation, we synthesize noisy 3D boxes by perturbing the
GT depth along the camera ray while keeping all other attributes fixed. Perturbation magnitudes are
tied to the object’s own size sobj to match the object-relative action design in Sec. Each box is parameterized as
b = (x, y, z, w, h, l, θ), where (x, y, z) is the object center in the camera coordinate frame, (w, h, l)
are the box dimensions, and θ is the yaw angle. We write d = z for the object depth, the component
of the center along the camera’s optical axis. Each box is parameterized as
b = (x, y, z, w, h, l, θ), where (x, y, z) is the object center in the camera coordinate frame, (w, h, l)
are the box dimensions, and θ is the yaw angle. We write d = z for the object depth, the component
of the center along the camera’s optical axis. 5, left): free-form generation introduces
unnecessary lexical variability, making the supervision target sparse and harder to align consistently
across samples. Joint tokens, which dedicate one special token to each action, recover most of the
gap on DirAcc (86. 5, left): free-form generation introduces
unnecessary lexical variability, making the supervision target sparse and harder to align consistently
across samples. Joint tokens, which dedicate one special token to each action, recover most of the
gap on DirAcc (86.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Safeguards and Ethical Considerations
The research conforms to the NeurIPS Code of Ethics. Our model and curated data pose no high risk for misuse beyond what is already enabled by existing publicly available 3D detection systems and pretrained VLMs. p. 2 In parallel, open-vocabulary methods instead generalize Mono3D beyond a
fixed category set by scaling 3D training data or transferring foundation-model priors. Specifically,
Omni3D [3] unifies existing 3D detection datasets into a single large-scale benchmark [29], autolabeling pipelines [69, 58] turn unlabeled images into pseudo-annotated 3D data via 3D-aware
foundation models [55], and methods such as OVMono3D [68], 3D-MOOD [67], and DetAny3D [72]
couple 2D vision foundation models [55, 51, 33] with depth foundation models [42, 55] to recognize
and localize novel categories in unseen domains [59]. In parallel, open-vocabulary methods instead generalize Mono3D beyond a
fixed category set by scaling 3D training data or transferring foundation-model priors. Specifically,
Omni3D [3] unifies existing 3D detection datasets into a single large-scale benchmark [29], autolabeling pipelines [69, 58] turn unlabeled images into pseudo-annotated 3D data via 3D-aware
foundation models [55], and methods such as OVMono3D [68], 3D-MOOD [67], and DetAny3D [72]
couple 2D vision foundation models [55, 51, 33] with depth foundation models [42, 55] to recognize
and localize novel categories in unseen domains [59].Where the paper says it
Where the paper says it
New Assets Introduced
We introduce two new assets: (i) a curated training dataset of ∼ 3M depthperturbation samples with chain-of-thought annotations; and (ii) the Refine3D benchmark, documented in Appendix E. Both will be released with documentation upon acceptance.
Conclusion
By improving the object-level depth precision of existing 3D detectors, RefineAny3D can make downstream perception systems safer and more reliable. In autonomous driving, more accurate object localization translates directly into better trajectory prediction and collision avoidance, particularly in the long-tail open-vocabulary regime where current detectors are weakest. The method’s plug-and-play design lowers the barrier to adopting these improvements, since practitioners can integrate it on top of existing detectors without retraining the underlying perception stack. p. 2 Because refinement depends only on the local alignment between a projected box and its target
object, RefineAny3D generalizes across categories, scenes, and cameras without retraining. We show
that RefineAny3D, applied as a plug-and-play module, consistently improves the depth accuracy of
closed-set 3D detectors, open-vocabulary 3D detectors, and 3D auto-labeling tools alike, establishing
3D box refinement as a practical capability that complements existing Mono3D systems.Where the paper says it
Pruning redundant reasoning tokens
The authors deliberately keep the chain-of-thought prefix that the model produces before emitting an action token. The CoT acts as a language-anchored intermediate signal: the model articulates which object it is looking at, what visual cue it is reading, and which direction the cue implies, before committing to a discrete action. p. 2 Although our RefineAny3D follows the broader idea of reasoning over an overlaid visual cue, we use
the projected 3D box as an alignment indicator of how the candidate depth should be refined. Vision-Language Models for 3D Understanding.Where the paper says it
How the training signal is built
The target string for each sample is the concatenation of the generated reasoning prose and the canonical Action line. The cross-entropy loss is applied uniformly over both. p. 5 Perturbation magnitudes are
tied to the object’s own size sobj to match the object-relative action design in Sec. 2, and stratified
so that all action-token outcomes are uniformly represented.Where the paper says it
The training data curation pipeline includes a two-stage filter, combining geometric and VLM-based semantic checks. The third stage synthetically perturbs each GT box along the camera-frame depth axis to generate approximately 3.0M (image, target-tokens) training samples that span all action-token outcomes.
The CoT acts as a language-anchored intermediate signal: the model articulates which object it is looking at, what visual cue it is reading, and which direction the cue implies, before committing to a discrete action.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Refinement of Monocular 3D Detection Pipelines: The system can act as a plug-and-play module that corrects object depth without retraining upstream detectors, consistently improving accuracy across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools.
-
Semantic Alignment over Metric Regression: The system reformulates depth refinement from a metric regression problem into a semantic alignment problem in image space by using action tokens to guide iterative visual alignment based on the
visual signature of depth error.
-
Handling Novel Categories and Scenes: By leveraging vision-language models for reasoning grounded in explicit visual evidence, the system generalizes to novel categories, scenes, and cameras without retraining, overcoming the lack of category-specific geometric priors in depth foundation models.
-
Robustness to Upstream Errors: The system can correct propagated bias from depth foundation models or noisy 2D predictions by applying refinement on top of an oracle-grade detector (like DetAny3D conditioned on ground-truth 2D boxes), demonstrating that
refinement signal does not require clean upstream input.
-
Enhanced Precision in Open-Vocabulary Detection: The system achieves object-level precision for open-vocabulary 3D detection by iteratively recovering fine-grained accuracy through a sequence of discrete action tokens, moving beyond the limitations of current depth foundation models which
lack the object-level precision 3D detection demands.
Abstract
Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry. We find that current depth foundation models, despite their strong zero-shot generalization, lack the object-level precision 3D detection demands: substituting a state-of-the-art depth foundation model for a strong detector's predicted depth degrades accuracy. Rather than pushing detectors or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model that corrects depth without ever predicting a numerical value. Our key insight is that depth error has a direct visual signature in image space: when projected onto the image, a correctly placed box tightly encloses the object, while a too-far box projects too small and a too-close box projects too large. Depth refinement thus reduces to a visual alignment problem rather than a metric regression problem, which we instantiate by extending the VLM's vocabulary with action tokens that replace numerical depth output with categorical decisions, and by supervising the model on a large-scale chain-of-thought dataset that grounds each decision in explicit visual evidence. Applied as a single post-hoc step, RefineAny3D delivers consistent gains across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools, and generalizes to novel categories, scenes, and cameras without retraining. Code is available at https://github.com/alanzhangcs/RefineAny3D.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
- G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
- 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
- OpenVLA: An Open-Source Vision-Language-Action Model
- Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- DINOv2: Learning Robust Visual Features without Supervision
- MonoDGP: Monocular 3D Object Detection with Decoupled-Query and Geometry-Error Priors
- VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
- EVA-CLIP: Improved Training Techniques for CLIP at Scale
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models
- Qwen3 Technical Report
- Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
- LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model
- Detect Anything 3D in the Wild
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs