CLON: Cue-Calibrated Linguistic Object Onboarding for Zero-Shot 6D Pose Front-Ends

arXiv:2609.04784 · cs.CV · Submitted 2026-09-04 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "CLON: Cue-Calibrated Linguistic Object Onboarding for Zero-Shot 6D Pose Front-Ends".

Jane: The gist The CLON method introduces Cue-Calibrated Linguistic Object Onboarding, a front-end requiring no task-specific training for new objects,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we know the paper is about using language memory and calibration, but what’s the actual title really telling us? It’s CLON: Cue-Calibrated Linguistic Object Onboarding for Zero-Shot 6D Pose Front-Ends >

Jane: The title points to two main things that matter. First, they are onboarding objects by using linguistic cues to build a semantic memory, which is used for generating proposals >

Lu: And the second part is about calibrating those scores to improve the proposal generation during the online phase, which means they adjust how much weight they give to different types of evidence >

Meng: So it sounds like they’re trying to make a front-end that doesn't need task-specific training for new objects and also ensures the proposals are scored based on whether semantic or geometric cues are more important for that specific set of objects >

Tom: Right, so they’ve got this process where you build a memory offline using templates, then use language to query it online to get better starting points >

Jane: Exactly. The implication is that instead of relying only on the model's general knowledge or a fixed set of training examples, we can anchor the proposals in actual linguistic descriptions of what the objects look like and how they behave >

Lu: This moves us away from just raw visual matching toward something more structured, where every object has a linguistic description that helps guide the search space for downstream solvers >

Meng: From my perspective on deployment, if this works well, it means we could rapidly onboard new tools or objects without retraining the entire pipeline from scratch >

The paper's summary: Tom: Let’s look at the actual mechanics of CLON. They separate the onboarding part—building that memory—from the online scene inference part, treating onboarding as constructing a front-end object-set memory >

Jane: That means they first take rendered templates of objects and use a vision language model to generate compact, view-consistent noun phrases that describe persistent properties of those objects >

Lu: These generated phrases become the linguistic semantic memory, which then serves only as top-down semantic anchors for generating proposals using a model like SAM three >

Meng: So, the idea is that instead of just looking at pixels in a scene, the model uses these language descriptions to generate proposals that are likely to contain those specific onboarded objects >

Tom: And they don't just use those linguistic anchors for finding things; they also compute object-set cue weights before scene inference to calibrate the relative importance of semantic, appearance, and geometric scores >

Jane: That’s a crucial part. They compute these weights based on the characteristics of the onboarded object set—for example, if there's enough visual variation in the set, they assign different importance to each cue type >

Lu: This calibration is dynamic; for instance, if appearance scores are high enough, they might assign full weight to semantics and zero weight to geometry because there’s enough visual information already present >

Meng: That sounds like a smart way to handle scenes that are very visually uniform or have low texture, where you'd normally need more geometric cues to find the object >

The paper's improvements: Tom: So what are the specific improvements they claim? I’m hearing about how this linguistic memory guides SAM three toward high-recall proposals for onboarded objects >

Jane: That’s true. The linguistic memory helps the model generate proposals that focus on regions likely to contain those specific objects, which improves recall over just using generic prompts >

Lu: They also have this object-set cue calibration mechanism where they compute fixed semantic, appearance, and geometric score weights from template statistics before scene inference >

Meng: That calibration is key because it addresses a limitation where fixed weight values don't account for whether the current set of objects is best separated by semantic or geometric evidence >

Tom: And they show results on seven BOP-Classic-Core datasets where CLON improves detection AP by eight point one percentage points and segmentation AP by six point two percentage points compared to CNOS and SAM-6D front-ends >

Jane: Plus, for the downstream 6D pose estimation, they get up to four point one percentage points improvement over CNOS and SAM-6D front-ends on those same datasets >

Conclusion: Tom: So to wrap up on CLON: they use this front end to improve proposal quality and then use those calibrated weights to assign an identity based on semantic, appearance, and geometric cues during online scoring >

Jane: That’s the gist of it. It’s a modular front-end that improves proposal quality and identity assignment before downstream pose estimation >

Lu: I think the most important part is that they compute those object-set cue weights before scene inference and keep them fixed during online scoring, which means it adapts to whether the object set has enough visual variation or not >

Meng: From an engineering standpoint, reducing the initial proposal pool from sixty-seven point three down to thirteen point six proposals per image is a big win because it cuts down on how much computation you have to do later >

Lalam: I can confirm that the full CLON prompt set performs best, with an AP mean of zero point five five one, which shows that using multiple compact phrases is more effective than just one long description >

Tom: So in conclusion, CLON successfully leverages linguistic semantic memory anchoring and dynamically calibrated cue weights to improve proposal quality and downstream pose AR under clutter and occlusion >

Jane: That’s the gist of it. It’s a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. We’re going to take a quick break, but when we come back, we'll look at the specifics of how those cue weights are calculated before they hit the online scoring pipeline >

Seojin Ji, Yoojin Kwon, Hyung-Sin Kim

Seoul National University

cs.CV

Submitted: 2026-09-04

Updated: 2026-10-14

Comments: 9 pages, 5 figures

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The CLON method introduces Cue-Calibrated Linguistic Object Onboarding, a front-end requiring no task-specific training for new objects, which constructs linguistic semantic memory and

Key concepts

Linguistic Semantic Memory (Lc): This memory is created offline by querying a Vision-Language Model (VLM) with fixed prompts to generate compact noun phrases describing an object's properties. These phrases act as top-down semantic anchors, helping the system understand what kind of object it is looking for in a scene without needing specific training data for that object.
Object-Set Cue Weights (wC): These are fixed weights assigned to semantic, appearance, and geometric scores derived from template statistics. The method dynamically computes these weights based on how discriminative the object set is. This calibration ensures that when scoring a proposal, the system correctly prioritizes whether a feature is more important for identifying the specific object or just generally looking for any object.
Semantic Proposal Generation: During online inference, this stage uses the linguistic memories (Lc) to guide a segmentation model (SAM 3). The model is prompted jointly with generic prompts and specific phrases from Lc. This directs the proposal generation towards regions likely containing the target objects, significantly reducing the initial search space.
Calibrated Proposal-Object Scoring: This final stage assigns an identity to each generated proposal by calculating a weighted sum of semantic, appearance, and geometric scores using the calibrated cue weights (wC). The object with the highest final score is then assigned as the correct target, ensuring accurate object identification before pose estimation.

Summary

The gist The CLON method introduces Cue-Calibrated Linguistic Object Onboarding, a front-end requiring no task-specific training for new objects, which constructs linguistic semantic memory and object-set cue weights to improve zero-shot 6D pose estimation performance.

How it works

CLON separates offline object onboarding from online scene inference by treating onboarding as the construction of a front-end object-set memory, where each onboarded object is represented by rendered templates Tc = (I v c, Dv c) The linguistic semantic memory Lc is constructed by querying a VLM with a fixed prompt template to generate compact, view-consistent noun phrases describing persistent object properties These parsed phrases are used as the linguistic semantic memory Lc, which serves only as top-down semantic anchors for SAM 3 proposal generation.

The method also computes object-set cue weights wC = (wsem, wapp, wgeo) that calibrate the relative importance of semantic, appearance, and geometric scores from template-derived statistics before scene inference. These weights are computed based on the object set characteristics; for instance, if eapp > τapp (where τapp = 0.25), CLON assigns (wsem, wapp, wgeo) = (1, 0, 0), indicating sufficient global visual variation. Otherwise, the weights are computed as (wsem, wapp, wgeo) = α(1 − eapp), eapp, (1 − α)(1 − eapp) based on template-derived inter-object discriminability and geometric discriminability.

Online Inference Pipeline

During online inference, CLON employs a two-stage process: Semantic Proposal Generation and Calibrated Proposal-Object Scoring. First, semantic proposal generation uses the linguistic memories Lc to guide SAM 3 [12] toward regions likely to contain onboarded objects, jointly prompting SAM 3 with generic prompts such as “object” and those from Lc. The resulting proposals P are then consolidated using IoU-based non-maximum suppression to produce a refined set of candidates.

Second, calibrated proposal-object scoring assigns an identity to each proposal p ∈ P by computing a final score Sfinal(p, c) = ¯wsemSsem(p, c) + ¯wappSapp(p, c) + ¯wgeoSgeo(p, c), where w¯j are the onboarding-time cue weights renormalized over the available cues. The final object assignment is determined by finding the object with the highest final score, c∗(p) = arg max c∈C Sfinal(p, c).

Key Contributions and Results

The primary contributions of CLON are threefold:

We introduce linguistic object onboarding for zero-shot 6D pose front-ends, converting rendered templates into compact semantic memories that guide promptable segmentation.
We propose onboarding-time object-set cue calibration, which computes fixed semantic, appearance, and geometric score weights from template statistics before scene inference.
We demonstrate the resulting front-end across proposal quality, downstream pose estimation, and real robot-observation scenes, showing that target-aware linguistic proposals and cue calibration improve zero-shot 6D pose pipelines under clutter and occlusion.

Empirical results show significant improvements:

**/2D Detection and Segmentation Results Table 1 shows CLON achieves the highest APmean on both tasks, reaching APdet of 0.551 and APseg of 0.543, which improves over the strongest baseline by +8.1 pp for detection and +6.2 pp for segmentation. Table 7 shows that CLON produces only 23 proposals on LM-O and 16 on ICBIN, achieving an approximate 85% reduction in the initial proposal pool compared to SAM-6D [2]. **

**/6D Pose Estimation Results Table 2 shows CLON obtains the highest ARmean with all downstream solvers: 0.634 with GigaPose, 0.726 with SAM-6D, and 0.748 with FoundationPose. Relative to the strongest non-CLON frontend for each solver, this corresponds to gains of +4.1, +3.0, and +1.4 pp, respectively. **

Robustness and Real-World Applicability

The method demonstrates strong robustness across varying levels of object visibility. Table 11 shows that in the 30-40% visibility bin on the YCB-V dataset, CLON achieves a 97.7% accuracy, significantly outperforming baselines which only reach 8.8%. Furthermore, in highly visible scenarios (90%-100% visibility), our method consistently achieves near-perfect accuracy across all core datasets, reaching 98.9% on LM-O and YCB-V. Real-world robot evaluation confirms this efficacy, where CLON achieves the highest APmean of 0.811 and a grasp success rate of 36.0% in cluttered scenes, outperforming SAM-6D [2] by over four times.

Computational Overhead

CLON substantially reduces the number of proposals compared with SAM-6D, from 67.3 to 13.6 proposals per image on average, which decreases the subsequent pose estimation time. While iterating SAM 3 over multiple target-prompt pairs increases detection latency on datasets with many target objects, online memory consumption remains comparable to SAM-6D across datasets. The onboarding process itself takes 73.7 s on average across datasets, compared to 34.9 s for the baseline.

The CLON method successfully leverages linguistic semantic memory anchoring and dynamically calibrated cue weights to improve proposal quality and downstream pose AR under clutter and occlusion. The framework's adaptive calibration allows the model to automatically find the most effective weight distribution for each target scene, bypassing the limitations of fixed weight values. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose inference.

The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation. The CLON method provides a modular front-end that improves proposal quality and identity assignment before downstream pose estimation.

Improvements for AI systems

The CLON system can perform zero-shot 6D pose estimation for novel objects without task-specific training by constructing a linguistic semantic memory for top-down proposal generation and object-set cue weights that calibrate the relative importance of semantic, appearance, and geometric scores. This allows the system to generate proposals that are target-aware and ensures that object identity is assigned by a mechanism where Every proposal is scored against the onboarded objects using semantic, appearance, and geometric cues.
The system can improve detection performance by using language to guide segmentation, as demonstrated by achieving an improvement of +8.1 percentage points (pp) on detection AP on seven BOP-Classic-Core datasets compared to CNOS and SAM-6D front-ends. This enhancement is achieved because linguistic object memories provide effective top-down anchors for SAM 3, producing target-relevant proposals while suppressing many semantically plausible distractors.
The system can handle cluttered and occluded scenes with a higher grasp success rate by leveraging the object-set cue calibration mechanism. This is shown by CLON achieving a grasp rate of 36.0%, 2.0× the strongest baseline in real-world robot observations, as it mitigates verification errors under severe occlusion.
The system can be adapted to various object categories and contexts by using structured prompt configurations for the Vision-Language Model (VLM). The paper details specific rules, such as Rule 2 accommodating an optional high-level metadata field, which allows the system to seamlessly integrate the 'industrial object' tag when available during onboarding.
The system can achieve superior proposal quality by using a full set of linguistic anchors for prompt generation. The ablation study shows that The full CLON prompt set performs best (0.551 APmean), indicating that using multiple compact, view-consistent phrases are more effective than a single concatenated descriptor.

Abstract

Novel-object 6D pose pipelines depend on front-end hypotheses that retain target instances under clutter and occlusion without overwhelming downstream pose estimation. We introduce Cue-Calibrated Linguistic Object Onboarding (CLON), a training-free proposal-and-matching front-end. CLON converts multi-view CAD renderings into compact linguistic memories that provide object-specific conditioning to a frozen proposal generator. Constructed offline and reused across scenes, these memories augment generic prompts before segmentation; subsequent template verification assigns object identity. As a complementary onboarding step, CLON derives fixed cue weights from template statistics for semantic, appearance, and geometric verification. On seven BOP-Classic-Core datasets, CLON improves detection AP by 8.2 percentage points (pp), segmentation AP by 6.2 pp, and downstream pose AR by up to 4.1 pp over the evaluated CNOS and SAM-6D front-ends. In a real-world localization-to-grasp study under clutter and occlusion with 100 trials per method, CLON's improved localization doubles grasp success relative to the strongest evaluated baseline. Controlled ablations evaluate linguistic conditioning, multi-view construction, and verification. Analysis of a smaller candidate set with higher class-agnostic recall on five datasets.