Representation Before Weighting: Semantic Retention in Frozen 3D Foundation-Model Fusion

arXiv:2609.12099 · cs.CV · Submitted 2026-09-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Representation Before Weighting".

Jane: Here's the gist of the paper. The paper investigates how much useful semantic information is lost when heterogeneous frozen foundation models are collapsed to a single class before they can interact This study provides a mechanistic diagnosis of premature semantic collapse as an information bottleneck in modern 3D foundation-model composition.

Tom: Mechanistic Formulation of Collapse The central distinction explored is the ordering choice between reducing each source to its most likely class and then deciding which source to trust versus keeping semantic alternatives available until after sources have interacted This matters because sources are heterogeneous, meaning a dense 3D model can assign mass to several related classes while a sparse 2D model provides complementary evidence only for a subset of points.

Jane: The study uses RegionPLC as the frozen dense source and SAM3 as the primary sparse source to test this mechanism.

Tom: Matched Semantic-Information Intervention The authors introduce a "same-input top-k retention ladder" that fixes upstream sources, masks, geometry, scene sets, class vocabularies, and fusion operators while varying only how much semantic support survives before interaction.

Paper summary: Lu: The authors frame this as a concrete diagnostic for modern foundation-model composition, reframing how we think about classical fusion principles in this new regime. They are suggesting that distribution-level operators should operate on an information-rich semantic representation rather than just using a common hard top one control.

Tom: So the title itself really nails it: "Representation Before Weighting." It’s not just about the weights you apply; it's about what happens to the data representation *before* you start applying any of those weights.

Jane: It boils down to this: when combining models, we have to be careful not to throw away useful semantic context simply because we want a single final answer quickly. The paper shows that retaining a compact set of plausible alternatives is key for recovering that useful information.

Meng: From my side, if I'm deploying this at scale, the implication is that we don't need to run the full complexity of finding the absolute best score immediately; we can stop searching once we have enough semantic context retained.

Lalam: The study confirms that what matters isn't just one decision, but maintaining a set of hypotheses long enough for those different model types—the dense and sparse ones—to actually inform each other.

Tom: It’s a very mechanistic look at why these systems fail to combine well under certain conditions. They are showing us exactly where the information bottleneck is in this new field.

Conclusion: Tom: So, we're wrapping up this look at their work on "Representation Before Weighting: Semantic Retention in Frozen three dee Foundation-Model Fusion." Basically, they found that how you handle the information before you start combining models makes a huge difference.

Jane: It boils down to this: when you take these frozen foundation models and try to mash them together, the way you choose which semantic information stays around really dictates how much useful stuff they actually share.

Lu: The core finding is that premature semantic collapse happens all the time in these systems, and it’s caused by a specific choice about how you handle those inputs before they interact.

Meng: So what does that mean for my production pipeline? It means if we just use a hard top one control to pick our best class right away, we're probably throwing away too much potential information early on.

Tom: Exactly. They showed that if you keep a small set of plausible alternatives alive for longer, the models actually get better results when they finally talk to each other.

Jane: And what’s the actual number? On ScanNet200, they saw a gain of six point four when they relaxed that hard top one constraint. That’s not just theory; that's measurable improvement.

Lu: It proves the principle: you don't need to trust the single most likely class immediately if you want the best fusion outcome from different model types.

Tom: And it works across different data sets and even different ways they stack their sources. That’s a strong indicator that this isn't just some fluke on one dataset.

Jane: So, as science communicators, what does this actually change for us in how we build these complex AI systems? It suggests we need to rethink our basic arbitration rules for model combination.

Lu: The paper argues that the decision you make about which information survives needs to be based on a rich semantic representation, not just a simple top one list.

Meng: If we adopt this, it means we can potentially keep our candidate lists smaller and cheaper without losing much performance in the end.

Tom: It shifts the focus from just getting the final number to managing that internal set of plausible ideas during the process.

University of Ghana · Department of Computer Science

cs.CV

Submitted: 2026-09-10

Updated: 2026-11-15

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The paper investigates how much useful semantic information is lost when heterogeneous frozen foundation models are collapsed to a single class before they can interact This study provides a…

Key concepts

Premature Semantic Collapse
This occurs when heterogeneous models are forced to choose their most likely class too early. This happens because different sources (like dense 3D models and sparse 2D models) provide complementary evidence, and collapsing them prematurely discards valuable alternative semantic hypotheses that could help the final result.
Same-Input Top-K Retention Ladder
This is an experimental method used to systematically test how much semantic support survives before interaction. By fixing all inputs except the retention level (top-1 vs. top-k), researchers found that improving performance happens most significantly when they keep a small set of plausible alternatives available for longer.
Heterogeneous Frozen Foundation Models
These are different types of pre-trained models used in 3D tasks, such as a dense 3D model and a sparse 2D model. They have different strengths; one might assign mass to several related classes, while the other only provides evidence for a subset of points, making their interaction crucial.
GroundingDINO–SAM2.1 Source-Replacement Diagnostic
This diagnostic tool was used to isolate where semantic collapse happens. By replacing one sparse source with another and observing the performance change, researchers confirmed that preserving secondary semantic hypotheses in the sparse model is much better than collapsing that evidence to just the top-one prediction.

Terminology

Summary

The paper investigates how much useful semantic information is lost when heterogeneous frozen foundation models are collapsed to a single class before they can interact This study provides a mechanistic diagnosis of premature semantic collapse as an information bottleneck in modern 3D foundation-model composition. p. 0

Mechanistic Formulation of Collapse

The central distinction explored is the ordering choice between reducing each source to its most likely class and then deciding which source to trust versus keeping semantic alternatives available until after sources have interacted This matters because sources are heterogeneous, meaning a dense 3D model can assign mass to several related classes while a sparse 2D model provides complementary evidence only for a subset of points. The study uses RegionPLC as the frozen dense source and SAM3 as the primary sparse source to test this mechanism. p. 1

Matched Semantic-Information Intervention

The authors introduce a same-input top-k retention ladder that fixes upstream sources, masks, geometry, scene sets, class vocabularies, and fusion operators while varying only how much semantic support survives before interaction. The resulting top-k curves reveal a consistent pattern where the largest improvement appears immediately after relaxing top-1 collapse. p. 1 The evidence supports the principle to preserve a small set of plausible alternatives long enough for heterogeneous sources to interact rather than relying on full probabilities.

Where the Gains Come From

The empirical results show that full-distribution HM improves over matched top-1 by 6.40 points on ScanNet200 and by 3. p. 248 points on ScanNet++ with paired scene-bootstrap confidence intervals excluding zero. p. 2 The largest gain occurs immediately from top-1 to top-2. p. 0 The pattern is robust across datasets and source stacks, suggesting that most useful information is recovered by retaining a compact set of plausible alternatives, while unrestricted low-probability tails can introduce noise.

Source-Independence and Failure Analysis

A controlled GroundingDINO–SAM2.1 source-replacement diagnostic reproduces the retention effect, increasing HM from 14. p. 277 at top-1 to 18. p. 075 with full sparse retention. p. 0 This isolates semantic collapse in the replacement sparse stack, confirming that preserving its secondary semantic hypotheses remains substantially better than collapsing that same evidence to top-1. The study also shows that calibration and reliability weighting do not eliminate the semantic-retention advantage. p. 0

Conclusion and Implications

The conclusion is that the repeatable failure mode is premature semantic collapse in heterogeneous frozen-model systems. The findings suggest that an arbitration rule should operate on an information-rich semantic representation, as distribution-level operators already outperform a common hard Top-1 control. p. 0 The study reframes a classical fusion principle into a concrete diagnostic for modern foundation-model composition. p. 1

TIE-POLICY ROBUSTNESS

The frozen lowest-index policy is the strongest of the three Top-1 variants, yielding the smallest full retention gap. The sparse-favouring rule exposes the lower quality of the sparse Top-1 source, showing a gain of +14. p. 986 HM compared to top-1. p. 9

B FUSION-WEIGHT SENSITIVITY

Continuous source reweighting degenerates into a hard switch when using one-hot sources, as the retained distributions preserve graded within-class evidence. Full-distribution performance is stable across the entire grid of sparse weights, demonstrating that retained distributions preserve graded within-class evidence.

C HELD-OUT CALIBRATION DIAGNOSTICS

Calibration changes confidence reliability without changing source accuracy, indicating that calibration error and semantic information loss are distinct problems. The strong ECE reductions demonstrate the fitted calibration machinery is effective, but calibrated probability fusion still does not exceed uniform full-distribution averaging. p. 8

D COMPUTE AND MEMORY ACCOUNTING

The raw dense float16 representation is large, but chunked fusion is inexpensive compared with compressed-cache loading. The observed retention saturation around moderate k also suggests a practical route to reduce storage further by keeping only short candidate lists.

E POE NUMERICAL-FLOOR SENSITIVITY

Geometric pooling requires a numerical floor because the sparse distribution contains exact zeros, and HM changes by only 0.16 points over the entire range for small epsilon values. p. 1 The geometric operator is numerically stable to the zero-probability floor over eight orders of magnitude. p. 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve composition by introducing a matched semantic-information intervention, specifically a same-input top-k retention ladder that varies only how much semantic support survives before interaction. This allows for a controlled diagnosis of premature semantic collapse as a repeatable information bottleneck in heterogeneous frozen-model composition.

  2. Enable systems to recover useful cross-source complementary evidence by preserving a compact set of plausible alternatives long enough for heterogeneous sources to interact, rather than collapsing both sources to top-1 before they can interact.

  3. Develop inference-time logic that allows for the preservation of secondary semantic hypotheses, as this is shown by the observation that the largest gain appears immediately after relaxing top-1 collapse on datasets like ScanNet200.

  4. Implement a diagnostic pipeline that tests source dependence by replacing primary sparse sources, such as SAM3 with GroundingDINO–SAM2.1, to isolate semantic collapse specifically within the sparse foundation-model stack.

  5. Create robust fusion rules that are insensitive to arithmetic averaging alone by testing distribution-level operators like class-wise max pooling and geometric/log pooling, which can outperform a common hard Top-1 control while remaining stable against numerical floor sensitivity.

Sources

Related papers