The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models
summary
The gist
Auxiliary signal pathways in Vision-Language Models are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit…
In short
Researchers tested adding learnable gates to auxiliary signal pathways in Vision-Language Models. They found that these gated pathways become 'behaviourally closed,' meaning accuracy doesn't drop when the signal is removed during testing, even if the gate is partially open. This suppression happens because the signals either cancel each other out or actively harm performance.
Key concepts
- Suppression Phenomenon
- This occurs when an auxiliary signal pathway, controlled by a learnable gate, becomes ineffective. It's caused by two main reasons: either the image-derived signal is so similar to the text that it cancels out (dead-gradient regime), or the auxiliary signal actually makes the model perform worse (negative-utility regime).
- Dead-Gradient Regime
- This mechanism happens when signals derived from images are largely invariant to changes in text. Because these image signals don't provide a strong, new direction for learning, they nullify the first-order gradient that would normally drive the model's improvement.
- Negative-Utility Regime
- In this scenario, the auxiliary signal is detrimental to the model's overall performance. Instead of helping, it actively pushes the loss function in a way that forces the gate parameter toward zero, effectively shutting down its influence on training.
Terminology used across episodes
This episode discusses
- The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models · Paper Radio
- HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
The paper
The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models · Read on arXiv
Moshiur Farazi, Sameera Ramasinghe, Bekir Sait Ciftler, Mahbub Ahmed, Turza, Shafin Rahman
University of Doha for Science and Technology · Pluralis Research Australia · North South University Dhaka
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "The Gate Always Closes".
Jane: Here's the gist of the paper. Auxiliary signal pathways in Vision-Language Models are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit; we find that across five injection designs, every gated pathway becomes behaviourally closed, with accuracy invariant to ablating the pathway at inference even when the gate parameter would nominally pass 30–45% of the signal <ref:2607.23335#pg2> The suppression phenomenon is attributed to two regimes: a dead-gradient regime formalised through the caption-invariance of image-derived signals, and a negative-utility regime in which the auxiliary signal actively hurts the loss <ref:2607.23335#pg3> Rather than fight suppression, we exploit it by regularising LoRA fine-tuning with geometric auxiliary losses from hyperbolic visual relational graphs <ref:2607.23335#pg2> This finding emerged from a broader investigation of how auxiliary signals interact with a frozen LLM when routed through learnable pathways <ref:2607.23335#pg2> The Suppression Phenomenon Every configuration in which a learnable scalar controls the strength of an auxiliary pathway ends up behaviourally closed: task accuracy is essentially invariant to ablating the pathway at inference <ref:2607.23335#pg2> This is attributed to two distinct mechanisms <ref:2607.23335#pg2>: 1.
Tom: A dead-gradient regime, where caption-invariance of image-derived signals nulls the first-order discriminative gradient <ref:2607.23335#pg2>.
Tom: A negative-utility regime in which the auxiliary signal actively hurts the loss and the gradient consistently drives the gate toward its magnitude-zero limit <ref:2607.23335#pg2>.
Paper summary: Tom: So we're looking at this paper today, "The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models." It’s about how these auxiliary signal pathways in vision language models are fitted with learnable gates so the optimizer decides how much of that signal to admit. The main finding here is pretty striking: across five different ways they inject this auxiliary signal, every single gated pathway ends up being behaviorally closed. This means the accuracy stays basically the same even when you turn off that pathway when you actually use the model for inference, even when the gate parameter would normally pass about thirty to forty-five percent of the signal <ref:2607.23335#pg1,inference, even when the gate parameter would>.
Jane: That sounds like something that would be very interesting for us to talk about because it directly addresses how these models use their auxiliary information during actual use. So, what does this paper claim is causing this suppression phenomenon? Because when you look at the abstract, it mentions two specific regimes driving why the gate almost always decides on zero.
Lu: It’s attributed to two distinct regimes and we need to unpack those for the listener. First, there's a dead-gradient regime where caption invariance of image derived signals just nullifies that first order discriminative gradient. That means the signal isn't strong enough to move the gate much at all because it’s already predictable from the image itself.
Meng: And then there’s the second regime, which is a negative utility regime where that auxiliary signal actually hurts the loss, and that loss consistently drives the gate toward its magnitude zero limit. That suggests that when you try to add extra info to help, it actually makes things worse for training.
Tom: So it sounds like adding this auxiliary information doesn't help much during training because of these two regimes. But wait, the paper doesn't stop there; they don't just report what happens during training. They suggest a way to exploit this suppression instead of fighting it.
Jane: That’s the key point, right? If you can’t use the auxiliary signal to improve accuracy by adding it during training, why bother putting it in there at all? What is their proposed solution then?
Lu: They propose regularizing the LoRA fine-tuning with geometric auxiliary losses. These are geometric auxiliary losses coming from hyperbolic visual relational graphs. This means they aren't using the signal to directly guide the main task loss in a way that helps, but rather shaping the underlying structure of how the model learns.
Paper summary: Meng: So this geometric loss participates in the forward pass during training, but it can be dropped completely at inference time. That sounds like a clever way to use that structural information without running into those dead or negative utility regimes we just talked about.
Tom: Exactly. The full objective they're using is L equals L task plus alpha of t lambda ang lang lambda ent lent plus lambda kappa lkappa, where alpha of t ramps linearly from zero point zero point zero one to max over one hundred steps. That tells us the regularization isn't just on; it’s being ramped up during training.
Jane: It sounds like they are using this geometric signal to act as a structural regularizer for LoRA fine-tuning, which preserves the accuracy that LoRA alone might degrade when there’s no structural signal present. That connects directly to what they found about task selectivity dissociation.
Lu: Right, because the paper points out a task selectivity dissociation where learned prompts trade relational accuracy for attribute accuracy, but these geometric losses avoid that trade-off by trading aggregate attribute heavy compositionality for relational preservation and out of distribution spatial signal.
Meng: That means instead of chasing every possible detail in the data, they are focusing on keeping the core relationships intact while providing some spatial context. But what about the actual deployment side? How does this translate to something practical for a system that needs to run fast?
Tom: Well, there’s another piece of engineering they address regarding prefix tokens being derived from a hyperbolic manifold and projected into LLM embedding space. They found that these can carry L two norms very far from the pre-trained text embeddings because of residual stream dominance.
Jane: That sounds like a problem for generation quality, right? If the geometry is too skewed, the model just starts producing nonsense even if it’s trained well. So what did they do to fix that issue?
Lu: They proposed something called the RMS-prefix configuration as a generation safe alternative. This applies RMS normalization to every prefix token so that the resulting L two norm distribution tracks that of the text embeddings exactly. It seems like this restores generation quality, and they report GQA accuracy reaching sixty point seventy-five percent on full testdev when they use this RMS normalization.
Meng: sixty point seven five percent is a solid number, but I’m curious about the limitation here. The paper mentions that the geometric losses regularize through forward coupling at training time, but can be dropped at inference. So what if the geometry we use for training isn't perfectly representative of what happens in a real deployment scenario?
Paper summary: Tom: That is a fair question. The authors do acknowledge that this finding emerged from a broader investigation into how auxiliary signals interact with frozen LLMs when routed through learnable pathways, which suggests this is an initial discovery. They also point out that the geometric signal doesn't even need to be useful at inference for the regularizer to work during training.
Jane: So, what’s the big picture implication for how we think about injecting information into these models? It seems like we can use structural geometry as a stable anchor during training, even if that structure isn't immediately necessary when the model is just generating text.
Lu: The implication is that instead of relying on learned prompts or task gradients to guide the signal, we can use external structured computation—like these hyperbolic relational graphs—to impose a desired shape onto the embedding space during fine-tuning, and this shape holds up better than task-only training.
Meng: From an engineering standpoint, it means we have a systematic way to inject structural knowledge without needing constant hyperparameter tuning for that auxiliary signal. It shifts the complexity from runtime inference checks to a more controlled training objective.
Tom: So we’ve seen how these gates almost always close, why they close—due to dead gradients or negative utility—and how geometric losses help us exploit that suppression to keep performance high during fine-tuning. That’s what this paper is showing us about injecting auxiliary signals into frozen vision language models.
Jane: And the title, "The Gate Always Closes," really captures the essence of it, doesn't it? It suggests that the system is designed to ignore that extra information unless it’s trained in a very specific structural way.
Lu: The authors are essentially showing us how to build a training objective that forces the model to learn structure even when we don't explicitly tell it how important that structure is during inference.
Meng: I think for deployment, this means we can be more confident that the model retains its core pretrained capabilities while adding helpful, but not always necessary, structural context.
Tom: That’s what we’re seeing—a method where the structure of the input guides the learning process without needing to interfere with every single output decision at inference time.
Jane: So, this work gives us a new lens on how we can engineer these models by looking at them less as black boxes and more as systems where specific structural pressures dictate what information is actually learned and what gets ignored.
Conclusion: Tom: It’s about these gated pathways in vision language models and finding out they almost always get closed, no matter how you design the injection.
Jane: So there’s this phenomenon where every pathway ends up being behaviorally closed, meaning the accuracy doesn't really change if you just turn off that pathway when you actually use the model.
Lu: The authors attribute this suppression to two main things during training: a dead gradient regime, and a negative utility regime.
Meng: That’s interesting because it means trying to add that extra signal usually just hurts the loss instead of helping it during fine-tuning.
Tom: Right, so instead of fighting that suppression, they suggest exploiting it by using geometric auxiliary losses from hyperbolic visual relational graphs for LoRA fine-tuning.
Jane: So they use these geometric losses to regularize the LoRA training in a way that preserves the model's accuracy even when we don't need the extra structural information at inference.
Lu: The core idea is that these geometric losses act as structural regularizers during training, and this effect doesn't require the signal to be useful when you actually run a test on it.
Meng: I’m curious about how they manage that scaling, because if we’re running this at scale, we need to know what the overhead is for these geometric losses.
Tom: The authors show a task selectivity dissociation where learned prompts trade relational accuracy for attribute accuracy, but their geometric training avoids that trade-off by focusing on spatial signal.
Jane: It sounds like they are trading off some specific types of detail to keep the core relationships strong and focused on spatial context instead of just chasing every little attribute.
Lu: They show three configurations without geometric losses lose a few points on relational questions but gain a bit on attribute questions, while the fourth configuration loses much more relational accuracy for very little gain in attributes.
Meng: That’s important because it shows that simply training with a lot of diverse data doesn't guarantee you get the right trade-off for what you actually need at runtime.
Tom: And they also address an issue with how prefix tokens are mapped into the model's embedding space, where they found a norm mismatch due to residual stream dominance.
Jane: So they propose using something called RMS normalization on every prefix token so that its length distribution matches the pre-trained text embeddings, which restores generation quality.
Lu: That norm matching fixes generation quality and brings GQA accuracy up to sixty point seven five percent on their test development set when that specific normalization is applied.
Tom: It seems the overall message here is that we can use structural geometry to anchor our fine-tuning without needing every single piece of auxiliary information to be actively useful during inference.
Jane: It really suggests a way to engineer these models by imposing a desired structure during training, rather than relying only on what the model learns directly from the task data.
Lu: The implication is that we can use external structured computation to shape the embedding space, and this shaping holds up better than just standard task-based fine-tuning.
Meng: For deployment, this means we might be able to deploy models with more confidence because we know the structural anchor is solid during inference.
Tom: So what’s next? We’re going to look at how these findings connect to things like the geometry they used for those hyperbolic graphs and if that structure generalizes across different tasks.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound