Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

summary

The gist

The gist The proposed approach outperforms unpruned baselines while storing 23.3% fewer language-model parameters by retrofitting depth recurrence into a pretrained vision-language model Retrofitting

In short

The work retrofits depth recurrence into a pretrained vision-language model to reduce its parameter count while maintaining or improving performance. By pruning layers and looping over a subset of remaining layers with a linear adapter, the model achieves 23.3% fewer language parameters than the unpruned version. A three-stage recovery training process restores performance through language pretraining, projector alignment, and visual instruction tuning.

Key concepts

Depth Recurrence Retrofitting: This technique involves taking a pretrained vision-language model (VLM), removing some of its layers (pruning), and then looping over the remaining subset. A linear adapter is used at each step to merge the output of the current block with the representations from previous layers, effectively reintroducing recurrence into a pruned structure.
Prune-then-Loop Retrofit: This describes the structural modification where layers are partitioned into four groups: a prelude (P), a recurrent block (B), and a coda (C). The model then applies the recurrent block 'r' times, using an adapter to connect it back to the prelude. This process modifies the original model's architecture by selectively keeping only necessary layers for recurrence.
Three-Stage Recovery Training: To fix performance loss from pruning, a three-step training curriculum is used. First, text-only pretraining repairs the language backbone. Second, the vision-language projector is retrained to ensure image and text align correctly. Finally, full fine-tuning with visual instruction tuning restores overall model capability.
Knowledge Distillation (KD): Knowledge distillation is used during recovery training stages to prevent the language backbone from drifting too far from its original pretrained state. The output of the large, non-recurrent teacher model is used to guide and constrain the training of the smaller, recurrent model.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Flow Matching — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 17 episodes

This episode discusses

The paper

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones · Read on arXiv

Madhav Iyengar, Andreas Geiger, Hilde Kuhne

University of Tubingen AI Center

Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Retrofitting Recurrent Vision-Language Models from Pretrained Backbones".

Jane: The gist The proposed approach outperforms unpruned baselines while storing 23.3% fewer language-model parameters by retrofitting depth recurrence into a pretrained vision-language model Retrofitting Mechanism The core of the work…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper titled "Retrofitting Recurrent Vision-Language Models from Pretrained Backbones." Basically, the authors are taking a vision-language model and trying to make it smaller by using depth recurrence.

Jane: That sounds like they're trying to take a big model, right, and strip away some of the unnecessary parts so it runs faster or uses less memory. What is their main idea for how they do that?

Tom: They adapt the idea of depth recurrence from language models—where you loop over certain layers—and apply it to visual understanding within a vision-language model. The goal is reducing the parameter footprint of this pretrained vision-language model, which is a big deal for deployment.

Lu: What they're doing specifically is pruning layers from the language backbone of the VLM and then looping over just a subset of those remaining layers. They call this strategy "prune-then-loop retrofit" <ref:2610.22075#pg3>.

Meng: So, if you prune some layers, you're essentially cutting out parts of the original model architecture, and then you try to make those remaining parts work in a loop instead of just passing through them once. How big is this parameter reduction they are aiming for?

Tom: They claim that by using this method, they achieve twenty-three point three percent fewer language parameters compared to the unpruned baseline when trained with the same recovery data. That's a solid number <ref:2610.22075#pg1>.

Jane: Twenty-three point three percent is significant, especially when you're talking about a vision-language model that already has a large language backbone. But how do they fix the performance drop that comes with pruning? That seems like the hardest part for any retrofitting process.

Lu: To fix it, they propose a three-stage recovery training curriculum <ref:2610.22075#pg1>. First, they use a language-only pretraining stage to recover the performance of the language backbone while freezing everything else.

Tom: That makes sense; you need to repair the language part before you mess with the visual alignment. After that, they move to aligning that vision-language projector and then do a full fine-tuning using visual instruction tuning.

Meng: So it's not just one training step; it’s a whole sequence of recovery stages designed to patch up the damage from pruning <ref:2610.22075#pg3>. What about how they handle the knowledge transfer between these stages?

Jane: The paper mentions that in the first and last stages, they distill the outputs of the pretrained non-recurrent model into their recurrent model, which is supposed to further improve recovery. That sounds like using what you already have to help fix what you've broken <ref:2610.22075#pg1>.

Lu: They also incorporated a knowledge distillation term into stages one and two to limit the drift of the language backbone from its original representation, using the pretrained non-recurrent model as a teacher. That helps keep things stable during training <ref:2610.22075#pg3>.

Tom: Now for some design choices, they did an ablation study on things like the adapter between recurrences. They found that a linear adapter yields the best performance among tested variants, outperforming gated and Parcae adapters <ref:2610.22075#pg3>.

Paper summary: Jane: So it's not just about *if* you loop, but *how* you loop—the specific mechanism connecting the layers in the loop matters a lot for how well it recovers performance. And they also found that a larger recurrent block performs better at equal compute <ref:2610.22075#pg3>.

Meng: From a deployment standpoint, that's important because it tells us we can tune the architecture to get better results without just throwing more compute at it. But what about inference? How does this change when the model is actually running in production?

Tom: They looked at KV cache sharing during inference and found that keeping a separate cache for each recurrence block is necessary, because sharing costs accuracy <ref:2610.22075#pg3>. They also introduced an adaptive halting criterion based on KL divergence to skip computation when it's not needed.

Jane: So, while the training recovery is complex and involves multiple steps, they are also thinking about how this model will behave once it’s actually in use for generating text or answering questions. That adds a layer of practical engineering <ref:2610.22075#pg3>.

Lu: The paper also found that a model trained at a fixed depth only works at the depth it was trained at, but one trained with sampled recurrence works at other depths, though its performance doesn't improve beyond the trained depth <ref:2610.22075#pg3>.

Tom: That means you can train it in a flexible way, but you still have to respect the original training depth for peak performance. But they also found that sharing KV cache across recurrences during inference costs accuracy, which is another practical constraint <ref:2610.22075#pg3>.

Jane: So the main point here is that this retrofitting approach outperforms a recurrent language model extended to a VLM on the same data, and it’s not just about adding a vision encoder to an existing recurrent model <ref:2610.22075#pg1>.

Tom: What's really interesting is where they see the gains. They show that this proposed model gains on reasoning-oriented benchmarks, specifically MMBench and RealWorldQA <ref:2610.22075#pg3>.

Meng: So it’s not a general boost across the board for every visual task, but it’s particularly helpful when the task requires complex reasoning capabilities. That helps us understand where this kind of efficiency gain is most valuable in real-world applications <ref:2610.22075#pg3>.

Jane: And on the other hand, they found that its performance on basic visual understanding benchmarks remains on par with the unpruned baseline, which they attribute to those datasets usually having single-step question types <ref:2610.22075#pg3>.

Tom: So in summary, this paper introduces the first prune-then-loop retrofit of a pretrained VLM that outperforms unpruned baselines while storing twenty-three point three percent fewer language parameters <ref:2610.22075#pg1>.

Lu: The conclusion is that retrofitting recurrence directly into a pretrained VLM results in better performance than introducing a vision encoder into a recurrent language model to create a recurrent VLM <ref:2610.22075#pg3>.

Jane: It seems like the core idea isn't just about size reduction, but finding a way to reuse the existing structure of the pretrained model in a more efficient way for visual tasks. That’s what this paper is about <ref:2610.22075#pg1>.

Conclusion: Tom: So we've seen how they took depth recurrence from language models and retrofitted it into a vision-language model to make it smaller, right?

Jane: It sounds like they’re trying to reuse an existing structure instead of building a new one from scratch. What are the authors of this paper, Tom?

Tom: The paper is "Retrofitting Recurrent Vision-Language Models from Pretrained Backbones," and I see Lu and Jane on today.

Lu: I'm Lu, and we looked at how they prune layers to create a recurrent block, which is kind of interesting because it’s a structural change.

Jane: So the main point here is that this method outperforms just adding a vision encoder to an old recurrent language model <ref:2610.22075#pg1>. What does that mean for someone who just listens to the show?

Tom: It means they found a way to get better results with less language parameter count, specifically twenty-three point three percent fewer parameters on TinyLLaVA-one point 4B, compared to the unpruned baseline <ref:2610.22075#pg1>.

Meng: Twenty-three point three percent is a solid number for efficiency, but I wonder if that performance gain is real or just on a specific test set, right?

Lu: The authors did ablation studies on the design choices, and they showed that the linear adapter between recurrences is actually the best one tested <ref:2610.22075#pg3>.

Jane: So it’s not just about looping layers; it’s about *how* you connect them in that loop that makes a difference, and they even found a larger recurrent block works better than a smaller one for the same amount of compute <ref:2610.22075#pg3>.

Tom: And they showed gains on reasoning-oriented benchmarks like MMBench and RealWorldQA, but basic visual understanding performance stayed the same, which makes sense given how those tasks are usually structured <ref:2610.22075#pg3>.

Jane: So what does this suggest about building future AI models?

Lu: It suggests that retrofitting recurrence directly into a pretrained vision-language model is better than just tacking on a vision encoder to an existing recurrent language model <ref:2610.22075#pg3>.

Tom: Exactly. If you want efficiency without completely starting over, reusing the backbone structure in this recurrent way seems like the path forward for some of us.

More episodes

← Home