Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
The work retrofits depth recurrence into a pretrained vision-language model to reduce its parameter count while maintaining or improving performance. By pruning layers and looping over a subset of remaining layers with a linear adapter, the model achieves 23.3% fewer language parameters than the unpruned version. A three-stage recovery training process restores performance through language pretraining, projector alignment, and visual instruction tuning.