LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation
LINGUDISTILL is an adapter-free method to restore native language skills in vision-language models by using a frozen language model as a teacher. It achieves this through sharing key memory caches between the student and teacher decoders and selectively distilling knowledge based on data type. This results in about 10% performance recovery on language tasks while keeping vision performance relatively stable.