Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

Name: Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
Uploaded: 2026-10-21T14:07:29.301125+00:00
Duration: 357 s
Description: This work systematically studies long-context continued pre-training for vision-language models by introducing MMProLong. The method extends a 7B model to 128K context using long-document VQA as primary data. Results show MMProLong boosts performance by 7.1% at longer contexts and maintains strong results beyond 128K without further training, proving a transferable recipe for large models.

This work systematically studies long-context continued pre-training for vision-language models by introducing MMProLong. The method extends a 7B model to 128K context using long-document VQA as primary data. Results show MMProLong boosts performance by 7.1% at longer contexts and maintains strong results beyond 128K without further training, proving a transferable recipe for large models.

Open the full episode