Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
DRAPE synthesizes continuous instance-specific soft prompts for Multimodal Continual Instruction Tuning by deriving prompts from text and visual features. It improves performance over static task-level prompting by creating unique prompts for every query. DRAPE mitigates catastrophic forgetting using null-space gradient projection and CLIP-based prototype routing, achieving state-of-the-art results on benchmarks.