TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
Multimodal LLMs often fail to produce consistent answers when given text and image inputs that mean the same thing. Researchers created TokenSwap, a method that mixes text with semantically aligned images to test this 'modality gap' on a new benchmark called TOKENSWAP-BENCH. The study found this gap exists across models but showed that incorporating TokenSwap during training effectively reduces it without hurting performance.