TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

arXiv:2607.28640 · cs.CL, cs.CV · Submitted 2026-05-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs".

Jane: The gist The modality gap is a systematic discrepancy where multimodal large language models fail to generate consistent responses given semantically equivalent inputs across textual and image modalities,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've looked at this paper, "TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs," and now we're getting to the conclusion of what all that work means.

Jane: Essentially, these researchers are tackling this problem where multimodal models just don't talk consistently when you swap out text for an image.

Tom: Right. They’ve built a specific test suite called TOKENSWAP-BENCH to prove how bad this inconsistency is across forty-two different models.

Lu: And the main finding there is that there's this big gap, performance dropping from four point two percent up to forty-seven point four percent when moving from pure text to interleaved inputs #pg1

Meng: That's a pretty significant drop, and it shows that just having both text and images in one prompt isn't automatically better for these models.

Jane: They also pointed out that reasoning models actually handle this gap much better, with them showing smaller performance drops compared to the non-reasoning ones #pg2

Tom: And what they did to fix it? They introduced TokenSwap, a training method where you augment the text data by swapping concepts with semantically aligned images.

Lu: The idea is that by training on these interleaved examples, the models learn to be more consistent across those different formats #pg2

Meng: So it’s like teaching the AI how to bridge that gap during its learning phase instead of just hoping it figures it out later.

Jane: They showed this training method works, actually reducing the modality gap from zero point two six eight down to zero point one six seven with generated images #pg2

Tom: That's a real win for making multimodal systems more reliable and consistent in the real world.

Lu: It suggests that data augmentation through TokenSwap is a solid way to improve cross-modal alignment without messing up their core language skills #pg2

Jane: But there are still some caveats they brought up, like the gap depends on how good the images are, and it needs in-distribution data for the training to really stick #pg3

Tom: So while this paper shows a clear path forward with TokenSwap training, we still need to watch those image quality issues and domain mismatches as we try to use this in practice.

Lu: Exactly. It’s a data-centric method, so how you construct that interleaved data is actually critical for the success of the whole thing #pg3

Meng: So for someone building these systems, it means focusing on creating high-quality image pairings during training could be a smart move to improve consistency across the board.

Jane: That’s right. It opens up a new direction for how we train these models to handle mixed inputs more gracefully than they are now #pg3

Conclusion: Tom: So we’re wrapping up on "TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs." This paper looks at how multimodal models struggle to give consistent answers when you feed them text and images that mean the same thing but are in different formats.

Jane: They show that this problem, which they call the modality gap, is widespread across forty-two different models tested on a benchmark called TOKENSWAP-BENCH.

Tom: The authors introduce TokenSwap as a training technique where you swap text concepts for images to teach the AI consistency during its learning phase.

Lu: This means they’re not just measuring the problem; they’re providing a method to actively fix it by augmenting the training data with these image-interleaved sequences.

Meng: It sounds like a practical solution because it lets you improve that cross-modal alignment without having to completely rewrite your whole model pipeline from scratch.

Jane: They also found a clear difference between reasoning models and others, showing that reasoning capabilities help them stay more consistent even with mixed inputs.

Tom: The paper’s main implication is that data augmentation through TokenSwap is a viable way to make multimodal systems more reliable in the real world by teaching them how to handle visual and textual information together better.

University of California, Santa Barbara

cs.CL, cs.CV

Submitted: 2026-05-20

Updated: 2026-10-08

Project page: https://dongxzz.github.io/TokenSwap

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The modality gap is a systematic discrepancy where multimodal large language models fail to generate consistent responses given semantically equivalent inputs across textual and image

Key concepts

Modality Gap
This is the systematic difference in model accuracy when the same information is presented as pure text versus when it is presented using multimodal inputs, like text paired with a corresponding image. It measures how well models handle semantic equivalence across different data types.
TokenSwap
A method that creates new training data by replacing specific words or concepts in a text with semantically matched images. This interleaves visual tokens directly into the text sequence, creating 'image-interleaved' inputs to systematically test model consistency.
TOKENSWAP-BENCH
A specialized benchmark created by applying TokenSwap to existing tasks like MMLU. It consists of text and image sequences where visual tokens are intentionally mixed with text tokens, allowing researchers to quantify the performance drop (the modality gap) across many different multimodal models.
Reasoning Models vs. Non-Reasoning Models
The study found that models capable of complex reasoning show a smaller modality gap than those that do not. This suggests that the ability to reason helps these models maintain consistency when switching between textual and visual representations, indicating greater robustness in multimodal understanding.

Terminology

Summary

The gist The modality gap is a systematic discrepancy where multimodal large language models fail to generate consistent responses given semantically equivalent inputs across textual and image modalities, and TokenSwap is a training-based method that effectively mitigates this gap while preserving strong text-only and vision-language performance. <ref:2607.28640#pg2>

Formalizing the Modality Gap

The modality gap is formally defined as the difference in model performance when semantically equivalent content is presented in textual versus multimodal form The authors introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens To systematically study this gap, they apply TokenSwap to MMLU [16] to construct TOKENSWAP-BENCH, a benchmark specifically designed to quantify the modality gap The modality gap is quantified as ∆Gap = Eval(Xtext) − Eval(Xinterleaved) where Eval(·) denotes the model’s performance metric (e.g., accuracy).

Benchmark Construction and Observations

The TOKENSWAP-BENCH benchmark consists of 1,516 samples with 6,946 total images, specifically designed to quantify the modality gap Across 42 MLLMs evaluated, a pervasive modality gap exists where performance decreases by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% ± 3.3% across models Notably, reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models. Furthermore, the authors find that common prompting strategies, such as chain-of-thought and few-shot prompting cannot reliably reduce the modality gap and that scaling training compute yields only marginal gains, with a 10× increase in FLOPs reducing the gap by approximately 2.8%.

Analysis of Model Families and Reasoning Capabilities

The evaluation reveals that all models exhibit a non-trivial modality gap where all evaluated models lie strictly below the y = x line, indicating that replacing text with semantically aligned images consistently degrades performance The modality gap varies across model families, with proprietary models like Claude exhibiting the smallest gaps (mean gaps 8.3%), while open-source models generally show larger gaps A clear distinction is observed between reasoning and non-reasoning models, as Reasoning models not only achieve stronger performance on both text and image-interleaved inputs, but also exhibit consistently smaller modality gaps This trend persists when considering the Relative Modality Gap, suggesting the advantage is not solely attributable to stronger text capabilities, and may be partly associated with reasoning-oriented training.

Mitigation Strategy: TokenSwap Training

The authors demonstrate that incorporating TokenSwap during training effectively mitigates this gap while preserving strong text-only and vision-language performance They show that TokenSwap reduces the modality gap and significantly mitigates the modality gap compared to both the baseline and text training, with a gap decreasing from 0.268 to 0.167 in post-training with generated images Crucially, TokenSwap preserves text-only and vision-language performance as it maintains comparable text accuracy and in some settings slightly improves it.

Benchmark Construction Details

The TOKENSWAP-BENCH construction involves three stages: 1) Visualizable Concept Extraction, where they use Gemini-2.0-Flash to identify concepts; 2) Visual Concept Rendering, synthesizing images using a text-to-image model like Gemini with the prompt generate an image of a c; and 3) Replacement Filtering, which includes Validity filtering to ensure replacements that preserve the original meaning are retained and Caption-Guided Validation, which uses a round-trip strategy where they verify semantic recoverability by replacing visual tokens with generated captions, retaining samples satisfying Eval(Xtext) = 1 and Eval(X˜text) = 1.

Limitations and Future Directions

The paper notes that the measured gap depends on image quality, where relatively lower-quality (e.g., retrieval-based) images can introduce a bigger modality gap, although model rankings remain consistent Additionally, TokenSwap training requires in-distribution data, and domain mismatch between training and evaluation (e.g., natural images vs. OCR data) may limit transfer The approach is presented as a data-centric method for constructing image-interleaved inputs that can be integrated into existing pipelines.

Conclusion

In summary, the paper proposes TOKENSWAP-BENCH to quantify the pervasive modality gap in MLLMs and introduces TokenSwap training as a training strategy that effectively reduces this gap while maintaining strong text performance The findings suggest that reasoning models are inherently more robust against this gap, and data augmentation via TokenSwap is a viable method for improving multimodal consistency.

--- Page 10 ---

The gist The modality gap is a systematic discrepancy where multimodal large language models fail to generate consistent responses given semantically equivalent inputs across textual and image modalities, and TokenSwap is a training-based method that effectively mitigates this gap while preserving strong text-only and vision-language performance.

How it works

The modality gap is formally defined as the difference in model performance when semantically equivalent content is presented in textual versus multimodal form TokenSwap operates at the concept level, replacing individual textual concepts with semantically aligned natural images while preserving the surrounding context and structure This process transforms text-only datasets into image-interleaved counterparts, resulting in TOKENSWAP-BENCH.

Benchmark Construction Details

The construction of TOKENSWAP-BENCH follows three stages: 1) Visualizable Concept Extraction, using Gemini-2.0-Flash to identify concepts; 2) Visual Concept Rendering, synthesizing images using a text-to-image model with the prompt generate an image of a c; and 3) Replacement Filtering, which includes Validity filtering to ensure replacements that preserve the original meaning are retained and Caption-Guided Validation, which uses a round-trip strategy where they verify semantic recoverability by replacing visual tokens with generated captions, retaining samples satisfying Eval(Xtext) = 1 and Eval(X˜text) = 1.

Mitigation Strategy: TokenSwap Training

To mitigate the modality gap, the authors apply TokenSwap during training by augmenting text-only data with image-interleaved counterparts constructed through visual concept substitution They compare Text training and TokenSwap training across pre-training and post-training settings using generated (Gen) or retrieved (Ret) images The results show that TokenSwap reduces the modality gap and significantly mitigates the modality gap compared to both the baseline and text training, while preserving text accuracy.

Limitations and Future Directions

The paper identifies limitations, stating that the measured gap depends on image quality, where relatively lower-quality (e.g., retrieval-based) images can introduce a bigger modality gap Furthermore, TokenSwap training requires in-distribution data, and domain mismatch between training and evaluation (e.g., natural images vs. OCR data) may limit transfer The approach is positioned as a data-centric method for constructing image-interleaved inputs that can be integrated into existing pipelines.

Conclusion

The paper concludes by summarizing that all models exhibit a non-trivial gap, with at least a 4% drop under image-interleaved inputs and proposes TokenSwap training as an effective strategy for reducing this gap while preserving text performance

--- Page 11 ---

The gist The modality gap is a systematic discrepancy where multimodal large language models fail to generate consistent responses given semantically equivalent inputs across textual and image modalities, and TokenSwap is a training-based method that effectively mitigates this gap while preserving strong text-only and vision-language performance.

Benchmark Construction Details

The construction of TOKENSWAP-BENCH follows three stages: 1) Visualizable Concept Extraction, using Gemini-2.

Improvements for AI systems

  1. Improvement of MLLM Consistency through TokenSwap Training: Implement TokenSwap during training by augment[ing] text-only data with image-interleaved counterparts constructed through visual concept substitution, which has been shown to effectively mitigat[e] the modality gap in both pre-training and post-training settings. This allows AI systems to generate consistent responses given semantically equivalent inputs across modalities, addressing the systematic discrepancy observed when moving from text-only to image-interleaved inputs.

  2. Development of TOKENSWAP-BENCH for Gap Quantification: Utilize the proposed benchmark, TOKENSWAP-BENCH, which is designed to measure the performance discrepancy when the same semantic information is presented in different modalities, enabling researchers to systematically quantify the modality gap using the formula ∆Gap = Eval(Xtext) − Eval(Xinterleaved). This provides a rigorous metric for identifying where models fail due to cross-modal variation.

  3. Enhanced Reasoning Model Performance: Focus on reasoning-oriented models, as they exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models. This suggests that tailoring training specifically for reasoning capabilities can yield superior cross-modal consistency.

  4. Targeted Training Strategy Selection: Employ TokenSwap training as a primary mitigation strategy, noting that TokenSwap training effectively reduces the modality gap at both pretraining and post-training stages, while preserving strong text-only and vision-language performance. This ensures that performance improvements in cross-modal consistency do not come at the cost of either text-only or vision-language performance.

  5. Robust Concept Extraction Pipeline: Integrate a multi-stage concept extraction pipeline involving Visualizable Concept Extraction, Visual Concept Rendering, and rigorous filtering steps like Validity filtering and Caption-guided validation to ensure that substituted images are semantically aligned, as demonstrated by the success of the human study where annotators achieved over 90% accuracy in identifying intended concepts.

Sources

Related papers