V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance

arXiv:2604.12966 · cs.CV · Submitted 2026-04-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance".

Jane: The gist V-GIFT proposes a simple and lightweight approach to augment visual instruction tuning with visually grounded self-supervised tasks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So Jane, this paper is about Multimodal Large Language Models or MLLMs that are pretty good on many vision language tasks, but they struggle when the task really needs fine-grained visual perception.

Jane: Exactly. They often just rely on language priors because a lot of those tasks can be solved with just language knowledge, and this paper argues the problem isn't weak vision representations <ref:2604.12966#pg3>.

Lu: It seems to suggest the issue is more about how the models are trained, not just what visual features they have learned <ref:2604.12966#pg2>.

Meng: So if it's not the representation itself, what is this V-GIFT approach actually doing? Is it just adding more data?

Tom: It’s a bit more specific than just adding data. They propose augmenting visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions <ref:2604.12966#pg1>.

Jane: So they are taking classic self-supervised learning problems, like rotation prediction or color matching, and turning them into image instruction response triplets <ref:2604.12966#pg1>.

Lu: That’s the key idea. They reformulate those pretext tasks so that the correct answer really can't be guessed just from language priors because it needs visual evidence <ref:2604.12966#pg3>.

Meng: So they claim this supervision forces the model to actually use its vision tokens instead of just guessing based on text cues?

Tom: Right. They don't change the training process itself, though <ref:2604.12966#pg3>. They keep the standard autoregressive cross-entropy loss and just adjust the distribution of instruction-following tasks to include these vision-only problems <ref:2604.12966#pg3>.

Jane: The paper claims this works without needing any human annotations or any extra architectural changes, which is a big deal for production pipelines <ref:2604.12966#pg1>.

Lu: And the claim is that injecting just a small fraction of these tasks—between three percent and ten percent—consistently improves performance on vision-centric evaluations across multiple models and benchmarks <ref:2604.12966#pg3>.

Meng: That level of consistency across different models and training regimes is what I'm looking for, because most methods only work on one setup <ref:2604.12966#pg3>.

Tom: It sounds like they found a lever for improving visual perception by simply adjusting the data distribution during instruction tuning <ref:2604.12966#pg1>.

Jane: So, to put it simply, V-GIFT is about making sure the model has to use its eyes more when it's following instructions <ref:2604.12966#pg1>.

Lu: It seems like they are encouraging a better coordination between visual perception and high-level linguistic reasoning without needing complicated auxiliary losses or reinforcement learning with verifiable rewards <ref:2604.12966#pg3>.

Meng: What about the specific tasks they use? Are they just random ones, or are they chosen carefully?

Tom: They use three classes of tasks: rotation prediction, which requires identifying the applied rotation and answering only with a degree value <ref:2604.12966#pg8>.

Jane: Then there’s point-wise colorization, where the model has to match labeled points to their original colors by outputting a text string <ref:2604.12966#pg9>.

Lu: And the third class is point correspondence, where given two images, it has to identify which point in the second image matches a query point on the first one <ref:2604.12966#pg9>.

Meng: Those tasks sound very grounded because they require things like recognizing objects and relating orientations to canonical poses <ref:2604.12966#pg8>.

Tom: That’s because as Lu points out, solving those problems requires recognizing objects and relating orientation to canonical poses, meaning the correct answer cannot be inferred from language priors alone <ref:2604.12966#pg9>.

Jane: So for someone just listening, this means the model isn't just reading words; it has to look at the image and use that visual information to figure out what’s going on <ref:2604.12966#pg1>.

Lu: And the results they showed are pretty solid across different benchmarks like CVB-2D, POPE, MMStar, and BLINK <ref:2604.12966#pg3>.

Tom: And on a practical note, the paper notes that for models like LLaVA-one point five-Vicuna they set the injection ratio at ten percent, and for LLaVA-OneVision-one point five they use three percent <ref:2604.12966#pg8>.

Meng: If we look at what this means for deployment, is it feasible to just inject these instructions without retraining the whole system from scratch?

Lu: The paper states that injecting these tasks directly into the visual instruction tuning phase yields measurable gains across multiple models and training regimes <ref:2604.12966#pg3>.

Jane: It seems like the main implication is that we don't need massive, complex auxiliary losses to push models to be better at vision-centric reasoning <ref:2604.12966#pg3>.

Tom: And they’ve shown that this approach is effective even when you only use a small fraction of these visually grounded instructions, between three percent and ten percent <ref:2604.12966#pg3>.

Lu: The authors also point out a limitation, though, which is that the method doesn't modify the optimization paradigm itself <ref:2604.12966#pg3>. It still uses standard autoregressive cross-entropy training <ref:2604.12966#pg8>.

Meng: So it’s a distribution shift problem, not a fundamental change in how the AI learns, which makes it way more deployable for us <ref:2604.12966#pg3>.

Jane: It really does sound like V-GIFT is just about giving the model enough visual experience during its instruction tuning phase to stop relying so heavily on linguistic shortcuts <ref:2604.12966#pg1>.

Tom: So, the paper's title, Boosting Visual Instruction Tuning with Self-Supervised Guidance, really captures that idea of using guidance from self-supervised tasks to boost visual instruction tuning <ref:2604.12966#pg1>.

Lu: It’s a simple way to encourage better use of visual tokens during decoding <ref:2604.12966#pg2>.

Meng: If you only listen to this, the main thing is that MLLMs can get significantly better at tasks that really require looking at the image details, without needing a huge compute budget or complex new training pipelines <ref:2604.12966#pg3>.

Conclusion: Tom: So we've been looking at V-GIFT, which is this new method for Multimodal Large Language Models that tries to get them better at vision tasks by adding some self-supervised training data.

Jane: Right, so the title is "V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance." It sounds a bit technical, but basically, they're suggesting that we can improve how these models use images when following instructions by giving them extra visual practice during their instruction tuning phase.

Lu: Exactly. The paper argues that the problem isn't just having bad vision representations; it’s that the models don't learn to actually *use* those visual cues enough when they are trying to answer questions.

Meng: So, what does this mean for us at a startup? If we can get better visual grounding without adding massive amounts of human-labeled data, that’s huge because annotation is slow and expensive.

Tom: It means they're taking standard tasks—like predicting rotation or matching colors—and turning them into instruction triplets, which forces the model to rely on visual evidence instead of just language shortcuts.

Jane: That’s a simple way to think about it: they’re injecting "vision-forcing" tasks directly into the training data so the AI has to show its work visually.

Lu: The key finding is that even a small fraction, maybe three percent to ten percent, of these guided instructions is enough across different models and benchmarks to consistently boost performance on vision-centric evaluations.

Tom: And they did this without changing the architecture or adding any extra training stages; it’s just about adjusting how you structure the existing instruction tuning data.

Meng: That’s good for deployment because it keeps the pipeline lightweight and doesn't require a complete overhaul of our infrastructure.

Jane: So, to wrap up this part, V-GIFT shows that we don't need a total redesign to make MLLMs better at seeing and reasoning visually.

Lu: It’s about finding the right way to guide the model during instruction tuning so it stops relying on language priors and starts grounding its answers in what it actually sees.

Tom: Now, this leads us into how much extra compute this actually costs compared to just training a standard MLLM from scratch.

Sorbonne University · Institut Universitaire de France

cs.CV

Submitted: 2026-04-14

Updated: 2026-10-15

Comments: Accepted to NeurIPS 2026

Code: https://github.com/sirkosophia/V-GIFT

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: The gist V-GIFT proposes a simple and lightweight approach to augment visual instruction tuning with visually grounded self-supervised tasks, which consistently improves performance on vision-centric

Key concepts

MLLMs
Multimodal Large Language Models are AI systems that can understand and process both text and images. They are used for tasks that require reasoning about how visual information relates to language, but they often struggle when the task demands very precise visual details.
Instruction Tuning
This is the training stage where a model learns to follow specific instructions given in natural language. In V-GIFT, this process is augmented by injecting specialized tasks that require looking closely at an image to answer correctly, rather than just using general knowledge.
Self-Supervised Learning (SSL) Pretext Tasks
These are tasks where the model learns useful representations from unlabeled data by creating its own training signals. V-GIFT takes standard visual tasks, like predicting an image's rotation or matching colors, and rephrases them as instructions that demand visual answers, making the learning process more visually grounded.

Terminology

Summary

The gist

V-GIFT proposes a simple and lightweight approach to augment visual instruction tuning with visually grounded self-supervised tasks, which consistently improves performance on vision-centric evaluations in Multimodal Large Language Models (MLLMs) by encouraging better use of visual information during instruction tuning<ref:2604.12966#pg6>.

Motivation

Multimodal Large Language Models (MLLMs) often struggle with vision-centric problems that require fine-grained visual reasoning because they tend to rely on language priors alone when many tasks can be partially solved using language priors<ref:2604.12966#pg2> This limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone<ref:2604.12966#pg3> The issue is not merely representational capacity, but training dynamics, as instruction-tuning examples are often automatically generated from generic captions without enforcing the use of vision cues for solving them<ref:2604.12966#pg4> The hypothesis is that if models are rarely required to depend on visual input in order to succeed, they will tend to default to language-based heuristics<ref:2604.12966#pg5>

Methodology

The core idea of V-GIFT is to reinterpret instruction tuning as a modality competition process by injecting inherently vision-forcing tasks into the instruction distribution<ref:2604.12966#pg6> The approach involves three main steps:

  1. Reformulating classical self-supervised learning (SSL) pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image–instruction–response triplets compatible with standard MLLM pipelines<ref:2604.12966#pg6> These tasks possess two critical properties: they are visually grounded by construction and the correct answer cannot be inferred from language priors alone<ref:2604.12966#pg6>

  2. Integrating these tasks directly into the visual instruction tuning phase by augmenting the original instruction dataset with automatically generated SSL instruction samples<ref:2604.12966#pg6> The final training dataset is formed as the union of the original visual instruction tuning dataset and Dssl, where Dssl is the set of automatically generated SSL instruction samples<ref:2604.12966#pg6>

  3. Controlling the relative amount of SSL supervision through a ratio ρ = 100 times the fraction of additional SSL instruction samples relative to the size of the original instruction tuning dataset<ref:2604.12966#pg8> The training is optimized using standard autoregressive cross-entropy loss in Eq. 2, without introducing auxiliary losses, architectural changes, or expensive RLVR pipelines<ref:2604.12966#pg8>

Visual Self-Supervised Tasks

V-GIFT utilizes three classes of visually grounded SSL tasks:

(a) Rotation prediction:

Given an image I, a rotated version ˜I = Rθ(I) is generated, and the instruction asks the model to identify the applied rotation and answer only with the degree value<ref:2604.12966#pg8> Solving this task requires recognizing objects and relating their orientation to canonical poses, meaning the correct answer cannot be inferred from language priors alone<ref:2604.12966#pg9>

(b) Point-wise colorization (color matching):

Given a color image I, the model is asked to match each labeled point to its original color index by outputting a short text string encoding the correct point-label to color-index pairs<ref:2604.12966#pg9> This task requires fine-grained visual discrimination and spatial grounding<ref:2604.12966#pg9>

(c) Point correspondence:

Given an image pair (I1, I2), the model is asked to identify which candidate point in the second image corresponds to the query point on the shared object in the first image<ref:2604.12966#pg9> This task requires identifying consistent visual features across viewpoints and reasoning about spatial correspondences between two images<ref:2604.12966#pg9>

Experimental Results

Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3–10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations<ref:2604.12966#pg6> For LLaVA-1.5 models, the injection ratio is set to ρ = 10% for LLaVA-1.5-Vicuna and ρ = 3% for LLaVA-OneVision-1.5<ref:2604.12966#pg8> Control experiments with matched training iterations confirm that the gains are not attributable to additional compute<ref:2604.12966#pg8> The performance peaks at ρ = 10% for LLaVA1.5-Qwen and ρ = 3% for LLaVA-OneVision-1.5, after which gains saturate or slightly decline<ref:2604.12966#pg9> Furthermore, results show that improvements are obtained only when SSL tasks are injected during instruction tuning, as applying them before or after instruction tuning yields performance comparable to the baseline

Conclusion

V-GIFT introduces a simple yet effective framework to reformulate classic self-supervised pretext tasks as visual instruction-following data, mitigating language shortcuts in MLLMs and encouraging the LLM to better utilize visual representations<ref:2604.12966#pg6> The method integrates seamlessly into the instruction-following pipeline without architectural modifications or additional training stages Through extensive experiments, V-GIFT shows consistent improvements across models, training regimes, and benchmarks while requiring minimal additional compute The approach enables models to better ground their reasoning in fine-grained visual information Future work will explore extending this approach to other modalities, such as 3D point clouds or audio inputs

--- Page 1 ---

The gist

V-GIFT proposes a simple and lightweight approach to augment visual instruction tuning with visually grounded self-supervised tasks, which consistently improves performance on vision-centric evaluations in Multimodal Large Language Models (MLLMs) by encouraging better use of visual information during instruction tuning<ref:2604.12966#pg6>.

Motivation

Multimodal Large Language Models (MLLMs) often struggle with vision-centric problems that require fine-grained visual reasoning because they tend to rely on language priors alone when many tasks can be partially solved using language priors<ref:2604.12966#pg2> The issue is not merely representational capacity, but training dynamics, as instruction-tuning examples are often automatically generated from generic captions without enforcing the use of vision cues for solving them<ref:2604.12966#pg4> The hypothesis is that if models are rarely required to depend on visual input in order to succeed, they will tend to default to language-based heuristics<ref:2604.12966#pg5>

Visual Self-Supervised Tasks

V-GIFT utilizes three classes of visually grounded SSL tasks:

(a) Rotation prediction:

Given an image I, a rotated version ˜I = Rθ(I) is generated, and the instruction asks the model to identify the applied rotation and answer only with the degree value<ref:2604.

Improvements for AI systems

  1. Bold header: Reinforcement of Visual Reasoning through Instruction Tuning Augmentation

The improved system can perform better on vision-centric evaluations by injecting visually grounded self-supervised tasks into instruction tuning, which consistently improves performance on vision-centric evaluations across multiple models and benchmarks.

  1. Bold header: Mitigation of Language Shortcuts in Multimodal Reasoning

The AI system will be less likely to default to language heuristics because the method addresses the bias where many instruction-tuning examples can be partially or fully solved using strong language priors alone, by injecting tasks that cannot be inferred from language priors alone.

  1. Bold header: Enhanced Fine-Grained Visual Discrimination and Spatial Grounding

By utilizing specific SSL tasks, such as Point-wise colorization (color matching) and Point correspondence, the model will develop abilities requiring fine-grained visual discrimination and spatial grounding to associate each labeled location with the correct color or cross-view feature matching and spatial reasoning.

  1. Bold header: Robust Cross-View Spatial Correspondence

The system can perform more reliably in tasks requiring multi-view understanding, such as identifying corresponding points across views, by utilizing the Point correspondence task which involves identifying consistent visual features across viewpoints and reasoning about spatial correspondences between the two images.

  1. Bold header: Efficient and Non-Invasive Training Integration

The improvement is achieved without modifying architecture or adding complex losses; specifically, it is integrated by augmenting the dataset with SSL samples, as described in Equation 6: The final training dataset is formed as the union D = D inst U D ssl.

Abstract

Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual perception. Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets, we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning with visually grounded SSL tasks as a powerful lever for improving visual perception in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT

Sources

Related papers