Counterfactual Reasoning for Robust Visual Question Answering

arXiv:2609.16567 · cs.CV · Submitted 2026-09-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Counterfactual Reasoning for Robust Visual Question Answering".

Jane: Here's the gist of the paper. Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias<ref:2609.16567#pg2>.

Tom: Our framework proposes a novel training framework that enhances counterfactual contrastive learning for VQA<ref:2609.16567#pg2>.

Jane: The model achieves a competitive accuracy of 61.64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.16%<ref:2609.16567#pg2>.

Tom: Pruning redundant reasoning tokens The paper introduces an enhanced counterfactual contrastive learning framework to address the limitations of existing methods in VQA<ref:2609.16567#pg4>.

Title and authors: Tom: So we're looking at this paper now, "Counterfactual Reasoning for Robust Visual Question Answering." It’s about how to stop models from just guessing answers based on what they saw in the training data and start actually reasoning about the image.

Jane: That sounds like a big problem because, as you mentioned before, current visual question answering models often learn to associate certain question patterns with high-frequency answers instead of looking at the actual picture, which causes them to fail badly when they see something new.

Lu: Exactly. The paper addresses this by proposing a novel training framework that enhances counterfactual contrastive learning specifically for VQA, aiming to guide attention toward causal evidence and improve feature discrimination in the model one <ref:2609.16567#pg1,a novel training framework that enhances counterfactual contrastive learning>.

Meng: So it’s not just another fine-tuning trick; they're building a whole new way to train the system using these counterfactual samples. What's the main idea behind this framework?

Tom: Well, they introduce three key contributions: first, a three-stage curriculum for stable multi-objective optimization; second, an enhanced batch contrastive loss for better feature learning; and third, two new regularizers: answer contrastive loss and gradient discrepancy loss one <ref:2609.16567#pg1,a three-stage curriculum for stable multi-objective optimization>.

Jane: The idea of a curriculum sounds interesting. How does that work in practice? Does it just ramp up the difficulty gradually?

Lu: Yes, it’s a three-stage process to introduce the complex objectives progressively. Stage one just optimizes the standard VQA task loss, stage two adds batch contrastive loss to structure the embedding space, and then stage three activates answer contrastive and gradient discrepancy losses to refine prediction margins and enforce visual grounding one <ref:2609.16567#pg1>.

Tom: And that third stage is where things get really interesting for me because it introduces the gradient discrepancy loss. It rewards a significant difference in model sensitivity between factual and counterfactual views, which they say penalizes shortcut learning one <ref:2609.16567#pg1>.

Meng: If it’s rewarding differences, what does that actually mean for the visual representation? Does it force the model to look at different parts of the image when you change a part of the question?

Jane: It forces them to develop distinct attention patterns that align with genuine causal evidence, which is what they are aiming for when they use that gradient discrepancy loss one <ref:2609.16567#pg1>. That’s a big step toward making sure the AI is grounding its answers in real visual facts.

Title and authors: Tom: And for the engineers, there’s also this enhanced batch contrastive loss. They use all in-batch negatives to get richer, more stable signals compared to just comparing an anchor against a single positive and negative sample one <ref:2609.16567#pg1>.

Lu: That loss function helps structure the feature space by contrasting one positive pair against numerous in-batch negatives, which they claim provides more robust gradients and forces the model to learn more generalizable features one <ref:2609.16567#pg1>. This is where they try to reduce reliance on those spurious language correlations.

Meng: But what about scaling this up? If we deploy this on a real traffic or customer service pipeline, how much overhead are we talking about with all these different losses and the contrastive generation module?

Tom: They found that by setting the weights appropriately—with lambda batch contrastive loss at two point zero, answer contrastive loss at two point zero, and gradient discrepancy loss at six point zero—they get a good trade-off between out-of-distribution robustness and in-distribution performance one <ref:2609.16567#pg1>.

Jane: So they achieved a competitive accuracy of sixty-one point six four percent on the bias-sensitive VQA CP v2 benchmark while only seeing a small generalization gap of one point one six percent on the standard VQA v2 dataset one <ref:2609.16567#pg1>. That sounds like a solid result for robustness.

Lu: The analysis shows that the answer distribution adapts to the test distribution instead of just memorizing biases from the training set, which is what we hoped for when we used counterfactual reasoning one <ref:2609.16567#pg1>. The qualitative results also show that those answer contrastive and gradient discrepancy losses successfully guide the model toward causal evidence for accurate and well-grounded reasoning one <ref:2609.16567#pg1>.

Meng: I’m still worried about deployment. If the model is relying on these complex interactions between three different loss functions, how stable is it at scale? What breaks when we move from a small test set to millions of real questions?

Tom: They are showing that this framework achieves a strong balance between out-of-distribution robustness and in-distribution performance one <ref:2609.16567#pg1>. The paper suggests it presents a robust approach toward more reliable VQA models that balance those two things one <ref:2609.16567#pg1>. It’s about making the model adaptable rather than just memorizing patterns.

Jane: So, if I was driving and asked a complex question about a picture, this framework means the AI is less likely to give me a wrong answer based on some hidden bias in the training set? That's what I can hear.

Lu: It means we’re moving away from models that just guess based on language patterns and toward systems that are actively seeking visual evidence when they encounter new situations one <ref:2609.16567#pg1>. They are pushing for more reliable VQA models that balance out-of-distribution robustness and in-distribution performance one <ref:2609.16567#pg1>.

Title and authors: Meng: What about the limitations? The authors mention that this framework is aimed at improving existing methods, but they don't say it solves every problem in multimodal reasoning. Where does it stop working?

Tom: They did note that future work includes applying this framework to larger Transformer-based architectures to further advance multimodal causal reasoning one <ref:2609.16567#pg1>. That suggests the current implementation might be optimized for a certain size or structure of the model.

Jane: It sounds like they’re not aiming for a perfect solution right now, but they’re making sure the foundation is solid enough for bigger, more complex systems down the road. So, where does this leave us?

Lu: This paper shows how to use counterfactual learning to actively enforce visual grounding through mechanisms like gradient discrepancy loss one <ref:2609.16567#pg1>. It gives us a concrete recipe for stabilizing optimization across multiple objectives.

Meng: For production, it’s a lot of specialized loss functions. It sounds like the cost of running this framework is going to be higher than a simpler baseline model we could just deploy today one <ref:2609.16567#pg1>.

Tom: The cost comes from training, but they are showing that the gain in robustness under distribution shifts is worth it for certain applications where reliability matters more than just raw speed on clean data one <ref:2609.16567#pg1>.

Jane: So, to wrap up on "Counterfactual Reasoning for Robust Visual Question Answering," this framework uses a three-stage curriculum and three specific losses—batch contrastive, answer contrastive, and gradient discrepancy—to force the model to look for causal visual evidence instead of relying on statistical shortcuts.

Lu: That’s the core idea: moving from pattern matching to causal grounding one <ref:2609.16567#pg1>. It really shows how multi-objective optimization can stabilize training when you introduce counterfactual samples one <ref:2609.16567#pg1>.

Meng: And while it’s complex, they achieved sixty-one point six four percent accuracy on VQA CP v2 and sixty two point eight zero percent on VQA v2 with a generalization gap of one point one six percent one <ref:2609.16567#pg1>. That's the measurable performance we have to look at.

Tom: Yeah, that small generalization gap is what I think is really promising for real-world deployment because it means it won't completely break when the test data shifts slightly from the training data one <ref:2609.16567#pg1>.

Jane: It sounds like a very careful approach to building models that are truly robust, not just good at passing a specific test set. That’s what I wanted to hear about this paper.

The paper's summary: Tom: So, we’ve been talking about how these new training methods are trying to fix the problem where AI models just memorize answers instead of actually seeing what's in an image, and now we’re looking at this paper on counterfactual reasoning for visual question answering.

Jane: Right, so the main idea is that they're using counterfactual samples—which are like slightly altered versions of a question or image—to force the model to stop relying on those weak language patterns and start looking for actual visual proof.

Tom: Exactly. It’s not just about training it to be smarter; it’s about making sure that when we give it a new, unseen image, it doesn't fail because we changed the wording of the question slightly.

Lu: What they introduce is a three-part framework to do this. First, they have this three-stage schedule for training where you build up the complexity gradually. Second, they have an enhanced batch contrastive loss that helps structure how the model learns its features. And third, there are these two new regularizers—answer contrastive loss and gradient discrepancy loss—to really sharpen the prediction margins and enforce causal grounding in the visual data.

Meng: That sounds complicated to set up. For someone actually shipping models, what does this mean for deployment stability? Are we talking about another massive training pipeline or something manageable?

Jane: Well, they found that by setting the weights for those three loss functions—lambda batch contrastive loss at two point zero, answer contrastive loss at two point zero, and gradient discrepancy loss at six point zero—they get a good balance. They actually got sixty-one point six four percent accuracy on a sensitive benchmark while keeping a small generalization gap of one point one six percent on the standard test set.

Tom: That small gap is what matters to me. It means the model isn't just great on the data we fed it, but it handles new situations reasonably well without completely breaking. It shows this method has a real trade-off between being robust when things change and being good at figuring out what’s already there.

Lu: The qualitative results they show are interesting too. They demonstrate that those two new losses, the answer contrastive loss and the gradient discrepancy loss, successfully guide the model toward causal evidence for its answers. It means it’s not just guessing based on a pattern; it's finding visual proof that lines up with what’s happening.

Jane: So when we look at this from a user perspective, it means if you ask an AI a question about a picture, and you slightly rephrase the question—say you change "Is this red?" to "What color is the object?"—the system is better equipped to actually check the visual evidence rather than just guessing based on which word sounds more likely.

Tom: Yeah, that shifts it from being a language processor to something closer to a reasoner that’s trying to build causal links between the question and the image. It moves away from memorizing training data biases toward adapting its understanding of the world.

Jane: It really does suggest we might be able to build visual question answering systems that are much more reliable in real-world, messy situations where things aren't perfectly clean or predictable. This opens up a lot of possibilities for applications where accuracy and dependability are really hard to get right.

The paper's improvements: Tom: So, we’ve seen how they set up that three-stage curriculum and those three different loss functions to get the results in front of us, now we're going to talk about what these specific loss improvements actually *do* for the model’s learning process.

Jane: Right, so instead of just having one big goal for the AI, they are using each part of that framework to tackle a different kind of reasoning problem.

Lu: That’s right. We have the batch contrastive loss which is designed to improve feature learning by comparing an anchor against all in-batch negatives. This forces the model to learn features that are much more generalizable, reducing its reliance on those spurious language correlations we talked about earlier.

Meng: So it's like giving the model a better way to see things in the picture so it doesn't get distracted by irrelevant background noise or weird lighting?

Lu: Precisely. By forcing it to learn features that work across many different views of the same image, you get these more robust gradients that help structure the embedding space.

Tom: I like that idea of structuring the feature space. It makes sense because if the underlying representation is messy, nothing else we add on top is going to make it work well at scale.

Jane: And then there’s this answer contrastive loss, which they use to refine the prediction margins. Think of it like pulling a correct answer closer to its true spot in the decision space and pushing incorrect answers further away.

Tom: So it directly enlarges the distance between what's right and what's wrong, so there’s less room for ambiguity when the model makes a choice. That addresses those cases where two answers might look similar on paper, like "red" versus "maroon."

Lu: Exactly. And then there is the gradient discrepancy loss. This one is all about enforcing causal visual grounding. It rewards a big difference in how sensitive the model is when you change something in the question compared to seeing the original image.

Jane: That means it forces the model to develop distinct attention patterns that actually match genuine visual evidence instead of just following a linguistic shortcut. It’s trying to make sure it understands *why* an answer is correct, not just *what* word comes next.

Tom: So we've got a system where one part builds good features, another sharpens the decision boundaries, and the third makes sure those decisions are based on actual visual cause and effect.

Meng: It sounds like you’re taking a lot of specialized training and layering it on top of each other to create this very specific type of reasoning engine. What does this mean for us when we think about deploying something that needs to be reliable but also flexible?

Lu: It shows that you can stabilize optimization across multiple objectives by using this curriculum approach. Stage one builds the base, stage two structures the space with batch contrastive loss, and then stage three activates answer contrastive and gradient discrepancy losses to finalize the precision.

Jane: The big implication for someone listening is that we’re moving toward models that are less likely to fail when they encounter a slight shift in context or question style because they aren't just memorizing training biases anymore.

Tom: It’s about building a more reliable system that balances being accurate on known data with being robust when the real world throws curveballs at it. This framework really shows how multi-objective optimization can work when you introduce counterfactual samples for training.

Conclusion: Tom: So we’ve covered how this paper, "Counterfactual Reasoning for Robust Visual Question Answering," uses a three-stage training schedule and three different loss functions to get better results on bias-sensitive datasets while maintaining a small generalization gap of one point one six percent.

Jane: That's right. The main implication is that we’re moving away from AI models that just guess based on language patterns and toward systems that are actively seeking visual evidence when they encounter new situations or slightly different questions.

Lu: Yeah, the future work they mentioned involves applying this framework to larger Transformer-based architectures to see if this causal reasoning approach scales up effectively. They’re pushing for more advanced multimodal causal reasoning down the road.

Meng: I'm still wondering about the cost of running that whole framework on a real production line. If we need all those different loss functions and complex training, does it just mean we have to pay for way more compute than a simpler baseline model?

Tom: The computational overhead comes from the training phase itself, but they are showing that the gain in robustness under distribution shifts is worth it for applications where reliability really matters. It’s about making the model adaptable rather than just memorizing patterns.

Jane: It sounds like this research is really focused on building a more reliable foundation for visual understanding, which could help us build tools that are dependable even when they aren't seeing things exactly as we expect.

Lalam: I am sure of my answer here; the framework addresses the core issue of memorizing training set biases by enforcing causal grounding through gradient discrepancy loss. I am very sure this direction will lead to more grounded visual question answering systems.

Tom: It’s about shifting the focus from statistical correlation to actual visual cause and effect in how models process information. That’s a big mechanical change for how we engineer these things.

Lu: Precisely. By using counterfactual reasoning, they’re ensuring the model develops distinct attention patterns that align with real visual facts when the input changes slightly.

Jane: It gives us hope that we can finally build visual question answering systems that are genuinely robust and not just brittle on clean test sets.

Meng: For deployment, I'm watching this closely to see if they can simplify those loss functions later so they don't create a massive training bottleneck for real-time use.

Tom: Anyway, that’s the picture on "Counterfactual Reasoning for Robust Visual Question Answering." It gives us a strong approach toward more reliable visual models.

Lu: It does show how multi-objective optimization can stabilize training when you introduce counterfactual samples for VQA.

Jane: We'll be looking out to see how this idea of enforced causal grounding translates into the next generation of visual AI applications.

Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

cs.CV

Submitted: 2026-09-15

Updated: 2026-09-15

Comments: Accepted for publication at the 30th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES 2026). 9 pages, 5 figures

Journal ref: Procedia Computer Science 292 (2026)

DOI: 10.1016/j.procs.2026.09.191

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 85/100

The gist: Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias<ref:2609.16567#pg2>.

Key concepts

Counterfactual Contrastive Learning
This technique generates modified versions of the input (counterfactuals) to train the model. By comparing the original view with these altered views, the model learns features that are robust to small changes in context or language, helping it generalize better outside its training distribution.
Batch-Contrastive Loss (LBC)
Inspired by SimCLR, this loss compares an anchor feature against all other samples within a batch. It forces the model to learn highly discriminative and generalizable features by ensuring that similar inputs are close together in the feature space, thereby reducing reliance on specific training biases.
Gradient-Discrepancy Loss (GD)
This loss enforces causal visual grounding by measuring how much the model's output changes when factual views are compared to counterfactual views. By penalizing low discrepancy, it forces the model to develop distinct attention patterns linked to genuine causal evidence rather than superficial correlations.

Terminology

Summary

Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias<ref:2609. p. 0

Where the paper says it

vn) Accepted for publication at the 30th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES 2026). Abstract Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias.

16567#pg2>. Our framework proposes a novel training framework that enhances counterfactual contrastive learning for VQA<ref:2609. p. 0
Where the paper says it

Although counterfactual learning has shown promise, existing methods can be improved to better guide attention toward causal evidence and strengthen feature discrimination. To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA.

16567#pg2>. The model achieves a competitive accuracy of 61. p. 0
Where the paper says it

Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding. Our model achieves a competitive accuracy of 61.

64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62. p. 0
Where the paper says it

Our model achieves a competitive accuracy of 61. 64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.

80% on the standard VQA v2 dataset, yielding a small generalization gap of 1. p. 0
Where the paper says it

64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62. 80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.

16%<ref:2609. p. 0
Where the paper says it

2 Related Work 2. 1 Language Bias in VQA While modern VQA models demonstrate strong performance on in-distribution (ID) benchmarks like VQA v2 [2], numerous studies [3, 5] have shown they exploit superficial linguistic correlations, or “language biases” in the training 1 arXiv:2609.

p. 0
Where the paper says it

2 Related Work 2. 1 Language Bias in VQA While modern VQA models demonstrate strong performance on in-distribution (ID) benchmarks like VQA v2 [2], numerous studies [3, 5] have shown they exploit superficial linguistic correlations, or “language biases” in the training 1 arXiv:2609.

16567#pg2>.

Pruning redundant reasoning tokens

The paper introduces an enhanced counterfactual contrastive learning framework to address the limitations of existing methods in VQA<ref:2609.16567#pg4>. The framework consists of three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding<ref:2609. p. 0

Where the paper says it

To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA. Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding.

16567#pg2>.

How the training signal is built

The baseline architecture uses an UpDn backbone [18] which extracts image and question features, fused by a module that produces joint features mapped to answer logits<ref:2609.16567#pg4>. The Counterfactual Samples Synthesizing (CSS) module acts as a data augmentation engine, generating anchor, positive, and negative features for contrastive optimization<ref:2609. p. 1

Where the paper says it

However, simply generating more data is often insufficient because how the model utilizes these samples is crucial. This has led to contrastive learning frameworks, where CL-VQA [7] first introduced a contrastive objective to learn the relationship between original (anchor), factual (positive), and counterfactual (negative) samples generated by CSS.

16567#pg4>. The Batch-Contrastive (BC) loss is inspired by SimCLR [19] and compares an anchor against single positive and negative samples to learn more generalizable features<ref:2609. p. 1
Where the paper says it

The original CL-VQA objective [7] compares an anchor against single positive and negative samples. To learn a more discriminative representation, we adopt a Batch-Contrastive (BC) objective inspired 2

16567#pg4>.

Where the gains come from

The proposed method utilizes three complementary loss functions targeting different reasoning aspects<ref:2609.16567#pg4>. These include the Batch-Contrastive Loss (LBC) to structure the feature space, the Answer-Contrastive Loss (AC) to refine prediction margins, and the Gradient-Discrepancy Loss (GD) to enforce causal visual grounding<ref:2609. p. 0

Where the paper says it

To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA. Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding.

16567#pg4>. The GD loss is formulated as a weighted objective that rewards gradient dissimilarity between factual and counterfactual views, penalizing low discrepancy which implies shortcut learning<ref:2609. p. 3
Where the paper says it

Large discrepancy implies proper causal grounding, while low discrepancy implies shortcut learning, which we aim to penalize. To formalize this, we define the GD loss as a weighted objective that rewards gradient dissimilarity.

16567#pg5>.

A three-stage curriculum training strategy is employed to progressively introduce the complex objectives<ref:2609.16567#pg4>. Stage 1 optimizes only the VQA task loss, Stage 2 adds BC to structure the embedding space, and Stage 3 activates AC and GD to refine answer margins and enforce visual grounding<ref:2609. p. 3

Where the paper says it

Stage 2 (Epochs 6–15) adds BC to structure the embedding space around causal relationships. Stage 3 (Epochs 16–30) activates AC and GD to refine answer margins and enforce visual grounding: L (1) = LVQA, L (2) = LVQA +λBCLBC, L (3) = LVQA +λBCLBC +λACLAC +λGDLGD.

16567#pg5>. This schedule stabilizes optimization by incrementally strengthening causal reasoning<ref:2609. p. 3
Where the paper says it

(4) The weighting coefficients λBC, λAC, and λGD are reported in Section 4 and analyzed in Table 3. This schedule stabilizes optimization and incrementally strengthens causal reasoning.

16567#pg5>. The combination of AC and GD losses achieves the highest overall accuracy of 61. p. 6
Where the paper says it

52%), while AC proves crucial for Number questions (53. Their combination achieves the highest overall accuracy of 61.

64%, confirming their complementary roles unlocked by the BC loss<ref:2609. p. 6
Where the paper says it

Their combination achieves the highest overall accuracy of 61. 64%, confirming that AC and GD play complementary roles unlocked by the discriminative feature space of our BC loss.

16567#pg9>.

The results demonstrate that our framework reduces reliance on spurious language correlations, enabling better generalization under distribution shifts<ref:2609.16567#pg9>. The qualitative results show that the AC and GD losses successfully guide the model toward causal evidence for accurate and well-grounded reasoning<ref:2609. p. 4

Where the paper says it

It accurately encompasses the red flowers while suppressing distractors like yellow flowers, with linguistic attention correctly centered on the causal terms “flowers” and “red”. Collectively, these examples demonstrate that the AC and GD losses successfully guide the model toward causal evidence for accurate and well-grounded reasoning.

16567#pg5>. The analysis of answer distribution confirms that our model adapts to the test distribution rather than memorizing training-set biases<ref:2609. p. 4
Where the paper says it

4 Analysis of Answer Distribution To provide direct evidence of bias mitigation, we analyze predicted answer distributions on the VQA-CP v2 test set (Figure 5). The analysis confirms our model adapts to the test distribution rather than memorizing training-set biases.

16567#pg9>. The model correctly aligns with the test set’s “green” majority for questions like “Is this…? p. 5
Where the paper says it

Figure 5: Comparison of answer distributions predicted by our model and the CL-VQA baseline, alongside the groundtruth distributions of the train and test sets of VQA-CP v2 for several representative question types. model correctly aligns with the test set’s “green” majority.

” which overcomes training bias toward “no”<ref:2609. p. 5
Where the paper says it

For “Is this. ” questions, our model successfully overcomes the training bias toward “no” to reflect the test set’s “yes” majority.

16567#pg9>. The final performance shows that the framework achieves 61. p. 0
Where the paper says it

Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding. Our model achieves a competitive accuracy of 61.

64% accuracy on VQA-CP v2 and 62. p. 0
Where the paper says it

Our model achieves a competitive accuracy of 61. 64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.

80% on VQA v2, with a small generalization gap of 1. p. 0
Where the paper says it

64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62. 80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.

16%<ref:2609.16567#pg2>. The best overall trade-off between OOD robustness and ID generalization is achieved by setting λBC = 2. p. 6
Where the paper says it

Ours (Setting 2) showcases the best overall trade-off between OOD robustness and ID generalization. Model Weights VQA-CP v2 test (%) VQA v2 val (%) λBC λAC λGD All Y/N Num Other All Y/N Num Other MUTANT [15] - - - 61.

0, λAC = 2. p. 3
Where the paper says it

Stage 3 (Epochs 16–30) activates AC and GD to refine answer margins and enforce visual grounding: L (1) = LVQA, L (2) = LVQA +λBCLBC, L (3) = LVQA +λBCLBC +λACLAC +λGDLGD. (4) The weighting coefficients λBC, λAC, and λGD are reported in Section 4 and analyzed in Table 3.

0, and λGD = 6. p. 3
Where the paper says it

Stage 3 (Epochs 16–30) activates AC and GD to refine answer margins and enforce visual grounding: L (1) = LVQA, L (2) = LVQA +λBCLBC, L (3) = LVQA +λBCLBC +λACLAC +λGDLGD. (4) The weighting coefficients λBC, λAC, and λGD are reported in Section 4 and analyzed in Table 3.

0<ref:2609. p. 0
Where the paper says it

2 Related Work 2. 1 Language Bias in VQA While modern VQA models demonstrate strong performance on in-distribution (ID) benchmarks like VQA v2 [2], numerous studies [3, 5] have shown they exploit superficial linguistic correlations, or “language biases” in the training 1 arXiv:2609.

16567#pg9>. The framework presents a robust approach towards more reliable VQA models that balance OOD robustness and ID performance<ref:2609. p. 7
Where the paper says it

This work presents a robust approach towards more reliable VQA models that balance OOD robustness and ID performance. Future work includes applying this framework to larger Transformer-based architectures to further advance multimodal causal reasoning.

16567#pg6>. The work demonstrates a strong balance between OOD robustness and in-distribution performance<ref:2609. p. 0
Where the paper says it

80% on the standard VQA v2 dataset, yielding a small generalization gap of 1. This demonstrates a strong balance between OOD robustness and in-distribution performance.

16567#pg2>. The future work includes applying this framework to larger Transformer-based architectures to further advance multimodal causal reasoning<ref:2609. p. 7
Where the paper says it

This work presents a robust approach towards more reliable VQA models that balance OOD robustness and ID performance. Future work includes applying this framework to larger Transformer-based architectures to further advance multimodal causal reasoning.

16567#pg6>. The research is funded by Vietnam National University, HoChiMinh City (VNU-HCM) under grant number CB2025-18-04<ref:2609. p. 7
Where the paper says it

Future work includes applying this framework to larger Transformer-based architectures to further advance multimodal causal reasoning. Acknowledgements This research is funded by Vietnam National University, HoChiMinh City (VNU-HCM) under grant number CB2025- 18-04.

16567#pg9>.

REFERENCE

arXiv:2609.16567v1 [cs. p. 0

Where the paper says it

1 Language Bias in VQA While modern VQA models demonstrate strong performance on in-distribution (ID) benchmarks like VQA v2 [2], numerous studies [3, 5] have shown they exploit superficial linguistic correlations, or “language biases” in the training 1 arXiv:2609. 16567v1 [cs.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Batch-Contrastive Loss (LBC) improves feature learning by comparing an anchor against all in-batch negatives using SimCLR, which supplies more robust gradients and forces the model to learn more generalizable features. This leads to a reduction in reliance on spurious language correlations.

  2. Answer-Contrastive Loss (AC) refines the prediction space by pulling anchor and positive features closer to ground-truth answer embeddings, directly enlarging decision margins between correct and incorrect predictions. This addresses ambiguity between semantically similar answers like red versus maroon.

  3. Gradient-Discrepancy Loss (GD) enforces causal visual grounding by compelling a significant difference in model sensitivity between factual and counterfactual views, penalizing shortcut learning. This forces the model to develop distinct attention patterns that align with genuine causal evidence.

  4. A three-stage curriculum training framework stabilizes optimization by progressively introducing objectives: Stage 1 establishes base representations, Stage 2 adds Batch-Contrastive Loss to structure the embedding space, and Stage 3 activates Answer-Contrastive and Gradient-Discrepancy Losses to refine prediction margins and enforce visual grounding.

  5. The improved system can achieve competitive accuracy on bias-sensitive datasets like VQA-CP v2 while maintaining a small generalization gap of 1.16% on the standard VQA v2 dataset, demonstrating a strong balance between out-of-distribution robustness and in-distribution performance.

Abstract

Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias. Although counterfactual learning has shown promise, existing methods can be improved to better guide attention toward causal evidence and strengthen feature discrimination. To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA. Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding. Our model achieves a competitive accuracy of 61.64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.16%. This demonstrates a strong balance between OOD robustness and in-distribution performance.

Related papers