Reward Design for Physical Reasoning in Vision-Language Models

arXiv:2604.13993 · cs.AI, cs.CL, cs.CV · Submitted 2026-04-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Reward Design for Physical Reasoning in Vision-Language Models".

Jane: The gist The reward design shapes physical reasoning behavior in Vision-Language Models by comparing four reward signals—format compliance, answer accuracy, a composite rubric reward,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Okay, so the title is "Reward Design for Physical Reasoning in Vision-Language Models," and honestly, it sets up this comparison of different training signals.

Jane: It’s about taking these models that can see images and read text, like Vision Language Models, and figuring out the best way to give them feedback to help them do physics reasoning better.

Lu: The authors are systematically testing four reward configurations—format compliance, answer accuracy, a composite rubric reward which includes correctness plus principle identification and unit consistency, and then this new internal attention-based reward.

Meng: What's interesting is that they aren't just looking at one setting; they’re checking five different training conditions to see how these signals interact with each other.

Lalam: Essentially, the goal is to find out which combination of these reward signals actually shapes the model into a better physical reasoning machine.

The paper's summary: Tom: The paper summarizes that accuracy-based rewards are generally providing the strongest overall gains across their tests for physics problems.

Jane: For example, when they looked at multiple choice questions, the combination of format, accuracy, and answer correctness emerged as a really strong configuration.

Lu: They found that GRPO training with format plus accuracy plus answer correctness scored zero point four six two overall on multiple choice questions and also ranked second in three other physics domains on the PhyX benchmark.

Meng: But it’s not always the same for open-ended questions, where things get a bit more nuanced.

Lalam: For open-ended problems, they found that format plus accuracy was their strongest configuration overall, giving a score of zero point zero two seven and showing a two times improvement over the standard supervised fine-tuned model.

The paper's improvements: Tom: The authors suggest that reward design isn't just about getting the right answer; it actually induces different types of reasoning behaviors in the model, which is a really important point.

Jane: They show that using an internal attention-based reward, which we call R4, specifically enhances spatial reasoning skills but it can degrade performance when dealing with symbolic domains like thermodynamics and wave or acoustics.

Lu: And they also found that using the composite rubric reward shows an eleven point eight percent accuracy decrease compared to just format plus accuracy for open-ended evaluations.

Meng: That suggests that when you try to optimize a multi-objective reward signal on a smaller model, you can actually destabilize the whole training process with GRPO optimization.

Lalam: So the paper points out that there isn't one single best way to train these models; the choice of which reward signal to use depends entirely on what you want it to do.

Conclusion: Tom: Wrapping up, this whole study suggests that reward design should be seen as a mechanism for controlling the nature of reasoning in these multimodal models.

Jane: It means we need to choose our reward signal based on what we need at deployment: do you want raw accuracy, or do you want interpretable reasoning chains?

Lu: The choice of reward signal should be guided by deployment needs, which means if you want visual grounding, use the attention-based reward, but if you prioritize symbolic reasoning capacity, that’s a different path.

Meng: It's clear that visual grounding supervision and symbolic reasoning capacity end up competing for the representational resources available at the two billion parameter scale.

Lalam: So for anyone working on this, it means matching the reward to the target behavior is just as important as picking the right training algorithm itself.

Tom: Exactly. The paper "Reward Design for Physical Reasoning in Vision-Language Models" shows that there isn't one configuration that dominates across all physics domains or reasoning types, and the choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding.

Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

Pacific Northwest National Laboratory

cs.AI, cs.CL, cs.CV

Submitted: 2026-04-15

Updated: 2026-10-14

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist The reward design shapes physical reasoning behavior in Vision-Language Models by comparing four reward signals—format compliance, answer accuracy, a composite rubric reward, and an

Key concepts

External Rewards: These are rewards based on outside verification, such as checking if an answer follows a specific format or getting a score from an LLM judge. They rely on human-defined rules or another model to grade the output, providing feedback without requiring labels directly from the model.
Internal Attention-Based Reward: This is a reward calculated using the model's own attention mechanism, looking at which parts of the image tokens it focuses on. It acts as an annotation-free signal that helps improve spatial reasoning by rewarding the model for paying attention to relevant visual regions.
Reward Design: This refers to choosing and combining different signals (like accuracy or attention) to guide a model's training. The paper shows that the specific reward chosen significantly shapes the type of reasoning behavior the VLM develops, not just what it gets right.
GRPO Training: Group Relative Policy Optimization (GRPO) is the specific reinforcement learning algorithm used to train the Vision-Language Model. It uses these various reward signals to iteratively adjust the model's policy, steering its behavior toward better physical reasoning outcomes.

Terminology

Accuracy — 5× in this paper · explained in 1 episode
GRPO — 3× in this paper · explained in 6 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Ablation — 1× in this paper · explained in 1 episode
Ablation Study — 1× in this paper · explained in 3 episodes
Consistency — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Group Relative Policy Optimization — 1× in this paper
Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 6 episodes
Structured Reasoning — 1× in this paper · explained in 3 episodes
Supervised Fine-Tuning — 1× in this paper
Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist The reward design shapes physical reasoning behavior in Vision-Language Models by comparing four reward signals—format compliance, answer accuracy, a composite rubric reward, and an internal attention-based reward—on a 3,000-problem physics benchmark.

System Overview

The paper presents a systematic reward ablation study for Group Relative Policy Optimization (GRPO)-based Vision Language Model (VLM) training on physical reasoning to understand how different reward signals shape the resulting behavior. The study evaluates IBM Granite Vision 3.3 (2B) on the PhyX benchmark. The main contributions include a systematic reward ablation study covering format, accuracy, rubric, and attention-based reward signals across five training conditions.

Reward Signal Designs

The researchers compare four distinct reward configurations of increasing semantic richness. The rewards are structured to supervise different aspects of the output. For the rubric reward (R3), they define components based on question type. For open-ended questions (OE), they replace rule-based components with a structured LLM judge call that jointly evaluates all rubric dimensions. Finally, the attention-weight reward (R4) is an internal reward derived from the model’s own attention distribution over image tokens.

Experimental Results and Findings

The experimental results show that accuracy-based rewards provide the strongest overall gains. The GRPO (Fmt + Acc + ASM) configuration emerged as the strongest overall for Multiple Choice Question Answering (MCQ), achieving an overall score of 0.462 and also posting second in three other physics domains. For Open-Ended (OE) questions, GRPO (Fmt + Acc) emerged as the strongest overall configuration with a score of 0.027 and achieved a 2× improvement (100%) in scores over the SFT model overall.

Impact of Reward Design

The study reveals that reward design induces domain-specific reasoning behaviors. The attention-based reward (ASM) enhances spatial reasoning while degrading performance in symbolic domains such as Thermodynamics and Wave/Acoustics. Furthermore, the composite rubric reward shows an 11.8% accuracy decrease relative to Fmt+Acc for Open Ended (OE) evaluations and suggests that optimizing a multi-objective reward in a small model can destabilize GRPO optimization.

Conclusion

The findings suggest that reward design should be viewed as a mechanism for controlling the nature of reasoning in multimodal models. The choice of reward signal should be guided by deployment needs: raw accuracy, interpretable reasoning chains, or visual grounding. No single reward configuration dominates across all physics domains or reasoning types. This work suggests that as reward complexity increases, so does the variance of the training signal. Reward design shapes not just what the model gets right, but how it reasons>. The results indicate that visual grounding supervision and symbolic reasoning capacity compete for representational resources at the 2B scale. Reward design shapes not just what the model gets right, but how it reasons>. The paper concludes that matching the reward to the target behavior is as important as the training algorithm itself. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>. No single reward configuration dominates across all physics domains or reasoning types. The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding>.

Improvements for AI systems

Improve physical reasoning accuracy by implementing a multi-reward strategy tailored to specific tasks, as Reward design does not uniformly improve performance. Instead, it induces domain-specific reasoning behaviors. For instance, using GRPO (Fmt + Acc + ASM) yields the strongest overall configuration for MCQ tasks (0.462), while the attention reward specifically boosts spatial reasoning accuracy from 0.27 to 0.50 on that specific task type.
Enhance structured reasoning quality by incorporating rubric rewards, but be mindful of optimization stability as noted: When the reward signal simultaneously scores answer correctness, principle identification, unit consistency, and reasoning quality, the gradient updates reflect a weighted combination of these objectives. This suggests that while GRPO (Rubric) achieves high scores in some domains (e.g., Thermodynamics 0.463), it can be less consistent than accuracy-based rewards.
Develop models capable of robust spatial grounding by leveraging internal attention weights, as the Attention-based reward enhances spatial reasoning while degrading performance in symbolic domains. This is achieved by using the Attention Score Mask (ASM) reward, which is derived from the fraction of attention mass on non-white foreground pixels, resulting in a high score when the model attends to meaningful image content.
Improve reasoning chain coherence by integrating explicit reasoning quality supervision, as the paper notes: The Rubric method shows an 11.8% accuracy decrease relative to Fmt+Acc (Figure 3a, right), but the Rubric model maintains coherent reasoning chains despite lower accuracy. This indicates that for open-ended tasks, supervising valid step-by-step derivation is crucial for producing logically sound outputs.
Implement dynamic reward selection based on deployment goals: The choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding. This allows practitioners to select the appropriate configuration—such as GRPO (Fmt + Acc + ASM) for high overall MCQ performance or a rubric-based approach for detailed open-ended explanations.

Abstract

Physical reasoning over images requires integrating perception, domain knowledge, and multi-step symbolic inference, yet Vision-Language Models (VLMs) fall far short of human performance on physics benchmarks. Reinforcement learning can improve reasoning via reward signals, but it remains unclear whether these signals must come from outside the model or can be derived from the model itself. We compare external rewards, which rely on an outside verifier (rule-based format and answer-correctness checks, or an LLM-judge rubric), with an internal attention-based reward computed from the model's own attention over image regions, requiring no labels or judge. We find that external rewards drive most of the accuracy gains, while preserving more coherent reasoning when rewarded for reasoning tokens. The internal attention-based reward, provides an annotation-free grounding signal that improves spatial-relation reasoning and is complementary to external rewards: adding it to format and accuracy rewards, at no additional annotation cost, improves average accuracy on Molmo 2 while generalizing to out-of-distribution tasks. Our results suggest that internal attention rewards can supplement external supervision and lead to improved physical reasoning.