PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis
Xiaomin He, Dongling Xiao, Jiahao Xie, Ruiqi Lu, Qianle Wang, Zhongbin Guo, Wanxuan Sun
cs.LG, cs.AI
Submitted: 2026-08-07
Updated: 2026-08-10
License: http://creativecommons.org/licenses/by/4.0/
The gist: Real-world multimodal instructions often bundle multiple requirements with unequal importance, yet most multimodal training data still reduce instruction following to answering one self-contained
Terminology
Abstract
Real-world multimodal instructions often bundle multiple requirements with unequal importance, yet most multimodal training data still reduce instruction following to answering one self-contained question. We study this gap through rubric comprehension, which casts the model not as a generator measured against rubrics but as an executor that follows them: given an image and a typed, prioritized rubric, the model must verify each rule before producing an overall judgment. To support this setting, we propose PRISM, a four-stage data synthesis framework that produces persona--task pairs, prefix-guided rule sets, quality-filtered rubrics, and structured verification traces. We further introduce PRISM-Eval, whose Loose and Strict metrics use deterministic matching against fixed labels and therefore require no inference-time judge model. With only 10K synthesized samples, PRISM lifts Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on PRISM-Eval while preserving average performance on general benchmarks, and the gains transfer to four additional open-source MLLMs across dense and MoE architectures, suggesting that structured rubric supervision is a scalable path toward multi-rule, priority-aware multimodal instruction following.
Sources
- Qwen3-VL Technical Report
- MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
- LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
- Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis
- Instruction-Following Evaluation for Large Language Models
- AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
- Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
- ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
- Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
- How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
- Visually Dehallucinative Instruction Generation: Know What You Don't Know
- Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models
- ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots
- LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks