RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning
cs.CV, cs.AI, cs.LG
Submitted: 2026-03-10
Updated: 2026-09-26
License: http://creativecommons.org/licenses/by/4.0/
The gist: Dense image captioning is critical for vision-language pretraining and for text-to-image generation, but scaling expert-quality annotations is prohibitively expensive.
Terminology
Abstract
Dense image captioning is critical for vision-language pretraining and for text-to-image generation, but scaling expert-quality annotations is prohibitively expensive. While synthesizing captions from strong vision-language models (VLMs) is a practical alternative, supervised distillation often yields limited output diversity and weak generalization. Reinforcement learning (RL) could overcome these, but its successes have so far been concentrated in verifiable domains that rely on deterministic checkers---a luxury not available in open-ended captioning. We address this verification bottleneck with RubiCap, an RL framework that derives fine-grained, sample-specific rewards from LLM-written rubrics. For each image, an LLM rubric writer compares captions from a diverse committee of VLMs to identify consensus strengths and diagnose the current policy's deficiencies. These findings are converted into explicit evaluation criteria, enabling an LLM judge to decompose quality assessment and replace coarse scalar rewards with structured, multi-faceted assessments. Across extensive benchmarks, RubiCap achieves the highest win rates on CapArena, outperforming supervised distillation, prior RL methods, human-expert annotations, and GPT-4V-augmented outputs. On CaptionQA, it shows superior word efficiency: our 7B model matches Qwen2.5-VL-32B-Instruct, and our 3B model surpasses its 7B counterpart. Remarkably, using a compact RubiCap-3B as a captioner produces stronger pretrained VLMs than those trained on captions from proprietary models.
Sources
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Text-only Synthesis for Image Captioning
- From Pixels to Prose: A Large Dataset of Dense Image Captions
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
- RL's Razor: Why Online Reinforcement Learning Forgets Less
- Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
- Kimi k1.5: Scaling Reinforcement Learning with LLMs
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
- CaptionQA: Is Your Caption as Useful as the Image Itself?
- DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
- VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
- Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
- Reinforced Visual Perception with Tools
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models