When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering
Rasul Khanbayov, Hasan Kurban
cs.LG
Submitted: 2026-08-06
Code: https://github.com/KurbanIntelligenceLab/rendeq
License: http://creativecommons.org/licenses/by/4.0/
The gist: A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness.
Terminology
Abstract
A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image perturbations preserve meaning only by assumption, and no exact answer key localizes errors. Scientific figures remove both obstacles, a figure is drawn from data by a program, so redrawing it yields images that are semantically equivalent by construction and share a programmatically exact answer. We build RENDEQ, a generator of such render-equivalence sets, and measure the coupling on three open-weight VLMs, checking every finding across three independent instantiations. Re-rendering beats resampling on both accuracy and reliability. Agreement beats an evidence-carrying baseline, mean token log-probability, on two of three models and ties on the third, reversing an intermediate, buggy replication traced to a rendering-pipeline failure. The dispersion behind this is concentrated in one style factor, the plotting library, more than double the next-largest factor and an order of magnitude above the noise floor. Fine-tuning on the model's own cross-render consensus inverts: accuracy falls in every one of five replication runs, the opposite sign to published results on natural images. Agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.
Sources
- VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation
- Questioning the Stability of Visual Question Answering
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
- ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation
- Losing the Plot: How VLM responses degrade on imperfect charts
- On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
- V-FAT: Benchmarking Visual Fidelity Against Text-bias
- CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
- Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Qwen2.5-VL Technical Report
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks