Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
Rasul Khanbayov, Hasan Kurban
cs.LG
Submitted: 2026-08-07
Updated: 2026-08-10
Code: https://github.com/KurbanIntelligenceLab/rend-equiv
License: http://creativecommons.org/licenses/by/4.0/
The gist: Label-free reliability for vision-language models rests on invariance: perturb the input and a faithful reader's answer should not change.
Terminology
Abstract
Label-free reliability for vision-language models rests on invariance: perturb the input and a faithful reader's answer should not change. This has a known blind spot, a systematic misreading survives the perturbation and gets certified wrong, which we show is computable, not just real: an error is invisible to an edit exactly when the two commute, so the errors a suite cannot reach form its joint centralizer, a set that shrinks as edits are added and can be written down rather than guessed at. We act on the complementary relation, equivariance: edit a figure's data and the correct answer must change by a computable amount. Two matched edits are provably complete for affine reading errors; no suite of swap edits is complete for label permutations, and cyclic relabeling closes most of that gap. We instantiate the theory as the Equivariance-Consistency Score, a label-free, training-free detector, and release REND-EQUIV, pairing matched invariance and equivariance sets over identical data. The predicted ordering holds across three models and a hand-labeled population immune to the one circularity in how it is selected; a second invariance-family method confirms the blind spot belongs to the relation, not to any implementation; and cyclic relabeling delivers its predicted gain on a matched real sample. The same characterization explains a reported inversion of this ordering in the classifier metamorphic-testing literature: detectability is a joint property of the relation and the fault class, never of the relation alone.
Sources
- Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines
- VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation
- MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
- Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey
- Minimum Complete MR Subsets under Semantic-Mutation Fault Models: A Support-Set Domination Boundary
- Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness
- Losing the Plot: How VLM responses degrade on imperfect charts
- Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
- On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
- Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication
- Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucination
- ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Qwen2.5-VL Technical Report
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks