Strategic Self-Consistency
cs.LG, cs.AI, stat.ML
Submitted: 2026-09-24
Updated: 2026-09-24
Code: https://github.com/Human-Centric-Machine-Learning/strategic-self-consistency
License: http://creativecommons.org/licenses/by/4.0/
The gist: Self-consistency has become a popular technique for enhancing the reasoning abilities of large language models by generating multiple reasoning paths and selecting the final answer through a majority
Terminology
Abstract
Self-consistency has become a popular technique for enhancing the reasoning abilities of large language models by generating multiple reasoning paths and selecting the final answer through a majority vote. However, because model providers typically charge users in proportion to the number of reasoning paths generated, they have a financial incentive to artificially increase the path count. In this work, we show that an unfaithful provider can exploit this incentive using a simple, efficient algorithm while avoiding detection by an auditor: by generating and strategically reordering additional reasoning paths, the algorithm makes every path appear necessary to reach the majority. To validate our algorithm, we conduct experiments with multiple instruct models from the Llama and Qwen families, as well as reasoning models distilled from DeepSeek-R1, on benchmark datasets spanning mathematics, science, and question answering. Our results suggest that the distribution of additional reasoning paths generated by our algorithm is heavy-tailed and that substantial capacity to overcharge remains even under the best possible audit designed to keep the false-positive rate below α= 0.1.
Sources
- Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs
- Training Verifiers to Solve Math Word Problems
- More Test-Time Compute Can Hurt: Overestimation Bias in LLM Beam Search
- Competition and Diversity in Generative AI
- Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
- Distribution-free changepoint localization after sequential change detection
- CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs
- Test-Time Compute Games
- BEACON: Bayesian Optimal Stopping for Efficient LLM Sampling
- Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks