Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models
cs.AI, cs.CL
Submitted: 2026-11-10
Updated: 2026-11-10
Terminology
Sources
- One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation
- Large Language Models Are Not Robust Multiple Choice Selectors
- Quantization Degradation in Large Language Models: A Signal-Noise Perspective
- Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
- When Quantization Affects Confidence of Large Language Models?
- A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation
- Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think
- Mitigating Selection Bias with Node Pruning and Auxiliary Options
- Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
- Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
- Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers
- Investigating Social Bias Changes in Quantized Language Models
- How Quantization Shapes Bias in Large Language Models
- QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
- Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
- A Comprehensive Evaluation of Quantization Strategies for Large Language Models
- The case for 4-bit precision: k-bit Inference Scaling Laws
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Measuring Massive Multitask Language Understanding
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents