Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
cs.AI
Submitted: 2026-11-10
Updated: 2026-11-10
Terminology
Sources
- Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
- BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
- QuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switching
- Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design
- Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
- MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification
- Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs
- Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference
- Matryoshka Quantization
- MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
- QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
- Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification
- TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification
- Faster Cascades via Speculative Decoding
- Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
- Confident Adaptive Language Modeling
- Entropy-Aware Token Rejection for Improving Speculative Decoding
- Training Verifiers to Solve Math Word Problems
- Qwen3 Technical Report
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents