Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs
cs.AI
Submitted: 2026-11-03
Updated: 2026-11-03
Terminology
Sources
- Phi-4 Technical Report
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Gemma 3 Technical Report
- Faster Inference of LLMs using FP8 on the Intel Gaudi
- FP8 Formats for Deep Learning
- gpt-oss-120b & gpt-oss-20b Model Card
- Qwen2.5 Technical Report
- Qwen3 Technical Report
- Microscaling Data Formats for Deep Learning
- Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
- Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis
- A Survey on Efficient Inference for Large Language Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents