Inference for Reasoning Models: Practical Lessons from Evaluation and Serving
cs.AI
Submitted: 2026-11-07
Updated: 2026-11-07
Code: https://github.com/ai-dynamo/nixl
Terminology
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
- SGLang: Efficient Execution of Structured Language Model Programs
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
- Training Verifiers to Solve Math Word Problems
- Measuring Massive Multitask Language Understanding
- Are We Done with MMLU?
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
- Evaluating Large Language Models Trained on Code
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- FP8 Formats for Deep Learning
- Microscaling Data Formats for Deep Learning
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents