The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures
cs.DC, cs.AI, cs.LG, cs.PF
Submitted: 2026-05-12
Updated: 2026-10-29
Comments: Accepted by PPAM 2026 (16th International Conference on Parallel Processing and Applied Mathematics)
Code: https://github.com/fla-org/flash-linear-attention
License: http://creativecommons.org/licenses/by/4.0/
The gist: Power capping is a common GPU energy lever in LLM serving, and it appears to work: the cap is set, the GPU reports drawing less than it, and the energy budget is met.
Terminology
Abstract
Power capping is a common GPU energy lever in LLM serving, and it appears to work: the cap is set, the GPU reports drawing less than it, and the energy budget is met. We show the cap does little for the phase that dominates LLM serving: autoregressive decode. Across four attention paradigms (GQA, MLA, Gated DeltaNet, and Mamba2) on an NVIDIA H200 under vLLM eager execution, decode kernels wait on HBM and the GPU idles between kernel launches, so decode draws less than half of the GPU's 700W rating and throughput stays within a few percent under every cap we set. The cap engages only when a large batch runs a long generation. Locking the SM clock instead lowers power without slowing memory-bound decode and recovers up to 32% of decode energy at minimal throughput loss. We identify three architecture-dependent frequency-scaling classes and find that, once prefill is measured without prefix caching, the recurrent architectures pay no prefill penalty: their linear-time mixing undercuts quadratic attention on long prompts, so their efficient decode adds to the prefill saving and lowers total request energy from the first output token at large batch.
Sources
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- TransMLA: Multi-Head Latent Attention Is All You Need
- Compact Language Models via Pruning and Knowledge Distillation
- POLCA: Power Oversubscription in LLM Cloud Providers
- Splitwise: Efficient generative LLM inference using phase splitting
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
Related papers
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends
- AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition