Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention
cs.LG
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/NVIDIA/TensorRT-LLM
Project page: https://nvidia.github.io/TensorRT-LLM
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-horizon reasoning and agentic workloads strain KV-cache capacity and decode attention throughput.
Terminology
Abstract
Long-horizon reasoning and agentic workloads strain KV-cache capacity and decode attention throughput. We present Jet-QSparse, a training-free method that combines compact NVFP4 cache storage with native FP4 query-key computation on Blackwell tensor cores. A dense scan scores the full key history and selects 128-token blocks for sparse FP16 value aggregation with FP32 accumulation, preserving higher precision in the output-forming step while enabling contiguous tile loads. Calibrated key centering and at most two BF16 pages per sequence and layer address quantization error; FP8 prefill uses a temporary arena while the persistent cache is packed directly from BF16 activations. Across four hybrid LLMs, a budget of 128 blocks per KV head shows no detected accuracy loss on the evaluated mathematical reasoning, coding, and agentic benchmarks, although some smaller budgets incur losses. With 64 blocks, Qwen3.8-27B achieves a 1.94 times end-to-end decode speedup over dense BF16 at 128K context and batch size 16 on one GB300, and the measured Qwen KV pools provide 2.58 - 2.76 times BF16's token capacity.
Sources
- CuTe Layout Representation and Algebra
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Hardware-Aware FP4 FlashAttention-4
- Pretraining Large Language Models with NVFP4
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis