Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression
cs.LG
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/ASzecsenyi/transformer-fsq
Terminology
Sources
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
- Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
- Kimi K3: Open Frontier Intelligence
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
- Steering Language Models With Activation Engineering
- VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
- Qwen3 Technical Report
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis