RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Anthony. Lui, Mohamed. Elsaied, N. P. Savani
cs.NE, cs.LG
Submitted: 2026-08-08
Updated: 2026-08-11
Comments: 9 Pages, 9 Tables, Initial submission into NeurIPS 2026
Code: https://github.com/ml-explore/mlx
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value
Terminology
Abstract
Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand. We present RotaryQuant, a three-axis compression system that addresses all three. Mixed-precision weight quantization assigns bit-widths by architectural role: 4-bit for dense layers, 2-bit for routed experts, and 8-bit for the shared expert whose high activation kurtosis resists aggressive compression. LRU expert offloading pages non-resident experts to disk under genuine memory pressure. The novel axis is IsoQuant, a KV cache compression method that applies a Walsh--Hadamard transform followed by block-diagonal SO(4) rotations to isotropize activation distributions before 3-bit scalar quantization, requiring O(d d) operations and 256 stored parameters per head versus O(d 2) and 16, 384 for dense rotation methods. A fused four-kernel Metal GPU pipeline performs attention directly on packed 3-bit tensors without materializing full-precision KV state---a different execution model, not just a quantization scheme. The combined system fits Gemma 4-26B-A4B and Qwen3-30B-A3B within a 16,GB budget and Nemotron-H 120B within 32,GB, running interactively at 9--19 tok/s with near-zero perplexity degradation (PPL at most +0.0012) and 100% retrieval accuracy at 32K context.
Sources
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- SpinQuant: LLM quantization with learned rotations
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
- TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
Related papers
- Evolutionary Ensemble of Agents
- Encoding and Decoding Temporal Signals with Spiking Bandpass Wavelets
- Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems
- Evolutionary Architecture Search for Chlorophyll- a Prediction in Lakes using Sentinel-2
- Learning Alzheimer's Disease Signatures by bridging EEG with Spiking Neural Networks and Biophysical Simulations
- Neuromorphic Satellite Intelligence: A Formal Framework for Event-Driven Architectures, Spike-Based Algorithms, and Quantitative Energy-Accuracy Trade-offs