AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement
cs.AI
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/NVIDIA/Model-Optimizer
Terminology
Sources
- DGEMM with Ozaki Scheme I/II on FP4 Tensor Cores: A Base-13 E2M1 Limb Representation
- FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
- SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning
- QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
- MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving
- Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
- ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling
- FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
- PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
- Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
- Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
- Matryoshka Quantization
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs
- Microscaling Data Formats for Deep Learning
- MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
- PhaseQuant: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics
- MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents