FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
cs.AI
Submitted: 2026-11-03
Updated: 2026-11-03
Terminology
Sources
- Pretraining Large Language Models with NVFP4
- signSGD: Compressed Optimisation for Non-Convex Problems
- Language Models are Few-Shot Learners
- Metis: Training LLMs with FP4 Quantization
- PaLM: Scaling Language Modeling with Pathways
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- Reducing Activation Recomputation in Large Transformer Models
- FP8-LM: Training FP8 Large Language Models
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Local SGD Converges Fast and Communicates Little
- Optimizing Large Language Model Training Using FP4 Quantization
- Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents