GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training

arXiv:2607.07494 · cs.DC, cs.LG · Submitted 2026-07-08 · Read on arXiv

cs.DC, cs.LG

Submitted: 2026-07-08

Updated: 2026-09-30

Project page: http://skylion007.github.io/OpenWebTextCorpus

Terminology

• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Quantization — 1× in this paper · explained in 12 episodes

Sources

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• FP8-LM: Training FP8 Large Language Models
• COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
• Decoupled Weight Decay Regularization
• SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Memory-Efficient Pipeline-Parallel DNN Training
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
• Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• FP8 Formats for Deep Learning
• LoCo: Low-Bit Communication Adaptor for Large-scale Model Training
• Mixed Precision Training

Related papers

• iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
• SAMM: Sharded Automated Market Maker
• InferScale: GPU-Native KV Injection for Personalized LLM Serving
• Vigil: Accountable Liveness against Selective Silence
• Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
• Democratizing MoE inference on commodity GPUs with CoMoE