Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference
cs.AI, cs.LG
Submitted: 2026-10-09
Updated: 2026-10-09
Sources
- EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Acceleration
- Does compressing activations help model parallel training?
- Numerical stability analysis of large language models
- FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3 Technical Report
- Towards Low-bit Communication for Tensor Parallel LLM Inference
- The Llama 3 Herd of Models
- Communication Compression for Tensor Parallel LLM Inference
- Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
- FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
- FP8 Formats for Deep Learning
- Pretraining Large Language Models with NVFP4
- Activations and Gradients Compression for Model-Parallel Training
- FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
- BitNet: Scaling 1-bit Transformers for Large Language Models
- Optimization Hyper-parameter Laws for Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems