Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
Seon Ho Kim, Ui Jeong Jeon, Su Hyeon Kim, Min Tae Hwang
cs.DC, cs.LG
Submitted: 2026-08-06
Comments: 13 pages, 5 figures. Experience report
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training
- Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
- Seeing is believing: The case for hardware-centric inefficiency detection and remediation
- Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
- When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry
- GPU Under Pressure: Estimating Application's Stress via Telemetry and Performance Counters
- The Energy Cost of Execution-Idle in GPU Clusters
- From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
Related papers
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- Load Balancing with Evolving Workloads in Large-Scale Language Model Serving
- Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
- SAMM: Sharded Automated Market Maker
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Vigil: Accountable Liveness against Selective Silence