SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
SlideDP is a synchronous runtime for fine-tuning large language models across shared-host multi-GPU systems. It enables training beyond GPU memory by coordinating parameter delivery, gradient aggregation, and CPU updates across ranks. The system uses an analytical model and policy selection to optimize communication routes and activation placement based on hardware topology.