BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels
BARTER is a new Dynamic Sparse Training framework that reduces training costs by keeping model tensors physically compact using standard dense kernels instead of specialized sparse ones. It achieves this by dynamically pruning and cloning attention heads and feed-forward channels based only on active unit statistics, allowing for significant speedups up to 6.29x on GPUs while maintaining competitive accuracy.