TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
TACO is a new first-order optimizer designed to drastically reduce memory overhead during full-parameter LLM fine-tuning. It achieves this by computing an exact steepest-descent direction using a ternary, column-wise, one-sparse approach based on the largest magnitude entries in weight matrices. This method reduces persistent optimizer state memory by 174 times compared to AdamW8bit while maintaining competitive accuracy.