Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

Name: Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces
Uploaded: 2026-10-18T22:12:42.841683+00:00
Duration: 936 s
Description: AdaSub-ZO is a novel forward-only fine-tuning method for large language models that uses learnable low-rank projection matrices to guide gradient directions. It improves standard zeroth-order optimization by learning task-aware perturbations and incorporating subspace momentum to smooth noisy estimates, leading to faster convergence and better performance on LLMs.

AdaSub-ZO is a novel forward-only fine-tuning method for large language models that uses learnable low-rank projection matrices to guide gradient directions. It improves standard zeroth-order optimization by learning task-aware perturbations and incorporating subspace momentum to smooth noisy estimates, leading to faster convergence and better performance on LLMs.

Open the full episode