On the Convergence of Stochastic Low-Rank Adaptation

arXiv:2607.21975 · cs.LG, math.OC · Submitted 2026-07-24 · Read on arXiv

cs.LG, math.OC

Submitted: 2026-07-24

Updated: 2026-09-26

Code: https://github.com/tatsu-lab/stanford_alpaca

License: http://creativecommons.org/licenses/by/4.0/

The gist: Low-rank adaptation (LoRA) optimizes J(B,A)= L(W base+sBA) over two adapters B in R m times r and A in R r times n that form a low-rank update to a frozen pretrained weight matrix W base in R m times

Terminology

Abstract

Low-rank adaptation (LoRA) optimizes J(B,A)= L(W base+sBA) over two adapters B in R m times r and A in R r times n that form a low-rank update to a frozen pretrained weight matrix W base in R m times n. The prior analysis shows LoRA-GD takes O(ε-2) oracle calls to find an ε-stationary point such that | grad J(B,A)| at most ε in the deterministic setting. We sharpen the analysis and show that O(ε-4) full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an ε-stationary point with O(ε-8) stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to O(ε-6).

Sources

Related papers