Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

arXiv:2610.15225 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces".

Jane: The gist: AdaSub-ZO, a novel forward-only fine-tuning framework,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this new work called "Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces". It sounds pretty technical, but it’s about making AI fine-tuning more stable when you only have access to forward passes.

Jane: That’s right, Tom. The main idea is that standard zeroth-order methods struggle with high variance because the random subspaces they use don't always line up with what the model actually needs to learn for a specific task, and this paper proposes learning those subspaces instead of just picking them randomly.

Lu: Exactly. They introduce this framework called AdaSub-ZO which alternates between learning these low-rank projection matrices and updating the actual model parameters, which is new for zeroth-order LLM fine-tuning because it makes the subspace learnable rather than fixed.

Meng: So, what’s the payoff here in terms of what this means for actually shipping models? Is this just a theoretical win on a benchmark, or does it help with real deployment headaches?

Tom: The paper shows that by learning these projection matrices through finite-difference feedback, the model gets task-aware perturbations that track those specific descent directions, which should dramatically reduce the gradient estimation variance we see in standard ZO methods.

Jane: And they don't stop there; they added a subspace momentum acceleration mechanism to smooth out those noisy gradient estimates across different learning cycles, which is crucial because when you update the projection matrices, you lose that historical information.

Lu: They manage this by maintaining the momentum buffer inside the low-rank coordinate space instead of trying to track it in the full parameter space, so they keep that memory efficiency while still stabilizing things.

Meng: From a deployment standpoint, smoothing noisy updates is important because if you’re training something on real traffic data, you can’t afford wildly unstable update directions that cause divergence or weird behavior at scale.

Tom: The experiments show this actually works well across seven natural language understanding tasks from SuperGLUE, and they reported a zero point nine five percentage-point advantage over SubZero on SST-two which is a solid win against existing zeroth-order baselines <ref:2610.15225#pg1>.

Title and authors: Jane: That difference in performance suggests that the learned subspace is indeed better aligned with the task's specific loss descent directions, which is exactly what they set out to prove when they compare it to random subspace methods.

Lu: The paper also provides some interesting hyperparameter sensitivity analysis, showing that performance depends on things like the low-rank dimension being around thirty-six and a momentum coefficient beta m being zero point nine for the best results.

Meng: I’m looking at those numbers—a low-rank dimension of thirty-six sounds manageable, but tuning that parameter means we have to spend time figuring out what configuration is actually robust enough for production use.

Tom: The authors suggest that this whole setup converges to a neighborhood of a first-order stationary point under the assumptions they’ve laid out, which gives us some theoretical confidence in the stability they’re achieving.

Jane: It really seems like this paper tackles the core weakness of learnable subspace zeroth-order fine-tuning, which is that even with learned matrices, you still have those noisy finite-difference estimates that make updates unstable without a way to smooth them out.

Lu: The real innovation here is combining the learnable projection matrices with the subspace momentum transfer across evolving bases, which solves the problem of momentum loss when you refresh those bases.

Meng: So, if I were deploying this, I’d be watching how quickly those projection matrices adapt to a completely new task versus how much they rely on that accumulated momentum from previous tasks.

Tom: This paper really shows that by making the low-rank structure learnable and adding this momentum acceleration, we can stabilize these memory-efficient methods without sacrificing accuracy, provided you tune those dimensions correctly.

Jane: It’s a big step because it moves beyond just using fixed random matrices to something that actively learns the task's geometry during fine-tuning.

Lu: The main thing I see is that this framework offers a way to track task-specific descent directions directly through finite-difference feedback, which is what makes the perturbation subspace adaptive.

Title and authors: Meng: It’s interesting how they manage to keep the computational overhead low by only adding these learning modules on top of the existing forward-only structure, which is good for running on billion-scale models.

Tom: The ablation studies confirm that both the learning of projection bases from finite-difference feedback and the subspace momentum really contribute positively, meaning neither component can be ignored if you want to get those performance gains.

Jane: So what does this mean for people who only listen to the show? It means when you use zeroth-order fine-tuning, you’re not just guessing a direction; you’re actively teaching the system how to steer toward the right answer by learning its own shortcuts.

Lu: And they are providing a convergence guarantee to back that up, showing that this combination of learnable subspaces and momentum actually leads to a stable point in parameter space.

Meng: For us in production, it means we have a more reliable tool for probing model updates during fine-tuning without needing massive amounts of data or complex second-order information.

Tom: So, to wrap up the "Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces," the paper provides a concrete way to make low-rank subspace optimization task-aware and robust against noisy gradient estimates through learnable projection matrices and momentum acceleration.

Jane: It’s a solid contribution because it directly addresses the alignment problem between fixed random subspaces and the actual direction of change required by the specific task.

Lu: The authors did a great job balancing the need for learnability with maintaining the memory efficiency that makes zeroth-order fine-tuning attractive in first place.

Meng: It’s a tool we can use to get better results faster on smaller datasets, though I still have my reservations about deploying it directly onto massive infrastructure without further stress testing.

Tom: That’s the balance we see—a very promising method that significantly improves the quality and stability of update directions by learning layer-wise projection bases and maintaining subspace momentum.

Jane: It's definitely a paper worth reading if you are working on memory-efficient fine-tuning for large language models because it shows how to make those zero-shot methods smarter, not just faster.

The paper's summary: Tom: So, to recap, this paper is about taking those memory-efficient zeroth-order fine-tuning methods and making them smarter by learning the low-rank subspaces instead of just using random ones for task alignment.

Jane: Right. The big idea here is that standard zero-order methods struggle because their fixed random subspaces don't really match the specific direction the model needs to move for a particular job.

Lu: What they did in AdaSub-ZO is introduce learnable projection matrices that adapt during the fine-tuning process, so these subspaces actually track those task-specific descent directions.

Meng: So it’s not just picking a subspace once at the start; it’s actively learning which directions are important layer by layer as the model adjusts.

Tom: Exactly. And they didn't stop there because if you learn those projection matrices, you still have noisy estimates from the forward passes, so they added something to smooth that out.

Jane: That something is this subspace momentum acceleration module which accumulates those gradient estimates using an exponential moving average and transfers that history across different learning cycles.

Lu: They keep the momentum buffer in a low-dimensional space, r by r coordinate space instead of the massive parameter space, which keeps it computationally light.

Tom: And when they refresh the projection bases, they have this mechanism to transfer historical momentum so you don't lose all that smoothing effect when you update the matrices again.

Jane: So what does this mean for someone just listening? It means we’re moving past just picking a random direction and instead teaching the AI how to find its own best path by learning those low-rank shortcuts.

Lu: The math shows that this whole setup converges to a neighborhood of a first-order stationary point, which gives them some theoretical backing that this isn't just some lucky heuristic.

Meng: I’m still thinking about the real cost—the hyperparameter tuning for things like the low-rank dimension r and that momentum coefficient beta m—it looks like you have to do a lot of work to get it running reliably at scale.

Tom: They did an ablation study that shows both learning the projection bases from finite-difference feedback and having that subspace momentum really add value, so neither part is optional if you want those gains.

Jane: It’s a solid step because it tackles the alignment problem between fixed random subspaces and the actual direction of change required by the specific task.

Lu: The paper also shows that this method preserves the forward-only computational order of standard zeroth-order methods, which is important for billion-scale LLMs.

Meng: If it keeps that complexity low, then it’s a much more practical tool for production environments than something with higher complexity.

Tom: So, AdaSub-ZO is this framework that improves the quality and stability of update directions by learning layer-wise projection bases and maintaining subspace momentum to smooth noisy gradient estimates.

Jane: It’s a big step because it moves beyond just using fixed random matrices to something that actively learns the task's geometry during fine-tuning.

Lu: I think this is a good direction for future work because they laid out how to make the subspace update cycle more dynamic, but for now, this provides a very stable foundation.

The paper's improvements: Tom: So, to wrap up what we just heard, this paper introduces two main improvements: learning those projection matrices from finite-difference feedback and adding that subspace momentum acceleration mechanism to smooth out the noisy estimates.

Jane: That’s right. The authors are suggesting that by having the system learn its own low-rank structure through that feedback, it can better track what the model actually needs to change for a specific task.

Lu: And they argue this is significant because random subspaces are just not aligned with those true descent directions of downstream tasks, so learning them fixes that misalignment issue directly.

Meng: It means instead of guessing a good direction and hoping it works, the AI learns the geometry of what’s important for that specific fine-tuning goal.

Tom: Exactly. And on top of that adaptive structure, they provide this momentum transfer mechanism when you refresh those bases so you don't lose all the smoothing benefit from previous updates.

Jane: That’s a clever way to manage the trade-off between adapting to new tasks and keeping track of where it was going before.

Lu: The theoretical bounds they show, Theorem two prove that this whole AdaSub-ZO system converges to a neighborhood of a first-order stationary point, which gives them good stability guarantees.

Tom: And the number on the performance is what really gets me; they beat SubZero on SST-two by zero point nine five percentage points, which is a solid win against other zeroth-order baselines we've seen.

Jane: It shows that this approach isn't just theoretical; it actually produces better results in real tasks when compared to methods using fixed, random low-rank structures.

Meng: But I still have to ask about the deployment cost—how much compute does learning those projection matrices add per fine-tuning run? That overhead could be significant if you’re running this on billion-scale models.

Lu: The authors address that in Appendix A, showing that the time complexity preserves the forward-only computational order of standard zeroth-order methods, so it's relatively efficient for those large models.

Tom: So it keeps the overhead low by only adding these lightweight learning modules on top of existing forward passes, which is good news for running things at scale.

Jane: It means this method isn't just a fancy way to get better results; it’s a more practical tool because it maintains the efficiency needed for large models.

Lu: The limitation they flag is that the performance is very sensitive to hyperparameters like the low-rank dimension r, which they find optimal around thirty-six so you do have to tune things carefully.

Meng: Tuning parameters adds complexity for my team; we don't want something that requires a whole grid search just to get it running well.

Tom: The conclusion is that AdaSub-ZO improves the quality and stability of update directions by learning layer-wise projection bases and maintaining subspace momentum to smooth noisy gradient estimates.

Jane: It’s a big step because it moves beyond just using fixed random matrices to something that actively learns the task's geometry during fine-tuning.

Lu: I think this is a good direction for future work because they laid out how to make the subspace update cycle more dynamic, but for now, this provides a very stable foundation.

Conclusion: Tom: So we’ve covered the mechanics of how AdaSub-ZO works, focusing on learning those projection matrices and using subspace momentum to stabilize the updates during fine-tuning.

Jane: Exactly. The main implication is that for zeroth-order methods, we can move away from relying on random perturbations and start steering the model toward better task alignment through learned structures.

Lu: It means we’re giving the AI a way to learn its own shortcuts, which should lead to more effective fine-tuning than just picking directions randomly.

Meng: From my side, what matters is that it doesn't introduce massive new complexity or require an impossible amount of training data to figure out those initial low-rank structures.

Tom: The theoretical convergence guarantee they provide means we have some confidence that this method actually leads to a stable point in the parameter space under the assumptions they set up.

Jane: It’s a solid result because it shows how to make memory-efficient methods smarter, not just faster, and it gets us closer to reliable fine-tuning for very large models.

Lu: The limitation is that performance is quite sensitive to those hyperparameter settings we talked about; you really need to find the right balance for r and beta m.

Meng: I still worry about deploying something where the stability depends so heavily on those specific tuning knobs, especially when we’re trying to ship something reliably at scale across different tasks.

Tom: The full AdaSub-ZO algorithm converges to a neighborhood of a first-order stationary point, which is what the authors are claiming.

Jane: It really does show that by learning these projection bases and adding momentum acceleration, we can improve the stability of those noisy gradient estimates significantly across different learning cycles.

Lu: This paper opens up a path for developing more sophisticated zeroth-order techniques where the subspace itself evolves with the model’s progress.

Meng: I think if we can figure out how to automate that subspace learning process, it could save us a lot of manual tuning time when we move to production.

Tom: That’s the potential gain—automating that learning loop would make this method much more deployable for real-world scenarios.

Jane: This paper is a good example of how researchers are pushing these memory-efficient techniques into more robust and practical solutions.

Lu: The full AdaSub-ZO framework shows that learnable low-rank subspace optimization can indeed be made task-aware and robust against noisy estimates.

Meng: We’ll keep an eye on how the community tunes those dimensions r, because that’s going to be the real bottleneck for us when we try to integrate this into our pipeline.

Tom: It's a lot of interesting stuff about learning structure and stability in LLM optimization with this paper, "Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces."

Jane: We’ll be looking at how these learned subspaces can interact with other techniques, like those we discussed earlier regarding alignment training.

Chuang Tong, Ling Chen

Zhejiang University

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Comments: 7 pages, 4 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist: AdaSub-ZO, a novel forward-only fine-tuning framework, introduces learnable low-rank projection matrices to track task-aware descent directions and incorporates subspace momentum

Key concepts

Zeroth-Order (ZO) Optimization
This technique allows fine-tuning large language models by estimating gradients using only forward-pass loss evaluations, avoiding the need for computationally expensive backward passes. However, standard ZO methods suffer from high variance in high dimensions because they perturb the entire parameter space.
Learnable Low-Rank Projection Matrices
These are matrices that are learned during training to define a low-dimensional subspace. AdaSub-ZO uses these learnable matrices to construct task-aware perturbations, meaning the search directions for updating model parameters align better with the actual loss descent path for a specific task.
Subspace Momentum Acceleration
This module smooths noisy gradient estimates by maintaining a momentum buffer in the low-dimensional subspace space. It accumulates historical ZO estimates and transfers this momentum across different subspace bases, stabilizing updates and improving convergence speed.

Terminology

Summary

The gist: AdaSub-ZO, a novel forward-only fine-tuning framework, introduces learnable low-rank projection matrices to track task-aware descent directions and incorporates subspace momentum acceleration to smooth noisy gradient estimates in zeroth-order LLM fine-tuning.

Motivation and Problem Statement

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) by estimating gradients only through forward-pass loss evaluations <ref:2610.15225#pg4>. However, standard ZO methods suffer from high variance in high-dimensional parameter spaces because they perturb the full parameter space, making the variance of gradient estimates scale with the number of trainable parameters <ref:2610.15225#pg4>. This leads to noisy update directions, slow convergence, and unstable fine-tuning for billion-scale LLMs <ref:2610.15225#pg4>. While restricting perturbations to a low-rank subspace helps, these subspaces are usually random and non-learnable, making their perturbation directions poorly aligned with task-specific loss descent directions <ref:2610.15225#pg4>.

Proposed Framework: AdaSub-ZO

AdaSub-ZO is proposed as a novel forward-only fine-tuning framework that alternates between learnable subspace construction and model parameter optimization <ref:2610.15225#pg5>. The primary contributions include:

  1. A low-rank projection matrix learning module that updates layer-wise projection matrices through forward-only finite-difference feedback to learn taskaware perturbations that better align with task-specific descent directions <ref:2610.15225#pg5>.

  2. A subspace momentum acceleration module for learnable low-rank subspaces, which accumulates ZO gradient estimates via an exponential moving average and transfers historical momentum across evolving subspace bases to smooth estimation noise <ref:2610.15225#pg5>.

Methodology Details

The framework operates in two stages: the subspace learning stage and the model update stage <ref:2610.15225#pg5>.

** Layer-Wise Low-Rank ZO Gradient Estimation**

For each layer, AdaSub-ZO maintains two projection bases Ul ∈ Rml×r and Vl ∈ Rnl×r, where r ≪ min(ml, nl) <ref:2610.15225#pg5>. The low-rank perturbation Zel is constructed as Zel = UlZlV⊤l (1) <ref:2610.15225#pg5>. The layer-wise ZO gradient estimate is formulated as ∇b θ(l)t = ρtZel = ρtUlZlV⊤l (3) <ref:2610.15225#pg5>.

** Forward-Only Learnable Subspace Update**

During the projection learning stage, AdaSub-ZO samples three Gaussian matrices: EA,l ∈ Rml×r, EB,l ∈ Rnl×r, and Zl ∈ Rr×r <ref:2610.15225#pg5>. It constructs two perturbed low-rank directions ∆+l and ∆−l using the learnable projection matrices Al and Bl <ref:2610.15225#pg5>. The scalar finite-difference signal sˆi is computed, and an exponential moving average (EMA) is used to filter noise in the projection matrices PA,l and PB,l <ref:2610.15225#pg5>. After M projection learning steps, QR decomposition is applied to obtain orthogonal factors Ul and Vl <ref:2610.15225#pg5>.

** Subspace Momentum Acceleration**

To stabilize noisy ZO estimates, AdaSub-ZO maintains a subspace momentum buffer Ml in the r × r coordinate space <ref:2610.15225#pg5>. The update rule is Ml ← βmMl + ρtZl (10) <ref:2610.15225#pg5>. Model parameters are then updated by projecting the momentum direction back to the original parameter space: θ(l)t+1 = θ(l)t − ηθq dl r2 UlMlV⊤l (11) <ref:2610.15225#pg5>. When projection bases are refreshed, historical momentum is transferred via Ml ← γ(Unew l)⊤Uold l Ml Vold l (Vnew l)⊤ (12) <ref:2610.15225#pg5>.

Experimental Results and Analysis

Experiments were conducted on seven natural language understanding tasks from SuperGLUE using the OPT-1.3B backbone model <ref:2610.15225#pg7>. AdaSub-ZO achieves the best overall result among zeroth-order baselines, obtaining a 0.95 percentage-point advantage over SubZero on SST-2 <ref:2610.15225#pg7>. The relative variance of ZO gradient estimates is substantially reduced by AdaSub-ZO compared to MeZO and SubZero, indicating that the learned subspace better aligns with task-specific descent directions <ref:2610.15225#pg8>. Training loss curves show that AdaSub-ZO converges faster and reaches a lower training loss on the CB task compared to MeZO <ref:2610.15225#pg9>.

Hyperparameter Sensitivity

The performance is sensitive to the low-rank dimension r and the subspace momentum coefficient βm <ref:2610.15225#pg9>. The best performance is achieved when r = 36, balancing subspace expressiveness and optimization stability <ref:2610.15225#pg9>. Furthermore, the best performance is achieved when βm = 0.9, as a small momentum coefficient provides limited smoothing while an excessively large coefficient adapts slowly to newly learned subspace bases <ref:2610.15225#pg9>.

Conclusion

AdaSub-ZO improves the quality and stability of update directions by learning layer-wise projection bases from finite-difference feedback and maintaining subspace momentum to transfer historical update information across basis refreshes <ref:2610.15225#pg10>. This design addresses the two key challenges of learnable subspace ZO fine-tuning, leading to improved convergence over random subspace methods <ref:2610.15225#pg10>. The full AdaSub-ZO algorithm converges to a neighborhood of a first-order stationary point under the specified assumptions <ref:2610.15225#pg10>.

How it works

AdaSub-ZO alternates between learnable subspace update and momentum-accelerated model parameter update <ref:2610.15225#pg5>. The projection matrices are learned from finite-difference feedback, enabling the perturbation subspace to adapt toward task-specific descent directions <ref:2610.15225#pg5>. Subspace momentum smooths noisy ZO estimates, while the momentum transfer mechanism keeps historical information consistent across evolving bases <ref:2610.15225#pg5>.

Subspace Momentum Acceleration

AdaSub-ZO maintains momentum in the low-dimensional r × r coordinate space instead of the full parameter space <ref:2610.15225#pg5>. The update rule is Ml ← βmMl + ρtZl (10) <ref:2610.15225#pg5>. Model parameters are then updated by projecting the momentum direction back to the original parameter space: θ(l)t+1 = θ(l)t − ηθq dl r2 UlMlV⊤l (11) <ref:2610.15225#pg5>. The momentum transfer mechanism preserves useful historical information while avoiding coordinate mismatch <ref:2610.15225#pg5>.

Ablation Study

The ablation study confirms the effectiveness of the proposed modules, showing that full AdaSub-ZO consistently outperforms variants without the learnable module and without subspace momentum <ref:2610.15225#pg8>. The learning of projection bases from finite-difference feedback helps construct more effective low-rank update subspaces <ref:2610.15225#pg8>. Subspace momentum can smooth noisy finite-difference updates and further improve the stability of learnable low-rank ZO finetuning <ref:2610.15225#pg8>.

Hyperparameter Settings

The hyperparameter search space for AdaSub-ZO was determined via a grid search on the validation set <ref:2610.15225#pg10>. The search space includes Model Learning Rate ηθ, Model Perturbation Scale ϵ, Low-Rank Dimension r, Projection Learning Cycle K, Projection Learning Steps M, Projection Learning Rate ηϕ, Projection Perturbation Scale µ, and Subspace Momentum Coefficient βm <ref:2610.15225#pg10>.

Improvements for AI systems

  1. textbfIncreased Task-Aware Perturbation Alignment in ZO Optimization: The AdaSub-ZO Module (Algorithm 1, Equations 4-9). This module learn[s] these bases from forward-only feedback, so that the subspace can better track task-specific descent directions, which is a significant improvement over fixed random subspaces that may remain misaligned with the true descent directions of downstream tasks.

  2. textbf Enhanced Gradient Stability via Subspace Momentum: The Subspace Momentum Acceleration Module (Algorithm 2, Equation 10). This module accumulat[es] ZO gradient estimates via an exponential moving average and ensures momentum consistency across learning cycles by transferring historical information to the new coordinate system, which smooth[s] noisy finite-difference updates.

  3. textbf Robust Convergence Guarantees: Improved Theoretical Bounds (Theorem 2). The analysis shows that full AdaSub-ZO converges to a neighborhood of a first-order stationary point, with the momentum term controlling the residual error by showing that subspace momentum smooths noisy ZO directions without maintaining full-dimensional optimizer states.

  4. textbf Optimized Training Efficiency: Reduced Computational Overhead (Appendix A). The time complexity analysis shows that AdaSub-ZO preserves the forward-only computational order of standard ZO methods, with only lightweight low-rank projection learning and subspace momentum overhead, making it efficient for billion-scale LLMs.

Sources

Related papers