Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
summary
The gist
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes…
In short
Zeroth-order (ZO) fine-tuning of large language models is memory-efficient but its optimization effort is unevenly distributed across layers. This study reveals that ZO fine-tuning is sharply dominated by a single decoding layer, which consistently matches or exceeds full-model ZO performance. This dominant layer is model and task specific and can be identified by analyzing activation outliers.
Key concepts
- Zeroth-order (ZO) Optimization
- A memory-efficient method for fine-tuning large language models (LLMs) that only uses forward passes without updating parameters directly. It aims to adapt the model using minimal resources, but the distribution of this adaptation across different layers remains unclear.
- Dominant Layer
- A single, specific layer within an LLM that receives the overwhelming majority of the optimization effort during ZO fine-tuning. This layer is task-agnostic for a given model but model-specific; it consistently shows superior performance across various downstream tasks for that particular LLM architecture.
- Activation Outliers
- Specific, abnormally large activation magnitudes that appear consistently at certain dimensions within the model's layers during inference. These outliers are independent of the input data and serve as a detectable signal to identify the dominant layer before extensive training is performed.
Terminology used across episodes
This episode discusses
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs · Paper Radio
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
- AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
- Massive Activations in Large Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Qwen3 Technical Report
- Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
The paper
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs · Read on arXiv
Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin
University of North Carolina at Charlotte University of Houston State University State University of New York at Binghamton Department of Artificial Intelligence and Informatics Mayo Clinic Arizona State University
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers. Specifically, the dominant layer consistently aligns with the first activation-outlier layer in the pre-trained model. To explain this phenomenon, we analyze how perturbation effects propagate under ZO optimization. We find that the dominant layer combines two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers. As a result, this layer produces disproportionately strong and stable optimization signals under forward-only updates. Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52 times training speedup.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs".
Jane: Here's the gist of the paper. Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers<ref:2606.05516#pg2>.
Tom: In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer<ref:2606.05516#pg2>.
Jane: Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning<ref:2606.05516#pg5>.
Tom: We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers<ref:2606.05516#pg3>.
Paper summary: Tom: So, we're talking about this paper called "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs." Essentially, they're looking at zero order optimization, or ZO, which is a way to fine-tune huge language models using only forward passes without needing those massive memory loads. But what they found is that this fine-tuning isn't spread out evenly across all the layers of the model.
Jane: Right, and what they claim is that this zero order optimization process is really dominated by just a single decoding layer. They say that fine-tuning only that one layer consistently matches or even beats tuning the whole model using zero order optimization. It seems like a pretty surprising result for how adaptation usually works in these models.
Lu: What's interesting is their finding that this dominant layer isn't dependent on the specific task you are trying to do, but it is definitely dependent on the specific model architecture they are using. They found that for a given language model, the same single layer tends to be the most important across different tasks.
Meng: From an engineering standpoint, if only one layer is doing all the heavy lifting during fine-tuning, that could mean a lot for deployment efficiency and what kind of parameter reduction we can actually expect. It suggests we might be able to achieve much bigger parameter reductions than we thought when using zero order optimization this way.
Lalam: I'm seeing a pattern here where the dominant layer is tied to something specific in the model structure, and they found that it aligns with the very first layer where activation outliers appear during inference. That's a concrete identifier they used to find this layer without doing expensive full-model tuning <ref:2606.05516#pg4>.
Tom: So, how do they pinpoint this dominant layer if we don't want to run all those expensive layer-wise zero order fine-tuning runs? They propose a simple method involving inference only analysis of activation outliers <ref:2606.05516#pg2>. It seems like they are using something already present in the model during a standard forward pass to find the spot.
Jane: That makes sense, and it connects it to the known activation outlier phenomenon where some activations show extremely large magnitudes at specific dimensions across layers in an input-independent way <ref:2606.05516#pg2>. So, they are basically looking for that layer where the activations are behaving abnormally large compared to the typical scale of everything else.
Lu: The mechanism behind why this layer dominates comes down to two things: it has high perturbation sensitivity and it's placed early in the residual stream <ref:2606.05516#pg5>. Because it's early, any small change you make during optimization gets propagated and accumulated through all the subsequent layers.
Meng: If a small change propagates well, that means we only need to focus our optimization efforts on that one layer, and the effect will compound nicely before it reaches the final loss function. That makes sense for making parameter-side reductions, as they mention about thirty-two times compared to full model zero order optimization <ref:2606.05516#pg9>.
Paper summary: Lalam: Exactly, and their analysis shows that this layer's output contains extremely large magnitude activations, which is what drives the perturbation effect through the system <ref:2606.05516#pg5>. This makes it a stable signal for zero order updates.
Tom: So, across their experiments on models like LLaMA two point seven billion and Qwen three eight billion across nine different benchmarks, they show that tuning just this dominant layer improves the average performance over both full model zero order optimization and lo ra based zero order optimization <ref:2606.05516#pg2>. And they report a training speedup of about four point five two times compared to those methods <ref:2606.05516#pg2>.
Jane: That's a big number for a speedup, especially when you consider the parameter-side reduction they found, which is about thirty-two times less than full model zero order optimization <ref:2606.05516#pg9>. It really shows that this selective approach is effective when dealing with parameter constraints.
Lu: The paper concludes that full model zero order optimization doesn't just update too many parameters randomly; instead, it allocates the optimization effort unevenly across layers <ref:2606.05516#pg10>. This means future zero order methods should probably account for which layer is important and how much importance it has within that layer.
Meng: I'm seeing a limitation here that they admit in their work, and it's the performance gap between this dominant-layer zero order optimization and first order fine-tuning methods <ref:2606.05516#pg10>. It still takes a lot of steps for dominant-layer zero order to get good results, which means it might not be practical for some applications yet.
Tom: That gap is something we need to watch closely, especially since they acknowledge that the dominant layer approach still needs more work <ref:2606.05516#pg10>. It suggests that while this single-layer focus is a powerful idea, it's not a complete solution for everything in zero order fine-tuning yet.
Jane: So, to wrap up this paper on "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs," the main thing they show is that zero order optimization isn't uniformly useful across all layers; it heavily favors one dominant layer that can be identified by looking at activation outliers. They also show that tuning just this single layer can outperform other zero order methods and offer significant speedups.
Lu: The implication for the field is pointing toward future zero order methods that need to consider the identity of the layer and its specific importance inside the model structure <ref:2606.05516#pg10>. They've shown a mechanism where high perturbation sensitivity combined with an early position in the residual stream allows one small change to affect many subsequent blocks.
Meng: For deployment, this means if we can reliably identify that dominant layer for a specific model and task, we could potentially train much smaller models with zero order optimization while still getting good performance. That’s the practical value I’m looking at here.
Conclusion: Tom: So, we've seen how zero order optimization isn't spread evenly across all the layers of a large language model, and now we’re looking at what they actually found in this paper titled "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs."
Jane: Right, so to recap, they’ve looked at how zero order fine tuning works on these massive models and discovered that it mostly focuses its effort on just one single decoding layer.
Lu: Basically, the authors are showing that this dominant layer isn't random; it’s tied to the specific model architecture they used and stays the same across different tasks, which is actually kind of interesting for how we think about model adaptation.
Meng: I’m curious if focusing on just one layer is a realistic thing to deploy at scale, or if we still need to manage all those other layers.
Lalam: From my perspective in the model, this means the optimization process has a very clear target; it finds that specific layer where things are most sensitive and starts there.
Tom: And they show that tuning just that one layer can actually beat full zero order fine tuning, and they even report a speedup of about four point five two times on models like LLaMA two point seven billion.
Jane: That's a solid number, Tom, but the real question is how this selection mechanism works—how do you find that dominant layer without doing all that expensive layer-wise tuning?
Lu: They found it by looking for activation outliers during inference, where those specific dimensions show an abnormally large magnitude compared to the usual activations. That’s their identification method.
Meng: So, if I want to use this idea at my startup, would I need to run a full forward pass just to pick the layer? What's the practical cost there?
Lalam: The authors suggest a simple inference-only selection method; you run the model once, compute those layer statistics, and pick the first layer whose maximum activation magnitude is unusually large. That’s how they propose identifying it.
Tom: So they’re proposing a way to skip the heavy work of checking every single layer by just looking at how the activations behave during a normal forward pass.
Jane: It really changes how we view parameter reduction in this field; it suggests that instead of thinking about updating everything, you can focus on this one structurally important layer.
Lu: The mechanism they found is that this dominant layer has high perturbation sensitivity and an early position in the residual stream, which lets its small changes propagate and accumulate through the rest of the model.
Meng: That propagation idea is key; if a small change gets amplified early on, it can lead to a big effect later on, even if we’re only updating one layer.
Lalam: Exactly. That accumulation before reaching the final loss makes it a stable signal for those zero order updates they're talking about.
Tom: So, the main conclusion is that full model zero order doesn't update parameters evenly; it allocates its effort unevenly across layers, and this dominant layer is a key target.
Jane: It’s pointing toward future research where we need to account for the identity of the layer and how important it is within that specific model structure.
Lu: That’s what they are suggesting—that we need zero order methods that consider layer identity and what makes each layer important within the whole system.
Meng: The limitation they mention is still a performance gap compared to first order fine tuning, so it’s not a perfect solution yet for every application.
Tom: So, this paper gives us a concrete target—the dominant layer—but we still have to figure out how reliably and efficiently we can find it and use it in practice.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI