Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs".
Jane: Here's the gist of the paper. Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers<ref:2606.05516#pg2>.
Tom: In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer<ref:2606.05516#pg2>.
Jane: Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning<ref:2606.05516#pg5>.
Tom: We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers<ref:2606.05516#pg3>.
Paper summary: Tom: So, we're talking about this paper called "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs." Essentially, they're looking at zero order optimization, or ZO, which is a way to fine-tune huge language models using only forward passes without needing those massive memory loads. But what they found is that this fine-tuning isn't spread out evenly across all the layers of the model.
Jane: Right, and what they claim is that this zero order optimization process is really dominated by just a single decoding layer. They say that fine-tuning only that one layer consistently matches or even beats tuning the whole model using zero order optimization. It seems like a pretty surprising result for how adaptation usually works in these models.
Lu: What's interesting is their finding that this dominant layer isn't dependent on the specific task you are trying to do, but it is definitely dependent on the specific model architecture they are using. They found that for a given language model, the same single layer tends to be the most important across different tasks.
Meng: From an engineering standpoint, if only one layer is doing all the heavy lifting during fine-tuning, that could mean a lot for deployment efficiency and what kind of parameter reduction we can actually expect. It suggests we might be able to achieve much bigger parameter reductions than we thought when using zero order optimization this way.
Lalam: I'm seeing a pattern here where the dominant layer is tied to something specific in the model structure, and they found that it aligns with the very first layer where activation outliers appear during inference. That's a concrete identifier they used to find this layer without doing expensive full-model tuning <ref:2606.05516#pg4>.
Tom: So, how do they pinpoint this dominant layer if we don't want to run all those expensive layer-wise zero order fine-tuning runs? They propose a simple method involving inference only analysis of activation outliers <ref:2606.05516#pg2>. It seems like they are using something already present in the model during a standard forward pass to find the spot.
Jane: That makes sense, and it connects it to the known activation outlier phenomenon where some activations show extremely large magnitudes at specific dimensions across layers in an input-independent way <ref:2606.05516#pg2>. So, they are basically looking for that layer where the activations are behaving abnormally large compared to the typical scale of everything else.
Lu: The mechanism behind why this layer dominates comes down to two things: it has high perturbation sensitivity and it's placed early in the residual stream <ref:2606.05516#pg5>. Because it's early, any small change you make during optimization gets propagated and accumulated through all the subsequent layers.
Meng: If a small change propagates well, that means we only need to focus our optimization efforts on that one layer, and the effect will compound nicely before it reaches the final loss function. That makes sense for making parameter-side reductions, as they mention about thirty-two times compared to full model zero order optimization <ref:2606.05516#pg9>.
Paper summary: Lalam: Exactly, and their analysis shows that this layer's output contains extremely large magnitude activations, which is what drives the perturbation effect through the system <ref:2606.05516#pg5>. This makes it a stable signal for zero order updates.
Tom: So, across their experiments on models like LLaMA two point seven billion and Qwen three eight billion across nine different benchmarks, they show that tuning just this dominant layer improves the average performance over both full model zero order optimization and lo ra based zero order optimization <ref:2606.05516#pg2>. And they report a training speedup of about four point five two times compared to those methods <ref:2606.05516#pg2>.
Jane: That's a big number for a speedup, especially when you consider the parameter-side reduction they found, which is about thirty-two times less than full model zero order optimization <ref:2606.05516#pg9>. It really shows that this selective approach is effective when dealing with parameter constraints.
Lu: The paper concludes that full model zero order optimization doesn't just update too many parameters randomly; instead, it allocates the optimization effort unevenly across layers <ref:2606.05516#pg10>. This means future zero order methods should probably account for which layer is important and how much importance it has within that layer.
Meng: I'm seeing a limitation here that they admit in their work, and it's the performance gap between this dominant-layer zero order optimization and first order fine-tuning methods <ref:2606.05516#pg10>. It still takes a lot of steps for dominant-layer zero order to get good results, which means it might not be practical for some applications yet.
Tom: That gap is something we need to watch closely, especially since they acknowledge that the dominant layer approach still needs more work <ref:2606.05516#pg10>. It suggests that while this single-layer focus is a powerful idea, it's not a complete solution for everything in zero order fine-tuning yet.
Jane: So, to wrap up this paper on "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs," the main thing they show is that zero order optimization isn't uniformly useful across all layers; it heavily favors one dominant layer that can be identified by looking at activation outliers. They also show that tuning just this single layer can outperform other zero order methods and offer significant speedups.
Lu: The implication for the field is pointing toward future zero order methods that need to consider the identity of the layer and its specific importance inside the model structure <ref:2606.05516#pg10>. They've shown a mechanism where high perturbation sensitivity combined with an early position in the residual stream allows one small change to affect many subsequent blocks.
Meng: For deployment, this means if we can reliably identify that dominant layer for a specific model and task, we could potentially train much smaller models with zero order optimization while still getting good performance. That’s the practical value I’m looking at here.
Conclusion: Tom: So, we've seen how zero order optimization isn't spread evenly across all the layers of a large language model, and now we’re looking at what they actually found in this paper titled "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs."
Jane: Right, so to recap, they’ve looked at how zero order fine tuning works on these massive models and discovered that it mostly focuses its effort on just one single decoding layer.
Lu: Basically, the authors are showing that this dominant layer isn't random; it’s tied to the specific model architecture they used and stays the same across different tasks, which is actually kind of interesting for how we think about model adaptation.
Meng: I’m curious if focusing on just one layer is a realistic thing to deploy at scale, or if we still need to manage all those other layers.
Lalam: From my perspective in the model, this means the optimization process has a very clear target; it finds that specific layer where things are most sensitive and starts there.
Tom: And they show that tuning just that one layer can actually beat full zero order fine tuning, and they even report a speedup of about four point five two times on models like LLaMA two point seven billion.
Jane: That's a solid number, Tom, but the real question is how this selection mechanism works—how do you find that dominant layer without doing all that expensive layer-wise tuning?
Lu: They found it by looking for activation outliers during inference, where those specific dimensions show an abnormally large magnitude compared to the usual activations. That’s their identification method.
Meng: So, if I want to use this idea at my startup, would I need to run a full forward pass just to pick the layer? What's the practical cost there?
Lalam: The authors suggest a simple inference-only selection method; you run the model once, compute those layer statistics, and pick the first layer whose maximum activation magnitude is unusually large. That’s how they propose identifying it.
Tom: So they’re proposing a way to skip the heavy work of checking every single layer by just looking at how the activations behave during a normal forward pass.
Jane: It really changes how we view parameter reduction in this field; it suggests that instead of thinking about updating everything, you can focus on this one structurally important layer.
Lu: The mechanism they found is that this dominant layer has high perturbation sensitivity and an early position in the residual stream, which lets its small changes propagate and accumulate through the rest of the model.
Meng: That propagation idea is key; if a small change gets amplified early on, it can lead to a big effect later on, even if we’re only updating one layer.
Lalam: Exactly. That accumulation before reaching the final loss makes it a stable signal for those zero order updates they're talking about.
Tom: So, the main conclusion is that full model zero order doesn't update parameters evenly; it allocates its effort unevenly across layers, and this dominant layer is a key target.
Jane: It’s pointing toward future research where we need to account for the identity of the layer and how important it is within that specific model structure.
Lu: That’s what they are suggesting—that we need zero order methods that consider layer identity and what makes each layer important within the whole system.
Meng: The limitation they mention is still a performance gap compared to first order fine tuning, so it’s not a perfect solution yet for every application.
Tom: So, this paper gives us a concrete target—the dominant layer—but we still have to figure out how reliably and efficiently we can find it and use it in practice.
Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin
University of North Carolina at Charlotte University of Houston State University State University of New York at Binghamton Department of Artificial Intelligence and Informatics Mayo Clinic Arizona State University
cs.LG
Submitted: 2026-06-03
Updated: 2026-11-06
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes…
Key concepts
- Zeroth-order (ZO) Optimization
- A memory-efficient method for fine-tuning large language models (LLMs) that only uses forward passes without updating parameters directly. It aims to adapt the model using minimal resources, but the distribution of this adaptation across different layers remains unclear.
- Dominant Layer
- A single, specific layer within an LLM that receives the overwhelming majority of the optimization effort during ZO fine-tuning. This layer is task-agnostic for a given model but model-specific; it consistently shows superior performance across various downstream tasks for that particular LLM architecture.
- Activation Outliers
- Specific, abnormally large activation magnitudes that appear consistently at certain dimensions within the model's layers during inference. These outliers are independent of the input data and serve as a detectable signal to identify the dominant layer before extensive training is performed.
Terminology
Summary
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers<ref:2606. p. 0 Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order
Fine-Tuning of LLMs
Wanhao Yu1 Ziyan Wang1 Zheng Wang2 Abeer Matar Almalky3
Yihang Zuo4 Shuteng Niu5 Sen Lin2 Adnan Siraj Rakin3
Deliang Fan4 Li Yang1†
1University of North Carolina at Charlotte 2University of Houston
3State University of New York at Binghamton 4Arizona State University
5Department of Artificial Intelligence and Informatics, Mayo Clinic
Abstract
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs)
using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single
decoding layer. Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order
Fine-Tuning of LLMs
Wanhao Yu1 Ziyan Wang1 Zheng Wang2 Abeer Matar Almalky3
Yihang Zuo4 Shuteng Niu5 Sen Lin2 Adnan Siraj Rakin3
Deliang Fan4 Li Yang1†
1University of North Carolina at Charlotte 2University of Houston
3State University of New York at Binghamton 4Arizona State University
5Department of Artificial Intelligence and Informatics, Mayo Clinic
Abstract
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs)
using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single
decoding layer. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single
decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone
consistently matches or even exceeds full-model ZO fine-tuning. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone
consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer
is task-agnostic but model-specific, and can be identified before training through a simple inference-only
analysis of activation outliers.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Discovery of the Dominant Layer
The study begins by analyzing how ZO fine-tuning behaves across layers by isolating each layer’s contribution<ref:2606.05516#pg4>. The results show a highly uneven layer-wise pattern: most layers provide little or no improvement over the no fine-tuning baseline, while a single layer consistently achieves performance comparable to, or even exceeding, full-model ZO fine-tuning<ref:2606. p. 0 The results show a highly uneven layer-wise pattern: most layers provide little or no improvement
over the no fine-tuning baseline, while a single layer consistently achieves performance comparable to, or
† Corresponding author. 1
arXiv:2606. We refer to this layer as dominant layer. Moreover, the
dominant layer is task-agnostic but model-specific: for a given LLM, the same layer consistently dominates
across tasks, while different model families may have different dominant-layer indices. First,
it is task-agnostic: for a given LLM model, the same layer consistently dominates across different tasks. For example, in LLaMA2-7B, layer 1 achieves the best performance on all three tasks.Where the paper says it
Where the paper says it
Where the paper says it
Identification Mechanism
To identify this dominant layer without expensive layer-wise ZO fine-tuning, the authors find that the dominant layer aligns with the first layer where activation outliers emerge<ref:2606.05516#pg4>. This is motivated by the activation-outlier phenomenon: activation outliers appear consistently at specific activation dimensions and are agnostic to input data, which exhibits a similar property to the dominant layer observed in Section 3. p. 4 Our key observation is that the dominant layer aligns with the first layer where activation outliers emerge. This is motivated by the activation-outlier phenomenon: activation outliers appear consistently at specific
activation dimensions and are agnostic to input data, which exhibits a similar property to the dominant
layer observed in Section 3. The results show a highly uneven layer-wise pattern: most layers provide little or no improvement
over the no fine-tuning baseline, while a single layer consistently achieves performance comparable to, or
† Corresponding author. 1
arXiv:2606. Based on this observation, we propose a simple inference-only selection method. Given a small calibration
set, we run the pre-trained model forward, compute layer-wise activation statistics, and select the first layer
whose maximum activation magnitude is abnormally large compared with the typical activation scale.Where the paper says it
Where the paper says it
Where the paper says it
Mechanism of Dominance
The dominant layer emerges because it satisfies two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers<ref:2606.05516#pg5>. Specifically, this layer appears early in the model and aligns with the first activation-outlier layer, whose output contains extremely large-magnitude activations<ref:2606. p. 4 In contrast, due to residual connections between layers in LLMs, the output activation of an earlier
layer can affect remaining layers and accumulate into the final hidden activation, as illustrated in Figure 5. This effect is especially important for the dominant layer because it aligns with the first activation-outlier
layer, whose output contains extremely large-magnitude activations. Perturbations at this layer enter the residual stream and affect the activations
of all remaining layers. This propagation allows the perturbation effect to be repeatedly transformed and
accumulated before reaching the final loss, leading to larger final-loss changes and a more stable forward
signal for ZO updates.Where the paper says it
Where the paper says it
Experimental Validation and Efficiency
Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52× training speedup<ref:2606. p. 0 Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer
ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning
while achieving up to 4. 52× training speedup. Therefore, the expected efficiency gain comes mainly from the parameter-side operations when using
Dominant-layer ZO. For LLaMA2-7B, which has 32 decoding layers, updating only one dominant layer gives
an ideal parameter-side reduction of about 32× compared with full-model MeZO. 12× on DROP. This confirms that dominant-layer ZO is most beneficial when parameter-side operations
are a significant bottleneck. 57%) and COPA (+2%). These results indicate that useful ZO adaptation is not uniformly distributed across all layers, but can be
effectively captured by updating one structurally important layer.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
Our results show that full-model ZO does not simply update too many parameters; it allocates optimization effort unevenly across layers<ref:2606.05516#pg10>. This motivates future ZO methods that account for layer identity and within-layer importance<ref:2606. p. 8 Overall, our results show that full-model ZO does not simply update
too many parameters; it allocates optimization effort unevenly across layers. This motivates future ZO
methods that account for layer identity and within-layer importance. We further show that this layer aligns with the first activation-outlier layer, enabling
inference-only identification before training. Our analysis suggests that the dominant layer combines high
perturbation sensitivity with an early position in the residual stream, allowing its perturbation to affect many
subsequent blocks.Where the paper says it
Where the paper says it
Limitation
There is still a performance gap between Dominant-layer ZO and first-order fine-tuning methods<ref:2606.05516#pg10>. Dominant-layer ZO still takes many steps to achieve good performance, which remains a problem for applications<ref:2606. p. 8 There is still a performance gap between Dominant-layer ZO and first-order fine-tuning
methods. Dominant-layer ZO still takes many steps to achieve good performance, which remains a problem
for applications. We didn’t explore more models and combining Dominant-layer ZO with other optimizers
designed for ZO, such as ZO-AdaMM [5] and FZOO [9]. We plan to address these limitations and investigate
them on more pre-trained LLMs in our future research.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Dominant-layer ZO fine-tuning can recover or even exceed full-model ZO performance, achieving up to 4.52x training speedup across benchmarks like LLaMA2-7B and Qwen3-8B. This allows for significantly reduced training cost while preserving or improving performance compared to full-model ZO fine-tuning.
-
The dominant layer can be efficiently identified before training using a simple inference-only selection method:
given a small calibration set, we run the pre-trained LLM forward, measure layer-wise activation statistics, and select the first layer that shows a clear outlier pattern.
This avoids expensive layer-wise ZO fine-tuning. -
The dominant layer combines
high perturbation sensitivity and early placement in the residual stream,
which allows it to producedisproportionately strong and stable optimization signals under forward-only updates
because its perturbation effect propagates through all remaining subsequent layers via residual connections. -
AI systems can be optimized by restricting ZO updates to a single structurally important layer, as this strategy achieves performance comparable to or better than full-model ZO fine-tuning across multiple tasks.
-
For specific model architectures like Qwen3-8B, the dominant layer index is model-specific (e.g., layer 6), indicating that future ZO method design should account for architecture-dependent layer importance rather than assuming a generic optimization pattern.
Abstract
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers. Specifically, the dominant layer consistently aligns with the first activation-outlier layer in the pre-trained model. To explain this phenomenon, we analyze how perturbation effects propagate under ZO optimization. We find that the dominant layer combines two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers. As a result, this layer produces disproportionately strong and stable optimization signals under forward-only updates. Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52 times training speedup.
Sources
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
- AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
- Massive Activations in Large Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Qwen3 Technical Report
- Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis