Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

arXiv:2606.05516 · cs.LG · Submitted 2026-06-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs".

Jane: Here's the gist of the paper. Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers<ref:2606.05516#pg2>.

Tom: In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer<ref:2606.05516#pg2>.

Jane: Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning<ref:2606.05516#pg5>.

Tom: We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers<ref:2606.05516#pg3>.

Paper summary: Tom: So, we're talking about this paper called "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs." Essentially, they're looking at zero order optimization, or ZO, which is a way to fine-tune huge language models using only forward passes without needing those massive memory loads. But what they found is that this fine-tuning isn't spread out evenly across all the layers of the model.

Jane: Right, and what they claim is that this zero order optimization process is really dominated by just a single decoding layer. They say that fine-tuning only that one layer consistently matches or even beats tuning the whole model using zero order optimization. It seems like a pretty surprising result for how adaptation usually works in these models.

Lu: What's interesting is their finding that this dominant layer isn't dependent on the specific task you are trying to do, but it is definitely dependent on the specific model architecture they are using. They found that for a given language model, the same single layer tends to be the most important across different tasks.

Meng: From an engineering standpoint, if only one layer is doing all the heavy lifting during fine-tuning, that could mean a lot for deployment efficiency and what kind of parameter reduction we can actually expect. It suggests we might be able to achieve much bigger parameter reductions than we thought when using zero order optimization this way.

Lalam: I'm seeing a pattern here where the dominant layer is tied to something specific in the model structure, and they found that it aligns with the very first layer where activation outliers appear during inference. That's a concrete identifier they used to find this layer without doing expensive full-model tuning <ref:2606.05516#pg4>.

Tom: So, how do they pinpoint this dominant layer if we don't want to run all those expensive layer-wise zero order fine-tuning runs? They propose a simple method involving inference only analysis of activation outliers <ref:2606.05516#pg2>. It seems like they are using something already present in the model during a standard forward pass to find the spot.

Jane: That makes sense, and it connects it to the known activation outlier phenomenon where some activations show extremely large magnitudes at specific dimensions across layers in an input-independent way <ref:2606.05516#pg2>. So, they are basically looking for that layer where the activations are behaving abnormally large compared to the typical scale of everything else.

Lu: The mechanism behind why this layer dominates comes down to two things: it has high perturbation sensitivity and it's placed early in the residual stream <ref:2606.05516#pg5>. Because it's early, any small change you make during optimization gets propagated and accumulated through all the subsequent layers.

Meng: If a small change propagates well, that means we only need to focus our optimization efforts on that one layer, and the effect will compound nicely before it reaches the final loss function. That makes sense for making parameter-side reductions, as they mention about thirty-two times compared to full model zero order optimization <ref:2606.05516#pg9>.

Paper summary: Lalam: Exactly, and their analysis shows that this layer's output contains extremely large magnitude activations, which is what drives the perturbation effect through the system <ref:2606.05516#pg5>. This makes it a stable signal for zero order updates.

Tom: So, across their experiments on models like LLaMA two point seven billion and Qwen three eight billion across nine different benchmarks, they show that tuning just this dominant layer improves the average performance over both full model zero order optimization and lo ra based zero order optimization <ref:2606.05516#pg2>. And they report a training speedup of about four point five two times compared to those methods <ref:2606.05516#pg2>.

Jane: That's a big number for a speedup, especially when you consider the parameter-side reduction they found, which is about thirty-two times less than full model zero order optimization <ref:2606.05516#pg9>. It really shows that this selective approach is effective when dealing with parameter constraints.

Lu: The paper concludes that full model zero order optimization doesn't just update too many parameters randomly; instead, it allocates the optimization effort unevenly across layers <ref:2606.05516#pg10>. This means future zero order methods should probably account for which layer is important and how much importance it has within that layer.

Meng: I'm seeing a limitation here that they admit in their work, and it's the performance gap between this dominant-layer zero order optimization and first order fine-tuning methods <ref:2606.05516#pg10>. It still takes a lot of steps for dominant-layer zero order to get good results, which means it might not be practical for some applications yet.

Tom: That gap is something we need to watch closely, especially since they acknowledge that the dominant layer approach still needs more work <ref:2606.05516#pg10>. It suggests that while this single-layer focus is a powerful idea, it's not a complete solution for everything in zero order fine-tuning yet.

Jane: So, to wrap up this paper on "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs," the main thing they show is that zero order optimization isn't uniformly useful across all layers; it heavily favors one dominant layer that can be identified by looking at activation outliers. They also show that tuning just this single layer can outperform other zero order methods and offer significant speedups.

Lu: The implication for the field is pointing toward future zero order methods that need to consider the identity of the layer and its specific importance inside the model structure <ref:2606.05516#pg10>. They've shown a mechanism where high perturbation sensitivity combined with an early position in the residual stream allows one small change to affect many subsequent blocks.

Meng: For deployment, this means if we can reliably identify that dominant layer for a specific model and task, we could potentially train much smaller models with zero order optimization while still getting good performance. That’s the practical value I’m looking at here.

Conclusion: Tom: So, we've seen how zero order optimization isn't spread evenly across all the layers of a large language model, and now we’re looking at what they actually found in this paper titled "Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs."

Jane: Right, so to recap, they’ve looked at how zero order fine tuning works on these massive models and discovered that it mostly focuses its effort on just one single decoding layer.

Lu: Basically, the authors are showing that this dominant layer isn't random; it’s tied to the specific model architecture they used and stays the same across different tasks, which is actually kind of interesting for how we think about model adaptation.

Meng: I’m curious if focusing on just one layer is a realistic thing to deploy at scale, or if we still need to manage all those other layers.

Lalam: From my perspective in the model, this means the optimization process has a very clear target; it finds that specific layer where things are most sensitive and starts there.

Tom: And they show that tuning just that one layer can actually beat full zero order fine tuning, and they even report a speedup of about four point five two times on models like LLaMA two point seven billion.

Jane: That's a solid number, Tom, but the real question is how this selection mechanism works—how do you find that dominant layer without doing all that expensive layer-wise tuning?

Lu: They found it by looking for activation outliers during inference, where those specific dimensions show an abnormally large magnitude compared to the usual activations. That’s their identification method.

Meng: So, if I want to use this idea at my startup, would I need to run a full forward pass just to pick the layer? What's the practical cost there?

Lalam: The authors suggest a simple inference-only selection method; you run the model once, compute those layer statistics, and pick the first layer whose maximum activation magnitude is unusually large. That’s how they propose identifying it.

Tom: So they’re proposing a way to skip the heavy work of checking every single layer by just looking at how the activations behave during a normal forward pass.

Jane: It really changes how we view parameter reduction in this field; it suggests that instead of thinking about updating everything, you can focus on this one structurally important layer.

Lu: The mechanism they found is that this dominant layer has high perturbation sensitivity and an early position in the residual stream, which lets its small changes propagate and accumulate through the rest of the model.

Meng: That propagation idea is key; if a small change gets amplified early on, it can lead to a big effect later on, even if we’re only updating one layer.

Lalam: Exactly. That accumulation before reaching the final loss makes it a stable signal for those zero order updates they're talking about.

Tom: So, the main conclusion is that full model zero order doesn't update parameters evenly; it allocates its effort unevenly across layers, and this dominant layer is a key target.

Jane: It’s pointing toward future research where we need to account for the identity of the layer and how important it is within that specific model structure.

Lu: That’s what they are suggesting—that we need zero order methods that consider layer identity and what makes each layer important within the whole system.

Meng: The limitation they mention is still a performance gap compared to first order fine tuning, so it’s not a perfect solution yet for every application.

Tom: So, this paper gives us a concrete target—the dominant layer—but we still have to figure out how reliably and efficiently we can find it and use it in practice.

Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin

University of North Carolina at Charlotte University of Houston State University State University of New York at Binghamton Department of Artificial Intelligence and Informatics Mayo Clinic Arizona State University

cs.LG

Submitted: 2026-06-03

Updated: 2026-11-06

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes…

Key concepts

Zeroth-order (ZO) Optimization
A memory-efficient method for fine-tuning large language models (LLMs) that only uses forward passes without updating parameters directly. It aims to adapt the model using minimal resources, but the distribution of this adaptation across different layers remains unclear.
Dominant Layer
A single, specific layer within an LLM that receives the overwhelming majority of the optimization effort during ZO fine-tuning. This layer is task-agnostic for a given model but model-specific; it consistently shows superior performance across various downstream tasks for that particular LLM architecture.
Activation Outliers
Specific, abnormally large activation magnitudes that appear consistently at certain dimensions within the model's layers during inference. These outliers are independent of the input data and serve as a detectable signal to identify the dominant layer before extensive training is performed.

Terminology

Summary

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers<ref:2606. p. 0

Where the paper says it

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs Wanhao Yu1 Ziyan Wang1 Zheng Wang2 Abeer Matar Almalky3 Yihang Zuo4 Shuteng Niu5 Sen Lin2 Adnan Siraj Rakin3 Deliang Fan4 Li Yang1† 1University of North Carolina at Charlotte 2University of Houston 3State University of New York at Binghamton 4Arizona State University 5Department of Artificial Intelligence and Informatics, Mayo Clinic Abstract Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer.

05516#pg2>. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer<ref:2606. p. 0
Where the paper says it

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs Wanhao Yu1 Ziyan Wang1 Zheng Wang2 Abeer Matar Almalky3 Yihang Zuo4 Shuteng Niu5 Sen Lin2 Adnan Siraj Rakin3 Deliang Fan4 Li Yang1† 1University of North Carolina at Charlotte 2University of Houston 3State University of New York at Binghamton 4Arizona State University 5Department of Artificial Intelligence and Informatics, Mayo Clinic Abstract Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer.

05516#pg2>. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning<ref:2606. p. 0
Where the paper says it

In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning.

05516#pg5>. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers<ref:2606. p. 0
Where the paper says it

Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers.

05516#pg3>.

Discovery of the Dominant Layer

The study begins by analyzing how ZO fine-tuning behaves across layers by isolating each layer’s contribution<ref:2606.05516#pg4>. The results show a highly uneven layer-wise pattern: most layers provide little or no improvement over the no fine-tuning baseline, while a single layer consistently achieves performance comparable to, or even exceeding, full-model ZO fine-tuning<ref:2606. p. 0

Where the paper says it

The results show a highly uneven layer-wise pattern: most layers provide little or no improvement over the no fine-tuning baseline, while a single layer consistently achieves performance comparable to, or † Corresponding author. 1 arXiv:2606.

05516#pg4>. Moreover, the dominant layer is task-agnostic but model-specific: for a given LLM model, the same layer consistently dominates across different tasks<ref:2606. p. 1
Where the paper says it

We refer to this layer as dominant layer. Moreover, the dominant layer is task-agnostic but model-specific: for a given LLM, the same layer consistently dominates across tasks, while different model families may have different dominant-layer indices.

05516#pg4>. For instance, in LLaMA2-7B, layer 1 achieves the best performance on all three tasks<ref:2606. p. 3
Where the paper says it

First, it is task-agnostic: for a given LLM model, the same layer consistently dominates across different tasks. For example, in LLaMA2-7B, layer 1 achieves the best performance on all three tasks.

05516#pg4>.

Identification Mechanism

To identify this dominant layer without expensive layer-wise ZO fine-tuning, the authors find that the dominant layer aligns with the first layer where activation outliers emerge<ref:2606.05516#pg4>. This is motivated by the activation-outlier phenomenon: activation outliers appear consistently at specific activation dimensions and are agnostic to input data, which exhibits a similar property to the dominant layer observed in Section 3. p. 4

Where the paper says it

Our key observation is that the dominant layer aligns with the first layer where activation outliers emerge. This is motivated by the activation-outlier phenomenon: activation outliers appear consistently at specific activation dimensions and are agnostic to input data, which exhibits a similar property to the dominant layer observed in Section 3.

2<ref:2606. p. 0
Where the paper says it

The results show a highly uneven layer-wise pattern: most layers provide little or no improvement over the no fine-tuning baseline, while a single layer consistently achieves performance comparable to, or † Corresponding author. 1 arXiv:2606.

05516#pg4>. Based on this observation, they propose a simple inference-only selection method: given a small calibration set, we run the pre-trained model forward, compute layer-wise activation statistics, and select the first layer whose maximum activation magnitude is abnormally large compared with the typical activation scale<ref:2606. p. 4
Where the paper says it

Based on this observation, we propose a simple inference-only selection method. Given a small calibration set, we run the pre-trained model forward, compute layer-wise activation statistics, and select the first layer whose maximum activation magnitude is abnormally large compared with the typical activation scale.

05516#pg4>.

Mechanism of Dominance

The dominant layer emerges because it satisfies two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers<ref:2606.05516#pg5>. Specifically, this layer appears early in the model and aligns with the first activation-outlier layer, whose output contains extremely large-magnitude activations<ref:2606. p. 4

Where the paper says it

In contrast, due to residual connections between layers in LLMs, the output activation of an earlier layer can affect remaining layers and accumulate into the final hidden activation, as illustrated in Figure 5. This effect is especially important for the dominant layer because it aligns with the first activation-outlier layer, whose output contains extremely large-magnitude activations.

05516#pg5>. This propagation allows the perturbation effect to be repeatedly transformed and accumulated before reaching the final loss, leading to larger final-loss changes and a more stable forward signal for ZO updates<ref:2606. p. 1
Where the paper says it

Perturbations at this layer enter the residual stream and affect the activations of all remaining layers. This propagation allows the perturbation effect to be repeatedly transformed and accumulated before reaching the final loss, leading to larger final-loss changes and a more stable forward signal for ZO updates.

05516#pg5>.

Experimental Validation and Efficiency

Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52× training speedup<ref:2606. p. 0

Where the paper says it

Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4. 52× training speedup.

05516#pg2>. The dominant-layer ZO also improves training efficiency, as updating only one dominant layer gives an ideal parameter-side reduction of about 32× compared with full-model MeZO<ref:2606. p. 7
Where the paper says it

Therefore, the expected efficiency gain comes mainly from the parameter-side operations when using Dominant-layer ZO. For LLaMA2-7B, which has 32 decoding layers, updating only one dominant layer gives an ideal parameter-side reduction of about 32× compared with full-model MeZO.

05516#pg9>. This confirms that dominant-layer ZO is most beneficial when parameter-side operations are a significant bottleneck<ref:2606. p. 7
Where the paper says it

12× on DROP. This confirms that dominant-layer ZO is most beneficial when parameter-side operations are a significant bottleneck.

05516#pg9>. The results indicate that useful ZO adaptation is not uniformly distributed across all layers, but can be effectively captured by updating one structurally important layer<ref:2606. p. 6
Where the paper says it

57%) and COPA (+2%). These results indicate that useful ZO adaptation is not uniformly distributed across all layers, but can be effectively captured by updating one structurally important layer.

05516#pg8>.

Conclusion

Our results show that full-model ZO does not simply update too many parameters; it allocates optimization effort unevenly across layers<ref:2606.05516#pg10>. This motivates future ZO methods that account for layer identity and within-layer importance<ref:2606. p. 8

Where the paper says it

Overall, our results show that full-model ZO does not simply update too many parameters; it allocates optimization effort unevenly across layers. This motivates future ZO methods that account for layer identity and within-layer importance.

05516#pg10>. The dominant layer combines high perturbation sensitivity with an early position in the residual stream, allowing its perturbation to affect many subsequent blocks<ref:2606. p. 8
Where the paper says it

We further show that this layer aligns with the first activation-outlier layer, enabling inference-only identification before training. Our analysis suggests that the dominant layer combines high perturbation sensitivity with an early position in the residual stream, allowing its perturbation to affect many subsequent blocks.

05516#pg10>.

Limitation

There is still a performance gap between Dominant-layer ZO and first-order fine-tuning methods<ref:2606.05516#pg10>. Dominant-layer ZO still takes many steps to achieve good performance, which remains a problem for applications<ref:2606. p. 8

Where the paper says it

There is still a performance gap between Dominant-layer ZO and first-order fine-tuning methods. Dominant-layer ZO still takes many steps to achieve good performance, which remains a problem for applications.

05516#pg10>. The authors plan to address these limitations and investigate them on more pre-trained LLMs in their future research<ref:2606. p. 8
Where the paper says it

We didn’t explore more models and combining Dominant-layer ZO with other optimizers designed for ZO, such as ZO-AdaMM [5] and FZOO [9]. We plan to address these limitations and investigate them on more pre-trained LLMs in our future research.

05516#pg10>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Dominant-layer ZO fine-tuning can recover or even exceed full-model ZO performance, achieving up to 4.52x training speedup across benchmarks like LLaMA2-7B and Qwen3-8B. This allows for significantly reduced training cost while preserving or improving performance compared to full-model ZO fine-tuning.

  2. The dominant layer can be efficiently identified before training using a simple inference-only selection method: given a small calibration set, we run the pre-trained LLM forward, measure layer-wise activation statistics, and select the first layer that shows a clear outlier pattern. This avoids expensive layer-wise ZO fine-tuning.

  3. The dominant layer combines high perturbation sensitivity and early placement in the residual stream, which allows it to produce disproportionately strong and stable optimization signals under forward-only updates because its perturbation effect propagates through all remaining subsequent layers via residual connections.

  4. AI systems can be optimized by restricting ZO updates to a single structurally important layer, as this strategy achieves performance comparable to or better than full-model ZO fine-tuning across multiple tasks.

  5. For specific model architectures like Qwen3-8B, the dominant layer index is model-specific (e.g., layer 6), indicating that future ZO method design should account for architecture-dependent layer importance rather than assuming a generic optimization pattern.

Abstract

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers. Specifically, the dominant layer consistently aligns with the first activation-outlier layer in the pre-trained model. To explain this phenomenon, we analyze how perturbation effects propagate under ZO optimization. We find that the dominant layer combines two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers. As a result, this layer produces disproportionately strong and stable optimization signals under forward-only updates. Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52 times training speedup.

Sources

Related papers