Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

arXiv:2610.15159 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Beyond Adjacent Layers".

Jane: The gist The FCM framework exploits full pairwise Centered Kernel Alignment similarity matrices to identify non-adjacent functional communities within large language models,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper, "Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models." Basically, they are testing the idea that just looking at next-door layers isn't enough for compressing these big models.

Jane: That makes sense. The core thesis here is that functional redundancy in these large language models isn't just local to adjacent layers; it’s actually spread out across non-adjacent ones too, and this paper shows how to find that structure.

Lu: What they claim is that when you compute full pairwise Centered Kernel Alignment similarity matrices for thirteen different LLMs from six different architectures, non-adjacent layers show a much higher degree of similarity than adjacent ones, with three point six to fifteen point nine times more high-similarity pairs than adjacent ones <ref:2610.15159#pg1>.

Meng: So they’re saying the assumption that functional redundancy is local is wrong, which means existing methods that only look at neighbors are missing a lot of potential compression opportunities.

Lalam: Exactly. The paper calls this structure a "structure that prior similarity-based layer merging has not systematically exploited" <ref:2610.15159#pg1>. They propose Fusion via Community Mining, or FCM, to handle this non-local relationship.

Tom: FCM builds the whole L by L similarity graph and then uses spectral community detection to find these functional clusters of layers that are equivalent, and then they keep the most important layer in each one.

Jane: That sounds like a pretty sophisticated way to group things, but what makes this approach different from just looking at simple layer importance scores?

Lu: They use Block Influence, or BI

Men et al., two thousand twenty-five: , which measures the cosine distance between each layer's input and output hidden states, quantifying how much a layer actually transforms its input <ref:2610.15159#pg4>. This score helps them select the most important member within each community.

Meng: I wonder what that means for deployment. If we are compressing to K layers, which one do they keep from that community? That choice seems critical for maintaining performance at scale.

Lalam: They select the layer with the highest Block Influence score in each detected community, which is called the most important layer in that cluster <ref:2610.15159#pg4>. This is then fed into a LoRA recovery fine-tuning process to reconstruct the model <ref:2610.15159#pg4>.

Tom: And there's this detail about how they decide whether to use cosine similarity or CKA similarity, which depends on some rule computed only on calibration data <ref:2610.15159#pg4>. That's a practical step that grounds the theory in actual application.

Jane: So, what are the actual numbers they show? Because I need to know if this isn't just theoretical fluff.

Tom: At two times compression, FCM lowers me perplexity below ShortGPT on twelve out of thirteen models, which is significant on ten of them, and it achieves gains up to fifty percent on Qwen3-4B Table three.

Meng: Fifty percent gain is substantial. But how does that translate when you think about the cost of running these compressed models compared to the original ones?

Lu: The paper shows that FCM attains the best perplexity on ten out of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table four <ref:2610.15159#pg1>.

Jane: So they're beating several existing methods in terms of accuracy on this specific benchmark setup. What does that imply for the real world application?

Tom: The gains carry over to downstream tasks Section four point six, and the compression sweep reveals a crossover point beyond which FCM’s advantage keeps growing up to three times and persists at the 14B scale Table three.

Meng: Three times that factor is big. But what about the limitations they admit? What's not covered by this method, or where does it break down?

Lalam: They state that FCM casts grouping as community detection on the full L by L similarity graph, which is a specific way of doing things <ref:2610.15159#pg3>. It also mentions that MKA groups greedily by pairwise similarity and merges weights rather than retaining a phase-covering representative <ref:2610.15159#pg3>.

Jane: So, the method relies heavily on the quality of that full similarity graph construction. If the initial profiling step is flawed, does the whole compression strategy fall apart?

Lu: The paper does mention that existing methods for depth compression score layers individually or group only adjacent or contiguous layers, which implicitly assumes that functional redundancy is local <ref:2610.15159#pg1>. FCM directly tests this assumption by computing those full pairwise CKA similarity matrices <ref:2610.15159#pg2>.

Tom: So the main thing here is moving from a local view of layers to a global view defined by these non-adjacent functional relationships, which leads to better compression and better results for the models.

Jane: This paper, "Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models," seems to suggest that we need a more holistic way of thinking about how these massive AI models are structured internally.

Lu: It pushes the idea that understanding the functional equivalence between layers, not just their physical adjacency, is key for effective model compression <ref:2610.15159#pg3>.

Meng: From an engineering standpoint, if we can reliably identify these functional communities across different model architectures using this graph approach, it could lead to much more efficient and accurate distillation techniques.

Lalam: I'm confident in the findings because they are based on testing thirteen LLMs across six families, which gives a broad view of how this structure behaves <ref:2610.15159#pg1>.

Conclusion: Tom: So we've seen how this paper uses full pairwise similarity matrices to find functional groups of layers in large language models, and now we're looking at what the authors are actually trying to claim with this "Beyond Adjacent Layers" work <ref:2610.15159#pg4>.

Jane: The title suggests they're moving past just looking at neighbors and realizing the important connections are spread out <ref:2610.15159#pg3>. But what does that actually mean for how we build these models?

Lu: They are showing that functional equivalence isn't just local; it happens across non-adjacent layers too, which is a structure that prior similarity-based layer merging hasn't systematically exploited <ref:2610.15159#pg3>. It suggests the compression gains can go way beyond what you get from only looking at adjacent layers.

Meng: So if this graph structure is real, it means we could potentially compress these models much further than current techniques that rely on local assumptions <ref:2610.15159#pg1>. But how do we know this structure holds up when you try to deploy the compressed version?

Lalam: The method involves finding communities through spectral clustering on the similarity graph, then picking the most important layer in each community based on its Block Influence score <ref:2610.15159#pg4>. That selection process is what dictates which layers we keep for the compressed model.

Tom: And they show that this approach doesn't just compress; it actually improves performance, with lower perplexity and better results on downstream tasks compared to other methods <ref:2610.15159#pg7>. That’s a measurable win here.

Jane: So the big idea is that by understanding these deep functional relationships, we get both smaller models and models that don't lose too much accuracy <ref:2610.15159#pg8>. But the authors also have to be careful about how they define those communities because if the initial similarity calculation is off, the whole grouping fails.

Lu: Yeah, they acknowledge that this process relies on building that full L by L similarity graph correctly, and it's a specific way of doing things <ref:2610.15159#pg3>. That initial step is where the main computational effort goes into making sure the structure is sound.

Meng: For us in production, the cost comes with that initial profiling step on a calibration dataset D, which you need to collect first <ref:2610.15159#pg2>. I wonder how much data that upfront profiling actually costs compared to just running a standard compression sweep.

Lalam: The paper shows that FCM uses cosine similarity by default unless it hits a certain degeneracy threshold, which is defined by specific fixed thresholds for beta and mu <ref:2610.15159#pg4>. That gives the grouping decision some concrete rules to follow.

Tom: So, in short, this paper proposes a method that looks at non-adjacent layers to find functional groups, picks the best representative layer from each group based on influence, and then recovers the model with fine-tuning <ref:2610.15159#pg7>. It's about finding better structure for smaller models.

Jane: So it’s less about making a single perfect model and more about finding a smarter way to represent the same knowledge in fewer, more meaningful parts <ref:2610.15159#pg4>. This makes sense for scaling up AI systems that are currently becoming massive.

Lu: It really pushes the idea that understanding how layers functionally relate to each other, rather than just their physical position on the stack, is key for effective model compression <ref:2610.15159#pg3>. That's a major shift in how we think about representation learning in these models.

Meng: If this technique scales well across different architectures, it changes the whole compression toolkit for deploying large language models to edge devices <ref:2610.15159#pg4>. It shows that structural understanding is a viable path to efficiency.

Lalam: The implication is that we can move from just cutting layers haphazardly to strategically fusing functionally related parts of the model, which leads to better compression and better results <ref:2610.15159#pg7>. That's what this work is pointing toward.

Muhammad Usama, Yeoreum Jung

Braindeck Inc

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Comments: Accepted at NeurIPS 2026. Code: https://github.com/Usama1002/fcm-layer-fusion

Code: https://github.com/Usama1002/fcm-layer-fusion

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The FCM framework exploits full pairwise Centered Kernel Alignment similarity matrices to identify non-adjacent functional communities within large language models, leading to compression

Key concepts

Centered Kernel Alignment (CKA): CKA measures how similar two activation matrices are by centering them first. It captures functional equivalence between different parts of the model, focusing on what the activations represent rather than just their weight differences. This allows the framework to group layers that perform similar functions together.
Block Influence (BI): Block Influence quantifies how much a specific layer transforms its input into its output. It is calculated as one minus the cosine similarity between a layer's input and output states across different dimensions. This metric helps determine which layer within a functional community is the most critical for that group.
Spectral Clustering: Spectral clustering is an algorithm used to partition data points based on their connectivity in a graph. In this context, it organizes layers into communities by treating the similarity matrix (derived from CKA) as a weighted graph. This process groups layers that are functionally similar together.
Layer Fusion and Recovery: The compressed model is formed by selecting one representative layer for each identified community. These selected layers are then reassembled, and LoRA recovery fine-tuning is applied to all linear projections of the resulting smaller model. This step ensures the compressed model retains high performance.

Summary

The gist The FCM framework exploits full pairwise Centered Kernel Alignment similarity matrices to identify non-adjacent functional communities within large language models, leading to compression gains that surpass existing methods and improve downstream performance.

How it works

Profiling: The process begins by collecting activation matrices at the output of each decoder block for a calibration dataset D, as described in Section 2.1 of page 3; this step is crucial for building the similarity structure Given a calibration dataset D = Hl = h¯(i)l,..., h¯(N)li⊤∈ R N×d (Equation 1), where h¯(i)l = 1/n Pn t=1 h(i)l,t ∈ R d is the mean-pooled hidden state across sequence positions for the i-th sample at layer l, as shown on page 3; this step is crucial for building the similarity structure
Similarity: The framework computes a full L×L pairwise similarity matrix S where Sij is given by Equation 2, which captures functional equivalence rather than weight-space similarity This computation involves measuring linear Centered Kernel Alignment [Kornblith et al., 2019], given centered activation matrices H˜i = Hi - 1/N 1⊤Hi and H˜j, CKA(Hi, Hj) = H˜ ⊤j H˜i2 / F H˜ ⊤i H˜iF · H˜ ⊤j H˜jF (Equation 2), as detailed on page 3; this computation involves measuring linear Centered Kernel Alignment [Kornblith et al., 2019>
Metric selection: The operational grouping decision is determined by evaluating mean cosine similarity as an alternative metric, and FCM uses cosine similarity by default unless it is degenerate, defined by Definition 1 on page 4; the partition is degenerate if B(C) µ∗, with thresholds fixed a priori to β∗ = 0.70 and µ∗ = 0.40, as shown on page 4; this computation involves evaluating mean cosine similarity as an alternative metric, and FCM uses cosine similarity by default unless it is degenerate, defined by Definition 1 on page 4; the partition is degenerate if B(C) µ∗, with thresholds fixed a priori to β∗ = 0.70 and µ∗ = 0.40, as shown on page 4
Importance: Within each detected community Ck = hl1,..., hlCk, the most important member is selected by measuring Block Influence (BI) [Men et al., 2025], which is the cosine distance between each layer’s input and output hidden states, BI(l) = 1 − cos(Hl−1, Hl), quantifying how much a layer transforms its input (Hl as in Equation 1), as shown on page 4; within each community Ck = hl1,..., hlCk, the most important member is selected by measuring Block Influence (BI) [Men et al., 2025], which is the cosine distance between each layer’s input and output hidden states, BI(l) = 1 − cos(Hl−1, Hl), quantifying how much a layer transforms its input (Hl as in Equation 1), as shown on page 4
Grouping: Spectral clustering [Ng et al., 2001] is applied to the similarity graph G = (V, E, w) with w(i, j) = Sij assigning each edge the pairwise similarity between corresponding layers, partitioning layers into communities C = h1,..., hK as shown on page 4; spectral clustering [Ng et al., 2001] is applied to the similarity graph G = (V, E, w) with w(i, j) = Sij assigning each edge the pairwise similarity between corresponding layers, partitioning layers into communities C = h1,..., hK as shown on page 4
Selection: For each community Ck ∈ C do select the layer with the highest Block Influence (BI) score as l∗ = arg maxl∈Ck BI(l), which is the most important layer in community, as shown on page 4; for each community Ck ∈ C do select the layer with the highest Block Influence (BI) score as l∗ = arg maxl∈Ck BI(l), which is the most important layer in community, as shown on page 4
Reassembly and Recovery: The compressed model Mˆ with K layers is reassembled by retaining only one representative per community, ordered by the minimum original layer index within each community, and then undergoes LoRA recovery fine-tuning applied to all linear projections (query, key, value, output, gate, up, down), as shown on page 4; The compressed model Mˆ with K layers is reassembled by retaining only one representative per community, ordered by the minimum original layer index within each community and then undergoes LoRA recovery fine-tuning applied to all linear projections (query, key, value, output, gate, up, down), as shown on page 4

Key Findings and Comparisons

**: FCM lowers mean perplexity below ShortGPT on twelve of thirteen models at 2× compression (significant on ten), with gains up to 50% on Qwen3-4B Table 3, page 7; it also attains the best perplexity on ten of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table 4, page 8; the gains carry over to downstream tasks Section 4.6, page 9; the compression sweep reveals a crossover beyond which FCM’s advantage keeps growing up to 3× and persists at the 14B scale Table 3, page 7; it attains the best perplexity on ten of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table 4, page 8; the gains carry over to downstream tasks Section 4.6, page 9; the compression sweep reveals a crossover beyond which FCM’s advantage keeps growing up to 3× and persists at the 14B scale Table 3, page 7; it attains the best perplexity on ten of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table 4, page 8; the gains carry over to downstream tasks Section 4.6, page 9; the compression sweep reveals a crossover beyond which FCM’s advantage keeps growing up to 3× and persists at the 14B scale Table 3, page 7; it attains the best perplexity on ten of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table 4, page 8; the gains carry over to downstream tasks Section 4.6, page 9; the compression sweep reveals a crossover beyond which FCM’s advantage keeps growing up to 3× and persists at the 14B scale Table 3, page 7; it attains the best perplexity on ten of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table 4, page 8; the gains carry over to downstream tasks Section 4.6, page 9; the compression sweep reveals a crossover beyond which FCM’s advantage keeps growing up to 3× and persists at the 14B scale Table 3, page 7; it attains the best perplexity on ten of thirteen models against FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and recovery Table 4, page 8; the gains carry over to downstream tasks Section 4.

Improvements for AI systems

Bold header: Non-adjacent functional similarity exploitation

The FCM framework exploits non-adjacent high-similarity layer pairs by building a full L × L similarity graph and using spectral community detection to find clusters that may span non-adjacent positions, ensuring the compressed model still spans all functional phases.

Bold header: Importance-guided layer selection

The system retains the most effective layer within each cluster by selecting the highest Block Influence (BI) score, which is defined as the cosine distance between each layer’s input and output hidden states to quantify how much a layer transforms its input.

Bold header: Robust metric choice

To ensure generalization across different model architectures, the system employs a dynamic similarity metric selection: it uses cosine similarity by default but falls back to Centered Kernel Alignment (CKA) when the cosine partition is degenerate, as defined by Definition 1.

Abstract

Existing depth-compression methods for large language models (LLMs) score layers individually or group only adjacent or contiguous layers, implicitly assuming that functional redundancy is local. We test this assumption by computing full pairwise Centered Kernel Alignment (CKA) similarity matrices for thirteen LLMs from six architecture families (1.6B-14B parameters). Non-adjacent layers have 3.6-15.9x more high-similarity pairs than adjacent ones, a structure that prior similarity-based layer merging has not systematically exploited. We propose Fusion via Community Mining (FCM), which builds the full L x L similarity graph, uses spectral community detection to find clusters of functionally equivalent layers, keeps the highest-Block-Influence layer in each cluster, and applies LoRA recovery; a pre-specified rule computed only on calibration data chooses between cosine and CKA similarity. At 2x compression, FCM lowers mean perplexity below ShortGPT on twelve of thirteen models, significantly on ten (Welch's t-test with Holm-Bonferroni correction; up to 50% on Qwen3-4B), with Mistral-7B the only loss (+13.4%). Against FlattenGPT, SWM, and DeltaLLM under matched calibration and recovery, FCM gives the best perplexity on ten of thirteen models (mean rank 1.15), and it also outperforms MKA on three anchor models. FCM matches or exceeds ShortGPT on 15 of 20 zero-shot reasoning cells and also improves MMLU, LAMBADA, HumanEval pass@1, and free-form generation quality. The 2x-compressed Qwen2.5-7B has 46% fewer parameters, 42% lower peak VRAM, and 84% higher decode throughput on a single A100. Spectral grouping outperforms Fisher contiguous segmentation by 41% and uniform partitioning by 51%, and FCM's advantage grows beyond a 2x crossover up to 3x and persists at the 14B scale. Code: https://github.com/Usama1002/fcm-layer-fusion