Force without transmission: a depth-induced rank collapse that no loss on the representation reopens

arXiv:2610.09958 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Force without transmission".

Jane: The gist The added loss term fails although its force was large, and they failed because almost none of it reached the weights that had to change.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re looking at this paper now, "Force without transmission: a depth-induced rank collapse that no loss on the representation reopens." Basically, the authors are asking if you can actually fix a network when it collapses into this weird state where all its token representations point in one direction and learning just stops.

Jane: That sounds really specific. So what’s the core idea here? What are they trying to prove or disprove?

Tom: They collapse small transformers by weakening the skip connection so that attention averages tokens across blocks, causing their representations to converge onto a single direction, which they call depth-induced rank collapse. And then they look at whether a loss term can repair this.

Jane: So the naive question is: does adding a corrective force actually help recover the network when it’s already collapsed?

Lu: The paper distinguishes between two kinds of corrective forces, vanishing and bounded force, which are measured by terms that differ only in their gradient behavior at the collapse.

Meng: That means they're testing if you need a force that doesn't vanish, or if a force that just stays big enough helps.

Lalam: The paper looks at two specific loss terms: Rpen(X˜) equals one minus Q squared over d minus one, and Rhinge(X˜) equals max of zero s0 minus Q, where srank is the rank.

Tom: And the key difference is that the penalty's gradient is proportional to Q and vanishes as the rank approaches one, whereas the hinge acts only below a healthy value, and its gradient keeps a constant size one over sigma one in the direction that would spread those tokens out.

Jane: That sounds like they’re setting up a comparison between forces that disappear quickly and ones that stay active in the right direction.

Tom: The collapse itself is produced by weakening a skip connection so that attention averages tokens across blocks, causing their representations to converge onto a single direction, which is depth-induced rank collapse. The driver of the collapse is the ramp where the skip connection scales down to alpha min equal zero point zero two by step three thousand.

Paper summary: Meng: So they are creating this intentional failure mode by scaling that connection down over time, right before they test if anything can stop it.

Jane: And then they tested several treatments including penalty one times, hinge one times, penalty four times, and hinge four times as well as cosine one times.

Lu: The results show that the added loss term fails although its force was large because almost none of it reached the weights that had to change.

Tom: That’s a crucial finding. It means even when you have a strong corrective force pushing, if that force doesn't actually arrive at the right place in the network, nothing happens.

Jane: So what does that imply about how we design these forces? If they don't reach the weights needing change, then what’s the real constraint here?

Tom: The paper measured block by block how much of the task gradient and of a hinge’s gradient reached the attention weights of each block. They found that the task gradient arrived weakly at blocks nearest the output and hardly at all at those nearest the input.

Jane: And for these collapsing blocks, like those near the output, they received almost nothing from either side because of this path issue?

Tom: Exactly. The query and key weights received less than one-zero to a minus six in every block of seventeen out of eighteen networks, against three point zero to a minus two to five point zero over per block in healthy ones.

Meng: That comparison is telling. It shows that the forces aren't distributed evenly across the model; they’re getting stuck on certain parts.

Jane: But what about the hinge gradient? Did it behave differently in those critical collapsing blocks?

Tom: The hinge's gradient had the opposite profile, being more than the healthy task gradient in the first block at depth twelve. So, because those collapsing blocks are near the output, they received almost nothing from either side.

Paper summary: Lu: Restoring the skip connection does exactly that without touching a weight. On the reversed ramp every network recovered its rank, eight out of eight in this main study and ten out of ten in the replication.

Jane: So restoring the path itself is what fixes things, even if no added loss term did anything while it was collapsed?

Tom: It does that without touching a weight. On the reversed ramp every network recovered its rank, eight out of eight in this main study and ten out of ten in the replication. Recovery did not happen where the collapse had begun, indicating a hysteresis.

Meng: So it’s like you have to push it past a certain point before it starts moving again?

Jane: And then they tested if they could predict when recovery would happen by looking at something before the skip connection was restored?

Tom: The predictor was the median over blocks of the task gradient on the query and key weights, read after the skip is restored and before any update. This predictor ranked recovery times after a switch to zero point three with a Spearman correlation of negative zero point four or lower.

Jane: So even predicting *when* it recovers is pretty messy, but they also tested if the loss gap at detection was a predictor, and that ordered recovery times better with a positive zero point seven nine.

Lu: The paper concludes that a loss term can only repair what its gradient reaches. Restoring the skip connection reopened that path and the network then learned its way out, only well above the scale of collapse, without its loss returning to that of a healthy network.

Tom: The rank recovered, but only far above the scale of collapse. The paper concludes that arrival was the binding constraint on whether a corrective force could repair the collapse.

Jane: So what does this mean for people who just listen to this show? It suggests that adding a loss term is less important than making sure your network can actually transmit information where it needs to go during failure.

Conclusion: Tom: So we’re wrapping up on "Force without transmission: a depth-induced rank collapse that no loss on the representation reopens." The main thing here is that they showed that a corrective force can only repair what it actually reaches in the network, and restoring the skip connection did the job without adding any new loss term.

Jane: So, to put it simply, even when a network collapses into this weird state where everything points in one direction because of depth, just fixing the connection itself is enough to get it back on track. That’s a big deal for how we think about training these huge models.

Lu: The authors are pointing out that the failure mode—this rank collapse—isn't something a simple loss function can fix by itself; it requires physically re-establishing the path through the network structure, not just applying more mathematical pressure.

Meng: From a deployment standpoint, this suggests that if we see these kinds of catastrophic failures in production, we shouldn't immediately throw in another complex regularization term and hope for the best; we should check if our skip connections are being weakened or averaged out by attention.

Lalam: I’m confident in this because the experiments showed that the forces, whether vanishing or bounded, didn't even get to most of the weights that needed changing. It was a transmission problem, not a magnitude problem.

Tom: Exactly what Lalam is saying—it was about where the force landed on those weights. So what does this mean for how we design these forces? If they can't reach the right place, they're just wasted computation and time spent tuning them.

Jane: It means that instead of chasing bigger numbers in a loss term, we need to focus on ensuring the information actually travels from the input to the output layer correctly during training. It shifts our focus from brute force correction to better network architecture or connection management.

Lu: And I think it opens up new avenues for studying representation learning failures, because it tells us *why* a specific type of collapse happens and what structure is missing.

Meng: If we can pinpoint the structural weakness that causes this rank collapse, we can build defenses against it directly into the architecture instead of patching after the fact.

Lalam: The paper suggests that arrival—the actual reaching of a weight—is the absolute constraint on whether any corrective force works. That’s a very concrete constraint to focus on for model stability.

Tom: Right, so we learned something really specific about how forces interact with network structure. Next up, we'll look at how they tried to predict when this recovery would happen after the connection was restored.

Martin Hofmann, Patrick Mäder

Technische Universität Ilmenau · German Centre for Integrative Biodiversity Research Halle–Jena–Leipzig · Faculty of Biological Sciences Friedrich Schiller University

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Comments: 16 pages, 8 figures, 2 tables

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist The added loss term fails although its force was large, and they failed because almost none of it reached the weights that had to change.

Key concepts

Rank Collapse: This occurs when a deep neural network's token representations converge onto a single direction, losing diversity. This is caused by weakening skip connections during training, which allows attention to average tokens across blocks.
Vanishing Force (Penalty): This corrective force measures how much the loss gradient reaches the attention weights. In this study, even a large penalty failed because its gradient was too weak; it never reached the critical weights that needed adjustment to correct the collapse.
Bounded Force (Hinge): This force acts differently, keeping a constant gradient size in directions that would spread tokens. However, like the penalty, its gradient also failed to reach the necessary attention weights during collapse events.

Summary

The gist The added loss term fails although its force was large, and they failed because almost none of it reached the weights that had to change.

Two Kinds of Corrective Force

The paper distinguishes between two kinds of corrective force, a vanishing and a bounded force. These forces are measured by terms that differ only in their gradient behavior at the collapse. The thing measured is how many directions the token representations of a block span.

The two loss terms are defined as Rpen(X˜) = 1 − Q 2/(d − 1) and Rhinge(X˜) = max(0, s0 − Q), where srank is the rank. The penalty's gradient is proportional to Q and vanishes as the rank approaches one. Conversely, the hinge acts only below the healthy value, and its gradient keeps a constant size 1/σ1 in the direction that would spread the tokens.

How the Collapse is Produced and Treated

The collapse is produced by weakening a skip connection so that attention averages tokens across blocks, causing their representations to converge onto a single direction, which is depth-induced rank collapse. The driver of the collapse is the ramp where the skip connection scales down to αmin = 0.02 by step 3,000.

The treatments tested include:

penalty 1×
hinge 1×
penalty 4×
hinge 4×
cosine 1×
untreated – control – reversed ramp – driver reversed.

The Collapse is Harmful and Does Not Heal

A collapsed network's loss lay 1.73–1.94 nats above that of a healthy network trained on the same data, at 7.55–7.63 nats. Left untreated for 4,000 steps, the rank stayed below the recovery threshold in 3/3 collapse events at depth 12 and 5/5 at depth 24. Attention in the collapsed state was close to uniform: every token attended to all tokens almost equally (normalised entropy 0.94–0.99, against 0.65–0.69 in healthy networks).

The Force Does Not Reach the Weights That Must Change

The paper measured block by block how much of the task gradient and of the hinge’s gradient reached the attention weights of each block. The task gradient arrived weakly at blocks nearest the output and hardly at all at those nearest the input. The query and key weights received less than 10−6 in every block of 17 of 18 networks, against 3 · 10−2 to 5 · 10−2 per block in healthy ones.

The hinge's gradient had the opposite profile, being more than the healthy task gradient in the first block at depth 12. The blocks that complete the collapse, near the output, therefore received almost nothing from either side.

Restoring the Path Recovers the Rank, with Hysteresis

Restoring the skip connection does exactly that without touching a weight. On the reversed ramp every network recovered its rank, 8/8 in the main study and 10/10 in the replication. Recovery did not happen where the collapse had begun, indicating a hysteresis.

Does the Path Predict Recovery?

The predictor was the median over blocks of the task gradient on the query and key weights, read after the skip is restored and before any update. The predictor ranked recovery times after a switch to 0.3 with a Spearman correlation of −0.4 or lower (−0.79 on the main study).

The loss gap at detection was also tested as a predictor, and it ordered recovery times better (+0.79; a larger gap went with slower recovery).

Conclusion

A loss term can only repair what its gradient reaches. Restoring the skip connection reopened that path and the network then learned its way out, only well above the scale of collapse, without its loss returning to that of a healthy network. The rank recovered, but only far above the scale of collapse.

The paper concludes that arrival was the binding constraint on whether a corrective force could repair the collapse.

Author Contributions

M.H conceived the study, wrote the protocol and the code, ran and analysed the experiments and wrote the manuscript. P.M supervised the work and revised the manuscript.

Reproducibility Statement

Code, preregistration, protocol, frozen constants, analysis scripts and the scripts that generate this manuscript are kept in one version-controlled repository and are available from the authors on request. Every measured quantity in this paper is a macro rendered from the analysis outputs; tables and figures are produced by the same scripts, and an audit checks hashes, placeholders and unbound decimals before every build. Training ran on NVIDIA A100-SXM4-40GB GPUs (54 GPU-hours of wall time for calibration and main study at the cluster share available). The data stream is the FineWeb-Edu sample prepared with the GPT-2 tokenizer and a SHA-256 manifest by a sibling project. The registrations of section 8 and appendix E were committed before the runs they predict (commits 6358641, b4b0963 and bf90737).

Use of AI-assisted Tools

Claude Code (Anthropic; Claude Fable 5.1 and Claude Opus 5.5, September and October 2026) assisted with research code, run orchestration, analysis scripts, literature lookup and drafting. The authors designed and verified the study and are responsible for its interpretation, citations and final text.

References

Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning.

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth.

Alessio Giorlandino and Sebastian Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation.

Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers.

Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesv´ari, and Dale Schuurmans. Escaping the gravitational pull of softmax.

Improvements for AI systems

Bold Header: Rank Collapse Repair Mechanism

Repair mechanisms should focus on whether the gradient still reaches the weights that must change, not on how strongly a loss term pushes, implying that corrective terms must target specific weights rather than relying solely on large forces at the loss function. This allows for targeted intervention during rank collapse, as demonstrated by how Restoring the skip connection, which changes no weight, reopened this path at once.

Bold Header: Gradient Path Diagnostics

Implement a block-by-block measurement of gradient arrival to determine if a corrective force is effective; specifically, measure how much of it reaches the attention weights of each block, distinguishing between forces that are vanishing and a bounded force. This diagnostic can differentiate between mechanisms like depth-induced rank collapse and those driven by other causes, such as a burst of high learning rate.

Bold Header: Path Prediction for Recovery

Utilize the measured gradient path as a predictor for network recovery; the paper suggests that measuring the path tells in advance which networks will recover, specifically identifying the median over blocks of the task gradient on the query and key weights, read after the skip is restored and before any update as a potential predictor.

Bold Header: Skip Connection Restoration as Primary Repair

Prioritize restoring structural elements over adding loss terms; since Restoring the skip connection does exactly that without touching a weight, this structural change is identified as the most effective way to reopen this path and the network then learned its way out.

Bold Header: Hysteresis in Recovery Thresholds

Model recovery using hysteresis based on the skip scale parameter alpha; since Recovery did not happen where the collapse had begun but occurred only at α between 0.25 and 0.97 at depth 12, recovery is contingent on reaching a threshold far above the scale of collapse.

Abstract

Training can drive a transformer into a rank collapse: all token representations point in one direction, and learning stops. In a related collapse of attention, a loss term with a bounded corrective force repairs the network during the run. We ask whether such a term repairs rank collapse. We collapse small transformers by weakening their skip connection and treat copies of the collapsed network. No added loss term repaired the collapse, although the stronger kind pushed with about a tenth of the task gradient. The reason was the path, not the strength. The task gradient no longer reached the query and key weights, which decide where attention looks, and the added term's gradient faded before the blocks where the collapse forms. Restoring the skip connection, which changes no weight, reopened this path at once. The rank then recovered, but only far above the scale of collapse. After a burst of high learning rate the path stayed open and the rank recovered untreated. Registered predictions from the path ranked recovery times but did not transfer to this cause. In every case the loss stayed above that of a healthy network after the rank recovered. Whether a collapsed network can be repaired depends on whether the gradient still reaches the weights that must change, not on how strongly a loss term pushes.