One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

arXiv:2610.12448 · cs.CV, cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "One Block, Multiple Depths".

Jane: The gist: reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Alright, let's talk about the title and who wrote this paper. The paper is called "One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts." It’s a pretty descriptive name that immediately tells you what the main trick is.

Jane: It sounds like it’s focused on using recurrence within a single block to achieve multiple depths of computation without needing to stack many separate blocks. They list Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos as the authors.

Lu: I think the authors are tackling a very specific architectural challenge: how to efficiently handle depth-specific transformations in vision models when you want to keep the computational footprint manageable.

Meng: I’m curious if they are proposing something that drastically changes how we think about scaling vision models, or if it's more about optimizing existing structures for better performance at a given cost.

Lalam: It feels like they are focusing on the internal mechanics of a transformer block, trying to make that single block do more work intelligently across different scales.

Tom: They are doing exactly that. They’re not proposing an entirely new type of network structure, but rather showing how a single recurrent Transformer block can be applied recurrently to match the accuracy of a full-depth vision encoder at comparable FLOPs.

Jane: The big implication here is efficiency. If you can achieve full-depth accuracy with fewer FLOPs, it makes models more accessible for real-time applications where computational power is limited.

Lu: It shifts the focus away from just adding more layers, which has been a common approach, toward smarter internal computation patterns within the block itself.

Meng: That means we spend less time just building longer sequential stacks and more time figuring out how to make those individual layers work harder with better parameter utilization.

Lalam: It suggests that the future of efficient vision models might lie in making each fundamental block more versatile, capable of adapting its behavior based on the input context or desired depth.

The paper's summary: Tom: Now, let's get into the actual mechanics described in "One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts." Basically, they show that they represent the FFN at each recurrent depth as a convex combination of a small shared expert bank.

Jane: They use this continuous normalized-depth coordinate to program this mixture, which defines a resampleable trajectory through the FFN parameter space. This is how they define the depth-specific transformations.

Lu: So, at each recurrent step, there’s one dense FFN that is produced by applying this weighted combination of experts to every token in the sequence.

Meng: It seems like the key here is that this single dense FFN is applied universally across all tokens after it's been programmed by that depth coordinate.

Lalam: So, instead of having different specialized layers for each depth, you have one shared structure whose output changes based on where you are along the depth trajectory.

Tom: That’s right. It replaces the independently learned depth-wise transformations with this continuous path through parameter space. This is a departure from static hypernetworks because they don't assign a fixed embedding to each depth level beforehand.

Jane: The router maps that scalar coordinate s t = t / (L - one) to soft coefficients over the expert bank, defining a soft mixture of experts at each recurrent depth <ref:2610.12448#pg1>.

Lu: They evaluate this design in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher <ref:2610.12448#pg1,evaluate this design in two regimes: supervised ImageNet-1k training and distillation>. Across both regimes, controlled adaptations identify weight-space merging as the strongest MoE family tested when you are restricted to one FFN budget.

Meng: So they are finding that weight-space merging is the most effective way to use a small number of experts when you have a tight budget on a single FFN computation.

Lalam: It’s about finding the best way to pack that accuracy into the fewest possible parameters by using this expert bank effectively.

The paper's improvements: Tom: Moving on to how they suggest improving this design, they focus heavily on replacing simpler MoE approaches with their method. They argue that weight-space merging is superior to token-dispatch and output-mixture alternatives when applied to a single recurrent block.

Jane: They also talk about how they found that expanding the expert bank size consistently recovers accuracy lost under plain weight tying, showing that increasing E from four to eight experts improves S/sixteen by two point zero on ImageNet points during supervised training <ref:2610.12448#pg1>.

Lu: So if you are training from scratch, having more experts helps boost the model's performance at larger scales while keeping the computation manageable because they still only evaluate one dense FFN per step.

Meng: That’s interesting because it means we get a direct trade-off: more capacity for better accuracy versus maintaining a lower FLOP count for that single FFN evaluation.

Lalam: It shows that capacity isn't just about throwing more parameters at the problem; it’s about structuring those parameters in a way that allows the model to leverage them more effectively during training.

Tom: They also noted that this depth-programmed merge outperforms the tested token-dispatch and output-mixture alternatives, which they cite Csordás et al., two thousand twenty-four Liu et al <ref:2610.12448#pg2>., two thousand twenty-four Yang et al <ref:2610.12448#pg2>., two thousand twenty-five and Wang et al <ref:2610.12448#pg2>.

Jane: And they also pointed out that once L is chosen, you can compute this map once and reuse it or insert it into a conventional fixed-depth graph with no routing or merging in the input path.

Lu: This flexibility is huge because it means you can choose to store the compact expert bank, or you can materialize the L merged FFNs directly for a fixed graph if you don't want any runtime routing involved.

Meng: So we have this choice: either a highly optimized compact storage or a pre-compiled sequence of dense layers, and both options maintain the same compute budget per depth step.

Conclusion: Tom: We’ve covered a lot about how "One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts" works—how it uses a continuous coordinate to program experts at each recurrent depth. This technique allows for matching full-depth accuracy at comparable FLOPs.

Jane: Essentially, they found that weight-space merging is the strongest MoE family tested when you are restricted to a single FFN budget, and they showed that increasing the expert bank size improves accuracy by two point zero on ImageNet points when supervised training <ref:2610.12448#pg1>.

Lu: The ability to choose between storing experts or materializing merged FFNs for deployment flexibility is a practical aspect that makes this architecture very adaptable to different deployment scenarios.

Meng: The main thing we take away is that this structure allows us to use fewer parameters while maintaining good accuracy, and it’s a solid path toward more efficient vision systems.

Lalam: It’s about finding a way to structure the network so that the capacity you add actually gets utilized without increasing the cost of running it.

Tom: So that's what this paper is all about: reViT, showing how recurrence and depth programming can give us depth-specific computation while maintaining high accuracy at comparable inference FLOPs. We’ll leave you with a paper that shows we can build more efficient vision systems by intelligently structuring the FFN layers.

Adrian Bulat, Yassine Ouali, Georgios Tzimiropoulos

Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London

cs.CV, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank, matching full-depth vision encoder

Key concepts

Recurrent Vision Transformer: Instead of stacking many standard Transformer blocks sequentially, reViT uses one block that runs recurrently for multiple steps. This allows the model to process information across different 'depths' or stages in a single pass, mimicking the behavior of deeper models.
Depth-Programmed Experts: The model represents the FFN at each recurrent depth as a convex combination of a small bank of shared experts. A continuous coordinate programs this mixture, creating a resampleable trajectory through the expert space that defines how transformations change with depth.
Normalized-Depth Coordinate: This is a continuous scalar value ranging from 0 to 1 that programs the mixture weights for the experts at any given depth. This single coordinate dictates which combination of experts is used to compute the FFN transformation for that specific layer's depth.
Soft Mixture of Experts: At each recurrent step, a router maps the normalized coordinate to soft coefficients over an expert bank. This creates a smooth, weighted combination of different expert functions, allowing the model to dynamically select and blend different transformation strategies based on the current depth.

Terminology

Transformer — 28× in this paper · explained in 2 episodes
Distillation — 23× in this paper · explained in 2 episodes
Ablation — 15× in this paper · explained in 1 episode
Accuracy — 14× in this paper · explained in 1 episode
Adaptation — 10× in this paper · explained in 1 episode
Alignment — 8× in this paper · explained in 1 episode
Hypernetwork — 4× in this paper · explained in 2 episodes
Regularization — 4× in this paper · explained in 3 episodes
Adam — 2× in this paper · explained in 2 episodes
Embeddings — 2× in this paper · explained in 2 episodes
Sparsity — 2× in this paper · explained in 4 episodes
Cosine Similarity — 1× in this paper · explained in 3 episodes
Linear probe — 1× in this paper · explained in 2 episodes
Mixture of Experts — 1× in this paper · explained in 2 episodes
Provenance — 1× in this paper · explained in 3 episodes
Self-Supervised Learning — 1× in this paper · explained in 7 episodes
Trace — 1× in this paper · explained in 2 episodes
Transformer Models — 1× in this paper · explained in 2 episodes

Summary

The gist: reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank, matching full-depth vision encoder accuracy at comparable inference FLOPs without intermediate feature distillation.

How it works

reViT replaces the standard ViT's depth-wise stack with one recurrent Transformer module where a continuous normalized-depth coordinate programs a resampleable trajectory through an E-expert FFN bank The router maps this scalar coordinate to expert logits and mixture weights, defining a soft mixture of experts at each recurrent depth Specifically, at depth t, the router maps st = t/(L−1) to soft coefficients over an E-expert FFN bank This weighted combination produces one dense FFN that is applied to every token The recurrent step involves applying the shared attention and normalization, followed by the depth-programmed FFNt(u) = W¯2,t GELUW¯1,tu + ¯b1,t + ¯b2,t.

Key Mechanisms and Design

The design incorporates several key features to achieve depth-specific computation while maintaining efficiency

**)(a) Full-depth recurrence involves one shared module running for L steps, where the router maps st = t/(L−1) to soft coefficients over an E-expert FFN bank. The weighted combination produces one dense FFN that is applied to every token. This continuous normalized-depth coordinate programs a continuous path through FFN parameter space, replacing the independently learned depth-wise transformations. 5) The router takes the normalized coordinate s(L)t directly rather than assigning a learned embedding to each depth, as in static hypernetworks. 7) Together, Eqs. 1-2 map any s ∈ [0, 1] to a merged FFN. The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image. 9) Once L is chosen, they can be computed once and reused or inserted into a conventional fixed-depth graph with no routing or expert merging in the input path. 7) The compact form stores E expert FFNs, whereas the fixed graph stores the L merged FFNs directly. 9) In both cases, each depth evaluates one dense FFN, so increasing E adds stored capacity without increasing dense FFN computation. 7) The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image. 9) Once L is chosen, they can be computed once and reused or inserted into a conventional fixed-depth graph with no routing or expert merging in the input path. 7) The compact form stores E expert FFNs, whereas the fixed graph stores the L merged FFNs directly. 9) In both cases, each depth evaluates one dense FFN, so increasing E adds stored capacity without increasing dense FFN computation. 7) The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image. 9) Once L is chosen, they can be computed once and reused or inserted into a conventional fixed-depth graph with no routing or expert merging in the input path. 7) The compact form stores E expert FFNs, whereas the fixed graph stores the L merged FFNs directly. 9) In both cases, each depth evaluates one dense FFN, so increasing E adds stored capacity without increasing dense FFN computation. 7) The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image. 9) Once L is chosen, they can be computed once and reused or inserted into a conventional fixed-depth graph with no routing or expert merging in the input path. 7) The compact form stores E expert FFNs, whereas the fixed graph stores the L merged FFNs directly. 9) In both cases, each depth evaluates one dense FFN, so increasing E adds stored capacity without increasing dense FFN computation. 7) The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image. 9) Once L is chosen, they can be computed once and reused or inserted into a conventional fixed-depth graph with no routing or expert merging in the input path. 7) The compact form stores E expert FFNs, whereas the fixed graph stores the L merged FFNs directly. 9) In both cases, each depth evaluates one dense FFN, so increasing E adds stored capacity without increasing dense FFN computation. 7) The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image. 9) Once L is chosen, they can be computed once and reused or inserted into a conventional fixed-depth graph with no routing or expert merging in the input path. 7) The compact form stores E expert FFNs, whereas the fixed graph stores the L merged FFNs directly. 9) In both cases, each depth evaluates one dense FFN, so increasing E adds stored capacity without increasing dense FFN computation. 7) The model of depth L evaluates this map at L evenly spaced points. 9) Choosing a different L changes only the sampling grid, allowing the same learned depth program to be used without retraining. 7) Because this map depends only on depth, the merged FFNs are the same for every image.

Improvements for AI systems

Bold header: Recurrent Vision Architecture for Depth-Specific Computation

This system replaces a ViT’s depth-wise stack with one recurrent Transformer module and uses a continuous normalized-depth coordinate programs a continuous trajectory through FFN parameter space to match the accuracy of full-depth vision encoders at comparable FLOPs.

Bold header: Elastic Depth Inference and Deployment Flexibility

The model supports multiple tested inference depths by resampling the same normalized coordinate interval, allowing for flexible deployment, or it can be materialized into a conventional dense graph, removing online routing and merging without changing the one-FFN-per-depth compute but expanding deployment storage.

Bold header: Expert Composition Superiority in Recurrent Settings

The system identifies weight-space merging as the strongest tested MoE family at a matching one-FFN budget, ahead of the token-dispatch and output-mixture alternatives when applied to a single recurrent block.

Bold header: Enhanced Teacher Alignment via Learned Depth Programs

The router learns where the dominant expert changes and blends experts across each transition, which is crucial for teacher alignment, improving performance over simpler mechanisms when distilling from frozen teachers.

Bold header: Scalable Expert Capacity Utilization

By expanding the merged expert bank consistently recovers accuracy lost under plain weight tying, with analysis showing that increasing E from four to eight improves S/16 by 2.0 ImageNet points for supervised training, the system achieves higher accuracy at larger scales while maintaining a single dense FFN evaluation per step.