One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
reViT aims to match full-depth vision encoder accuracy using only a single recurrent Transformer block applied repeatedly. It achieves this by representing the Feed-Forward Network (FFN) at each depth as a mixture of experts, controlled by a continuous normalized-depth coordinate. This method restores depth-specific transformations without needing intermediate feature distillation.