Adaptive-Depth Is a Finishing Stage
The fixed-first training method optimizes adaptive-depth looped transformers by adjusting model depth only during a short final training phase. This approach significantly cuts compute and wall-clock time while preserving performance across various serving depths, balancing specialization with efficiency.