Compressing History into Memory: Distilling Transformers into Recurrent Transformers
The work proposes a distillation method to improve recurrent transformers by transferring compression strategies from full-history transformers. A teacher model compresses observation history into a fixed memory bottleneck, and a student recurrent model learns to mimic this compression using supervised learning. This results in an efficient recurrent model that achieves performance close to computationally expensive full-history transformers.