Distilling Token-Trained Models into Byte-Level Models
The paper proposes a two-stage distillation process to convert existing token-trained Large Language Models into Byte Language Models (BLMs). This method aligns byte-level representations with teacher embeddings and then fine-tunes the model entirely in the byte space. The result is a highly efficient BLM that achieves comparable performance using only about 125 billion bytes of training data, significantly lowering the cost of building scalable BLMs.