SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
On-policy distillation (OPD) fails when teacher and student models use different tokenizers because they cannot compare tokens directly. SimCT solves this by enlarging the supervision space to include 'minimal aligned units'—the finest text segments both tokenizers can realize. This allows the method to recover crucial teacher signals that are lost due to vocabulary mismatches, leading to consistent performance gains.