Geometric Self-Distillation for Reasoning Generalization
GEOSD addresses mismatches in AI radio's self-distillation by using a geometric approach. It modifies teacher guidance by weighting preferences based on token overlap and penalizing updates that drift too far from recent checkpoints. This method improves out-of-distribution reasoning while keeping the model's in-distribution performance high.