Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
The paper characterizes complex reasoning as a two-phase process: broad semantic shortlisting followed by fine-grained refinement over a small set of options. It shows that large models succeed through these complementary steps. A new mechanistic distillation strategy, which supervises these specific phases separately, outperforms standard methods by teaching smaller models to replicate this superior two-stage reasoning trajectory.