Cliff Tokens: Analyzing Failure Trigger Tokens in LLM Mathematical Reasoning
This research identifies 'cliff tokens'—specific points where a reasoning trace has a high probability of failure—and categorizes them into deterministic, uncertain, and sampled-off types. By analyzing these triggers across multiple models and benchmarks, the study shows that deleting cliff tokens improves performance significantly. Training specifically on uncertain and sampled-off cliffs provides the most effective signal for improving mathematical reasoning.