The Stepwise Informativeness Assumption: When Do Entropy Dynamics Track Reasoning Correctness in LLMs?
The Stepwise Informativeness Assumption (SIA) explains why internal entropy dynamics correlate with external correctness in autoregressive models. The paper formalizes this by proposing that reasoning prefixes accumulate information about the true answer in expectation. Empirical validation shows that training induces a distribution satisfying SIA, and correct traces exhibit specific entropy signatures like early lock-in, distinguishing them from incorrect reasoning.