Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning
The episode discusses a paper titled "Are We Measuring Strategy or Phrasing?" which addresses a gap in measuring diversity within Large Language Model (LLM) math reasoning. The hosts explain how conventional metrics fail by capturing only surface-level variation, and propose using human-calibrated LLM judges to formally define approach-level diversity as strategy variation among correct solutions.