An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
The Generalization–and–Perturbation (GAP) framework systematically tests LLMs' mathematical reasoning by presenting problems that are mathematically equivalent but linguistically or parametrically varied. Results show that high scores on standard benchmarks can collapse when simple changes, like renaming symbols or altering numbers, are introduced. This reveals a weakness in models' true generalization beyond memorized patterns.