Distinguishing among reasoning levels of GPT 5.5 Pro via a grammar lens benchmark

arXiv:2611.00555 · cs.CL · Submitted 2026-10-31 · Read on arXiv

cs.CL

Submitted: 2026-10-31

Updated: 2026-10-31

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

The gist: Does the level of test-time processing effort affect a language model's computational capability? A formal grammar-lens benchmark method, derived from the theory of computation…

Terminology

Abstract

Does the level of test-time processing effort affect a language model's computational capability? A formal grammar-lens benchmark method, derived from the theory of computation, generates novel data at specified levels, each successively more computationally powerful than the levels below. Using this data, we tested GPT-5.5, and GPT-5.5-Pro-Medium, -High, and-XHigh, showing incrementally increased acceptance of test data: 16, 40, 43, and 52%, respectively, illustrating the benchmark's ability to discern fine-grained differences among models' added test-time reasoning levels. Additionally, as test-time compute effort increases, model responses diverge from human responses to the same data. This increasing disagreement may turn out to be broadly characteristic of more capable language models, introducing the potential of a general capability-versus-alignment conflict.

Sources

Related papers