CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models
CombEval is a dynamic benchmark framework designed to rigorously test Large Language Models' ability to perform combinatorial counting. It uses a formal language called Cofola to create complex, structured counting problems from specifications. This allows researchers to systematically control problem difficulty and diagnose exactly where LLMs fail in reasoning, such as with ordered objects or deep dependencies.