Estimating problem difficulty without ground truth using Large Language Model comparisons

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Estimating problem difficulty without ground truth using Large Language Model comparisons".

Jane: Here's the gist of the paper. The gist: LLM compare is a new method for estimating problem difficulty that does not rely on ground truth data, using an LLM to perform pairwise difficulty comparisons and then computing Bradley-Terry scores based on the outcomes.

Tom: Problem Difficulty Estimation Context Recent advances in the finetuning of large language models (LLMs) have significantly improved their performance on established benchmarks, emphasizing the need for increasingly difficult, synthetic data The ultimate goal is to create synthetic out-of-distribution data, i.e.

Jane: problems that are currently unsolvable by both humans and LLMs Current approaches, such as human calibration or performance-based scoring, fail to generalize to out-of-distribution problems because they are not scalable, time-consuming, and ground truth dependent Traditional problem difficulty was measured in a human-centric way, such as Classical Test Theory (CTT) and Item Response Theory (IRT) However, these methods fundamentally depend on human responses and do not generalize to newly generated problems.

Tom: The LLM Compare Methodology LLM compare consists of two parts First, an LLM plays a fixed number of matches per problem in a dataset we want to score by letting it decide which out of two problems is more difficult with a basic prompt.

Paper summary: Tom: So, looking at "Estimating problem difficulty without ground truth using Large Language Model comparisons," the authors are proposing this LLM compare method as a way to estimate problem difficulty by having an LLM do pairwise comparisons and then using those outcomes to calculate Bradley-Terry scores.

Jane: It really highlights that we can get a measure of difficulty that is dynamic and continuous, and it doesn't require any ground truth data for the initial estimation.

Tom: And the key thing they showed is that this method isn't just good at ranking problems; it can actually identify distinct tiers in a benchmark because they found clear separation between easy problems and hard problems with a Cohen’s d of two point five six on their synthetic set <ref:2512.14220#pg3>.

Jane: For someone who only listens to the show, this means we are looking at tools where AI can help structure the difficulty of new content in a way that is interpretable, unlike some other ranking methods that might just bin things into groups.

Tom: The authors are positioning this as a fine-grained and reliable alternative to existing measures like CTT or IRT because it’s robust to hallucinations and shows high correlation with human labels.

Jane: It opens up possibilities for using AI in curriculum design or adaptive testing systems where the difficulty of problems needs to be constantly evaluated without relying on slow, time-consuming human annotations.

Tom: This paper is a significant step toward creating synthetic data that has a structure, not just random noise.

Lu: It’s important because it moves us past the ground truth dependency that has limited how much we can generate challenging problems for large language models to learn from.

Meng: From my side, it shows that AI can be used as an evaluator in a way that is scalable and doesn't require us to rely on constantly running human annotation pipelines.

Lalam: So the paper essentially gives us a tool where the difficulty itself is derived from how LLMs compare problems against each other, and it’s a method that seems quite solid for moving forward with synthetic data generation.

Conclusion: Tom: So, we're wrapping up this discussion on "Estimating problem difficulty without ground truth using Large Language Model comparisons." Basically, these researchers developed a new way to figure out how hard a problem is by just having one AI compare two problems against each other.

Jane: It’s about ditching the need for human labels or perfect answers when we want to measure difficulty. They use the AI to do pairwise comparisons and then calculate scores using something called Bradley-Terry models.

Lu: The core idea is that you don't need ground truth data upfront; you just need a set of problems, and the LLM iteratively estimates how strong one problem is compared to another based on those matches.

Meng: It’s interesting that this method doesn't rely on any pre-existing human calibration, which is a big deal for scaling up difficulty testing without needing massive annotation teams.

Tom: What I find compelling is that it’s dynamic and continuous, meaning we can test new problems as they come in without having to re-calibrate the whole system every time.

Jane: And they showed that these scores actually align well with human judgments, showing a Pearson correlation of over zero point eight for a large set of problems.

Lu: That alignment is strong because it means the AI’s internal estimation process isn't completely detached from what humans consider hard or easy.

Meng: But the paper also points out that this method can actually reveal tiers in a benchmark, meaning it can tell you there are distinct groups of problems that are significantly harder than others.

Tom: That’s significant because traditional measures often just give you a single score, not the nuance of having clear difficulty levels within your test set.

Jane: So, what this means for us is we might be able to build synthetic data sets that have a more structured difficulty curve rather than just random problems.

Lu: Exactly. This moves us toward generating out-of-distribution problems that are truly challenging because the AI has learned a consistent relative measure of difficulty across the entire set.

Tom: It’s a step toward making those synthetic problems more useful for training better models, and it’s certainly less reliant on manual labeling than we've seen before.

Jane: Next up, we're going to look at how these scores actually separate the hard problems from the easy ones in practice.