Nine Raters, One Index: Carrying LLM Disagreement into Labour-Market Estimates

arXiv:2507.22748 · econ.GN, q-fin.EC · Submitted 2025-07-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Nine Raters, One Index".

Warren: Here's the gist of the paper. When a large language model supplies research annotations, the choice of model becomes an analytic degree of freedom The paper develops GAISI, a task-based measure of UK jobs’ exposure to generative AI, using nine large language models to rate 1,100 task–occupation cells This distinction matters for the use of LLMs as measurement instruments The Core Methodology The study constructs GAISI by rating 1,100 task–occupation cells with nine LLM raters from three vendors and several model generations The ratings are weighted by worker-reported task importance in the British Skills and Employment Surveys to yield a worker-level index The analysis employs a multiverse approach where the rater is treated as one dimension of a multiverse, reporting each rater-specific estimate alongside a Rubin-style summary that incorporates their observed dispersion This procedure exposes and carries observed LLM-rater disagreement into empirical estimates Properties of the Index The study finds that rankings are substantially more stable across raters than levels, with pairwise rank correlations ranging from 0.74 to 0.92 Conversely, the share of British jobs scoring above 0.5 ranges from under 0.1% to 38% across raters The between-rater share of the Rubin-combined variance accounts for about a twentieth of the uncertainty in the pay estimate, which is negative under all nine raters The index behaves as an instrument, showing that discrimination is measured by how reproducibly it ranks jobs when the rating process is varied External Evidence and Validation The external evidence gives reasons to take the resulting ordering seriously, while leaving relevant questions unresolved GAISI’s association with worker-reported AI use falls from 0.226 to 0.154 after nine existing exposure measures enter together, including human-rated generative-AI indices At the task level, exposure rank is associated with observed assistant traffic under both generative and embedding-based mappings of conversation clusters The paper also shows that the UK ranking transfers better than the levels when applied to the OECD Survey of Adult Skills (PIAAC) workers Labour Market Implications The labour-market applications warrant the same restraint as other measures, showing that mean worker exposure rank rose by 4.5 percentile points between 2017 and 2023/24, almost entirely because employment shifted between occupations rather than because task profiles changed within them Relative pay and job adverts also moved against more exposed occupations after 2022, but the vacancy association disappears once teleworkability is allowed its own post-2022 path The paper concludes that neither pay nor labour demand identifies an effect of generative AI on these outcomes Practical Limits and Future Work GAISI has practical limits, as the SES task battery covers about one-third of working time, and treating all unobserved work as unexposed lowers discrimination from 0.72 to 0.

Paper summary: Ruth: So we're looking at this paper called Nine Raters, One Index: Carrying LLM Disagreement into Labour-Market Estimates. Basically, they’re asking how much it matters which large language model you use to rate jobs when you're trying to measure a UK job's exposure to generative AI.

Warren: And the core idea is that the choice of model actually becomes an analytic degree of freedom in your measurement. They put one thousand one hundred task occupation cells through nine different large language models from three different vendors and weighted those ratings by how important the tasks were according to worker surveys on British Skills and Employment Surveys <ref:2507.22748#pg1,1,100 task occupation cells>.

Iris: So what they claim is that this method exposes the disagreement between these models directly into the final estimate. They treat each rater as its own dimension in a multiverse, reporting both the specific estimate for each model and a summary that incorporates how much those estimates vary across all nine raters.

Ruth: That’s what it suggests about how we should view this data. It means we can't just pick one model and assume we have the definitive answer about job exposure. We need to see how the rater choice impacts the results.

Warren: And they found that rankings are actually pretty stable across the different raters, with pairwise rank correlations between zero point seven four and zero point nine two, which is quite high for this kind of comparison <ref:2507.22748#pg1>. But levels—just checking if a job scores above a certain threshold—are much more volatile, with the share of British jobs scoring above zero point five ranging from under one point zero percent up to thirty-eight percent across the raters.

Iris: That difference between rankings and levels is important for understanding stability versus uncertainty. What this implies for us is that while we might disagree on exactly where a job falls in a list, we have a much clearer picture of its relative position.

Ruth: And then they looked at how this relates to the actual pay estimates, because that’s where the real labour market questions come in. They found that the between-rater share of the uncertainty from those nine models accounts for about one twentieth of what we think is going into our pay estimate.

Warren: That uncertainty is negative across all nine raters when looking at the pay estimate, which suggests a tendency for less exposed jobs to have lower pay, but it’s important to remember that this uncertainty is much smaller when you control for teleworkability.

Iris: The paper also found that the association between job exposure and actual AI use in Britain falls from zero point two two six to zero point one five four after adding in nine other existing exposure measures. That shows how much this new measure contributes to our existing picture of AI adoption.

Paper summary: Ruth: And on the task level, they found that job exposure rank is linked to observed assistant traffic under both generative and embedding based mappings of conversation clusters. That gives us a look at what kind of tasks are actually being rated differently by these models.

Warren: It’s interesting because the paper also shows that when applying this ranking system to the OECD Survey of Adult Skills workers, the UK ranking transfers better than just looking at levels.

Iris: So what this means for practical application is that if you're trying to use AI as a measurement instrument, treating it as a single entity doesn't capture the complexity of how different models weigh things differently.

Ruth: Absolutely. And when we look at labour market shifts post-two thousand twenty-two the paper notes that mean worker exposure rank rose by four point five percentile points between two thousand seventeen and two thousand twenty three, but that’s almost entirely because employment shifted between occupations rather than because the tasks within those occupations changed.

Warren: And even though relative pay and job adverts moved against more exposed occupations after two thousand twenty-two that association disappears once you allow for teleworkability to have its own separate path.

Iris: The paper also points out a limitation here, saying that the task battery used in the British Skills and Employment Surveys only covers about one third of working time. That means treating all the work not covered by that battery as unexposed lowers their discrimination measure from zero point seven two to zero point five.

Ruth: So while this index gives us a way to carry rater disagreement into our estimates, we have to be careful about what it covers and where its assumptions break down.

Warren: Exactly. The paper concludes that neither pay nor labour demand identifies an effect of generative AI on these outcomes, which is a significant finding because it suggests the shift we're seeing might be driven by other factors entirely.

Iris: So, in the end, this work shows us that when you measure technology exposure using large language models, you have to account for the fact that different models will give you different results.

Ruth: And the title of this paper is Nine Raters, One Index: Carrying LLM Disagreement into Labour-Market Estimates. It really highlights that the choice of model is not just a technical detail; it's a fundamental part of how we calculate exposure.

Warren: It’s about showing how to handle that uncertainty, whether you're looking at job rankings or actual pay estimates across different models.

Iris: It underscores the need for these kinds of studies when we are trying to use AI as an instrument to understand labour markets because the results depend on which model you choose.

Conclusion: Ruth: So we're wrapping up on this paper called "Nine Raters, One Index: Carrying LLM Disagreement into Labour-Market Estimates." Essentially, they’ve built a way to measure how much uncertainty you have when you use different large language models to figure out job exposure to AI.

Warren: It’s about taking that mess of different model opinions and turning it into one single index that captures the disagreement. And they did this by rating eleven hundred job tasks using nine separate large language models from three different vendors.

Iris: The core idea is treating each rater as a dimension in a multiverse, so you see how much the estimates vary across all those raters, not just one number. That lets you see if the rankings are stable or if the uncertainty explodes depending on which model you pick.

Ruth: And what they found is that job rankings tend to be quite stable between different raters. The correlation for ranks is high, between point seven four and zero point nine two. But when you look at actual job scores, those levels are way more spread out, with the share of jobs scoring above a certain threshold ranging from less than one percent up to thirty-eight percent across the models.

Warren: That tells us that while we might disagree on where a specific job lands in a list, there’s a lot of overlap in the general picture. They also found that this uncertainty from the nine raters accounts for about twenty percent of the uncertainty in pay estimates.

Iris: And they show how this exposure rank relates to actual AI use and other existing measures; it drops after you include nine other exposure metrics, falling from zero point two two six down to zero point one five four. That suggests this new measure adds some weight but isn't the only thing driving the picture.

Ruth: The practical implication is that when we look at labour market shifts, like changes in mean worker exposure rank between two thousand seventeen and two thousand twenty three, they found that employment actually shifted between occupations a lot more than tasks changed inside them.

Warren: And even with relative pay and job adverts moving against more exposed jobs after two thousand twenty-two that connection disappears once you allow for teleworkability to factor in separately. That suggests the drivers are different from what we first expected.

Iris: But they did flag a limitation, saying the task battery they used only covers about one third of working time. And if you don't count that other half of the work as unexposed, that uncertainty in discrimination goes up quite a bit, from zero point seven two to zero point five.

Ruth: So the title really points to this idea: you have to be careful how you measure exposure because your choice of tool changes what you see. And it makes us wonder what other kinds of measurement tools we should be testing next.

Golo Henseke, Rhys Davies, Alan Felstead, Duncan Gallie, Francis Green, Ying Zhou

University College London · Cardiff University · Nuffield College, University of Oxford, Surrey Business School

econ.GN, q-fin.EC

Submitted: 2025-07-30

Updated: 2026-08-06

Comments: 47 pages, 4 figures

DOI: 10.1016/j.techfore.2026.124933

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: When a large language model supplies research annotations, the choice of model becomes an analytic degree of freedom The paper develops GAISI…

Key concepts

GAISI
A task-based measure designed to gauge how exposed UK jobs are to generative AI. It uses nine large language models to rate 1,100 specific job tasks, weighted by worker importance.
Multiverse Approach
A statistical method used in the analysis where each LLM rater is treated as a separate dimension in a 'multiverse.' This allows researchers to see how much disagreement exists between raters and incorporates that disagreement into the final estimates.
Rater Disagreement
The variation or difference in ratings produced by different large language models. The study found that while individual ratings vary, the pairwise rankings of jobs are highly stable across these different raters.

Terminology

Summary

When a large language model supplies research annotations, the choice of model becomes an analytic degree of freedom The paper develops GAISI, a task-based measure of UK jobs’ exposure to generative AI, using nine large language models to rate 1,100 task–occupation cells This distinction matters for the use of LLMs as measurement instruments

The Core Methodology

The study constructs GAISI by rating 1,100 task–occupation cells with nine LLM raters from three vendors and several model generations The ratings are weighted by worker-reported task importance in the British Skills and Employment Surveys to yield a worker-level index The analysis employs a multiverse approach where the rater is treated as one dimension of a multiverse, reporting each rater-specific estimate alongside a Rubin-style summary that incorporates their observed dispersion This procedure exposes and carries observed LLM-rater disagreement into empirical estimates

Properties of the Index

The study finds that rankings are substantially more stable across raters than levels, with pairwise rank correlations ranging from 0.74 to 0.92 Conversely, the share of British jobs scoring above 0. p. 05 ranges from under 0. p. 01% to 38% across raters The between-rater share of the Rubin-combined variance accounts for about a twentieth of the uncertainty in the pay estimate, which is negative under all nine raters The index behaves as an instrument, showing that discrimination is measured by how reproducibly it ranks jobs when the rating process is varied

External Evidence and Validation

The external evidence gives reasons to take the resulting ordering seriously, while leaving relevant questions unresolved GAISI’s association with worker-reported AI use falls from 0.226 to 0.154 after nine existing exposure measures enter together, including human-rated generative-AI indices At the task level, exposure rank is associated with observed assistant traffic under both generative and embedding-based mappings of conversation clusters The paper also shows that the UK ranking transfers better than the levels when applied to the OECD Survey of Adult Skills (PIAAC) workers

Labour Market Implications

The labour-market applications warrant the same restraint as other measures, showing that mean worker exposure rank rose by 4.5 percentile points between 2017 and 2023/24, almost entirely because employment shifted between occupations rather than because task profiles changed within them Relative pay and job adverts also moved against more exposed occupations after 2022, but the vacancy association disappears once teleworkability is allowed its own post-2022 path The paper concludes that neither pay nor labour demand identifies an effect of generative AI on these outcomes

Practical Limits and Future Work

GAISI has practical limits, as the SES task battery covers about one-third of working time, and treating all unobserved work as unexposed lowers discrimination from 0.72 to 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement a nine-rater ensemble approach to estimate generative AI exposure, treating each model as an analytic degree of freedom and reporting each rater-specific estimate alongside a Rubin-style summary that incorporates their observed dispersion. This procedure allows for exposing and carrying observed LLM-rater disagreement into empirical estimates.

  2. Develop conditional predictive content by assessing how the index behaves as an instrument by running downstream analyses under every rater's version of the index, reporting the rater-specific estimates and their minimum–maximum range alongside a pooled summary that incorporates their dispersion. This addresses the consequence depends on the downstream design.

  3. Incorporate sensitivity checks to design choices by systematically varying elements such as the assessment rubric, task importance weights, and threshold settings to determine which facets of the rating process actually move the measure. This helps identify what reproduces and what depends on design choices in terms of reliability.

  4. Design systems that integrate competing exposure measures, such as pre-generative-AI constructs and human-derived generative-AI measures, to assess whether the index adds anything beyond existing exposure measures by comparing rank correlations across construct distance blocks. This tests whether the index adds anything beyond existing measures.

  5. Build a task-level measurement system that classifies job tasks into four exposure levels (E0 to E3) based on meaningful time reduction thresholds, using probabilistic distributions across levels rather than mutually exclusive categories. This captures the high-level nature of SES work activities where exposure-level distinctions are blurred.

  6. Create a mechanism to assess circularity by examining how the association between exposure and reported AI use changes when competing explanations are considered, such as conditioning reported AI use on the worker's use of other digital tools. This helps determine if the relationship is unbiased or confounded by narrative commonality.

Sources

Related papers