Execution and Evaluation: Task Content, Labor Demand, and the Evaluator Pipeline
summary
The gist
Artificial intelligence automates execution more readily than evaluation: producing output is cheap, judging whether it is correct is not <ref:2607.20807#pg2> The execution share is reproducible…
In short
The study created a new measure called 'execution share' to distinguish between tasks where AI produces output and tasks where humans judge that output. Findings show execution-heavy white-collar jobs have lower employment growth since 2012, suggesting this trend is long-run rather than just due to AI. The capability gradient steepens post-2022, but the core execution trend predates generative AI.
Key concepts
- Execution Share
- This is a new measure that scores tasks based on whether AI can perform the actual work (producing output) versus whether a human must judge or evaluate that output. It is a model-based score, not something humans have directly validated, allowing researchers to separate production from judgment tasks.
- AI Capability Gradient
- This tracks how much AI's ability to perform tasks improves over time. The paper finds this gradient steepens after 2022, but it argues this change is a dated trend rather than a sudden event caused by generative AI itself.
- Execution vs. Evaluation
- The paper separates work into two distinct parts: execution (the act of producing the initial output) and evaluation (the act of judging if that output is correct or useful). This distinction helps model how human labor differs economically between creating something and verifying it.
Terminology used across episodes
This episode discusses
- Execution and Evaluation: Task Content, Labor Demand, and the Evaluator Pipeline · Paper Radio
- Automation, AI, and the Intergenerational Transmission of Knowledge
- Some Simple Economics of AGI
The paper
Execution and Evaluation: Task Content, Labor Demand, and the Evaluator Pipeline · Read on arXiv
Li Gan
Department of Economics, Texas A&M University
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Execution and Evaluation".
Warren: Here's the gist of the paper. Artificial intelligence automates execution more readily than evaluation: producing output is cheap, judging whether it is correct is not <ref:2607.20807#pg2> The execution share is reproducible across model coders and OOTNET vintages and distinct from AI capability and routine-task intensity; it is a model-based measure, not human-validated <ref:2607.20807#pg4> In a harmonized panel, employment growth is lower in execution-heavy white-collar occupations in every window since 2012, and equality of slopes cannot be rejected: the gradient is a secular trend rather than an AI-era event, largely between occupational families <ref:2607.20807#pg5> The vintage-valid capability gradient steepens after 2022, a change that is dated but not causally attributable <ref:2607.20807#pg5> The Core Measurement and Distinction The paper introduces an occupation-level execution share derived by scoring OOTNET task statements under a fixed rubric to measure whether AI can perform the work, distinct from AI capability <ref:2607.20807#pg4> This distinction separates tasks into execution—producing output—and evaluation—judging the output's correctness or fit for use <ref:2607.20807#pg2> The paper constructs an occupation-level execution share and separately scores the same statements for AI capability to permit direct comparisons <ref:2607.20807#pg2> This new measure is strongly negatively related to existing OOTNET evaluation composite, which is precisely what construct validity requires <ref:2607.20807#pg4> Empirical Findings on Labor Demand The main empirical contribution is a set of employment facts with an explicit chronology <ref:2607.20807#pg2> In harmonized OEWS data, employment growth was lower in execution-intensive white-collar occupations not only after 2022 but also in 2012–2016 and 2016–2019 Equality of the execution slopes through 2025 cannot be rejected on an estimator-consistent panel The vintage-valid capability measure is also significantly negative before 2022 The execution gradient predates generative AI, shows no detected break, and is largely between occupational families Model Structure and Causal Limitations The model posits that human labor per unit of occupational output is determined by conditional human labor quantity H(At) = ej s∗j(At) + (1 − ej)r Proposition 1 states that under Assumption 1, both the level decline and the proportional decline of conditional human employment as At rises are increasing in ej, holding aj fixed The interaction term is difficult to estimate in a single cross-section because ej and aj are moderately correlated, and their product is highly collinear with the levels The paper states that the long-run pipeline interpretation remains a hypothesis rather than a result Robustness and Future Directions The execution share as a task-level measure is reproducible across model coders; distinct from capability and routine intensity; not human validated The empirical signature is therefore a sequence rather than a single coefficient Stronger evidence would require predetermined or plausibly exogenous adoption, hiring and vacancy flows rather than smoothed occupation stocks The task-level design makes each of these extensions feasible because the execution measure can be attached to occupations, firms, postings, or worker histories without changing its conceptual definition The paper concludes that the execution gradient is a long-run occupational trend: it predates generative AI, shows no detected break, and is largely between broad occupational families The vintage-valid capability gradient is the one place where the post-2022 period is statistically distinguishable, and even there the design dates the change without attributing it The labor-market facts are correspondingly disciplined, and Table 7 pairs each with the interpretation it supports The execution share adds modest, control-dependent explanatory power The value is not that it dominates OOTNET’s existing descriptors.
Title and authors: Ruth: The paper we’re looking at today is called "Execution and Evaluation: Task Content, Labor Demand, and the Evaluator Pipeline." It really drills down into how these specific measurement tools affect labor demand trends across different windows since two thousand twelve > <ref:2607.20807#pg1> <ref:2607.20807#pg1>.
Warren: It seems like the main focus is on employment growth in execution-heavy white-collar jobs and whether there’s a consistent pattern, or if it's just noise from one big event like generative AI > <ref:2607.20807#pg5>.
Iris: The authors argue that they found employment growth was lower in those execution-intensive white-collar occupations not only after two thousand twenty-one but also between two thousand twelve and two thousand sixteen and again from two thousand sixteen to two thousand nineteen > <ref:2607.20807#pg5> <ref:2607.20807#pg1>.
Ruth: That suggests the pattern isn't just tied to the arrival of generative AI, but it’s a longer trend that predates it, which they call a secular trend rather than an AI-era event > <ref:2607.20807#pg5>.
Warren: And they emphasize that the equality of these slopes across different occupational families can’t be rejected on their panel data, meaning this isn't just about one group getting hit harder than another > <ref:2607.20807#pg5>.
Iris: They also point out that the vintage-valid capability measure shows a significant negative trend before two thousand twenty-one but they stress that the steepening of that capability gradient after two thousand twenty-one is dated and not something they can causally attribute to anything specific > <ref:2607.20807#pg5> <ref:2607.20807#pg5>.
Ruth: So what this means for us is that we need to be careful about timing our predictions based solely on the post-two thousand twenty-one AI boom, because the underlying labor demand trends are more complex and predate that event > <ref:2607.20807#pg1> <ref:2607.20807#pg1>.
Warren: It’s a warning against treating every change as an immediate cause, especially when you have these long-run occupational gradients that are already in motion > <ref:2607.20807#pg5>.
The paper's summary: Ruth: To summarize the main point of "Execution and Evaluation: Task Content, Labor Demand, and the Evaluator Pipeline," it’s about establishing this new execution share as a way to measure if AI can produce output versus if a human supplies that execution or evaluation > <ref:2607.20807#pg4>.
Warren: The paper sets up two separate scores for these things—the execution share and the AI capability—so they can directly compare them, and they note this new measure is strongly related to existing OOTNET evaluation composites in a negative way > <ref:2607.20807#pg2>.
Iris: The methodology involves scoring all nineteen thousand two hundred sixty-five O*NET task statements under fixed rubrics to build these occupation-level shares, and they confirm the execution share is reproducible across different model coders and older O*NET vintages > <ref:2607.20807#pg1> <ref:2607.20807#pg1,all 19,265 O*NET task statements under fixed rubrics to build>.
Ruth: They use this new measure to find employment facts that show growth is lower in execution-heavy white-collar jobs across several windows since two thousand twelve reinforcing that long-run trend idea > <ref:2607.20807#pg2> <ref:2607.20807#pg1>.
Warren: It also points out a specific finding about younger workers, where employment of younger workers has weakened relative to more experienced workers in highly exposed occupations, which is something they link to the pattern > <ref:2607.20807#pg2>.
Iris: They are careful not to claim that their measure leaves a large predictive gap compared to existing OOTNET activities; instead, they focus on providing a reproducible axis aligned statement by statement with AI capability > <ref:2607.20807#pg3> <ref:2607.20807#pg3>.
Ruth: So in simple terms, the paper is saying we have a better way to isolate the act of doing versus the act of checking that work, and this helps us see employment patterns that are longer-term than just the immediate hype cycle > <ref:2607.20807#pg1> <ref:2607.20807#pg1>.
Warren: It frames AI exposure not as a sudden shock but as part of a pre-existing trend in how labor is demanded by occupation type > <ref:2607.20807#pg5>.
The paper's improvements: Ruth: The paper suggests that the primary improvement is this task-level execution share because it allows for an occupation classification based on whether AI can perform the task versus whether a human supplies execution or evaluation > Ruth This lets us distinguish between tasks requiring production and those requiring judgment.
Warren: That distinction is powerful because it helps model how human input differs economically; execution produces the first-pass output, while evaluation supplies verification, certification, responsibility, and judgment > Warren It’s about separating the creation of something from the checking of that thing.
Iris: They also suggest that learning by doing mechanisms can be modeled to predict future workforce quality specifically noting that entry-level displacement exceeds senior displacement at a given AI-capable share > Iris And they point out that downstream evaluation degradation follows with a cohort lag > <ref:2607.20807#pg2> <ref:2607.20807#pg2>.
Ruth: They are also refining the model structure by positing how human labor per unit of occupational output is determined by conditional human labor quantity H of At using an equation involving ej and r > Ruth This lets them test different assumptions about how people respond when their job output changes.
Warren: The paper also suggests that the execution share as a task-level measure is reproducible across different model coders, which is a big deal for getting consistent results from these types of models > <ref:2607.20807#pg3> <ref:2607.20807#pg3>.
Iris: They are also looking at how alternative task weights produce correlations of zero point nine eight to zero point nine nine with the baseline measure, which suggests the underlying structure of what constitutes execution is quite robust across different ways you define it > <ref:2607.20807#pg3> <ref:2607.20807#pg3>.
Conclusion: Ruth: So to wrap up, "Execution and Evaluation: Task Content, Labor Demand, and the Evaluator Pipeline" concludes that this execution gradient is a long-run occupational trend that predates generative AI > <ref:2607.20807#pg1>.
Warren: It shows no detected break in that trend, and it’s largely between broad occupational families rather than being caused by AI itself > <ref:2607.20807#pg5>.
Iris: The vintage-valid capability gradient is the only place where the post-two thousand twenty-one period is statistically distinguishable, but even there, they say it dates the change without attributing it to anything specific > <ref:2607.20807#pg5> <ref:2607.20807#pg5>.
Ruth: They also state that the labor market facts are disciplined by this approach, and Table seven pairs each fact with what interpretation it supports > <ref:2607.20807#pg1>.
Warren: The execution share adds modest explanatory power overall, but its value isn't in dominating the existing OOTNET descriptors > <ref:2607.20807#pg3> <ref:2607.20807#pg3>.
Iris: I just want to add that their main limitation is that the long-run pipeline interpretation remains a hypothesis rather than a proven result > <ref:2607.20807#pg5> <ref:2607.20807#pg5>.
Ruth: That sounds like where we leave off for this paper. It’s solid evidence on chronology and measurement, but they are being careful not to overstate the causal link to AI adoption itself > <ref:2607.20807#pg1> <ref:2607.20807#pg1>.
Warren: Agreed, it’s a measure that provides structure to the existing data without claiming it solves all the mysteries of labor demand > <ref:2607.20807#pg3> <ref:2607.20807#pg3>.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19