Signaling in the Age of AI: Evidence from Cover Letters
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Signaling in the Age of AI: Evidence from Cover Letters".
Warren: Here's the gist of the paper. We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform <ref:2509.25054#pg14> The correlation between cover letters’ textual alignment and callbacks fell by 51%, consistent with what theory predicts if the AI technology reduces the signal content of cover letters <ref:2509.25054#pg5> Research Question and Methodology The study investigates how generative AI affects labor market signaling by tracking access to an AI-powered cover letter writing tool on Freelancer.com and analyzing usage at the application level <ref:2509.25054#pg14> The researchers observe access to the tool and usage of the tool at the application level, linking this information to submitted cover letters and hiring outcomes <ref:2509.25054#pg14> They use a difference-in-differences design comparing workers with and without access to measure the effect of the platform-native AI writing tool on cover letter tailoring and callback rates <ref:2509.25054#pg14> Impact on Cover Letter Tailoring The introduction of the AI tool increased cover-letter tailoring by 0.16 standard deviations when access was gained, while actual usage raised tailoring by 1.36 standard deviations <ref:2509.25054#pg3> The researchers find that AI substitutes for workers’ pre-AI cover letter tailoring skills, as workers who previously wrote more tailored cover letters experienced smaller gains in cover letter tailoring after gaining access to the AI tool <ref:2509.25054#pg4> Market-Level Signaling Shifts The correlation between cover letter tailoring and receiving a callback fell by 51% after the launch of the AI tool, consistent with what theory predicts if the AI technology reduces the signal content of cover letters <ref:2509.25054#pg5> Employers shifted toward alternative signals, including workers’ prior work histories, as cover letters became less correlated with hiring after AI is introduced <ref:2509.25054#pg5> The correlation between callbacks and workers’ review scores rose by 5% <ref:2509.25054#pg5> Human-AI Interaction Effects Workers with higher pre-AI writing ability tend to spend more time editing AI drafts, and controlling for worker fixed effects, a one–standard deviation increase in editing time is associated with a 0.31 percentage point rise in the probability of receiving a job offer <ref:2509.25054#pg8> This indicates that human revision enhances the effectiveness of AI-generated cover letters, contributing to a growing body of evidence on human–AI collaboration <ref:2509.25054#pg8> The study finds no evidence of changes in overall market outcomes such as hiring, interviewing, and job completion rates following the tool’s introduction <ref:2509.25054#pg7> This suggests that while AI disrupted one channel of signaling—cover letters—employers successfully shifted toward other signals, particularly past reputation scores <ref:2509.25054#pg7> However, workers with access benefit from AI, with moderately productive workers gaining more than the most productive, while control workers face a decline in hiring probability <ref:2509.25054#pg8> The tool’s availability raises hiring probabilities for treated workers whereas without access experience a decrease <ref:2509.25054#pg8> References Galdin and Silbert (2025) first measured similarity between cover letters and job posts on Freelancer.com, interpreted it as a signal of worker ability, and showed it predicts hiring <ref:2509.25054#pg14> Cowgill et al.
Ruth: (2024) examine the impact of generative AI on job applications and startup pitches through a survey experiment <ref:2509.25054#pg14> Wiles and Horton (2025) study generative AI assistance to employers for crafting job posts in a similar context and find that while the tool increased the number of job posts, it did not increase the number of matches <ref:2509.25054#pg14> Agrawal et al.
Warren: (2019), Noy and Zhang (2023), Brynjolfsson et al.
Ruth: (2025), and Acemoglu (2025) show how AI affects productivity and innovation Agarwal et al.
Paper summary: Ruth: So we’re looking at this paper today, "Signaling in the Age of AI: Evidence from Cover Letters." The core idea is tracking how using an AI-powered cover letter writing tool on a big online labor platform changes how much information those letters actually signal about the worker.
Warren: Right. It seems like they're testing if this new technology actually makes our applications better or worse when we’re trying to get hired. The abstract says the main finding is that when employers use this AI tool, the correlation between how well a cover letter matches a job description and getting a callback drops by fifty-one percent.
Iris: That drop in correlation, that's significant because it suggests the AI is reducing the quality of that signal, which is what theory predicts when generative AI reduces signal content <ref:2509.25054#pg5>. But the paper isn't just looking at correlation; they’re tracking access and usage at the application level on Freelancer.com <ref:2509.25054#pg14>.
Ruth: Exactly. They use a difference-in-differences design to see how this tool affects tailoring and callback rates by comparing workers who have access to the tool against those who don't <ref:2509.25054#pg14>. What they claim is that access to the tool increased textual alignment between cover letters and job posts, while actual usage raised that tailoring up by one point three six standard deviations <ref:2509.25054#pg3>.
Warren: Okay, so access makes the letters look better, but using it actually makes them even more tailored. That’s interesting because it means the AI isn't just making things worse; in some ways, it might be boosting the signal initially <ref:2509.25054#pg3>.
Iris: But then they immediately show that workers who used the tool before—before they had access—experienced smaller gains in tailoring after gaining access to the AI tool, suggesting the AI might substitute for those pre-existing skills <ref:2509.25054#pg4>. That’s a key part of their argument about whether workers are actually improving their skill set or just getting an easier template <ref:2509.25054#pg4>.
Paper summary: Ruth: And that leads into what they call market-level signaling shifts, where they find that the correlation between those cover letter tailoring scores and receiving a callback falls by fifty-one percent after the AI tool is launched <ref:2509.25054#pg5>. This supports the idea that employers are shifting their focus away from cover letters as a primary signal <ref:2509.25054#pg7>.
Warren: So, if cover letters become less informative, what do employers look at instead? The paper suggests they start looking toward other signals, specifically workers’ prior work histories <ref:2509.25054#pg7>. That makes sense; if the letter isn't a good signal anymore, they go back to traditional things like past performance reviews <ref:two thousand five hundred nine point two five zero five four#pg7.
Iris: They also noticed something else happening at the market level, which is that the correlation between callbacks and workers’ review scores went up by five percent after the AI tool came out <ref:2509.25054#pg5>. That's a bit of a counter-intuitive finding when you think about how much cover letters used to matter for those initial signals.
Ruth: It is interesting because it shows that while the letter signal got weaker, some other established signals, like review scores, became relatively stronger <ref:two thousand five hundred nine point two five zero five four#pg5. But we have to be careful about what this means for hiring outcomes overall.
Warren: The paper says they found no evidence of changes in big market outcomes like overall hiring or job completion rates following the tool’s introduction <ref:2509.25054#pg7>. So, despite the signal change, it doesn't seem to have messed with the total volume of jobs being filled <ref:two thousand five hundred nine point two five zero five four#pg7.
Iris: But there are still differences for workers based on their access. The paper points out that workers who had access to the AI benefit, and moderately productive workers gain more than the most productive ones, while control workers face a decline in hiring probability <ref:two thousand five hundred nine point two five zero five four#pg8. So it’s not a universal win for everyone <ref:two thousand five hundred nine point two five zero five four#pg8.
Ruth: That's where the human-AI interaction comes in, which they look at next. Workers who already had higher pre-AI writing ability tend to spend more time editing the AI drafts <ref:two thousand five hundred nine point two five zero five four#pg8. And this human revision actually helps; a one standard deviation increase in editing time is associated with a zero point three one percentage point rise in the probability of getting a job offer <ref:two thousand five hundred nine point two five zero five four#pg8.
Paper summary: Warren: So, it sounds like AI isn't just replacing the worker entirely; it’s changing how the work gets done, and human skill is still valuable for refining those AI outputs <ref:two thousand five hundred nine point two five zero five four#pg8. That’s a different story than if the tool just made everything easier without any human input <ref:two thousand five hundred nine point two five zero five four#pg8.
Iris: They are looking at this collaboration between human and AI as something that enhances effectiveness, which is a growing area of evidence in this field <ref:two thousand five hundred nine point two five zero five four#pg8. But the paper also flags a limitation, and they state that their study only looks at the introduction of this specific tool on Freelancer.com <ref:two thousand five hundred nine point two five zero five four#pg1.
Ruth: That's true; it’s specific to that platform and that one feature, so we don't know if this effect holds for other types of AI tools or other labor markets <ref:two thousand five hundred nine point two five zero five four#pg1. But the broader implication is that employers are learning to look at different data points now, like those review scores they mentioned earlier <ref:two thousand five hundred nine point two five zero five four#pg7.
Warren: So, for someone listening who isn't in a labor market, what does this paper actually mean for their day-to-day life? It means that if you use an AI tool to write a cover letter, the way you tailor it might not be as effective at getting you noticed by employers anymore <ref:two thousand five hundred nine point two five zero five four#pg1.
Iris: And because of that, the paper suggests that if workers want to keep their hiring chances up, they should probably focus more on building those prior work histories and review scores rather than just relying on a perfectly tailored cover letter <ref:two thousand five hundred nine point two five zero five four#pg7.
Ruth: That’s the central tension here: the AI makes crafting a perfect cover letter easier to do, but it also weakens that specific signal because it becomes too generic, forcing employers to rely on other things <ref:two thousand five hundred nine point two five zero five four#pg1. It’s about adapting your strategy when a tool fundamentally changes how you send your message to the market <ref:two thousand five hundred nine point two five zero five four#pg1.
Warren: It’s a real shift in what counts as a good signal in the job search landscape, driven by the introduction of this specific AI writing tool on Freelancer.com <ref:two thousand five hundred nine point two five zero five four#pg1. We'll keep an eye on how these other platforms and tools evolve over time <ref:two thousand five hundred nine point two five zero five four#pg1.
Conclusion: Ruth: So we're wrapping up our look at this study on cover letters and AI signaling, titled "Signaling in the Age of AI: Evidence from Cover Letters."
Warren: Yeah, basically, they tracked how using an AI cover letter tool on a big job site affects how much the letter actually signals about a worker.
Iris: The main thing is that when employers use this tool, the link between how well a cover letter matches the job description and getting a callback drops by fifty-one percent.
Ruth: That’s because the AI seems to reduce the actual signal content of those letters, which is what theory predicted back in twenty twenty-five.
Warren: And it shows that employers are shifting their focus away from cover letters toward other signals like past work histories when AI is introduced.
Iris: But they also found that workers with access to the tool benefit, and control workers face a drop in hiring probability, so it's not a universal win.
Ruth: And there’s this finding about human interaction: if you already wrote good letters before, spending more time editing the AI draft actually helps you get an offer by zero point three one percentage points.
Warren: So the paper suggests that while the AI changes how we signal, employers are learning to look at other things like review scores instead of just how well a letter is tailored.
Iris: It also notes that overall hiring rates don't change much, which tells us this isn't a huge market disruption yet.
Ruth: But it does show that the way we communicate our skills through these specific documents is being altered by the technology in ways we need to watch closely.
Yale Department of Economics
econ.GN, q-fin.EC
Submitted: 2025-09-29
Updated: 2026-11-10
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform <ref:2509.25054#pg14> The correlation…
Key concepts
- Labor Market Signaling
- This refers to any information a job seeker provides (like a resume or cover letter) that helps an employer assess their qualifications and abilities. The paper investigates how AI tools change the quality and effectiveness of these signals in the hiring process.
- Cover Letter Tailoring
- This is the degree to which a cover letter is specifically customized for a particular job application. The research found that AI usage reduced this tailoring, suggesting that workers rely less on their own pre-AI skills when using automated tools.
- Signal Content Reduction
- This concept describes how the introduction of an AI tool might make the information in a cover letter less informative for employers. The study found this reduction occurred because the AI substituted for human tailoring skills, weakening the traditional signal.
Terminology
Summary
We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform <ref:2509. p. 0 Signaling in the Age of AI: Evidence from Cover Letters ∗
Jingyi Cui, Gabriel Dias, and Justin Ye †
November 13, 2025
Abstract
We study the impact of generative AI on labor market signaling using the introduction
of an AI-powered cover letter writing tool on a large online labor platform. Our data track
both access to the tool and usage at the application level. Time spent editing AI-generated cover letter drafts is positively
correlated with hiring success. After the tool’s introduction, the correlation between cover
letters’ textual alignment and callbacks fell by 51%, consistent with what theory predicts if
the AI technology reduces the signal content of cover letters.Where the paper says it
Where the paper says it
Research Question and Methodology
The study investigates how generative AI affects labor market signaling by tracking access to an AI-powered cover letter writing tool on Freelancer.com and analyzing usage at the application level <ref:2509. p. 0 Signaling in the Age of AI: Evidence from Cover Letters ∗
Jingyi Cui, Gabriel Dias, and Justin Ye †
November 13, 2025
Abstract
We study the impact of generative AI on labor market signaling using the introduction
of an AI-powered cover letter writing tool on a large online labor platform. Our data track
both access to the tool and usage at the application level. 3 We observe access to the tool and
usage of the tool at the application level. We link access and usage information to the submitted
cover letters and to callback and hiring outcomes to analyze the effect of generative AI at both
the individual level and the market level. We first examine the effect of gaining access to the generative AI tool on cover letter tailoring
and on workers’ chances of receiving a callback. We use a difference-in-differences design comparing
workers with and without access.Where the paper says it
Where the paper says it
Where the paper says it
Impact on Cover Letter Tailoring
The introduction of the AI tool increased cover-letter tailoring by 0.16 standard deviations when access was gained, while actual usage raised tailoring by 1. p. 2 We find that access to the generative AI writing tool increased cover-letter tailoring by 0. 16
standard deviations, while actual usage raised tailoring by 1. We find that access to the generative AI writing tool increased cover-letter tailoring by 0. 16
standard deviations, while actual usage raised tailoring by 1. We compute workers’ average tailoring scores prior to the tool’s introduction
and study the heterogeneous effects of access to the tool by pre-existing writing ability. We find
that workers who previously wrote more tailored cover letters experienced smaller gains in cover
letter tailoring—indeed, the best writers gained half as much as the weakest ones.Where the paper says it
Where the paper says it
Where the paper says it
Market-Level Signaling Shifts
The correlation between cover letter tailoring and receiving a callback fell by 51% after the launch of the AI tool, consistent with what theory predicts if the AI technology reduces the signal content of cover letters <ref:2509.25054#pg5> Employers shifted toward alternative signals, including workers’ prior work histories, as cover letters became less correlated with hiring after AI is introduced <ref:2509. p. 0 After the tool’s introduction, the correlation between cover
letters’ textual alignment and callbacks fell by 51%, consistent with what theory predicts if
the AI technology reduces the signal content of cover letters. In response, employers shifted
toward alternative signals, including workers’ prior work histories. Instead, employers shifted toward other signals less
susceptible to AI influence, such as workers’ past work experience. The correlation between callbacks
and workers’ review scores—the platform’s proprietary metric summarizing past work experiences
on the platform and determining the default ranking of applications—rose by 5%.Where the paper says it
Where the paper says it
Human-AI Interaction Effects
Workers with higher pre-AI writing ability tend to spend more time editing AI drafts, and controlling for worker fixed effects, a one–standard deviation increase in editing time is associated with a 0.31 percentage point rise in the probability of receiving a job offer <ref:2509. p. 3 Examining the relationship between editing and outcomes, we show
that—controlling for worker fixed effects—a one–standard deviation increase in editing time is
associated with a 0. 31 percentage point rise in the probability of receiving a job offer, equal to
about 52% of the average offer rate among AI-assisted applications. human revision enhances the effectiveness of AI-generated cover letters, contributing to a growing
body of evidence on human–AI collaboration. Related Literature Galdin and Silbert (2025) offer complementary evidence on the effects of
generative AI on signaling in the same labor market.Where the paper says it
Where the paper says it
Conclusion on Market Outcomes
The study finds no evidence of changes in overall market outcomes such as hiring, interviewing, and job completion rates following the tool’s introduction <ref:2509.25054#pg7> This suggests that while AI disrupted one channel of signaling—cover letters—employers successfully shifted toward other signals, particularly past reputation scores <ref:2509. p. 3 Instead, employers shifted toward other signals less
susceptible to AI influence, such as workers’ past work experience. The correlation between callbacks
and workers’ review scores—the platform’s proprietary metric summarizing past work experiences
on the platform and determining the default ranking of applications—rose by 5%. We then extend the model to
employers making multinomial choices among applicants. Numerical simulations show that workers
with access to AI experience an increase in hiring probability while workers without access experience a decrease. We then extend the model to
employers making multinomial choices among applicants. Numerical simulations show that workers
with access to AI experience an increase in hiring probability while workers without access experience a decrease.Where the paper says it
Where the paper says it
Where the paper says it
References
Galdin and Silbert (2025) first measured similarity between cover letters and job posts on Freelancer.com, interpreted it as a signal of worker ability, and showed it predicts hiring <ref:2509. p. 0 Galdin and Silbert (2025) first measured similarity between cover
letters and job posts on Freelancer. com, interpreted it as a signal of worker ability, and showed it predicts hiring. -Generated R´esum´es,” The New York Times, June 21, 2025). 2Notable exceptions include Cowgill et al. Another closely related paper is Cowgill et al. (2024), who examine the impact of generative
AI on job applications and startup pitches through a survey experiment. While we also investigate
the role of AI—although of a different nature—in workers’ employment outcomes, we are additionally able to speak to the market-level outcomes associated with the introduction of generative AI. Another related paper is Wiles and Horton (2025), who study generative AI assistance to employers
for crafting job posts in a similar context and find that while the tool increased the number of job
posts, it did not increase the number of matches. the informativeness of signals on these platforms. More broadly, our contribution also connects to
research on AI’s impact on labor markets: Agrawal et al. More broadly, our contribution also connects to
research on AI’s impact on labor markets: Agrawal et al. (2019), Noy and Zhang (2023), Brynjolfsson et al. (2019), Noy and Zhang (2023), Brynjolfsson et al. (2025), and Acemoglu (2025) show how AI affects productivity and innovation, while we
highlight an alternative channel through labor market signaling. In economics more broadly, Agarwal et al. (2023) study
collaboration between AI and radiologists in medical diagnosis, while in computer science, Hosanagar and Ahn (2024) and Chakrabarty et al. (2024) and Allman et al. (2025), who analyze frictions and signaling mechanisms in
matching markets. (2025), and Acemoglu (2025) show how AI affects productivity and innovation, while we
highlight an alternative channel through labor market signaling. Also related is Brinatti et al. Also related is work that examines the functioning of online labor markets and their broader
economic implications. Horton (2010), Chen and Horton (2016), and Stanton and Thomas (2020)
document how these markets operate, the behavioral responses of workers to wage changes, and the
significance of platform-mediated employment. (iv) Does AI adoption affect overall hiring
1This trend has received widespread media attention. Recent headlines include “AI is supposed to make
applying to jobs easier—but it might be creating another problem” (NBC News, November 17, 2024) and “AI
Is Taking Over College Admissions” (The Nation, October 11, 2024). The AI Bid Writer tool was not the only generative AI tool that workers had access to in
the sample period. One alternative is GPT-4, a general-purpose AI tool that OpenAI released
on March 14, 2023, just over a month before the launch of the AI Bid Writer. Our work is complementary to theirs as both
sides of the labor market—firms and workers—increasingly adopt generative AI tools in matching. Also related is work that examines the functioning of online labor markets and their broader
economic implications. Also related is work that examines the functioning of online labor markets and their broader
economic implications. Horton (2010), Chen and Horton (2016), and Stanton and Thomas (2020)
document how these markets operate, the behavioral responses of workers to wage changes, and the
significance of platform-mediated employment. edu, and
justin. arXiv:2509. Horton (2010), Chen and Horton (2016), and Stanton and Thomas (2020)
document how these markets operate, the behavioral responses of workers to wage changes, and the
significance of platform-mediated employment. Stanton and Thomas (2025) provide evidence on
the distributional consequences of gig platforms, showing how access to online markets affects both
workers and firms, and highlighting heterogeneity in who ultimately benefits from platform participation. In a similar labor market, Wiles et al. (2023) study the effect of a non-generative algorithmic
writing assistance on employment through an experiment. (2025) show that generative AI reshapes productivity, information processing, and work patterns among knowledge workers. Studies closer to our context, such as
Cao and Zhu (2025a), Cao and Zhu (2025b), and Chen et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Increase cover letter tailoring by 0.16 standard deviations when access to generative AI is gained, and raise callback rates by 3.56 percentage points when usage is observed in response to the tool's introduction.
-
Implement a system that accounts for worker heterogeneity in AI adoption, recognizing that
workers with higher pre-AI writing ability experienced smaller gains in cover letter tailoring—indeed, the best writers gained half as much as the weakest ones.
-
Develop an AI assistant capable of identifying and utilizing alternative signals when cover letters lose informativeness, specifically by prioritizing
workers’ past work experience
over tailored text similarity. -
Integrate a mechanism for human-AI collaboration where the system tracks editing time, recognizing that
a one–standard deviation increase in editing time is associated with a 0.31 percentage point rise in the probability of receiving a job offer.
Abstract
We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform. Our data track both access to the tool and usage at the application level. Difference-in-differences estimates show that access to the tool increased textual alignment between cover letters and job posts and raised callback rates. Time spent editing AI-generated cover letter drafts is positively correlated with hiring success. After the tool's introduction, the correlation between cover letters' textual alignment and callbacks fell by 51%, consistent with what theory predicts if the AI technology reduces the signal content of cover letters. In response, employers shifted toward alternative signals, including workers' prior work histories.
Sources
- From signaling to interviews in random matching markets
- Measuring Massive Multitask Language Understanding
- Designing Human and Generative AI Collaboration
- Evaluating Large Language Models in Theory of Mind Tasks
- conSultantBERT: Fine-tuned Siamese Sentence-BERT for Matching Jobs and Job Seekers
- Embedding-based Recommender System for Job to Candidate Matching on Scale
Related papers
- Huge If True: Learning from the Size of the Surprise
- The Rebundling of Freelance Work in the AI Age
- Dynamic investment and coordination in teams: Theory and experimental evidence
- Does Frontier AI Strengthen Cyber Defense? Evidence from Project Glasswing
- Generative AI at Work
- What is Work For? Labor, Politics, and Meaning in the Age of AI