Signaling in the Age of AI: Evidence from Cover Letters
summary
The gist
We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform <ref:2509.25054#pg14> The correlation…
In short
The study examined how an AI-powered cover letter writing tool on Freelancer.com affects labor market signaling. Introducing the tool reduced the correlation between tailored cover letters and callbacks by 51%, suggesting AI diminishes the signal value of these documents. Employers shifted focus to other signals like past work history.
Key concepts
- Labor Market Signaling
- This refers to any information a job seeker provides (like a resume or cover letter) that helps an employer assess their qualifications and abilities. The paper investigates how AI tools change the quality and effectiveness of these signals in the hiring process.
- Cover Letter Tailoring
- This is the degree to which a cover letter is specifically customized for a particular job application. The research found that AI usage reduced this tailoring, suggesting that workers rely less on their own pre-AI skills when using automated tools.
- Signal Content Reduction
- This concept describes how the introduction of an AI tool might make the information in a cover letter less informative for employers. The study found this reduction occurred because the AI substituted for human tailoring skills, weakening the traditional signal.
Terminology used across episodes
This episode discusses
- Signaling in the Age of AI: Evidence from Cover Letters · Paper Radio
- From signaling to interviews in random matching markets · Paper Radio
- Measuring Massive Multitask Language Understanding
- Designing Human and Generative AI Collaboration
- Evaluating Large Language Models in Theory of Mind Tasks
- conSultantBERT: Fine-tuned Siamese Sentence-BERT for Matching Jobs and Job Seekers
- Embedding-based Recommender System for Job to Candidate Matching on Scale
The paper
Signaling in the Age of AI: Evidence from Cover Letters · Read on arXiv
Yale Department of Economics
We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform. Our data track both access to the tool and usage at the application level. Difference-in-differences estimates show that access to the tool increased textual alignment between cover letters and job posts and raised callback rates. Time spent editing AI-generated cover letter drafts is positively correlated with hiring success. After the tool's introduction, the correlation between cover letters' textual alignment and callbacks fell by 51%, consistent with what theory predicts if the AI technology reduces the signal content of cover letters. In response, employers shifted toward alternative signals, including workers' prior work histories.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Signaling in the Age of AI: Evidence from Cover Letters".
Warren: Here's the gist of the paper. We study the impact of generative AI on labor market signaling using the introduction of an AI-powered cover letter writing tool on a large online labor platform <ref:2509.25054#pg14> The correlation between cover letters’ textual alignment and callbacks fell by 51%, consistent with what theory predicts if the AI technology reduces the signal content of cover letters <ref:2509.25054#pg5> Research Question and Methodology The study investigates how generative AI affects labor market signaling by tracking access to an AI-powered cover letter writing tool on Freelancer.com and analyzing usage at the application level <ref:2509.25054#pg14> The researchers observe access to the tool and usage of the tool at the application level, linking this information to submitted cover letters and hiring outcomes <ref:2509.25054#pg14> They use a difference-in-differences design comparing workers with and without access to measure the effect of the platform-native AI writing tool on cover letter tailoring and callback rates <ref:2509.25054#pg14> Impact on Cover Letter Tailoring The introduction of the AI tool increased cover-letter tailoring by 0.16 standard deviations when access was gained, while actual usage raised tailoring by 1.36 standard deviations <ref:2509.25054#pg3> The researchers find that AI substitutes for workers’ pre-AI cover letter tailoring skills, as workers who previously wrote more tailored cover letters experienced smaller gains in cover letter tailoring after gaining access to the AI tool <ref:2509.25054#pg4> Market-Level Signaling Shifts The correlation between cover letter tailoring and receiving a callback fell by 51% after the launch of the AI tool, consistent with what theory predicts if the AI technology reduces the signal content of cover letters <ref:2509.25054#pg5> Employers shifted toward alternative signals, including workers’ prior work histories, as cover letters became less correlated with hiring after AI is introduced <ref:2509.25054#pg5> The correlation between callbacks and workers’ review scores rose by 5% <ref:2509.25054#pg5> Human-AI Interaction Effects Workers with higher pre-AI writing ability tend to spend more time editing AI drafts, and controlling for worker fixed effects, a one–standard deviation increase in editing time is associated with a 0.31 percentage point rise in the probability of receiving a job offer <ref:2509.25054#pg8> This indicates that human revision enhances the effectiveness of AI-generated cover letters, contributing to a growing body of evidence on human–AI collaboration <ref:2509.25054#pg8> The study finds no evidence of changes in overall market outcomes such as hiring, interviewing, and job completion rates following the tool’s introduction <ref:2509.25054#pg7> This suggests that while AI disrupted one channel of signaling—cover letters—employers successfully shifted toward other signals, particularly past reputation scores <ref:2509.25054#pg7> However, workers with access benefit from AI, with moderately productive workers gaining more than the most productive, while control workers face a decline in hiring probability <ref:2509.25054#pg8> The tool’s availability raises hiring probabilities for treated workers whereas without access experience a decrease <ref:2509.25054#pg8> References Galdin and Silbert (2025) first measured similarity between cover letters and job posts on Freelancer.com, interpreted it as a signal of worker ability, and showed it predicts hiring <ref:2509.25054#pg14> Cowgill et al.
Ruth: (2024) examine the impact of generative AI on job applications and startup pitches through a survey experiment <ref:2509.25054#pg14> Wiles and Horton (2025) study generative AI assistance to employers for crafting job posts in a similar context and find that while the tool increased the number of job posts, it did not increase the number of matches <ref:2509.25054#pg14> Agrawal et al.
Warren: (2019), Noy and Zhang (2023), Brynjolfsson et al.
Ruth: (2025), and Acemoglu (2025) show how AI affects productivity and innovation Agarwal et al.
Paper summary: Ruth: So we’re looking at this paper today, "Signaling in the Age of AI: Evidence from Cover Letters." The core idea is tracking how using an AI-powered cover letter writing tool on a big online labor platform changes how much information those letters actually signal about the worker.
Warren: Right. It seems like they're testing if this new technology actually makes our applications better or worse when we’re trying to get hired. The abstract says the main finding is that when employers use this AI tool, the correlation between how well a cover letter matches a job description and getting a callback drops by fifty-one percent.
Iris: That drop in correlation, that's significant because it suggests the AI is reducing the quality of that signal, which is what theory predicts when generative AI reduces signal content <ref:2509.25054#pg5>. But the paper isn't just looking at correlation; they’re tracking access and usage at the application level on Freelancer.com <ref:2509.25054#pg14>.
Ruth: Exactly. They use a difference-in-differences design to see how this tool affects tailoring and callback rates by comparing workers who have access to the tool against those who don't <ref:2509.25054#pg14>. What they claim is that access to the tool increased textual alignment between cover letters and job posts, while actual usage raised that tailoring up by one point three six standard deviations <ref:2509.25054#pg3>.
Warren: Okay, so access makes the letters look better, but using it actually makes them even more tailored. That’s interesting because it means the AI isn't just making things worse; in some ways, it might be boosting the signal initially <ref:2509.25054#pg3>.
Iris: But then they immediately show that workers who used the tool before—before they had access—experienced smaller gains in tailoring after gaining access to the AI tool, suggesting the AI might substitute for those pre-existing skills <ref:2509.25054#pg4>. That’s a key part of their argument about whether workers are actually improving their skill set or just getting an easier template <ref:2509.25054#pg4>.
Paper summary: Ruth: And that leads into what they call market-level signaling shifts, where they find that the correlation between those cover letter tailoring scores and receiving a callback falls by fifty-one percent after the AI tool is launched <ref:2509.25054#pg5>. This supports the idea that employers are shifting their focus away from cover letters as a primary signal <ref:2509.25054#pg7>.
Warren: So, if cover letters become less informative, what do employers look at instead? The paper suggests they start looking toward other signals, specifically workers’ prior work histories <ref:2509.25054#pg7>. That makes sense; if the letter isn't a good signal anymore, they go back to traditional things like past performance reviews <ref:two thousand five hundred nine point two five zero five four#pg7.
Iris: They also noticed something else happening at the market level, which is that the correlation between callbacks and workers’ review scores went up by five percent after the AI tool came out <ref:2509.25054#pg5>. That's a bit of a counter-intuitive finding when you think about how much cover letters used to matter for those initial signals.
Ruth: It is interesting because it shows that while the letter signal got weaker, some other established signals, like review scores, became relatively stronger <ref:two thousand five hundred nine point two five zero five four#pg5. But we have to be careful about what this means for hiring outcomes overall.
Warren: The paper says they found no evidence of changes in big market outcomes like overall hiring or job completion rates following the tool’s introduction <ref:2509.25054#pg7>. So, despite the signal change, it doesn't seem to have messed with the total volume of jobs being filled <ref:two thousand five hundred nine point two five zero five four#pg7.
Iris: But there are still differences for workers based on their access. The paper points out that workers who had access to the AI benefit, and moderately productive workers gain more than the most productive ones, while control workers face a decline in hiring probability <ref:two thousand five hundred nine point two five zero five four#pg8. So it’s not a universal win for everyone <ref:two thousand five hundred nine point two five zero five four#pg8.
Ruth: That's where the human-AI interaction comes in, which they look at next. Workers who already had higher pre-AI writing ability tend to spend more time editing the AI drafts <ref:two thousand five hundred nine point two five zero five four#pg8. And this human revision actually helps; a one standard deviation increase in editing time is associated with a zero point three one percentage point rise in the probability of getting a job offer <ref:two thousand five hundred nine point two five zero five four#pg8.
Paper summary: Warren: So, it sounds like AI isn't just replacing the worker entirely; it’s changing how the work gets done, and human skill is still valuable for refining those AI outputs <ref:two thousand five hundred nine point two five zero five four#pg8. That’s a different story than if the tool just made everything easier without any human input <ref:two thousand five hundred nine point two five zero five four#pg8.
Iris: They are looking at this collaboration between human and AI as something that enhances effectiveness, which is a growing area of evidence in this field <ref:two thousand five hundred nine point two five zero five four#pg8. But the paper also flags a limitation, and they state that their study only looks at the introduction of this specific tool on Freelancer.com <ref:two thousand five hundred nine point two five zero five four#pg1.
Ruth: That's true; it’s specific to that platform and that one feature, so we don't know if this effect holds for other types of AI tools or other labor markets <ref:two thousand five hundred nine point two five zero five four#pg1. But the broader implication is that employers are learning to look at different data points now, like those review scores they mentioned earlier <ref:two thousand five hundred nine point two five zero five four#pg7.
Warren: So, for someone listening who isn't in a labor market, what does this paper actually mean for their day-to-day life? It means that if you use an AI tool to write a cover letter, the way you tailor it might not be as effective at getting you noticed by employers anymore <ref:two thousand five hundred nine point two five zero five four#pg1.
Iris: And because of that, the paper suggests that if workers want to keep their hiring chances up, they should probably focus more on building those prior work histories and review scores rather than just relying on a perfectly tailored cover letter <ref:two thousand five hundred nine point two five zero five four#pg7.
Ruth: That’s the central tension here: the AI makes crafting a perfect cover letter easier to do, but it also weakens that specific signal because it becomes too generic, forcing employers to rely on other things <ref:two thousand five hundred nine point two five zero five four#pg1. It’s about adapting your strategy when a tool fundamentally changes how you send your message to the market <ref:two thousand five hundred nine point two five zero five four#pg1.
Warren: It’s a real shift in what counts as a good signal in the job search landscape, driven by the introduction of this specific AI writing tool on Freelancer.com <ref:two thousand five hundred nine point two five zero five four#pg1. We'll keep an eye on how these other platforms and tools evolve over time <ref:two thousand five hundred nine point two five zero five four#pg1.
Conclusion: Ruth: So we're wrapping up our look at this study on cover letters and AI signaling, titled "Signaling in the Age of AI: Evidence from Cover Letters."
Warren: Yeah, basically, they tracked how using an AI cover letter tool on a big job site affects how much the letter actually signals about a worker.
Iris: The main thing is that when employers use this tool, the link between how well a cover letter matches the job description and getting a callback drops by fifty-one percent.
Ruth: That’s because the AI seems to reduce the actual signal content of those letters, which is what theory predicted back in twenty twenty-five.
Warren: And it shows that employers are shifting their focus away from cover letters toward other signals like past work histories when AI is introduced.
Iris: But they also found that workers with access to the tool benefit, and control workers face a drop in hiring probability, so it's not a universal win.
Ruth: And there’s this finding about human interaction: if you already wrote good letters before, spending more time editing the AI draft actually helps you get an offer by zero point three one percentage points.
Warren: So the paper suggests that while the AI changes how we signal, employers are learning to look at other things like review scores instead of just how well a letter is tailored.
Iris: It also notes that overall hiring rates don't change much, which tells us this isn't a huge market disruption yet.
Ruth: But it does show that the way we communicate our skills through these specific documents is being altered by the technology in ways we need to watch closely.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19