Can Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection

arXiv:2606.04924 · cs.CL · Submitted 2026-06-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Can Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection".

Jane: The gist The widespread use of large language models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Okay, so the paper introduces this community survey to get a better handle on the situation. It sets up three main questions they wanted to answer about LLM usage in crowdsourcing studies.

Jane: The core idea is figuring out two things first: how much of a concern is LLM-generated text in these studies, and then what are the actual strategies researchers are using to deal with it.

Lu: They also want to know what practical considerations come out of all this experience for running crowdsourcing studies when we’re in the LLM era. It tries to consolidate all these different viewpoints from practitioners into one place.

Meng: So it's mapping out the landscape: the problem itself, how researchers feel about it, and what they are doing to fight it. I hope they get some concrete ideas on how to keep their data sound for real-world applications.

Lalam: The paper essentially provides an overview of these challenges and strategies, aiming to give practitioners a clearer picture of the implications for data quality. It’s trying to answer the question of whether this is a big deal or something they can manage.

The paper's summary: Tom: Let's look at what they found regarding how often LLMs are being used and what specific signals researchers are looking for when they spot that usage.

Jane: They noted that the observation rate actually went up with experience, with researchers having over five years of crowdsourcing experience reporting a sixty point zero percent rate of observed LLM use, compared to just one to two years at forty-four percent.

Lu: The signals they found for LLM usage are pretty specific: AI-like writing style shows up in sixty-three point six percent of cases, and overly fluent or generic responses account for forty-three point two percent of the observations. They also mentioned high similarity across multiple workers at thirty-one point eight percent and markup formatting also showed up at that same rate.

Meng: That suggests that the way we detect it is tied to specific linguistic patterns rather than just a general feeling that something is off with the text. It’s about looking for these particular traits in the output.

Lalam: And they pointed out something tricky: careful prompt engineering can actually make LLM output look pretty human, which makes those subtle cases much harder for people to catch. So detecting it isn't just about finding obvious errors anymore.

The paper's improvements: Tom: Now let’s talk about the concerns the survey revealed regarding how this impacts the data itself, because that’s where things get serious for data validity.

Jane: One big theme they found was "Changes in the data," where nearly forty-five percent of respondents felt that LLM-generated responses tend to be more homogeneous and less reflective of diverse human perspectives <ref:2606.04924#pg1>.

Lu: Another major concern was "Downstream consequences," which showed up in fifty-three point five percent of responses, specifically worrying about the validity and reliability of the results and even distorted study outcomes.

Meng: So it’s not just that the data looks different; it’s that those changes can mess up whatever conclusions researchers try to draw later on, which is a huge worry for deployment scenarios.

Lalam: In terms of mitigation, researchers are trying a few things: excluding instances from the final dataset fifty-six point eight percent of the time, labeling them as potentially LLM-generated in twenty-seven point three percent of cases, or taking action against the workers themselves.

Conclusion: Tom: So to wrap up what this paper on "Can Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection" tells us is that crowdsourcing can only keep going if researchers stop using it just as a quick way to collect or validate data.

Jane: The survey provides these practical considerations for future studies, pushing researchers to think about how they design and document their crowdsourcing studies differently now. It’s not about stopping the method entirely, but changing how we use it.

Lu: The paper suggests looking at three stances on LLM acceptability: fully restricting its use, allowing controlled use, or allowing unrestricted use for tasks. This frames the future research direction nicely.

Meng: For deployment, this means if you’re running a study today, you need to be very clear about your filtering procedures and what precautions you took when dealing with observed LLM text. It puts the documentation burden squarely on the researcher now.

Lalam: Indeed, it’s a call for researchers to be more transparent about their process so that these studies can continue to provide useful data in this new environment. We hope this survey helps guide those next steps in data collection research.

University of Stuttgart · Ghent University · Harvard University · IT University of Copenhagen

cs.CL

Submitted: 2026-06-03

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 79/100

The gist: The gist The widespread use of large language models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models.

Key concepts

LLM Usage Observation: Researchers surveyed if they see LLMs being used by crowdworkers in their free-text data. Only 44% reported observing this usage, though rates were higher for experienced researchers. This highlights the growing challenge of distinguishing human from machine-generated text.
Detection Signals: Methods used to spot AI-generated text include looking for distinctive writing styles, unusually fast completion times, and high similarity across multiple workers. These signals help practitioners identify potential LLM involvement in their datasets.
Data Validity Concerns: Participants worried that LLM text makes data less valid because it can be more homogeneous and less reflective of diverse human viewpoints. This impacts the reliability and accuracy of research outcomes derived from crowdsourced free-text responses.

Terminology

Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The widespread use of large language models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models.

Community Survey Overview

This paper presents a systematic community survey of 155 researchers in NLP and related disciplines regarding their experiences and opinions on collecting free-text responses via crowdsourcing in the LLM era. The survey aimed to provide an overview of practitioners’ challenges, mitigation strategies, and the foreseen implications on data quality. Forty-four percent of respondents reported observing LLM usage in their crowdsourced data. While ninety-three percent had anticipated this, half were unsure what precautions to take. The most prevalent detection strategies are distinctive textual style patterns and unusually fast completion times. Overall, survey responses show that the research community is aware of the problem and taking measures, but existing efforts remain insufficient to fully address it.

Prevalence of LLM Usage and Detection Signals

The survey revealed that only 44 of the 100 researchers who conducted free-text crowdsourcing studies reported observing LLM-generated responses. Observation rates increased with experience, as researchers with over five years of crowdsourcing experience reported higher rates of observed LLM use (60.0%) than those with one to two years (χ2 = 17.64, p = 0.007, df = 6). Signals of LLM usage included AI-like writing style (63.6%), overly fluent or generic responses (43.2%), high similarity across multiple workers (31.8%), markup formatting (31.8%), and close resemblance to known LLM output (29.5%). Furthermore, participants noted that careful prompt engineering can make LLM output closely resemble human writing, making subtle cases difficult to detect.

Perceived Impact on Data Validity and Quality

Respondents expressed significant concerns regarding the impact of LLM-generated text on data validity. One major theme was Changes in the data, where 44.9% (n = 57) of responses conveyed this, characterizing LLM-generated responses as more homogeneous (n = 24) and less reflective of diverse human perspectives (n = 16) Another prevalent theme was Downstream consequences, appearing in 53.5% (n = 68) of responses, which included concerns about result validity (n = 29), result reliability (n = 16), and distorted study outcomes (n = 10)

Mitigation Strategies and Future Considerations

Researchers employed various mitigation strategies, including explicit instructions, adding technical hurdles such as disabling copy-paste or adding CAPTCHA tasks. When detecting LLM-generated responses, researchers described strategies such as excluding instances (56.8%), labeling them as potentially LLM-generated (27.3%), or taking actions against workers, with 29.5% taking no action. The paper derives considerations for future studies based on three stances on LLM acceptability: fully restricting use, allowing controlled use, and allowing unrestricted use.

Conclusion

Crowdsourcing can only ‘survive’ in the LLM era if researchers no longer use it as a quick data collection or validation method. The paper concludes by providing a set of practical considerations to guide future research designing and documenting crowdsourcing studies in the LLM era.

Limitations

The study acknowledges limitations, noting that researchers who perceived LLM usage as a problem were more likely to participate, potentially skewing responses toward greater concern and awareness. Furthermore, the dominance of engineering sciences in the respondent pool suggests views are more representative of the NLP community than adjacent fields.

Acknowledgments

The authors express gratitude to everyone who participated in and distributed their survey. This work received support from the Belgian American Educational Foundation (BAEF) and the Research Foundation Flanders (FWO-Vlaanderen) under grant number 1S96324N, and the SoftwareCampus project Placebo, funded by German Federal Ministry of Research, Technology and Space (BMFTR) under the grant 01IS23072.

Ethics Statement

The study adhered to ethical standards, involving voluntary, anonymous participation with no sensitive personal data collected beyond professional background and demographics<ref:2606.

Improvements for AI systems

Bold header: Adaptive Data Collection Protocols

This improvement involves designing study protocols that explicitly account for LLM usage scenarios by implementing layered mitigation strategies based on task sensitivity, such as disabling copy-paste, presenting stimuli as images and adding CAPTCHA for high-risk tasks.

Bold header: Context-Aware Task Design

The improved system can dynamically adjust task instructions based on the study's goals to determine LLM acceptability, specifically by employing task-dependent acceptability where the impact of LLM use depends on the goal of the study.

Bold header: Transparent Data Documentation

The AI system should enforce rigorous documentation protocols for any observed or filtered data, ensuring that if filtering occurs, researchers clearly disclose the exclusion criteria and filtering procedures used, and that any action or precaution taken in this regard should be well document[ed].

Bold header: Controlled LLM Integration Framework

For tasks where LLM assistance is partially acceptable, the system can integrate LLMs as a verified aid by requiring participants to candidly report the method used for producing text for a study, and documenting the extent of reliance on LLM-generated content as a contextual variable during analysis.

Bold header: Verification-Focused Study Design

The AI system should prioritize human oversight workflows where annotators verify LLM generated drafts when studying tasks requiring high fidelity, such as those focusing on personal experience or interpersonal, demographic, or cultural variance, to ensure the authentic human signal is very crucial for the study.

Abstract

The widespread use of Large Language Models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models. To better understand how this is addressed, we surveyed 155 researchers in NLP and related disciplines about their experiences and opinions on collecting free-text responses via crowdsourcing. This paper provides an overview of practitioners' challenges, mitigation strategies, and the foreseen implications on data quality. 44% of respondents reported observing LLM usage in their crowdsourced data. While 93% of them had anticipated this, half were unsure what precautions to take. The most prevalent detection strategies are distinctive textual style patterns and unusually fast completion times. Overall, survey responses show that the research community is aware of the problem and taking measures, but existing efforts remain insufficient to fully address it. Finally, we derive a set of considerations to guide future crowdsourced free-text data collection in the era of LLMs.