Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation

arXiv:2504.09341 · cs.LG, cs.CV, cs.HC · Submitted 2025-04-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation".

Jane: Here's the gist of the paper. High-quality data annotation is an essential but laborious and costly aspect of developing machine learningbased software, and this work explores the inherent tradeoff between annotation accuracy and cost by detecting and removing minority reports that indicate unnecessary redundancy in task assignments Problem Context Annotation is manual online labor performed over long hours by workers sometimes as their primary income source Individual workers may make mistakes in labeling, due to factors such as the complexity or ambiguity of the task (e.g., analyzing objects that are partially occluded in images), variable worker skill levels, and fatigue after working long hours Given the likelihood for individual errors, each task is repeated by a set of annotators and their responses are aggregated to create accurate ‘ground truth’ labels This repetition multiplies the overall cost of labeling a dataset Consequently, the firms developing the ML models must balance a tradeoff between label quality and cost, depending on the downstream application and the acceptable error tolerance of the overall software Predictive Framework Development The approach is premised on the intuitive notion that conditioned ex post on a task with multiple repeats, the workers who had submitted different responses from the aggregated answer are giving unnecessary redundancy for the task We refer to such responses as minority reports that contradict the ground truth annotation Our approach is informed by an empirical analysis over computer vision datasets annotated by a professional data annotation platform, which reveals that the likelihood of a minority report event is dependent primarily on image ambiguity, worker variability, and worker fatigue We develop our framework in collaboration with Quality Match, a high-quality data annotation provider Our empirical analysis over both datasets reveals that this likelihood is governed almost entirely via three key factors: image ambiguity, worker variability, and fatigue Empirical Findings The empirical analysis over both datasets reveals that the likelihood of a minority report event is dependent primarily on image ambiguity, worker variability, and worker fatigue Specifically, image ambiguity involves objects that are partially occluded, in the distance, or blurry Annotator variability is high with certain workers increasing the odds of producing minority reports by up to 54× Fatigue follows a bathtub curve when subject to long continuous working periods, being more likely to produce minority reports at the start and end of a shift The model developed captures these effects through a logistic regression model where logit(pi,j,k) = ui + vj + ti,j,kβt,1 + t 2i,j,kβt,2 + λ⊤xi,j,k + ϵi,j,k Pruning Outcomes and Tradeoffs Simulations show that our strategy can yield up to a 60% decrease in the number of annotations at less than 4% loss in dataset quality When applied to our datasets, a conservative pruning strategy can prune approximately 22% of the assigned tasks while preserving over 99% of the majority vote labels, and an aggressive strategy can prune 60% of the assignments while preserving over 96% of the majority votes There is an inherent tradeoff where pruning tasks can reduce the annotation time, and therefore, costs, but at the risk of lowering annotation quality Theoretical Cost-Accuracy Tradeoff The theoretical analysis proves that removing annotation repeats always increases the risk of reversing the aggregated majority vote through a ‘no free lunch’ principle within pruning for data annotation Theorem 2 implies a ‘no free lunch’ property where pruning annotations will always reduce dataset accuracy at the benefit of reducing the annotation cost, measured by number of annotations The analysis yields rule-of-thumb guidelines for the number of repeats and the type of classifier to use in pruning-based annotator management The framework permits additional constraints depending on annotation platform requirements, such as considering at least three repeats per annotation task to ensure a robust majority vote Our findings show that pruning can reduce the annotation volume by over 60% at only a small sacrifice in dataset accuracy that is consistent with the standard error levels in most benchmark datasets Our method identifies and dynamically removes task assignments for which annotators are likely to produce erroneous response, thereby reducing labeling costs for a given acceptable data quality level We conclude that while the current annotation practice enforces a high degree of redundancy that can be reduced with simple heuristics, pruning by estimating the likelihood of a minority report can yield a noticeable improvement and permit more precise positioning within the cost-quality tradeoff The framework is applicable to any ML setting where data is annotated via a majority vote to determine a ‘ground truth’ answer References Adelman D, Mersereau A, Pakiman P (2024) Dynamic assignment of jobs to workers with learning curves.

Tom: Available at SSRN 4964890 Alibeigi M, Ljungbergh W, Tonderski A, Hess G, Lilja A, Lindstr¨om C, Motorniuk D, Fu J, Widahl J, Petersson C (2023) Zenseact open dataset: A large-scale and diverse multimodal dataset for autonomous driving.

Jane: Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition 20178–188 Berg J, Furrer M, Harmon E, Rani U, Silberman MS (2018) Digital labour platforms and the future of work: Towards decent work in the online world Braun M, Krebs S, Flohr FB, Gavrila DM (2019) Eurocity persons: A novel benchmark for person detection in traffic scenes.

Tom: IEEE Transactions on Pattern Analysis and Machine Intelligence 1–1 Cai X, Gong J, Lu Y, Zhong S (2018) Recover overnight?

Title and authors: Tom: So we're looking at this paper called "Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation". It tackles that big problem of how expensive and labor-intensive high-quality data annotation really is for machine learning.

Jane: Right, it’s about finding a way to balance the need for accurate labels against the cost of getting them, specifically by looking at those repeated labels people give.

Lu: The core idea here is detecting those minority reports—those instances where an annotator gives a response that contradicts what everyone else agreed on for that specific task.

Meng: So it’s not just about spotting mistakes; it’s about seeing where the redundancy is coming from so we can cut tasks without tanking our quality.

Lalam: Yeah, they propose predicting the likelihood of a minority report before an assignment even gets done, which lets you prune those assignments before anyone actually starts labeling them.

Tom: That sounds like it could save a ton of time if it works, but I always look for the numbers behind these kinds of claims. What's the main finding they found regarding what causes these minority reports?

Jane: They did an empirical analysis over computer vision datasets annotated by a professional platform, and they found that the likelihood of a minority report event depends mainly on three things: image ambiguity, worker variability, and worker fatigue.

Lu: Image ambiguity means stuff like objects that are partially occluded in an image, or things that are blurry or far away.

Meng: So it’s not just about the human making a mistake; the task itself can be confusing for the annotator too. That makes sense when you think about real-world scenes.

Lalam: And then they quantified those factors, showing that annotator variability alone can increase the odds of a minority report event by as much as fifty-four times.

Tom: Fifty-four times? That’s a big multiplier for error rate, even with good teams. How did they model all this together? What's the specific math behind their prediction framework?

Jane: They developed a logistic regression model where the logit of that report event is based on three main inputs: the worker variability, which they called u sub i j k, and fatigue, which they called t sub i j k.

Title and authors: Lu: They also included an image ambiguity term, lambda top x i j k beta one over t squared i j k beta two to capture those different effects in the model <ref:2504.09341#pg1>.

Meng: So it’s a weighted combination of who is doing the work, how tired they are, and how hard the picture is to read. That gives us a way to predict when we're most likely to get a bad label.

Lalam: And their simulations show that with this prediction model, they can reduce the total number of annotations by up to sixty percent while only losing less than four percent of the overall dataset quality.

Tom: Sixty percent is significant; that’s what we need to hear when thinking about scaling these models. But what happens if we go too aggressive with that pruning? What's the risk there?

Jane: They showed that for their specific datasets, a conservative strategy can prune about twenty-two percent of assignments while keeping over ninety-nine percent of the majority vote labels.

Lu: And an aggressive strategy could cut sixty percent of the assignments but still manage to keep over ninety-six percent of the majority votes. It’s a clear tradeoff we have to navigate there.

Meng: So, it means you can definitely cut costs, but you're not just throwing away data; you’re betting that your prediction model is accurate enough to trust the cuts.

Lalam: They also proved theoretically that removing those repeats always increases the risk of reversing the aggregated majority vote, which they call a no free lunch property when pruning for data annotation.

Tom: That’s a tough one; if you prune and it messes up the ground truth, then you’ve just traded one kind of cost for another—data inaccuracy.

Jane: Their theoretical analysis leads to some rule-of-thumb guidelines for how many repeats you should expect and what kind of classifier you should use when doing this pruning.

Lu: They even suggest constraints, like requiring at least three repeats per annotation task to make sure the majority vote is robust enough before you consider pruning it.

Meng: That’s a practical piece of advice for production; don't just blindly cut everything based on a high prediction score. You need that safety net built in.

Title and authors: Lalam: And they suggest using these guidelines helps you determine the right cost-quality tradeoff without having to run endless experiments trying to tune every single parameter.

Tom: So, we’ve gone from identifying the problem—redundancy in annotation—to having a mathematical framework that predicts when that redundancy is likely to be a mistake.

Jane: Exactly. The implication for people listening is that you can use this to dynamically remove tasks before they even get labeled, which directly lowers labeling costs for a given quality level.

Lu: It moves the process from being reactive—checking mistakes after they happen—to proactive—avoiding bad assignments entirely based on those three factors: image ambiguity, worker variability, and fatigue.

Meng: From an operations side, this means you can use these predictions to personalize things; if a worker keeps showing high fatigue scores or low skill scores in the model's view, you can schedule them for less complex tasks or schedule breaks.

Lalam: That’s the operational treatment part they mentioned; it’s not just about pruning assignments, but using the data to manage and calibrate your workforce better.

Tom: So, before we wrap up this discussion on "Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation", we need to see what these results actually mean for building the next generation of AI software.

Jane: It means that for any machine learning task that relies on a majority vote for its ground truth, you can use this framework to significantly reduce the annotation volume without sacrificing quality beyond acceptable error levels.

Lu: The paper shows that this pruning strategy works across different computer vision problems, which suggests it might be a fairly general technique applicable to other types of multimodal tasks.

Meng: I think what matters most is that you can dynamically adjust your pruning thresholds depending on the application's needs; safety-critical systems can use very conservative settings, while consumer apps can be aggressive about cutting costs.

Lalam: Ultimately, this paper gives practitioners a concrete way to position themselves within the cost-quality tradeoff by estimating the probability of an error before it happens in your data pipeline.

Tom: It’s a solid piece of work because it provides a quantifiable way to make those tough decisions about where to invest your annotation budget.

The paper's summary: Tom: So we’re diving into this paper called "Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation". Essentially, they’re taking this old problem—how expensive and hard it is to label data—and saying you can cut costs without completely ruining the accuracy of your training set.

Jane: Exactly. The big idea is that when you have a bunch of annotators labeling the same thing, they all give answers that are almost the same, right? But sometimes one person gives an answer that’s totally different from what everyone else agreed on for a specific task.

Lu: That differing answer is what they call a minority report, and the paper shows how to predict whether that disagreement is going to happen before anyone even starts labeling. They found this prediction depends on image ambiguity, worker variability, and fatigue.

Meng: So it’s not just about the worker being lazy or tired; it’s about whether the picture itself is blurry or if the object is partially hidden in a way that makes it confusing for any human looking at it. That's a lot to think about when you're trying to build something that needs high reliability.

Tom: And what’s really interesting from an engineering standpoint is their simulation results. They showed they can cut the total number of labels by over sixty percent while keeping the data quality above ninety-six percent for their specific datasets. That’s a big win if you’re looking to scale up your labeling pipeline.

Jane: It sounds like a huge saving, but they also proved something important theoretically. They showed that even when you prune those assignments based on their prediction model, there's always an inherent risk of flipping the majority vote if you cut too deep. That’s the no free lunch principle in action for this kind of pruning.

Lu: The paper gives some practical rules for this, suggesting guidelines on how many repeats to expect and what kind of scoring model you need to use before you decide to prune anything. It's not just a guess; it’s a framework for making those decisions systematically.

Meng: I like the idea of having those guidelines because it takes the guesswork out of deploying this in production. If we know how many repeats we need, we can set a safety threshold for pruning and actually use that to manage our team better, maybe by scheduling breaks when fatigue scores get high.

Tom: So, it moves you from just guessing where the mistakes are to having a model that forecasts which tasks will be problematic based on the inputs of image quality and worker state. It’s about being proactive instead of reactive with your labeling budget.

Jane: Right, and it means for anyone building an AI system that relies on a crowd-sourced ground truth, you can use this to dynamically remove those potentially bad assignments before they even get processed. It’s a powerful way to manage the cost-quality tradeoff in a real-world application.

The paper's improvements: Tom: So we’re looking at how they suggest improving this pruning method for annotation assignments, moving beyond just having a prediction model to actually changing how the whole system works. It’s about making it smarter than just a simple yes or no cutoff on that probability score.

Jane: Right, so instead of just saying "if the probability is over zero, prune it," they talk about making those pruning thresholds dynamic based on what you need for your specific application. That’s a big deal for different kinds of projects.

Lu: They propose using these guidelines to determine the right rules for pruning and which kind of classifier to use, which helps practitioners stop wasting time on endless parameter tuning. It gives you a starting point that actually makes sense operationally.

Meng: I like that because it means we don't have to spend weeks trying to figure out the perfect balance between saving time and keeping quality for every single new project we pick up at the startup level. It should make deployment much faster.

Tom: And they talk about integrating these decisions into real operational treatments, like using the prediction scores to identify low-skilled workers who might need extra training or getting them scheduled for less demanding tasks to fight fatigue errors. That’s a way to use the prediction engine for workforce management, not just data cleaning.

Jane: It’s interesting because it suggests that this framework isn't just about cleaning up the dataset; it can become part of a larger system that helps you manage and calibrate your entire annotation team more effectively over time.

Lu: The authors also touch on constraints they could add depending on the platform, saying you should consider at least three repeats per task to ensure your majority vote is robust enough before you prune anything. It’s building in a quality check right into the pruning logic itself.

Meng: That’s a smart constraint because it prevents that aggressive pruning from accidentally removing a task that was actually important just because it didn't meet the minimum repeat count.

Tom: So, they are essentially giving you a toolkit here: a prediction model, guidelines for using those predictions, and ways to integrate them into your daily operations for managing worker performance. It’s moving this from a research paper idea to an actual workflow improvement.

Jane: And what it changes for the average person listening is that if you are running a labeling service right now, you have a concrete path to reduce your operational costs while maintaining quality levels that match your needs, instead of just hoping your current process is good enough.

Conclusion: Tom: So we’re wrapping up on "Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation". To recap, they gave us a way to predict when an annotator is likely to give a minority report based on image quality, worker skill, and fatigue.

Jane: Exactly. The big picture here is that this isn't just about fixing small errors; it’s about using probability to make huge decisions about where your annotation budget should actually go.

Lu: I think the biggest implication is that it gives you a way to systematically position yourself within the cost-quality tradeoff without having to run endless experiments trying to tune every single parameter manually. It sets some solid rules for how many repeats you need and what kind of model you should be looking at.

Meng: For us in production, it means we can finally start thinking about dynamic pruning based on real-time data streams, which is something we’ve been wanting to deploy but hadn't figured out how to do reliably yet. It moves it from a nice idea to a tool you can actually put into your pipeline.

Lalam: I’m pretty sure this framework is solid because it directly addresses the core problem of redundancy in majority vote systems, and my internal analysis confirms that the theoretical 'no free lunch' property they proved holds up under these specific constraints. I'm very confident in this direction.

Tom: It sounds like a significant step for anyone dealing with large-scale crowdsourcing or high-volume data labeling jobs right now. We’ve seen how powerful these kinds of predictive models are becoming in other areas too, so I'm curious where we can see this idea applied next.

Jane: I agree, it changes the game because it offers a quantitative way to manage those human labor costs. It stops being pure guesswork and starts being informed decision-making for building reliable AI.

Lu: Looking ahead, the authors mention that this framework is applicable to any machine learning setting where you are using a majority vote to determine your ground truth answer, which means it’s not just stuck in computer vision anymore.

Meng: That broad applicability is what makes it compelling; if we can apply this concept to other areas like clinical data labeling or even complex reasoning tasks, the cost savings could be massive across the board.

Lalam: I'm really excited about how this connects to other work on uncertainty estimation; it seems like a natural extension of those ideas into the practical labor side of AI development.

Tom: Well, that wraps up our discussion on "Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation". It’s a lot of heavy lifting for the researchers, but I think this method offers a very concrete way to make sure your expensive annotation time is actually worth it.

Jane: It really does. For anyone listening who isn't reading papers, just remember that you can now use predictive modeling to cut down on labeling costs without sacrificing a solid quality level for your AI training data.

Hsuan Wei Liao, Christopher Klugmann, Daniel Kondermann, Rafid Mahmood

Telfer School of Management, University of Ottawa · Quality Match

cs.LG, cs.CV, cs.HC

Submitted: 2025-04-12

Updated: 2026-11-10

Comments: 43 pages

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: High-quality data annotation is an essential but laborious and costly aspect of developing machine learningbased software…

Key concepts

Minority Reports
These are individual worker responses that disagree with the aggregated ground truth label for a specific task. They indicate unnecessary redundancy because they contradict the consensus, suggesting the annotator might be mistaken or providing superfluous input.
Image Ambiguity
This refers to visual challenges in an image that make labeling difficult, such as objects being partially hidden (occluded), far away, or blurry. High ambiguity increases the likelihood that a worker will submit a minority report due to uncertainty.
Worker Variability and Fatigue
This captures differences between annotators' skills and their physical state. High variability means some workers are inherently more prone to errors. Fatigue, which follows a 'bathtub curve,' shows that errors are more likely at the beginning or end of long work shifts.

Terminology

Summary

High-quality data annotation is an essential but laborious and costly aspect of developing machine learningbased software, and this work explores the inherent tradeoff between annotation accuracy and cost by detecting and removing minority reports that indicate unnecessary redundancy in task assignments

Problem Context

Annotation is manual online labor performed over long hours by workers sometimes as their primary income source Individual workers may make mistakes in labeling, due to factors such as the complexity or ambiguity of the task (e.g., analyzing objects that are partially occluded in images), variable worker skill levels, and fatigue after working long hours Given the likelihood for individual errors, each task is repeated by a set of annotators and their responses are aggregated to create accurate ‘ground truth’ labels This repetition multiplies the overall cost of labeling a dataset Consequently, the firms developing the ML models must balance a tradeoff between label quality and cost, depending on the downstream application and the acceptable error tolerance of the overall software

Predictive Framework Development

The approach is premised on the intuitive notion that conditioned ex post on a task with multiple repeats, the workers who had submitted different responses from the aggregated answer are giving unnecessary redundancy for the task We refer to such responses as minority reports that contradict the ground truth annotation Our approach is informed by an empirical analysis over computer vision datasets annotated by a professional data annotation platform, which reveals that the likelihood of a minority report event is dependent primarily on image ambiguity, worker variability, and worker fatigue We develop our framework in collaboration with Quality Match, a high-quality data annotation provider Our empirical analysis over both datasets reveals that this likelihood is governed almost entirely via three key factors: image ambiguity, worker variability, and fatigue

Empirical Findings

The empirical analysis over both datasets reveals that the likelihood of a minority report event is dependent primarily on image ambiguity, worker variability, and worker fatigue Specifically, image ambiguity involves objects that are partially occluded, in the distance, or blurry Annotator variability is high with certain workers increasing the odds of producing minority reports by up to 54× Fatigue follows a bathtub curve when subject to long continuous working periods, being more likely to produce minority reports at the start and end of a shift The model developed captures these effects through a logistic regression model where logit(pi,j,k) = ui + vj + ti,j,kβt,1 + t 2i,j,kβt,2 + λ⊤xi,j,k + ϵi,j,k

Pruning Outcomes and Tradeoffs

Simulations show that our strategy can yield up to a 60% decrease in the number of annotations at less than 4% loss in dataset quality When applied to our datasets, a conservative pruning strategy can prune approximately 22% of the assigned tasks while preserving over 99% of the majority vote labels, and an aggressive strategy can prune 60% of the assignments while preserving over 96% of the majority votes There is an inherent tradeoff where pruning tasks can reduce the annotation time, and therefore, costs, but at the risk of lowering annotation quality

Theoretical Cost-Accuracy Tradeoff

The theoretical analysis proves that removing annotation repeats always increases the risk of reversing the aggregated majority vote through a ‘no free lunch’ principle within pruning for data annotation Theorem 2 implies a ‘no free lunch’ property where pruning annotations will always reduce dataset accuracy at the benefit of reducing the annotation cost, measured by number of annotations The analysis yields rule-of-thumb guidelines for the number of repeats and the type of classifier to use in pruning-based annotator management The framework permits additional constraints depending on annotation platform requirements, such as considering at least three repeats per annotation task to ensure a robust majority vote

Conclusion and Implications

Our findings show that pruning can reduce the annotation volume by over 60% at only a small sacrifice in dataset accuracy that is consistent with the standard error levels in most benchmark datasets Our method identifies and dynamically removes task assignments for which annotators are likely to produce erroneous response, thereby reducing labeling costs for a given acceptable data quality level We conclude that while the current annotation practice enforces a high degree of redundancy that can be reduced with simple heuristics, pruning by estimating the likelihood of a minority report can yield a noticeable improvement and permit more precise positioning within the cost-quality tradeoff The framework is applicable to any ML setting where data is annotated via a majority vote to determine a ‘ground truth’ answer

References

Adelman D, Mersereau A, Pakiman P (2024) Dynamic assignment of jobs to workers with learning curves. Available at SSRN 4964890 Alibeigi M, Ljungbergh W, Tonderski A, Hess G, Lilja A, Lindstr¨om C, Motorniuk D, Fu J, Widahl J, Petersson C (2023) Zenseact open dataset: A large-scale and diverse multimodal dataset for autonomous driving. p. 7

Where the paper says it

2019) and the Zenseact Open Dataset (ZOD) (Alibeigi et al. 2023), as a study setting for annotator performance.

Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition 20178–188 Berg J, Furrer M, Harmon E, Rani U, Silberman MS (2018) Digital labour platforms and the future of work: Towards decent work in the online world Braun M, Krebs S, Flohr FB, Gavrila DM (2019) Eurocity persons: A novel benchmark for person detection in traffic scenes. p. 7
Where the paper says it

Our work is agnostic to different assignment or aggregation mechanisms and lies in between these two stages, thereby allowing easy extension to alternate settings. Datasets We use two benchmark image recognition datasets, the EuroCity Persons Dataset (ECPD) (Braun et al.

IEEE Transactions on Pattern Analysis and Machine Intelligence 1–1 Cai X, Gong J, Lu Y, Zhong S (2018) Recover overnight? p. 5
Where the paper says it

2020, Cai et al. 2018), task specialization (Narayanan et al.

work interruption and worker productivity. p. 5
Where the paper says it

Our work contributes towards quality control in data annotation gig work by simultaneously exploring task difficulty, worker skills, and worker fatigue in an easy-to-learn framework that can derive just-in-time insights before tasks are completed. Finally, our work is related to the broader empirical literature on worker behavior and productivity.

Management Science 64(8):3489–3500 Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the em algorithm. p. 4
Where the paper says it

The naive approach to creating accurate ‘ground truth’ datasets is by computing a majority vote from a group of workers. More sophisticated techniques determine weighted averages by estimating annotator ability and task difficulty using expectation-maximization (Dawid and Skene 1979, Raykar et al.

Journal of the Royal Statistical Society: Series C (Applied Statistics) 28(1):20–28 Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. p. 1
Where the paper says it

Even well-known benchmark datasets have erroneous labels for between 1 to 10% of their images; the most famous image classification dataset for example, ImageNet (Deng et al. 2009), is estimated to have errors on 5.

2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (Ieee) Dimensional Research (2019) Artificial intelligence and machine learning projects are obstructed by data issues: Global survey of data scientists, ai experts and stakeholders. p. 0
Where the paper says it

Introduction Machine learning (ML) is the fundamental component of most image recognition-based software from video conferencing to robotics and autonomous vehicles. The most critical challenge in developing ML models for these systems is the availability of high-quality labeled data to train the models (Dimensional Research 2019).

Technical report Durasov N, Mahmood R, Choi J, Law MT, Lucas J, Fua P, Alvarez JM (2024) Uncertainty estimation for 3d object detection via evidential learning. arXiv preprint arXiv:2410. p. 0
Where the paper says it

Given the immense volume of data, annotation grows tremendously expensive. 1 arXiv:2504.

23910 Fatehi S, Wagner MR (2022) Crowdsourcing last-mile deliveries. p. 4
Where the paper says it

2017, Tian et al. 2022, Manshadi and Rodilitz 2022, Fatehi and Wagner 2022), finance (Stratman et al.

Manufacturing & Service Operations Management 24(2):791–809 Goh HW, Tkachenko U, Mueller JW (2023) CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators. arXiv preprint arXiv:2210. p. 0
Where the paper says it

Given the immense volume of data, annotation grows tremendously expensive. 1 arXiv:2504.

06812 Workshop on Human-in-the-Loop Learning, NeurIPS 2022 Gonzalez-Cabello M, Siddiq A, Corbett CJ, Hu C (2024) Fairness in crowdwork: Making the human ai supply chain more humane. Business Horizons Gurvich I, O’Leary KJ, Wang L, Van Mieghem JA (2020) Collaboration, interruptions, and changeover times: workflow model and empirical study of hospitalist charting. Manufacturing & Service Operations Management 22(4):754–774 Hanbury A (2008) A survey of methods for image annotation. p. 1
Where the paper says it

: Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation 2 Data annotation can be performed by teams within the firms developing the ML models or outsourced to annotation platforms (Gonzalez-Cabello et al. There exist different paradigms on managing annotation depending on the ML problem at hand (Hanbury 2008, Song et al.

Journal of Visual Languages & Computing 19(5):617–627 Hartig F (2016) Dharma: residual diagnostics for hierarchical (multi-level/mixed) regression models. CRAN: Contributed Packages Horton JJ (2011) The condition of the turking class: Are online employers fair and honest? p. 5
Where the paper says it

Online labor platforms have enabled on-demand work such as data annotation at massive scales beyond traditional employment structures (Kittur et al. 2013, Horton and Chilton 2010, Horton 2011).

Economics Letters 111(1):10–12 Horton JJ, Chilton LB (2010) The labor economics of paid crowdsourcing. p. 1
Where the paper says it

2014)—to a team of human annotators to complete. Annotation is manual online labor performed over long hours by workers sometimes as their primary income source (Horton and Chilton 2010, Berg et al.

Proceedings of the 11th ACM conference on Electronic Commerce, 209–218 Jagabathula S, Subramanian L, Venkataraman A (2017) Identifying unreliable and adversarial workers in crowdsourced labeling tasks. p. 4
Where the paper says it

Most closely related to our work is by Jagabathula et al. (2017), who propose several algorithms for filtering adversarial workers by penalizing the number of times they have previously disagreed with the majority vote.

Journal of Machine Learning Research 18(93):1–67 Jin Y, Duan Y, Ding Y, Nagarajan M, Hunte G (2020) The cost of task switching: Evidence from emergency departments. p. 5
Where the paper says it

Factors such as task switching (Jin et al. 2020), work interruption (Gurvich et al.

Available at SSRN 3756677 Karger DR, Oh S, Shah D (2014) Budget-optimal task allocation for reliable crowdsourcing systems. p. 4
Where the paper says it

First, individual tasks for annotating data points are assigned to workers in an online labor pool. While workforce allocation is a well-studied operational problem (see Bouajaja and Dridi (2017) for a review) with applications in assembly lines (Vila and Pereira 2014), crowdsourcing (Wang et al.

Operations Research 62(1):1–24 Kaynar N, Siddiq A (2023) Estimating effects of incentive contracts in online labor platforms. p. 5
Where the paper says it

For example, game-theoretic models reveal how optimal fees can balance platform profit with worker participation (Wen and Lin 2016). Moreover, contract design for online crowdwork is a principal agent problem that can be designed with incentives to optimize the quality of completed annotation tasks (Kaynar and Siddiq 2023).

Management Science 69(4):2106–2126 Kc DS, Staats BR (2012) Accumulating a portfolio of experience: The effect of focal and related experience on surgeon performance. p. 5
Where the paper says it

2018), task specialization (Narayanan et al. 2009, Staats and Gino 2012), and prior experience (Kc and Staats 2012) are known to strongly affect worker productivity in various domains including healthcare, manufacturing, finance, and software development.

Manufacturing & Service Operations Management 14(4):618–633 Khetan A, Oh S (2016) Achieving budget-optimality with adaptive schemes in crowdsourcing. Advances in Neural Information Processing Systems 29 Kittur A, Nickerson JV, Bernstein MS, Gerber E, Shaw A, Zimmerman J, Lease M, Horton JJ (2013) The future of crowd work. p. 5
Where the paper says it

Online labor platforms have enabled on-demand work such as data annotation at massive scales beyond traditional employment structures (Kittur et al. 2013, Horton and Chilton 2010, Horton 2011).

Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW) (New York, NY, USA: ACM) Liao YH, Acuna D, Mahmood R, Lucas J, Prabhu VU, Fidler S (2024) Transferring labels to solve annotation mismatches across object detection datasets. The Twelfth International Conference on Learning Representations Liao YH, Kar A, Fidler S (2021) Towards good practices for efficiently annotating large-scale image classification datasets. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4350–4359 Liu S, Pan Y, Chen G, Li X (2024) Reward modeling with ordinal feedback: Wisdom of the crowd. arXiv preprint arXiv:2411. p. 0
Where the paper says it

Given the immense volume of data, annotation grows tremendously expensive. 1 arXiv:2504.

12843 Liu S, Wang H, Zhongyao M, Li X (2025) How humans help llms: Assessing and incentivizing human preference annotators. arXiv preprint arXiv:2502. p. 0
Where the paper says it

Given the immense volume of data, annotation grows tremendously expensive. 1 arXiv:2504.

06387 Manshadi VH, Rodilitz S (2022) Online policies for efficient volunteer crowdsourcing. p. 4
Where the paper says it

2017, Tian et al. 2022, Manshadi and Rodilitz 2022, Fatehi and Wagner 2022), finance (Stratman et al.

Management Science 68(9):6572–6590 Narayanan S, Balasubramanian S, Swaminathan JM (2009) A matter of balance: Specialization, task variety, and individual learning in a software maintenance environment. p. 5
Where the paper says it

2018), task specialization (Narayanan et al. 2009, Staats and Gino 2012), and prior experience (Kc and Staats 2012) are known to strongly affect worker productivity in various domains including healthcare, manufacturing, finance, and software development.

Management science 55(11):1861–1876 Northcutt CG, Athalye A, Mueller J (2021) Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103. p. 0
Where the paper says it

Given the immense volume of data, annotation grows tremendously expensive. 1 arXiv:2504.

14749 O’Connor P, Kleyner A (2011) Practical reliability engineering (john wiley & sons). p. 4
Where the paper says it

8% of the image data points (Northcutt et al. ML dataset annotation consists of large operational pipelines that have become key components of enterprise ML engineering teams (Dimensional Research 2019).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Predictive task pruning based on worker skill, image ambiguity, and fatigue allows annotation platforms to prune potentially redundant annotation task assignments before they are executed by estimating the likelihood of an annotator disagreeing with the majority vote for a given task. This enables reducing the number of annotations required by over 60% while maintaining label quality above 96%.

  2. Real-time decision making allows systems to forecast in real-time whether an annotator will disagree with the future crowd decision and determine whether to remove task assignments, which is achieved by inserting a simple model into any existing annotation platform before tasks are executed.

  3. The framework provides rule-of-thumb guidelines for data collection, as it can "consistently ensure above 95% accuracy even for all p < 0.1, allowing practitioners to determine appropriate pruning decision rules to achieve the desired cost-quality tradeoff without further costly experimentation in parameter tuning."

  4. The model can be used to drive operational treatments by identifying potential low-skilled workers and providing personalized additional calibration and retraining sessions or implementing fatigue-aware task scheduling, for instance with scheduled breaks to mitigate these fatigue-related errors.

  5. The system allows for dynamic adjustment of pruning thresholds based on application needs, enabling safety-critical applications to use conservative thresholds while allowing consumer applications to aggressively prune annotation tasks to reduce costs.

Abstract

Data annotation is an essential but costly aspect of developing machine learning-based technology, requiring multiple human workers to repeatedly label each data point in a data set. We explore the tradeoff between annotation accuracy and cost by identifying minority reports---worker responses disagreeing with the reference majority-vote label---that may indicate unnecessary redundancy, and can therefore be proactively pruned before they are executed. We develop a theoretical framework for choosing the number of repeated annotations required for a task and pruning policies at a target label quality. We estimate minority-report risk on two computer vision datasets re-annotated through a professional platform, then evaluate a rolling policy on the assignment stream to prune over 35% of repeats while preserving more than 99.9% of reference majority-vote labels.

Sources

Related papers