Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

arXiv:2606.21069 · cs.CL · Submitted 2026-06-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Quality and Agreement in Multilabel Emotion Annotation".

Jane: The gist The paper presents a multilabel emotion annotation case study and uses it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So to get into it, the authors are Emily Ohman and Anna Koufakou from Waseda University and Florida Gulf Coast University in Tokyo. They’ve put this case study on emotion annotation in text, specifically using Hemingway’s The Old Man and the Sea as their example. Jane That sounds like a very specific setup for studying how subjective things like emotions are handled by technology.

Lu: It’s an interesting choice because it gives a concrete text to work with, and they use an expanded version of Plutchik’s wheel of emotions, which includes anger, anticipation, disgust, fear, joy, sadness, surprise and trust for the core set. Meng So they aren't just looking at one simple emotion; they're looking at a whole spectrum.

Lalam: And they also let annotators add free-text notes for any other emotions that pop up as "other" emotions or labels. Tom That adds another layer of complexity because it means the labels aren't just simple yes or no for those core feelings.

Jane: It suggests that just picking one main emotion isn't enough to capture what a piece of text might actually convey, and they are trying to figure out how to handle that richness. Lu Exactly. This paper is looking at how we can move beyond just finding the majority vote and start seeing the actual range of interpretations people have.

The paper's summary: Tom: They set up three annotators, A1, A2, and A3, and they look at how much they agree on different emotion labels across that whole dataset. Jane They found that preliminary agreement scores using Krippendorff’s alpha were low across the board when looking at the full set of labels.

Meng: That suggests that if we just take a single hard label, we’re going to get pretty noisy results because people aren't totally aligned on what they feel. Tom And then they introduce some things to make it more nuanced than just counting how many people picked a label.

Lu: They define targets as "hard labels," which is just one way to think about it, where a hard label is one if at least one annotator gave a positive score, and zero otherwise <ref:2606.21069#pg1>. But they also introduce something called an "intensity-weighted soft target" for modeling purposes. Jane That’s where things get interesting because they modulate the vote shares by those intensity scores that the annotators provide, scaling them from zero to one.

Lalam: These soft targets are interpreted as per-label Bernoulli probabilities, which means they give us a way to represent uncertainty rather than just a single guess. Tom That’s smart because it allows the downstream models to see the entire distribution of opinions, not just the loudest one.

The paper's improvements: Jane: The paper suggests that we shouldn't treat those hard labels as gospel for everything, especially when building classifiers. Tom They show that hard-label training can actually give you the highest F1 score in some scenarios, like when you include a specific annotator they flagged as reliable.

Lu: But the authors found that using those hard labels sometimes masks real uncertainty about what the annotators are thinking. They also introduce a whole evaluation pipeline designed to test this trade-off directly. Jane And that pipeline includes comparing traditional thresholded metrics, like macro-F1, against these probabilistic alignment metrics, which is called Bernoulli cross-entropy SoftBCE.

Meng: The results showed that for BERT under different conditions—using all annotators versus just the most reliable ones—the hard labels gave a macro-F1 of zero point six one six when Annotator two was included, but the soft labels actually yielded higher macro- and micro-F1 scores in both settings. Tom And they specifically pointed out that SoftBCE, which looks at the full predicted probability vector against those soft targets without thresholding, produced lower SoftBCE while still being competitive in F1.

Lalam: So the message is that the benefits of using soft supervision really depend on what you are trying to optimize for in your model. Jane It moves us away from just picking one 'gold' label and treating it as perfect, which is a big shift in how we approach training these systems.

Conclusion: Tom: So, to wrap up the main finding of this paper on "Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework," they are showing that disagreement is structured and leaves measurable traces in model behavior. Jane They found that hard labels might maximize F1 metrics, but soft supervision often yields predictions that better reflect the actual variance in annotator opinions.

Lu: The authors also point out a limitation, which they admit is that one annotator exhibited systematically sparse labeling behavior, which shows the need for clearer guidelines when we are looking at annotation workflows. Meng And another thing they flag is that intensity ratings aren't automatically helpful; their impact depends on how consistently annotators use the scale. Lalam It’s a methodological contribution because they use downstream models as a diagnostic tool to surface those design trade-offs between annotation choices and modeling choices. Jane That really frames annotation and modeling as two coupled decisions, not just sequential steps.

Tom: So we can see that treating majority-voted labels as perfect gold can actually mask uncertainty and overstate how reliable a model is. Lu It’s about using the paper "Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework" to show how to distinguish ambiguity in the data from behavior specific to the annotators.

Emily Ohman, Anna Koufakou

Waseda University · Florida Gulf Coast University

cs.CL

Submitted: 2026-06-19

Updated: 2026-06-19

Comments: Published in the Proceedings of the 1st Workshop on Computational Affective Science, CAS 2026, co-located with LREC 2026. This version corresponds to the published workshop paper

Journal ref: Proceedings of the 1st Workshop on Computational Affective Science (CAS) @ LREC 2026. pp. 1-15

DOI: 10.18653/v1/2026.cas-1.1

Code: https://github.com/esohman/SoftBCE_Emotions

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist The paper presents a multilabel emotion annotation case study and uses it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion

Key concepts

Soft Vote-Share Labels: Instead of forcing annotators to choose one label, this method calculates a probability (between 0 and 1) for each emotion based on how many annotators selected it. This captures the uncertainty and consensus level among human raters for each specific emotion.
Bernoulli Cross-Entropy (SoftBCE): This is a metric used to measure how well a model's predicted probability distribution aligns with the calculated soft targets. A lower SoftBCE score means the model's predicted probabilities are closer to what the annotators collectively indicated, reflecting better alignment with human uncertainty.
Disagreement Diagnostics (Dvar and DJac): These are data-derived measures computed directly from the raw annotator labels to quantify how much annotators disagree. Dvar measures variance in vote shares for a single label, while DJac measures the overall pairwise disagreement between different annotators' sets of labels.
Hard vs. Soft Targets: Hard targets are binary (1 or 0) based on whether any annotator selected an emotion. Soft targets are continuous values (like vote shares or intensity weights) that allow models to learn from the degree of consensus, not just the presence or absence of a label.

Terminology

Aggregation — 1× in this paper
Alignment — 1× in this paper
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist The paper presents a multilabel emotion annotation case study and uses it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers.

Case Study Overview

The study presents a case study of three annotators annotating Hemingway’s The Old Man and the Sea (in English) using an expanded version of Plutchik’s wheel of emotionsTable 1: Resource card for the TOMATS emotion annotation case study>. The resource name is The Old Man and the Sea emotion annotations (TOMATS)Table 1: Resource card for the TOMATS emotion annotation case study>. The label set (core) includes anger, anticipation, disgust, fear, joy, sadness, surprise, trustTable 1: Resource card for the TOMATS emotion annotation case study>. Additionally, optional free-text “other” emotions/notes are allowedTable 1: Resource card for the TOMATS emotion annotation case study>.

Annotator Behavior and Disagreement

The annotators were asked to first read the text in full before considering the emotions and then start from the beginning to label each sentence in contextSection 2.1: Dataset and Annotation Protocol>. Annotator 2 presents a different case, as they only considered roughly 10% of the data to have any emotion-association and almost no secondary or tertiary emotion associations (13 and 3 respectively)Table 2: Annotation counts for each annotator>. Preliminary agreement scores calculated using Krippendorff’s α in Table 3 indicate low agreementTable 3: Krippendorff’s α for binary presence/absence of each Plutchik emotion>. The analysis shows that agreement between Annotator 2 and the other annotators is extremely low (macroF1 ≈ 0.08) on the full dataset, but this rises sharply (macroF1 ≈ 0.38-0.40) when restricted to the subset of items that Annotator 2 actually annotated with a genuine emotion labelTable 4: Summary comparison of pairwise annotator agreement in the full vs. filtered subsets>.

Aggregating Multi-Annotator Labels

To construct hard targets, the paper defines y hard i,k = 1[pi,k > 0] (4)Section 3.1: Targets and disagreement diagnostics>. To incorporate intensity information, a intensity-weighted soft target is defined where vote shares are modulated by annotator-provided intensity scores (scaled to [0, 1])Section 3. Aggregating multi-annotator labels>. These soft targets y˜i,k ∈ [0, 1] can be interpreted as per-label Bernoulli probabilitiesSection 3. Targets and disagreement diagnostics>.

Evaluation Framework

The evaluation pipeline is designed to examine how annotation behavior and disagreement structure affect both agreement estimates and downstream emotion classificationSection 4: Evaluation Pipeline>. The pipeline consists of four stages: (1) Label extraction and harmonization, (2) Multilabel representation, (3) Agreement analysis with missing-label filtering, and (4) Downstream modeling and evaluationSection 4.

Downstream Modeling Results

The study compares four classifier families under two annotation conditions: using all available annotators or only the two annotators identified as reliable by the filtered agreement analysisTable 7: Downstream results for BERT under different supervision targets and annotator-inclusion regimes>. The results show that hard-label training yields the highest F1 overall when Annotator 2 is included (macroF1 = 0.616; microF1 = 0.625), but the intensity-weighted soft labels yield higher macro- and micro-F1 than uniformly weighted soft labels in both settingsTable 7: Downstream results for BERT under different supervision targets and annotator-inclusion regimes>. SoftBCE, in contrast, evaluates the full vector of predicted probabilities against the aggregated soft targets without thresholdingSection 4.1: Training objectives>.

Conclusion

The downstream results suggest that the benefits of soft supervision depend on what is being optimizedSection 5: Conclusions>. SoftBCE produced markedly lower SoftBCE while remaining broadly competitive in F1, indicating better alignment between predicted probabilities and the aggregated soft targetsTable 7: Downstream results for BERT under different supervision targets and annotator-inclusion regimes>. The main contribution of this paper is methodological, using downstream models as a diagnostic tool to surface design trade-offs in emotion annotationSection 5: Conclusions>. The results support treating annotation and modeling as coupled choices rather than sequential stepsSection 5: Conclusions>. The paper cautions that treating majority-voted labels as “gold” can mask uncertainty and overstate model reliabilitySection 5: Conclusions>.

Limitations

The work is intentionally framed as a case study, limiting the generalizability of absolute agreement values and downstream performanceSection 5: Conclusions>. One limitation is that one annotator exhibited systematically sparse labeling behavior, underscoring the need for clearer guidelines regarding abstentions in emotion annotation workflowsSection 5: Conclusions>. Finally, the paper suggests that intensity ratings are not automatically helpful and their impact depends on how consistently annotators use the scaleSection 5: Conclusions>. The main contribution of this paper is methodological, using downstream models as a diagnostic tool to surface design trade-offs in emotion annotation and to distinguish ambiguity in the data from annotator-specific behaviorSection 5: Conclusions>. The paper anticipates that applying this pipeline to shortform or informal text will increase both overlap among emotions and annotator disagreement due to reduced context and higher ambiguitySection 5: Conclusions>. The main contribution of this paper is methodological, using downstream models as a diagnostic tool, together with disagreement measures computed directly from the labels, provides a practical way to surface design trade-offs in emotion annotation and to distinguish ambiguity in the data from annotator-specific behaviorSection 5: Conclusions>. The paper anticipates that applying this pipeline to shortform or informal text will increase both overlap among emotions and annotator disagreement due to reduced context and higher ambiguitySection 5: Conclusions>. The main contribution of this paper is methodological, using downstream models as a diagnostic tool, together with disagreement measures computed directly from the labels, provides a practical way to surface design trade-offs in emotion annotation and to distinguish ambiguity in the data from annotator-specific behaviorSection 5: Conclusions>. The paper anticipates that applying this pipeline to shortform or informal text will increase both overlap among emotions and annotator disagreement due to reduced context and higher ambiguitySection 5: Conclusions>. The main contribution of this paper is methodological, using downstream models as a diagnostic tool, together with disagreement measures computed directly from the labels, provides a practical way to surface design trade-offs in emotion annotation and to distinguish ambiguity in the data from annotator-specific behaviorSection 5: Conclusions>. The paper anticipates that applying this pipeline to shortform or informal text will increase both overlap among emotions and annotator disagreement due to reduced context and higher ambiguitySection 5: Conclusions>. The main contribution of this paper is methodological, using downstream models as a diagnostic tool, together with disagreement measures computed directly from the labels, provides a practical way to surface design trade-offs in emotion annotation and to distinguish ambiguity in the data from annotator-specific behaviorSection 5: Conclusions>. The paper anticipates that applying this pipeline to shortform or informal text will increase both overlap among emotions and annotator disagreement due to reduced context and higher ambiguitySection 5: Conclusions>.

Improvements for AI systems

Bold label extraction: The system should extract only explicit Plutchik emotions, discarding “none”, blanks, and non-Plutchik entries to yield a mapping (i) → e1, e2,... that reflects only positive emotion assignments. This ensures that agreement analysis is performed on positive emotion assignments.
Disagreement handling: Implement missingness-aware aggregation by treating missing annotations as missing rather than as explicit negative votes, which allows the system to distinguish between structural disagreement caused by missing labels and genuine divergence in emotional interpretation, as described in Section 4.
Soft supervision modeling: Train models using vote-share soft targets where the target is defined as the empirical vote share among contributing annotators, which produces multilabel targets y˜i,k ∈ [0, 1] that can be interpreted as per-label Bernoulli probabilities.
Intensity integration: Incorporate intensity information by defining an intensity-weighted soft target where vote shares are modulated by annotator-provided intensity scores (scaled to [0, 1]), preserving graded signal about strength of affect.
Model evaluation shift: Shift evaluation focus from solely thresholded metrics to probabilistic alignment using Bernoulli cross-entropy (binary cross-entropy; BCE) between model predictions and the aggregated soft targets, as this metric evaluates the full predicted probability vector against the aggregated soft targets without thresholding.
Diagnostic monitoring: Integrate data-derived disagreement measures like mean per-label Bernoulli variance, Dvar(i), which is maximized when annotators split evenly on a label and minimized when they agree, to monitor annotation regimes during training.