Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework
The study examined how annotator disagreement affects NLP model performance when labeling multiple emotions. It introduced soft labels that represent vote shares instead of hard labels, showing that models trained on these softer targets better reflect human uncertainty and variance. This suggests preserving disagreement can lead to more reliable probability estimates for emotion recognition.