Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts
cs.CL, cs.CY
Submitted: 2026-11-09
Updated: 2026-11-11
Code: https://github.com/octner/nips-tae-definition-sensitivity
Terminology
Sources
- Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
- Are we done with ImageNet?
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
- When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding
- Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
- Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation
- Automated Annotation with Generative AI Requires Validation
- AI and the Everything in the Whole Wide World Benchmark
- Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data