Evaluating LLM Creativity through Long-Tail Performance
cs.CL
Submitted: 2026-10-09
Updated: 2026-10-09
Comments: 26 pages, including references and appendices. Code and data: https://github.com/YichenZW/longtail-creativity
Code: https://github.com/YichenZW/longtail-creativity
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Phi-4 Technical Report
- CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints
- Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- PIQA: Reasoning about Physical Commonsense in Natural Language
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
- Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator
- Art or Artifice? Large Language Models and the False Promise of Creativity
- Probing the Creativity of Large Language Models: Can models produce divergent semantic association?
- DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Training Verifiers to Solve Math Word Problems
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Gemma 3 Technical Report
- The Llama 3 Herd of Models
- A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing
- CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
- Creative Preference Optimization
- SimulBench: Evaluating Language Models with Creative Simulation Tasks
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering