CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text
cs.CL
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: Accepted to Findings of EMNLP 2026
Project page: https://smrmodares.github.io/CLiN
License: http://creativecommons.org/licenses/by/4.0/
The gist: Evaluating creativity in large language model (LLM) outputs remains challenging because creativity is multidimensional and human-centered.
Terminology
Abstract
Evaluating creativity in large language model (LLM) outputs remains challenging because creativity is multidimensional and human-centered. We examine how reliably LLMs evaluate short literary text in Persian, a low-resource language, across multiple evaluation strategies and prompt formulations. We find that LLM-human agreement varies substantially across dimensions: alignment is stronger for structured TTCT-derived properties such as Originality, Fluency, and Elaboration, but considerably weaker for more subjective dimensions, particularly Emotion and Attractiveness. Judgments are also sensitive to prompt formulation, while few-shot prompting, ensembling, and multi-agent debate provide no consistent improvement. Motivated by this dimension-dependent behavior, we investigate whether structured creativity dimensions can instead be approximated using simple, interpretable proxy metrics. We introduce CLIN, which evaluates three TTCT-derived dimensions separately using topic-aware novelty for Originality, contextual lexical clustering for Fluency, and lexical diversity for Elaboration. These proxies achieve human alignment comparable to or better than the strongest zero-shot LLM judge in our setting while requiring substantially lower evaluation cost.
Sources
- CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints
- Art or Artifice? Large Language Models and the False Promise of Creativity
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
- On the Creativity of Large Language Models
- Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering