Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Characterizing Human-Likeness in AI Generated Poetry".
Title and authors: Tom: Now let's look at what the authors suggest we do next, because they lay out some clear ways to improve this research <ref:2607.261#pg10>.
Jane: They have a few specific ideas centered around making the detection system stronger and more insightful.
Lu: First, they propose training Gemma four-31B with hard negative examples <ref:2607.261#pg1>. They argue that doing this will reduce the margin where the model gets confused when it tries to classify something, which should give us a clearer picture of what makes poetry truly human <ref:2607.261#pg9>.
Meng: That makes sense from an engineering side; giving a zero-shot model some examples of "this is definitely not human" helps constrain its decision space.
Tom: And they also suggest implementing a mechanism to explicitly check for style guide violations during the generation process itself <ref:2607.261#pg10>.
Jane: They found that because AI can be so good at "humanizing" text, this means we should look for those violations as potential human-like attributes that might be helping the evasion <ref:2607.261#pg10>.
Lu: Another idea is to enhance the Gemma four zero-shot pipeline by incorporating reasoning factor analysis from Algorithm two <ref:2607.261#pg3>.
Meng: That means instead of just looking at the final classification, we should look at *how* Gemma four reached its decision across those five criteria to figure out which criterion is most influential <ref:2607.261#pg9>.
Tom: And they also touched on a problem related to how prompts are handled during generation—specifically the disconnection between the user's theme and the poem's actual theme <ref:2607.261#pg3>.
Jane: They noted that this separation makes it difficult for the system to maintain a clean structure when it has to juggle a few different instructions at once.
Lu: So, they’re suggesting developing a more robust system to mitigate bias towards using few-shot examples over the actual user prompts during generation <ref:2607.261#pg3>.
Meng: That’s crucial because if the theme gets tangled up, we lose control over what the AI is actually producing versus what the user asked for.
Tom: And finally, they pointed out that Qwen three-32B was particularly good at this evasion <ref:2607.261#pg10>, so they suggest adding new detectors specifically to see how those models are fooling the system <ref:2607.261#pg10>.
Jane: It sounds like the next step is to build a more layered defense against these sophisticated generative techniques.
Lu: Basically, they’re moving from just classifying the final output to understanding the entire generation process and how those attributes interact <ref:2607.261#pg3>.
Meng: I see that as needing better accountability in how we build these tools if we want them to be reliable for production use.
The paper's summary: Tom: So wrapping up the study on "Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study," the main thing is that while Gemma four performed best among their zero-shot tests, human classification was still pretty poor with only zero point seven zero precision and a recall of zero point four five when looking at the AI generated poems <ref:2607.261#pg9>.
Jane: The study concludes that both the human-like attributes and the AI-like attributes really contribute to those classification errors <ref:2607.261#pg10>.
Lu: It’s a complex situation where the very traits we consider human versus machine are what confuse the current detection methods <ref:2607.261#pg3>.
Meng: So the implication is that we can't just rely on one feature; it’s a combination of rigid structure and unique literary choices that matters <ref:2607.261#pg10>.
Tom: The future work they outlined points toward training Gemma four-31B with hard negative examples and testing those high-evasion models like Qwen three-32B with new detection methods <ref:2607.261#pg10>.
Jane: They are trying to move past simple word counts and get down to the actual structural reasoning behind the poetry <ref:2607.261#pg5>.
Lu: It’s a lot of work, but it’s necessary because if we don't account for these subtle differences, our tools will keep getting outsmarted by increasingly capable generative models <ref:2607.261#pg5>.
Meng: Yeah, we need to keep pushing the limits of what these models can produce and what our detectors can see.
Tom: That’s the gist of it for this paper, "Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study." It shows us that the line between human and AI is really becoming blurred, but it also shows us exactly where those blur points are located.
Jane: It’s a fascinating piece of research because it tries to define the very essence of human writing through measurable attributes.
Lu: I think this work helps us understand the underlying representation learning issues in models like Gemma four-31B, especially how they handle style and thematic consistency <ref:2607.261#pg5>.
Meng: It gives us a roadmap for what kind of data we need to feed into models so they don't just memorize patterns but actually understand the concept of human-like composition.
The paper's improvements: Tom: So we're wrapping up on "Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study." Jane Basically, they tested how well different AI models and traditional tools could tell the difference between human and machine written poetry without any specific training data.
Lu: The main idea is that we can’t just look at simple things like word count or word overlap; the real distinction lies in these specific attributes, what they call defining attributes versus misleading ones.
Meng: It seems like those unique literary devices, like metaphors and imagery, are the key things that help the AI hide its identity when it’s trying to evade detection.
Lalam: I'm pretty sure that Qwen three-32B was the worst at hiding its machine nature because it used those misleading attributes really well against Gemma four <ref:2607.261#pg1>.
Tom: And they found that even when they tried to use their own zero-shot pipeline with Gemma four the results were still messy, with human classification accuracy staying really low <ref:2607.261#pg9>.
Jane: That's a big signal because it means the boundary between what we think is human and what AI produces is incredibly blurry right now.
Lu: The paper suggests that both those traits that make something look human and the traits that make it look like AI actually cause problems when you try to classify them reliably <ref:2607.261#pg10>.
Meng: From a deployment standpoint, it means we can't just rely on a single detection method; we need to consider how these attributes interact during generation itself.
Tom: They also pointed out that if an AI model is really good at sounding human, it can use those stylistic elements to confuse the very systems designed to spot them.
Jane: It makes me wonder what happens when we start building systems that can flag potential style guide violations *during* the creation process.
Lu: That's a solid direction for future work, trying to catch those evasive techniques before they even finish writing a poem <ref:2607.261#pg3>.
Meng: I’m interested in how much it would cost to implement such a check at scale, though that’s a whole different conversation.
Tom: Anyway, that’s what they found in this study about "Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study." Jane It gives us a clearer picture of the complexity we're dealing with when trying to automate the classification of creative writing.
Conclusion: Tom: So we've looked at "Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study." Basically, they tested how well different AI models and traditional tools could tell the difference between human and machine written poetry without any specific training data.
Jane: They used a zero-shot detection pipeline to figure out if the text looked human just by reading it, which is really interesting because the line between AI and human writing is getting so blurry.
Lu: The main idea there was that you can't just look at simple stuff like word count or word overlap; you need to look at those specific attributes, what they call defining versus misleading ones.
Meng: It seems like those unique literary devices, like metaphors and imagery, are the key things that help the AI hide its identity when it’s trying to evade detection.
Lalam: I'm pretty sure that Qwen three-32B was the worst at hiding its machine nature because it used those misleading attributes really well against Gemma four <ref:2607.26221#pg1>.
Tom: And they found that even when they tried to use their own zero-shot pipeline with Gemma four the results were still messy, with human classification accuracy staying around zero point seven zero and a recall of zero point four five.
Jane: That's a big signal because it means the boundary between what we think is human and what AI produces is incredibly blurry right now.
Lu: The paper suggests that both those traits that make something look human and the traits that make it look like AI actually cause problems when you try to classify them reliably.
Meng: From a deployment standpoint, it means we can't just rely on a single detection method; you need to consider how these attributes interact during generation itself.
Tom: They also pointed out that if an AI model is really good at sounding human, it can use those stylistic elements to confuse the very systems designed to spot them.
Jane: It makes me wonder what happens when we start building systems that can flag potential style guide violations *during* the creation process.
Lu: That's a solid direction for future work, trying to catch those evasive techniques before you even finish writing a poem.
Meng: I’m interested in how much it would cost to implement such a check at scale, though that's a whole different conversation.
Tom: Anyway, that’s what they found in "Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study." Jane It gives us a clearer picture of the complexity we're dealing with when trying to automate the classification of creative writing.
Jane: It confirms that while there are measurable differences, those differences are subtle enough that they can be easily masked by clever AI generation techniques.
Lu: This work helps us understand the underlying representation learning issues in models like Gemma four-31B, especially how they handle style and thematic consistency <ref:2607.26221#pg1>.
Meng: It gives us a roadmap for what kind of data we need to feed into models so they don't just memorize patterns but actually understand the concept of human-like composition.
Lalam: The core idea is that we need to stop looking at surface words and start looking at the underlying structure and style.
Tom: Alright, that’s our time on this paper. We’ll be right back after the break with another look at some new arXiv submissions on how AI handles video forgery detection.
Angshu Nirmegh Biswas, Taspiha Tabassum, Afia Abida Shohid, Razia Marzan Mou, Ayeeshah Akter Esha
BRAC University
cs.CL
Submitted: 2026-07-28
Updated: 2026-11-15
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 67/100
The gist: With the advancement of AI technologies, Generative AI (GenAI) and human written text have become nearly indistinguishable…
Key concepts
- Defining Attributes
- These are the specific characteristics of poetry—both human and non-human—that reliably help classify whether a poem is written by a person or an AI. The study found that the rigidity and clichés typical of AI composition often serve as these defining attributes for detection.
- Misleading Attributes
- These are poetic features, often unique literary devices like metaphors and imagery, that can trick detection systems. When GenAI incorporates these unique elements, it can successfully evade classification tools designed to spot common AI patterns.
- Zero-shot Detection Pipeline
- This is a method where a model is asked to classify new data (a poem) without being specifically trained on examples of that exact task beforehand. In this study, Gemma 4 was used in this pipeline to try and sort human versus AI poems based on five criteria.
- Weighted F1 Average
- This is a performance metric used to measure how well the classification model performs. A high Weighted F1 score indicates that the model is accurate in both identifying human poetry and correctly classifying AI poetry, showing strong overall classification ability.
Terminology
Summary
With the advancement of AI technologies, Generative AI (GenAI) and human written text have become nearly indistinguishable, making verification of GenAI poems difficult due to their black-box
nature <ref:2607.26221#pg2>. The main objective of the research is to deduce the attributes of English poetry that contribute to classification and misclassification of both human and AI poems and provide corroborating or contradicting evidence to the poetry distinguishability claim <ref:2607. p. 026221#pg2>.
Dataset Construction
The methodology involves creating a dataset consisting of both human and AI poems to verify the distinguishability of human and AI creation, which is achieved through a zero-shot detection pipeline <ref:2607.26221#pg4>. The dataset generation pipeline involves collecting 1,513 human poems from three sources including social media, personal archives, and open-access literary websites <ref:2607. p. 126221#pg4>. For poem generation, three LLMs were selected based on novelty: GPTOSS-120B, Llama 3. p. 13-70B, and Qwen 3-32B <ref:2607. p. 126221#pg5>. The generation process utilizes a Job Queue Formulation with Resume Logic
algorithm to package incomplete generation requests with a Universal Style Guide and a Global Thematic Pool <ref:2607.26221#pg4>.
Feature Extraction and Analysis
Feature extraction is performed on the dataset using vocabulary- and phrase-based features to compare AI-generated and human poems, including total word count, unique words, type-token ratio (TTR), bigrams, trigrams, semantic consistency, prompt repetition, vocabulary overlap, and TFIDF <ref:2607.26221#pg5>. Analysis of word diversity shows that the human corpus achieved the highest TTR compared to the AI models <ref:2607. p. 426221#pg6>. Furthermore, bigram and trigram overlap values remained low across GPT, Llama, and Qwen, indicating that phrase combinations are both model-specific and human-specific <ref:2607. p. 626221#pg8>.
Classification Performance
Gemma 4-31B was used as a zero-shot pipeline to classify human vs AI using five criteria and a Likert scale, which performed best by the holding best Weighted F1 Average <ref:2607.26221#pg9>. Traditional detectors like LogLikelihood [11] performed better than others, but Gemma 4 achieved stronger classification performance than the next best model by 3. p. 75% in Weighted F1 score <ref:2607. p. 726221#pg10>. Survey results showed that paired human evaluators converged on identical classification for only 35. p. 671% of the mutually reviewed corpus <ref:2607. p. 626221#pg9>.
Defining Attributes and Evasion
The research identified Human-like
and Non-Human-like
attributes, which are termed Defining Attributes, contrasting them with Misleading attributes <ref:2607.26221#pg10>. In most cases, the rigid and the cliche nature of GenAI composition aided significantly in detection <ref:2607. p. 826221#pg10>. Conversely, GenAI composition with unique usage of literary devices (Mainly, Metaphors and Imageries) aided in evasion <ref:2607. p. 826221#pg10>. Qwen leveraged this misleading attribute to evade Gemma 4 the most <ref:2607. p. 826221#pg10>.
Conclusion
The key findings indicate that while Gemma 4 performed best, human classification was also poor with only 0.70 precision and a recall of 0. p. 645 in the AI generated portion <ref:2607. p. 026221#pg9>. The study concludes that both human-like GenAI attributes and AI-like human attributes contribute to classification errors <ref:2607. p. 826221#pg10>. Future work includes training Gemma 4-31B with hard negative examples and investigating Qwen 3-32B’s high evasiveness via new detectors <ref:2607. p. 926221#pg10>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
The proposed pipeline should be trained with hard negative examples to reduce training margin, as
Training the model only on such attributes will not only reduce the training margin needed but also gives us a clearer picture of which framework is truly better while detecting GenAI poetry.
-
Implement a mechanism to explicitly check for style guide violations during generation and flag them as potential human-like attributes that might aid evasion, addressing the finding that
humanizing GenAI texts is crucial to evade the detection tools.
-
Enhance the Gemma 4 zero-shot pipeline by incorporating reasoning factor analysis from Algorithm 2 to better understand which of the five criteria most influences classification decisions, aiding in identifying
Defining Attributes
andMisleading attributes.
-
Develop a robust system to mitigate bias towards few-shot examples over user prompts during generation by ensuring that theme separation is maintained even when style guide constraints conflict with few-shot examples, as noted by the observation that
the disconnection between the user given poetry theme and the generated poem’s theme made such separation dificult achieve.
-
For Qwen 3-32B, investigate adding new detectors to its classification pipeline due to its high evasiveness, as suggested by the finding that
Qwen leveraged this misleading attribute to evade Gemma 4 the most.
Sources
- GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge
- Evaluating Diversity in Automatic Poetry Generation
- MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
- POEMetric: The Last Stanza of Humanity
- Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data