GENIE: A Fine-Grained Measure for Novelty

summary

The gist

The gist Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks, and this paper proposes GENIE, a fine-grained evaluation metric that measures response

In short

Large Language Models lack creativity, so this work proposes GENIE, a fine-grained metric to measure response novelty based on task-specific features. It works by automatically discovering features like SETTING and PLOT for creative writing and quantifying how novel a response is across these specific dimensions compared to a reference population.

Key concepts

GENIE: A fine-grained evaluation metric designed to quantify the nuances that differentiate equally novel responses. It measures novelty along task-specific features by comparing a target document against a defined population using generated questions and dissimilarity scores.
Feature Discovery and Mapping: The process of automatically identifying relevant axes for novelty within a specific task, such as creative writing. This is achieved by prompting an LLM multiple times to generate feature descriptions, which are then clustered to reveal recurring characteristics like SETTING or PLOT.
Novelty Score (Gfeat): A score that measures the novelty of a target document along a single task-specific feature. It is calculated as the average novelty across all prompt-specific questions related to that feature, comparing the target document's response against a population of responses using a 4-point Likert scale.
Holistic Metrics vs. GENIE: Traditional holistic metrics struggle to capture the high dimensionality of novelty and fail to show which properties they target. GENIE is shown to be more robust, being sensitive when features are intervened upon and showing better performance against lexical edits like paraphrasing.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Flow Matching — explained in 21 episodes
Quantum Fisher Information (QFI) — explained in 21 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

GENIE: A Fine-Grained Measure for Novelty · Read on arXiv

Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng, Greg Durrett, Junyi Jessy Li

University of Texas at Austin · New York University

Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks. Prior work has focused on addressing whether models are capable of generating creative outputs. Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner. We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features with respect to a population of responses. We show that unlike GENIE, holistic metrics struggle to capture the high-dimensionality of novelty and do not provide insight on which properties they target. Finally, we use GENIE to measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "GENIE: A Fine-Grained Measure for Novelty".

Jane: Here's the gist of the paper. The gist Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks, and this paper proposes GENIE, a fine-grained evaluation metric that measures response novelty along task-specific features with respect to an explicitly defined population.

Jane: GENIE is designed to quantify the fine-grained nuances that differentiate equally novel responses by extracting features present in the responses with respect to a pre-defined population The framework is inspired by alternative semantics of questions literature, where it generates and maps questions to automatically discovered features relevant to the task.

Paper summary: Tom: So we’ve seen that this paper introduces GENIE, which is this fine-grained evaluation metric designed to measure response novelty across task-specific features using an explicitly defined population.

Jane: The main thesis is that unlike holistic metrics, GENIE captures the high dimensionality of novelty and tells you exactly which properties the model targets.

Lu: They investigate what makes content novel or not novel in a way that is tied directly to the task at hand, rather than just looking at a single overall score.

Meng: It sounds like they're trying to get granular instead of general, which is what we need if we want to actually improve model control.

Lalam: They found that by defining novelty as relative dissimilarity across features for every document in the population, they can pinpoint exactly where the model is diverging from what’s already known.

Tom: The methodology involves generating prompt-specific questions for a target document and mapping those questions to automatically discovered features.

Jane: Those features are task-specific axes, found by prompting an LLM several times and then clustering the resulting characteristics, which they say can be as simple as setting and plot in creative writing.

Lu: So they map these discovered features to theoretically defined taxonomies in the domain they are studying so it gives it some real meaning.

Meng: But what is the actual mechanism for this mapping? How do you get those features to align with established concepts?

Lalam: The mapping happens by generating questions for each document and seeing which feature an answer relates to, and then averaging the novelty score across all those questions.

Tom: And they use Qwen2 point 5-32B-Instruct on a four-point Likert scale to compute that dissimilarity between documents along those features.

Jane: The goal here is to get this vector of feature novelty scores, G(dt), which summarizes how novel the document is across all discovered axes.

Lu: The paper shows that this fine-grained approach successfully detects differences in novelty across pairs of responses that minimally differ along these features.

Meng: That means if two outputs are almost identical but differ on a key feature like 'mood' or 'setting', GENIE should flag it immediately.

Lalam: Plus, the authors demonstrate robustness when the responses only differ via paraphrasing, meaning GENIE doesn't flag noise that isn't structural.

Conclusion: Tom: So to wrap up, this paper presents "GENIE: A Fine-Grained Measure for Novelty", which is a fine-grained evaluation metric for novelty in LLM responses against a defined population.

Jane: The key implication is that it moves us away from holistic metrics that don't show us which properties the model is targeting when it generates content.

Lu: By breaking down novelty into task-specific features, we gain insight into the specific creative elements—like plot or characterization—the model is succeeding or failing at.

Meng: For deployment, this gives us a tool to test mitigation methods more effectively because we can see precisely where an alteration actually boosts novelty scores.

Lalam: So in simple terms, GENIE tells you not just that something is novel, but *why* it's novel by breaking it down into measurable parts.

Tom: The authors showed that this approach is sensitive to feature changes when tested independently and doesn't get fooled by superficial wording changes.

Jane: It suggests a way forward for evaluating generative AI where we stop relying on one big score and start looking at the components of creativity separately.

More episodes

← Home