Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responses

arXiv:2606.06788 · cs.CL, cs.HC · Submitted 2026-06-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Explain Like I'm 5 or Whatever I Choose".

Jane: The gist The researchers propose a new evaluation framework to test large language models' ability to generate multiple,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So they start by setting up this study design. They found that testing models in this way requires looking beyond just one static output and instead focusing on the relationship between different versions of a response.

Jane: Right, and the setup involves generating five responses for ninety-eight scientific queries across five different AI models—GPT-five point one, GPT-five mini, Claude Sonnet four point five + Thinking, and DeepSeek-V3 point 1—to see how they handle that complexity shift <ref:2606.06788#pg1,GPT-5.1, GPT-5 mini, Claude Sonnet 4.5 + Thinking, and>.

Lu: They also developed three specific ways to measure this complexity: JARGON for the proportion of text made up of less familiar words, INFORMATION for identifying independent facts, and LENGTH for the total number of characters in the response.

Meng: I wonder about those measures. Measuring jargon is easy enough, but how reliable are those metrics when you're dealing with different models? That sounds like a big challenge for scaling this kind of testing up to real traffic.

Lalam: The paper defines performance by looking at whether the models increase these complexity measures in the direction you ask them to. They found that while models can vary complexity, most of those changes are inconsistent across all levels.

The paper's summary: Tom: So what they actually found is that despite the ability to vary responses, most of the changes they made weren't reliable or consistent when you moved from one level of complexity to another.

Jane: That inconsistency is a big deal because it means if you ask for a simpler answer, you can't be sure it will reliably get simpler, or that the information density will predictably change.

Lu: They specifically pointed out that the best model for changing jargon, Claude Sonnet four point five, only changed jargon in the right direction forty-six percent of the time across all five responses for a single query <ref:2606.06788#pg1>.

Meng: Forty-six percent is low when you're trying to build something that needs to be predictable at scale. It suggests that even the top performers aren't perfectly reliable when it comes to controlling output style based on complexity requests.

Lalam: They also found that in higher complexity levels, the changes in those measures actually started to approach chance level for most models. They showed this trend holds true even when they test with a bigger sample size of four hundred fifty-nine participants.

The paper's improvements: Tom: The authors suggest a few ways to improve this framework, focusing on making the complexity control more robust, especially since the initial results showed inconsistencies.

Jane: They want to move beyond just measuring changes and focus on a criterion where models must show positive changes across all measures when you increase the intended complexity level.

Lu: They also suggest testing this framework across different audience labels, like those from the WIRED five Levels series, to see if that trend of complexity moving in the correct direction holds true for different user groups <ref:2606.06788#pg1>.

Meng: Testing with audience labels makes sense because it grounds it in real-world usage. If the behavior changes based on who you are talking to, we need to know if this framework still works when those contexts change.

Lalam: They also pointed out a decoupling effect: sometimes an increase in length doesn't automatically mean an increase in complexity; for instance, length can go up while jargon and information actually decrease.

Conclusion: Tom: So to wrap up the paper "Explain Like I'm five or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responses," it seems the main implication is that we need a better way to evaluate these models when we ask them to generate multiple versions of an answer <ref:2606.06788#pg1,Explain Like I'm 5 or Whatever I Choose: Evaluating the>.

Jane: The paper concludes that while these models can vary complexity, the current evaluation methods don't capture how often those changes are actually reliable across different levels. They need a framework that demands consistency in the direction of change for all relevant measures.

Lu: The authors argue that by focusing on a criterion where models must consistently increase all intended complexity measures, we get a clearer picture of their true capability in this area.

Meng: For deployment, this means we can't just look at one model's performance; we need to test the entire response generation process against these specific directional goals to make sure it meets user expectations for varying detail.

Lalam: And as a final point, the paper suggests that models might be good at length but fail when complexity shifts in ways that reduce jargon or information, which is an important limitation to keep in mind.

Tom: So they're pushing for a more rigorous way to test these interactive capabilities, making sure the changes they see are actually meaningful and predictable for the user.

Indu Panigrahi, Tal August

Siebel School of Computing and Data Science University of Illinois Urbana-Champaign

cs.CL, cs.HC

Submitted: 2026-06-05

Updated: 2026-10-14

Comments: Preprint

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 88/100

The gist: The gist The researchers propose a new evaluation framework to test large language models' ability to generate multiple, distinct versions of a response across an interpretable axis of language

Key concepts

Direct Manipulation: This refers to allowing a user to directly control or change an output generated by an AI, such as editing text. In this study, it means users could select or request specific levels of complexity for the AI's response.
Complexity Measures: These are quantifiable ways to measure how difficult or simple a piece of text is. The study defined three: JARGON (how many hard words are used), INFORMATION (how many distinct facts are present), and LENGTH (the total number of characters). These measures help researchers track if the model is successfully changing the text's difficulty.
Within-Subjects User Study: This type of study involves having every participant experience every condition being tested. In this case, participants explored scientific topics and interacted with a system that let them manipulate response complexity. This allowed researchers to reliably determine what users consider a good or desired level of complexity.

Terminology

Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes

Summary

The gist The researchers propose a new evaluation framework to test large language models' ability to generate multiple, distinct versions of a response across an interpretable axis of language complexity for scientific information seeking tasks.

How it works

A formative study was conducted with 16 participants who explored unfamiliar STEM topics using a prototype chat interface that enabled direct manipulation of response language complexity
The framework evaluates models by generating 5 responses at different levels of language complexity to 98 scientific queries across five recent models: GPT-5.1, GPT-5 mini, Claude Sonnet 4.5 + Thinking, and DeepSeek-V3.1
Three complexity measures were curated from prior work and grounded in the user study: JARGON (proportion of text that consists of less familiar words), INFORMATION (identifying independent facts), and LENGTH (total number of response characters)
Model performance is defined by the relationship between versions of a response based on the criterion that models increase the measures as intended complexity levels increase

Key Findings from Evaluation

While models are able to vary complexity across responses, most changes remain inconsistent

the best performing model for jargon change (Claude Sonnet 4.5) only changed jargon in the correct direction 46% of the time

most changes remain inconsistent, with the best performing model (Claude Sonnet 4.5) only shifting reliable complexity measures in the correct direction 46% of the time

"In fact, 7 participants (4 of whom had not yet seen the interactive condition) ended up describing their preferred level of complexity in follow-up prompts, indicating a strong desire for control over perceived complexity of responses"

As participants decreased complexity, they prioritized three primary, desired trends: decreases in jargon (16/16), amount of information (13/16), and length (12/16)

Model Performance Trends

All models vary between increasing and decreasing complexity measures when generating multiple levels

LENGTH generally increases with increasing complexity, as is the intended trend

Models increase length across the 5 levels for the majority of their inputs (e.g., DeepSeek-V3.1 increases length across all levels for 98.98% of the inputs)

**"When directly examining responses, we notice that cases where length increases on its own (i.e.

Improvements for AI systems

Improved Evaluation Framework for Multi-Response Generation: The system can be evaluated based on the relationship between versions of a response, rather than a single output, to test models’ ability to generate multiple, distinct versions of a response across a dimension of interest.
Enhanced Complexity Control via Direct Manipulation: The AI system can be equipped with an interface that allows users to directly manipulate language complexity, as demonstrated by the study where participants found the responses from the conventional chat interface inconsistent in terms of complexity and preferred having a slider mechanism to adjust the language complexity of chat responses.
Refined Complexity Measurement Criterion: The system's performance should be judged by its ability to adhere to a specific criterion: Models that are better at generating distinct levels of complexity will produce positive changes in all measures, operationalized as measuring the direction of changes in the measures between the 5 levels of text that models generate for each query.
Differentiated Complexity Measure Focus: The system should be optimized to reliably adjust complexity across responses, focusing on JARGON, INFORMATION, and LENGTH while acknowledging that models often fail to reliably adjust complexity measures in the correct direction across versions.
Adaptive Audience Label Testing: The AI system can be tested across different audience labels—such as those from the WIRED 5 Levels series—to identify performance variations, noting that the same trend [of complexity measures going in the correct direction] holds while exploring how audience labels affected the similarity between levels.
Length-Complexity Decoupling: The system must be trained to recognize when increased length does not equate to increased complexity, as evidenced by instances where an increase in length may not always indicate an increase in complexity, especially when JARGON and INFORMATION decrease.

Abstract

Given the increased use of large language models (LLMs) for reading and seeking information and the varying needs and backgrounds of users, direct manipulation of text can be a valuable mechanism for end-user control. However, enabling direct manipulation requires that models generate text consistent with user-requested changes. We focus on one such setting, manipulating the complexity of scientific information, and ask if models can generate multiple summaries along a continuum of text complexity that enable effective user selection and control. Through a within-subjects user study, we identify reliable complexity measures and a criterion for how models should conform to these measures. We evaluate five LLMs of different families, sizes, and reasoning abilities on 557 scientific queries. We find that while models can vary complexity across responses, most changes remain inconsistent, with models shifting the complexity measures in the correct direction at most 33.67 % of the time.