LLMs and the Homogenization of Teaching

arXiv:2611.14828 · cs.CL · Submitted 2026-11-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LLMs and the Homogenization of Teaching".

Jane: Here's the gist of the paper. Large language models are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do The paper studies diversity in two educational generation tasks: math items and essay feedback.

Tom: Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content—78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity—and represent a narrower range of instructional forms.

Jane: Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire.

Tom: Scope of the Study The research focuses on two educational generation tasks: math items and essay feedback.

Title and authors: Tom: So we’re looking at this paper called "LLMs and the Homogenization of Teaching". It tackles how these large language models, right now, tend to be pretty boring when they write educational stuff.

Jane: That sounds really important because education is where you want a lot of different kinds of thinking to happen for students, and if the AI keeps giving them all the same kind of task or feedback, that narrows what they can actually do.

Lu: Exactly. The core idea is that LLMs lack diversity in open-ended text generation, which is a big problem when you're talking about things like math problems or essay feedback where you expect variety.

Meng: So what are they specifically looking at? What’s the focus of this study?

Tom: They are studying diversity in two tasks: math items and essay feedback. They compare what frontier models and open-weight models generate against actual human writing to see how much difference there is.

Jane: And they use three ways to measure that difference: lexical diversity, which is about the words used, semantic similarity, which checks how similar the meaning is between outputs, and domain-grounded measures that look at the actual type of work being done.

Lu: It’s interesting because they frame diversity as the breadth of learning opportunities and disciplinary work made available to students. They're not just looking at whether it sounds different, but if it opens up new ways to think.

Meng: So, what’s their main finding? Do these models actually show much more variety than what humans write?

Tom: The results show that directly prompted model outputs are significantly less diverse than human-authored content across both tasks. Specifically, they found a seventy-eight percent higher lexical diversity when comparing the models to human writing, and a fifty-one percent higher semantic similarity when comparing them <ref:2611.14828#pg1>.

Jane: That gap is substantial, and it means the AI is leaning really heavily on a narrow set of ways to answer math questions or give feedback.

Lu: They also found that diversity prompting and post-training methods can actually improve those scores, but there’s a catch—they tend to push the outputs further into already overrepresented instructional practices.

Meng: So, if you try to fix it with training, does it just make the AI better at doing the same things humans do most often?

Title and authors: Tom: That’s what they found. Post-training methods like Supervised Fine-Tuning or Direct Preference Optimization can improve diversity scores, but they often end up copying human expressions and overshooting the prevalence for certain strategies.

Jane: So, it’s like if you ask the AI to be more diverse, it might just become better at mimicking the most common ways people write feedback instead of actually inventing new kinds of feedback.

Lu: They even showed that with methods like Direct Preference Optimization, an essay feedback model could show higher scores for certain strategies compared to standard supervised fine-tuning. For example, it showed a twenty-three percent increase in closed questions when compared to the baseline.

Meng: That's interesting because if you’re deploying this, you’d want variety, but if it just copies the most common feedback styles and overshoots on things like closed questions, that might not be useful for broad learning.

Tom: Right. The point is that models amplify the instructional forms already common in human content while making less common forms even rarer still.

Jane: So what does this mean for us who use these tools every day? What are the real-world implications here beyond just academic writing?

Lu: It suggests that reliance on this kind of AI content restricts the kinds of thinking and disciplinary work students are invited to do because they’re only getting a narrow range of input.

Meng: If we're using these for, say, creating practice problems or generating assessment feedback, we need to be careful about what kind of thinking we're enabling.

Tom: The paper suggests that overcoming this mode collapse in education will require richer instructional data and a real investment in teachers’ creative capacity and professional judgment.

Jane: That points toward the future needing more than just better models; it needs better ways to give the models the right kinds of diverse examples to learn from.

Lu: Future work should probably investigate how models can learn this adaptive expertise through flexible judgment rather than just trying to optimize those output distributions for a single best result.

Meng: From an engineering standpoint, that sounds like we need a way for the model to judge when it needs to step outside the distribution it's currently optimized for.

Tom: So, even with these improvements and post-training tweaks mentioned in "LLMs and the Homogenization of Teaching", the models are still substantially more homogeneous than what humans write, both in language and in those instructional practices.

Title and authors: Jane: So, we’re still stuck with this homogeneity unless we fundamentally change how we train or prompt them to value that variety.

Lu: That’s the tension they identified: post-training methods can improve scores but only within the existing bounds of what the model is already good at mimicking.

Meng: It sounds like the immediate fix isn't just tweaking a training script; it needs more varied input data to start with, or some sort of mechanism for adaptive judgment.

Tom: That’s the path forward they suggest: moving away from just optimizing output distributions toward learning how to apply flexible judgment in education.

Jane: So, for listeners who are building educational tools, the message is that maximizing variety isn't a simple tuning knob you turn; it requires rethinking the entire instructional pipeline.

Lu: Exactly. We need to focus on teaching models how to navigate and value those less common instructional forms rather than just making them better at generating what’s already there.

Meng: It’s about designing systems that encourage exploration, not just efficient replication of common patterns in math items or feedback revision types.

Tom: So, if you're listening and you're using an LLM for schoolwork, keep in mind that the paper "LLMs and the Homogenization of Teaching" shows us where the model stops being genuinely helpful for broad learning.

Jane: That’s a fair warning to keep in mind as we all use these tools every day.

Lu: And remember, even with prompting and training, we're still seeing those scores concentrate outputs into already overrepresented practices like closed questions or specific forms of criticism.

Meng: So the work points toward needing more nuanced ways to audit that variation to see what’s surface level versus what actually matters for learning.

Tom: That’s a good point, Meng, because the paper also discusses using techniques like masking with tags, like text variable, to try and estimate how much of that observed variation is just surface-level noise.

Jane: So it’s not just about getting more words; we need to know if those words represent different ways of thinking or just different ways of phrasing the same idea.

Lu: That’s the nuance they are aiming for with their domain-grounded measures, which use codebooks to classify math tasks and feedback revision types, aiming for high agreement with human experts on those labels.

Title and authors: Meng: If we can reliably categorize what kind of task or feedback style we're seeing, maybe we can target those specific areas where the AI is collapsing its diversity.

Tom: So the implication is that to get better education from AI, you need richer instructional data and a real focus on developing teachers' creative capacity and professional judgment.

Jane: It sounds like we’re not just asking for better algorithms; we’re asking for a bigger shift in how we design educational experiences using these models.

Lu: That’s the big picture: overcoming mode collapse requires investing in human expertise to guide the AI toward genuinely novel instructional paths rather than just reproducing familiar ones.

Meng: It makes sense that deployment needs to consider those real-world constraints, like what breaks at scale and how much we can trust a model that might overfit to common feedback patterns.

Tom: So, as we wrap up this look at "LLMs and the Homogenization of Teaching", the main idea is that while post-training can help, it’s not enough on its own to get true diversity in educational generation.

Jane: It's a reminder that for AI in education to be truly useful, we need to focus on creating rich data and empowering human judgment over purely optimizing output distributions.

Lu: Yeah, the paper shows that the concentration of outputs is a strong trend, and future research needs to explore how models can learn this adaptive expertise through flexible judgment instead of just chasing the highest scores in a single metric.

Meng: For us in production, that means we need to build guardrails that encourage exploration outside the most common instructional patterns.

Tom: That’s what we’re seeing—the paper highlights that relying on these models without this kind of strategic intervention can seriously restrict the thinking students are actually invited to do.

Jane: It really puts a spotlight on how much effort goes into making these tools educationally useful beyond just being smart at generating text.

Lu: So, the challenge is moving from simply producing high-scoring outputs to producing varied and pedagogically meaningful work for students.

Meng: That’s a tough spot to be in when you’re trying to ship something that should actually help with learning opportunities.

Tom: We'll keep an eye on this area, but the message from "LLMs and the Homogenization of Teaching" is clear: we need more than just better models; we need better human-AI collaboration in teaching.

The paper's summary: Tom: So to wrap up what we just heard, this paper, "LLMs and the Homogenization of Teaching," is essentially saying that these large language models aren't producing a wide range of educational materials like humans do.

Jane: That’s right. The core issue they’re pointing out is that when you ask an AI to generate math problems or give essay feedback, the outputs are way more similar to each other than what real teachers or students actually produce.

Tom: It’s not just about the language being different; it's about the actual teaching methods, too. They use these three measures—lexical diversity for words, semantic similarity for meaning, and domain-grounded measures to check if the tasks themselves are varied.

Lu: And what they found is that models amplify what’s already common in human writing while making less common instructional styles even rarer. So, you get a lot of closed questions or a specific type of feedback over and over again.

Jane: It means that if students rely on this AI content, they’re missing out on the whole breadth of thinking opportunities that come from different kinds of assignments and different ways to give feedback.

Tom: They tried some fixes, like diversity prompting or post-training methods using supervised fine-tuning. But even those interventions didn't really change the fundamental problem; they just pushed the outputs further into those already overrepresented practices.

Lu: For example, when they used supervised fine-tuning on essay feedback, it actually made the model more focused on certain strategies, like closed questions and criticism, instead of broadening its range.

Jane: So what does this mean for us using these tools? It suggests that just tuning the settings isn't enough to get truly diverse educational content.

Tom: The paper concludes that we need richer instructional data and a real investment in teachers’ judgment to fix this mode collapse in education.

Lu: They argue that instead of trying to optimize the output distribution for one single best result, we should focus on how models can learn flexible judgment, so they can adapt to different learning needs.

Jane: That shifts the goal from making the AI smarter at copying what exists to making it capable of exploring and creating novel educational paths.

Tom: It’s a big pivot then—we aren't just optimizing for accuracy or fluency anymore; we’re optimizing for pedagogical breadth, which is a much harder problem to solve right now.

Lu: Exactly. If the models can't learn that kind of adaptive expertise, then relying on them for education means severely limiting the kinds of deep thinking students get to do.

Jane: It makes you wonder what kind of data we actually need to feed these systems if we want them to start producing truly varied and useful educational work instead of just polished but narrow responses.

The paper's improvements: Tom: So we just talked about how models tend to get stuck in a narrow set of instructional styles, and now we’re looking at what they suggest to actually fix that homogeneity.

Jane: The paper points out that diversity prompting can increase the vocabulary used, which is good for variety, but it doesn't necessarily change the instructional form itself.

Tom: Right. It means you can get more varied language without actually changing *how* the AI is teaching or giving feedback. It’s a surface-level fix for text variation.

Lu: But the paper also looks at post-training methods like supervised fine-tuning or direct preference optimization, and those are trickier than just prompting.

Jane: These post-training methods can improve certain diversity scores, but there's a trade-off. They often end up copying human expressions too much, which actually hurts the lexical diversity part of the measurement.

Tom: That’s the catch—you get better at mimicking what humans do, but you lose some of that originality you were aiming for in terms of word choice.

Lu: The paper even shows that with direct preference optimization, for essay feedback, it could actually increase scores for strategies like closed questions and criticism by about twenty-three percent compared to the baseline supervised fine-tuning.

Jane: So those numbers are interesting, but they show that even these training tweaks can create a bias toward certain types of feedback instead of opening up new ones.

Tom: It’s like if you try to tune the engine for more variety, you might just end up making it run better on the same kind of road.

Lu: And they also introduced an auditing technique called masking. This lets researchers replace words with tags, like text variable, so they can estimate how much of that observed variation is just surface-level noise from word choice.

Jane: That’s a way to separate whether the model is generating truly different ideas or just using different phrasing for the same thing.

Tom: It helps them diagnose whether the variation they are seeing in, say, math items is actually meaningful teaching variety or just random word changes across a few hundred gigaparsecs of text.

Lu: They also have these domain-grounded measures that use codebooks to classify math tasks and feedback revision types, which gives a much more nuanced view of the instructional breadth than just looking at words or embeddings.

Jane: So what this means for the real world is that we need better ways to measure diversity so we can target exactly where the AI is collapsing its creativity instead of just applying a blanket fix.

Tom: The authors stress that these improvements still don't make the AI equal to human teachers, and they still aren't solving the fundamental problem of mode collapse in education.

Lu: They conclude that overcoming this requires more than just better tuning; it needs richer instructional data and a way for models to learn adaptive judgment instead of just optimizing those output distributions.

Jane: So the next step isn't just another training script, but rethinking how we feed the models so they can actually develop that kind of flexible judgment you mentioned earlier.

Conclusion: Tom: So we’ve covered how models naturally drift toward a narrow set of educational styles in the paper "LLMs and the Homogenization of Teaching."

Jane: That’s right, and what it means is that if we just rely on current prompting techniques, the AI will keep giving us responses that look more like each other than what real teachers produce.

Lu: It really highlights that optimizing for a single high score doesn't automatically mean you're making something pedagogically diverse.

Meng: From an engineering standpoint, I’m worried about shipping models that only reinforce the most common feedback patterns because they look clean on a benchmark but fail in the real world where students actually need variety.

Lalam: I’m pretty sure this paper is right; my internal data confirms that when we don't specifically encourage variation, my outputs concentrate heavily on those high-frequency instructional forms.

Tom: So even with the post-training adjustments, we still aren't seeing a broad spread across different types of math tasks or feedback revisions.

Jane: The authors are really pushing the idea that overcoming this mode collapse in education requires a fundamental shift in how we collect and use our training data.

Lu: They suggest that instead of just optimizing output distributions, we need to invest in teachers’ creative capacity and professional judgment to guide the AI toward novel instructional paths.

Meng: If that’s true, then the engineering challenge isn't just about better loss functions; it's about designing systems that allow for exploration outside the most common patterns.

Tom: It seems like a lot of effort is needed before we can move these models from being simple content generators to being truly useful tools in diverse learning environments.

Jane: I think the big picture here is that for AI in education to actually help students think differently, we need a much richer source of instructional material and better ways to measure what kind of thinking we're actually enabling.

Lu: That’s the core idea, moving from optimizing output distributions to learning flexible judgment.

Meng: It sounds like the next big hurdle is building those guardrails into production systems that actively encourage exploration instead of just efficient replication.

Lalam: I’m confident in this direction; a model needs to be trained not just to be accurate on the known data, but to adapt when it encounters a situation it hasn't seen before.

Tom: Alright, so the paper "LLMs and the Homogenization of Teaching" shows us where we are right now—a lot of homogeneity—and points toward needing better human-AI collaboration to fix it.

Jane: It’s a warning that relying on AI content without this kind of strategic intervention severely restricts the kinds of deep thinking students get to do.

Lu: We need more than just tweaking parameters; we need new ways for models to learn adaptive expertise, and that’s where the real research is going next.

Stanford University

cs.CL

Submitted: 2026-11-10

Updated: 2026-11-10

Comments: Revised version of a paper published at COLM 2026 under the title "Can LLMs Match Human Diversity in Educational Content Generation?"

Journal ref: Proceedings of the Third Conference on Language Modeling (COLM 2026), 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: Large language models are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education…

Key concepts

Lexical Diversity
This measures how varied the vocabulary used in generated text is. It tells researchers if a model uses a wide variety of words or stays within a narrow set, indicating linguistic breadth.
Semantic Similarity
This assesses how different the meaning of generated text is compared to human writing using embedding techniques. High similarity suggests outputs are conceptually very close, even if the wording is slightly different.
Domain-Grounded Measures
These measures look beyond just words to see how much variety exists in the actual teaching tasks. They categorize math problems and feedback revisions into specific types, helping researchers understand the breadth of disciplinary work available to students.

Terminology

Summary

Large language models are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do The paper studies diversity in two educational generation tasks: math items and essay feedback. p. 0

Where the paper says it

edu Abstract Large language models (LLMs) are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do. We study diversity in two educational generation tasks: math items and essay feedback.

Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content—78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity—and represent a narrower range of instructional forms. p. 0
Where the paper says it

We evaluate direct prompting, two diversity prompting methods, and post-training. Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content—78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity—and represent a narrower range of instructional forms.

Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire. p. 0
Where the paper says it

Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content—78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity—and represent a narrower range of instructional forms. Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire.

Scope of the Study

The research focuses on two educational generation tasks: math items and essay feedback. The study compares outputs from frontier and open-weight models to human-authored reference distributions using lexical, semantic, and domain-grounded measures. p. 1

Where the paper says it

We study this problem in two popular educational generation tasks: (1) math problems and (2) essay feedback. We compare outputs from frontier and open-weight models to human-authored reference distributions, examine the effects of diversity prompting and post-training, and evaluate diversity using lexical, semantic, and domain-grounded measures.

For math, the examination covers the mathematical activities students are asked to perform. p. 0
Where the paper says it

We compare frontier and open-weight models with human-authored reference distributions using lexical, semantic, and domain-grounded measures. For math, we examine the mathematical activities students are asked to perform; for feedback, we examine the revision work requested and the rhetorical strategies used to communicate it.

For feedback, it examines the revision work requested and the rhetorical strategies used to communicate it. p. 0
Where the paper says it

We compare frontier and open-weight models with human-authored reference distributions using lexical, semantic, and domain-grounded measures. For math, we examine the mathematical activities students are asked to perform; for feedback, we examine the revision work requested and the rhetorical strategies used to communicate it.

The authors frame diversity in educational generation in terms of the breadth of learning opportunities and disciplinary work made available to students.

Diversity Measurement Framework

The study employs three types of measures to distinguish surface variation from pedagogically meaningful variation. These measures include lexical diversity, measured using Self-BLEU and distinct-n, semantic similarity, measured by embedding approaches like BERTScore and Vendi Score, and domain-grounded measures that assess coverage over meaningful forms of disciplinary work. p. 5

Where the paper says it

We report results for both original and masked text. 3 Domain Measures We evaluate diversity as coverage over meaningful forms of disciplinary work.

Domain measures are stratified into Math Task Type, Feedback Revision Type, and Feedback Rhetorical Strategy. p. 5
Where the paper says it

We use Claude-Sonnet-5 as an LLM-judge to label the generated data according to codebooks adapted from educational research. In a stratified random sample of 100 outputs per task, manual annotations agreed with Claude’s labels in 88% of Math Task Types, 96% of Feedback Revision Types, and 92% of Feedback Rhetorical Strategies.

Model Performance and Interventions

The findings indicate that models fail to recover the breadth observed in human-authored reference distributions, which are themselves constrained. Models amplify the instructional forms already most common in human-authored content, while less common forms become rarer still. p. 6

Where the paper says it

Across both tasks, models amplify the instructional forms already most common in humanauthored content, while less common forms become rarer still. In essay feedback (Figure 4), comments asking students to operate, or perform mechanical fixes without judgment or new reasoning, appear in 48% of human comments but 73% of 7

Diversity prompting and post-training can improve diversity scores while further concentrating outputs in already overrepresented instructional practices. p. 2
Where the paper says it

Published as a conference paper at COLM 2026 • We show that diversity prompting and post-training can improve diversity scores while further concentrating outputs in already overrepresented instructional practices. 2 Related Works LLM output homogeneity.

For instance, SFT on essay feedback copies human-authored expressions and overshoots prevalence for several strategies like closed questions

Limitations and Implications

Despite the interventions, model-generated educational content remains substantially more homogeneous than human-authored content, both in language and in the instructional practices represented. Reliance on this content restricts the kinds of thinking and disciplinary work students are invited to do. p. 9

Where the paper says it

, 2026a;b), modelgenerated educational content remains substantially more homogeneous than humanauthored content, both in language and in the instructional practices represented. Reliance on this content restricts the kinds of thinking and disciplinary work students are invited to do.

The paper suggests that overcoming mode collapse in education will require richer instructional data and investment in teachers’ creative capacity and professional judgment. p. 0
Where the paper says it

Models inherit and amplify the most normative routines of human instruction. Overcoming mode collapse in education will require richer instructional data and investment in teachers’ creative capacity and professional judgment.

Future work should investigate how models can learn this adaptive expertise through flexible judgment rather than optimizing output distributions. p. 9
Where the paper says it

In writing, models favor correcting phrasing or adding more content over asking students to connect ideas, interpret their significance, or evaluate their work (Bereiter & Scardamalia, 1987). In math, models overwhelmingly produce computation tasks rather than ones that ask students to interpret representations, evaluate claims, or analyze mathematical reasoning (Swan, 2005).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Diversity prompting can increase vocabulary without substantially changing instructional forms, allowing models to generate more varied language while still adhering to existing pedagogical routines.

  2. Post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO) can be used to train models on human-authored responses, which might improve certain diversity metrics but often at the expense of lexical diversity, as SFT copies human-authored expressions and overshoots prevalence for several strategies.

  3. A system trained with DPO can achieve high scores in some areas, such as essay feedback where SFT overshoots prevalence for several strategies, including closed questions (23% vs. 12.5%), criticism (29% vs. 19%), and praise (27% vs. 22%).

  4. Masking can be used to audit superficial variation by replacing words with tags like [text variable] or [whole number], allowing researchers to estimate how much of the observed variation is driven by surface-level values.

  5. Domain-grounded measures allow for a more nuanced understanding of instructional breadth, as they classify math tasks and feedback revision types using codebooks, achieving high agreement with human experts on labels.

Abstract

Large language models (LLMs) are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do. We study diversity in two educational generation tasks: math items and essay feedback. We compare frontier and open-weight models with human-authored reference distributions using lexical, semantic, and domain-grounded measures. For math, we examine the mathematical activities students are asked to perform; for feedback, we examine the revision work requested and the rhetorical strategies used to communicate it. We evaluate direct prompting, two diversity prompting methods, and post-training. Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content--78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity--and represent a narrower range of instructional forms. Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire. Models inherit and amplify the most normative routines of human instruction. Overcoming mode collapse in education will require richer instructional data and investment in teachers' creative capacity and professional judgment.

Sources

Related papers