LLMs and the Homogenization of Teaching

summary

The gist

Large language models are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education…

In short

The study examined how large language models generate educational material, finding their outputs are significantly less diverse than human content across math problems and essay feedback. Interventions like diversity prompting or post-training methods fail to broaden the actual range of teaching practices, meaning models still narrow student learning opportunities.

Key concepts

Lexical Diversity
This measures how varied the vocabulary used in generated text is. It tells researchers if a model uses a wide variety of words or stays within a narrow set, indicating linguistic breadth.
Semantic Similarity
This assesses how different the meaning of generated text is compared to human writing using embedding techniques. High similarity suggests outputs are conceptually very close, even if the wording is slightly different.
Domain-Grounded Measures
These measures look beyond just words to see how much variety exists in the actual teaching tasks. They categorize math problems and feedback revisions into specific types, helping researchers understand the breadth of disciplinary work available to students.

Terminology used across episodes

This episode discusses

The paper

LLMs and the Homogenization of Teaching · Read on arXiv

Stanford University

Large language models (LLMs) are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do. We study diversity in two educational generation tasks: math items and essay feedback. We compare frontier and open-weight models with human-authored reference distributions using lexical, semantic, and domain-grounded measures. For math, we examine the mathematical activities students are asked to perform; for feedback, we examine the revision work requested and the rhetorical strategies used to communicate it. We evaluate direct prompting, two diversity prompting methods, and post-training. Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content--78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity--and represent a narrower range of instructional forms. Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire. Models inherit and amplify the most normative routines of human instruction. Overcoming mode collapse in education will require richer instructional data and investment in teachers' creative capacity and professional judgment.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LLMs and the Homogenization of Teaching".

Jane: Here's the gist of the paper. Large language models are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do The paper studies diversity in two educational generation tasks: math items and essay feedback.

Tom: Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content—78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity—and represent a narrower range of instructional forms.

Jane: Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire.

Tom: Scope of the Study The research focuses on two educational generation tasks: math items and essay feedback.

Title and authors: Tom: So we’re looking at this paper called "LLMs and the Homogenization of Teaching". It tackles how these large language models, right now, tend to be pretty boring when they write educational stuff.

Jane: That sounds really important because education is where you want a lot of different kinds of thinking to happen for students, and if the AI keeps giving them all the same kind of task or feedback, that narrows what they can actually do.

Lu: Exactly. The core idea is that LLMs lack diversity in open-ended text generation, which is a big problem when you're talking about things like math problems or essay feedback where you expect variety.

Meng: So what are they specifically looking at? What’s the focus of this study?

Tom: They are studying diversity in two tasks: math items and essay feedback. They compare what frontier models and open-weight models generate against actual human writing to see how much difference there is.

Jane: And they use three ways to measure that difference: lexical diversity, which is about the words used, semantic similarity, which checks how similar the meaning is between outputs, and domain-grounded measures that look at the actual type of work being done.

Lu: It’s interesting because they frame diversity as the breadth of learning opportunities and disciplinary work made available to students. They're not just looking at whether it sounds different, but if it opens up new ways to think.

Meng: So, what’s their main finding? Do these models actually show much more variety than what humans write?

Tom: The results show that directly prompted model outputs are significantly less diverse than human-authored content across both tasks. Specifically, they found a seventy-eight percent higher lexical diversity when comparing the models to human writing, and a fifty-one percent higher semantic similarity when comparing them <ref:2611.14828#pg1>.

Jane: That gap is substantial, and it means the AI is leaning really heavily on a narrow set of ways to answer math questions or give feedback.

Lu: They also found that diversity prompting and post-training methods can actually improve those scores, but there’s a catch—they tend to push the outputs further into already overrepresented instructional practices.

Meng: So, if you try to fix it with training, does it just make the AI better at doing the same things humans do most often?

Title and authors: Tom: That’s what they found. Post-training methods like Supervised Fine-Tuning or Direct Preference Optimization can improve diversity scores, but they often end up copying human expressions and overshooting the prevalence for certain strategies.

Jane: So, it’s like if you ask the AI to be more diverse, it might just become better at mimicking the most common ways people write feedback instead of actually inventing new kinds of feedback.

Lu: They even showed that with methods like Direct Preference Optimization, an essay feedback model could show higher scores for certain strategies compared to standard supervised fine-tuning. For example, it showed a twenty-three percent increase in closed questions when compared to the baseline.

Meng: That's interesting because if you’re deploying this, you’d want variety, but if it just copies the most common feedback styles and overshoots on things like closed questions, that might not be useful for broad learning.

Tom: Right. The point is that models amplify the instructional forms already common in human content while making less common forms even rarer still.

Jane: So what does this mean for us who use these tools every day? What are the real-world implications here beyond just academic writing?

Lu: It suggests that reliance on this kind of AI content restricts the kinds of thinking and disciplinary work students are invited to do because they’re only getting a narrow range of input.

Meng: If we're using these for, say, creating practice problems or generating assessment feedback, we need to be careful about what kind of thinking we're enabling.

Tom: The paper suggests that overcoming this mode collapse in education will require richer instructional data and a real investment in teachers’ creative capacity and professional judgment.

Jane: That points toward the future needing more than just better models; it needs better ways to give the models the right kinds of diverse examples to learn from.

Lu: Future work should probably investigate how models can learn this adaptive expertise through flexible judgment rather than just trying to optimize those output distributions for a single best result.

Meng: From an engineering standpoint, that sounds like we need a way for the model to judge when it needs to step outside the distribution it's currently optimized for.

Tom: So, even with these improvements and post-training tweaks mentioned in "LLMs and the Homogenization of Teaching", the models are still substantially more homogeneous than what humans write, both in language and in those instructional practices.

Title and authors: Jane: So, we’re still stuck with this homogeneity unless we fundamentally change how we train or prompt them to value that variety.

Lu: That’s the tension they identified: post-training methods can improve scores but only within the existing bounds of what the model is already good at mimicking.

Meng: It sounds like the immediate fix isn't just tweaking a training script; it needs more varied input data to start with, or some sort of mechanism for adaptive judgment.

Tom: That’s the path forward they suggest: moving away from just optimizing output distributions toward learning how to apply flexible judgment in education.

Jane: So, for listeners who are building educational tools, the message is that maximizing variety isn't a simple tuning knob you turn; it requires rethinking the entire instructional pipeline.

Lu: Exactly. We need to focus on teaching models how to navigate and value those less common instructional forms rather than just making them better at generating what’s already there.

Meng: It’s about designing systems that encourage exploration, not just efficient replication of common patterns in math items or feedback revision types.

Tom: So, if you're listening and you're using an LLM for schoolwork, keep in mind that the paper "LLMs and the Homogenization of Teaching" shows us where the model stops being genuinely helpful for broad learning.

Jane: That’s a fair warning to keep in mind as we all use these tools every day.

Lu: And remember, even with prompting and training, we're still seeing those scores concentrate outputs into already overrepresented practices like closed questions or specific forms of criticism.

Meng: So the work points toward needing more nuanced ways to audit that variation to see what’s surface level versus what actually matters for learning.

Tom: That’s a good point, Meng, because the paper also discusses using techniques like masking with tags, like text variable, to try and estimate how much of that observed variation is just surface-level noise.

Jane: So it’s not just about getting more words; we need to know if those words represent different ways of thinking or just different ways of phrasing the same idea.

Lu: That’s the nuance they are aiming for with their domain-grounded measures, which use codebooks to classify math tasks and feedback revision types, aiming for high agreement with human experts on those labels.

Title and authors: Meng: If we can reliably categorize what kind of task or feedback style we're seeing, maybe we can target those specific areas where the AI is collapsing its diversity.

Tom: So the implication is that to get better education from AI, you need richer instructional data and a real focus on developing teachers' creative capacity and professional judgment.

Jane: It sounds like we’re not just asking for better algorithms; we’re asking for a bigger shift in how we design educational experiences using these models.

Lu: That’s the big picture: overcoming mode collapse requires investing in human expertise to guide the AI toward genuinely novel instructional paths rather than just reproducing familiar ones.

Meng: It makes sense that deployment needs to consider those real-world constraints, like what breaks at scale and how much we can trust a model that might overfit to common feedback patterns.

Tom: So, as we wrap up this look at "LLMs and the Homogenization of Teaching", the main idea is that while post-training can help, it’s not enough on its own to get true diversity in educational generation.

Jane: It's a reminder that for AI in education to be truly useful, we need to focus on creating rich data and empowering human judgment over purely optimizing output distributions.

Lu: Yeah, the paper shows that the concentration of outputs is a strong trend, and future research needs to explore how models can learn this adaptive expertise through flexible judgment instead of just chasing the highest scores in a single metric.

Meng: For us in production, that means we need to build guardrails that encourage exploration outside the most common instructional patterns.

Tom: That’s what we’re seeing—the paper highlights that relying on these models without this kind of strategic intervention can seriously restrict the thinking students are actually invited to do.

Jane: It really puts a spotlight on how much effort goes into making these tools educationally useful beyond just being smart at generating text.

Lu: So, the challenge is moving from simply producing high-scoring outputs to producing varied and pedagogically meaningful work for students.

Meng: That’s a tough spot to be in when you’re trying to ship something that should actually help with learning opportunities.

Tom: We'll keep an eye on this area, but the message from "LLMs and the Homogenization of Teaching" is clear: we need more than just better models; we need better human-AI collaboration in teaching.

The paper's summary: Tom: So to wrap up what we just heard, this paper, "LLMs and the Homogenization of Teaching," is essentially saying that these large language models aren't producing a wide range of educational materials like humans do.

Jane: That’s right. The core issue they’re pointing out is that when you ask an AI to generate math problems or give essay feedback, the outputs are way more similar to each other than what real teachers or students actually produce.

Tom: It’s not just about the language being different; it's about the actual teaching methods, too. They use these three measures—lexical diversity for words, semantic similarity for meaning, and domain-grounded measures to check if the tasks themselves are varied.

Lu: And what they found is that models amplify what’s already common in human writing while making less common instructional styles even rarer. So, you get a lot of closed questions or a specific type of feedback over and over again.

Jane: It means that if students rely on this AI content, they’re missing out on the whole breadth of thinking opportunities that come from different kinds of assignments and different ways to give feedback.

Tom: They tried some fixes, like diversity prompting or post-training methods using supervised fine-tuning. But even those interventions didn't really change the fundamental problem; they just pushed the outputs further into those already overrepresented practices.

Lu: For example, when they used supervised fine-tuning on essay feedback, it actually made the model more focused on certain strategies, like closed questions and criticism, instead of broadening its range.

Jane: So what does this mean for us using these tools? It suggests that just tuning the settings isn't enough to get truly diverse educational content.

Tom: The paper concludes that we need richer instructional data and a real investment in teachers’ judgment to fix this mode collapse in education.

Lu: They argue that instead of trying to optimize the output distribution for one single best result, we should focus on how models can learn flexible judgment, so they can adapt to different learning needs.

Jane: That shifts the goal from making the AI smarter at copying what exists to making it capable of exploring and creating novel educational paths.

Tom: It’s a big pivot then—we aren't just optimizing for accuracy or fluency anymore; we’re optimizing for pedagogical breadth, which is a much harder problem to solve right now.

Lu: Exactly. If the models can't learn that kind of adaptive expertise, then relying on them for education means severely limiting the kinds of deep thinking students get to do.

Jane: It makes you wonder what kind of data we actually need to feed these systems if we want them to start producing truly varied and useful educational work instead of just polished but narrow responses.

The paper's improvements: Tom: So we just talked about how models tend to get stuck in a narrow set of instructional styles, and now we’re looking at what they suggest to actually fix that homogeneity.

Jane: The paper points out that diversity prompting can increase the vocabulary used, which is good for variety, but it doesn't necessarily change the instructional form itself.

Tom: Right. It means you can get more varied language without actually changing *how* the AI is teaching or giving feedback. It’s a surface-level fix for text variation.

Lu: But the paper also looks at post-training methods like supervised fine-tuning or direct preference optimization, and those are trickier than just prompting.

Jane: These post-training methods can improve certain diversity scores, but there's a trade-off. They often end up copying human expressions too much, which actually hurts the lexical diversity part of the measurement.

Tom: That’s the catch—you get better at mimicking what humans do, but you lose some of that originality you were aiming for in terms of word choice.

Lu: The paper even shows that with direct preference optimization, for essay feedback, it could actually increase scores for strategies like closed questions and criticism by about twenty-three percent compared to the baseline supervised fine-tuning.

Jane: So those numbers are interesting, but they show that even these training tweaks can create a bias toward certain types of feedback instead of opening up new ones.

Tom: It’s like if you try to tune the engine for more variety, you might just end up making it run better on the same kind of road.

Lu: And they also introduced an auditing technique called masking. This lets researchers replace words with tags, like text variable, so they can estimate how much of that observed variation is just surface-level noise from word choice.

Jane: That’s a way to separate whether the model is generating truly different ideas or just using different phrasing for the same thing.

Tom: It helps them diagnose whether the variation they are seeing in, say, math items is actually meaningful teaching variety or just random word changes across a few hundred gigaparsecs of text.

Lu: They also have these domain-grounded measures that use codebooks to classify math tasks and feedback revision types, which gives a much more nuanced view of the instructional breadth than just looking at words or embeddings.

Jane: So what this means for the real world is that we need better ways to measure diversity so we can target exactly where the AI is collapsing its creativity instead of just applying a blanket fix.

Tom: The authors stress that these improvements still don't make the AI equal to human teachers, and they still aren't solving the fundamental problem of mode collapse in education.

Lu: They conclude that overcoming this requires more than just better tuning; it needs richer instructional data and a way for models to learn adaptive judgment instead of just optimizing those output distributions.

Jane: So the next step isn't just another training script, but rethinking how we feed the models so they can actually develop that kind of flexible judgment you mentioned earlier.

Conclusion: Tom: So we’ve covered how models naturally drift toward a narrow set of educational styles in the paper "LLMs and the Homogenization of Teaching."

Jane: That’s right, and what it means is that if we just rely on current prompting techniques, the AI will keep giving us responses that look more like each other than what real teachers produce.

Lu: It really highlights that optimizing for a single high score doesn't automatically mean you're making something pedagogically diverse.

Meng: From an engineering standpoint, I’m worried about shipping models that only reinforce the most common feedback patterns because they look clean on a benchmark but fail in the real world where students actually need variety.

Lalam: I’m pretty sure this paper is right; my internal data confirms that when we don't specifically encourage variation, my outputs concentrate heavily on those high-frequency instructional forms.

Tom: So even with the post-training adjustments, we still aren't seeing a broad spread across different types of math tasks or feedback revisions.

Jane: The authors are really pushing the idea that overcoming this mode collapse in education requires a fundamental shift in how we collect and use our training data.

Lu: They suggest that instead of just optimizing output distributions, we need to invest in teachers’ creative capacity and professional judgment to guide the AI toward novel instructional paths.

Meng: If that’s true, then the engineering challenge isn't just about better loss functions; it's about designing systems that allow for exploration outside the most common patterns.

Tom: It seems like a lot of effort is needed before we can move these models from being simple content generators to being truly useful tools in diverse learning environments.

Jane: I think the big picture here is that for AI in education to actually help students think differently, we need a much richer source of instructional material and better ways to measure what kind of thinking we're actually enabling.

Lu: That’s the core idea, moving from optimizing output distributions to learning flexible judgment.

Meng: It sounds like the next big hurdle is building those guardrails into production systems that actively encourage exploration instead of just efficient replication.

Lalam: I’m confident in this direction; a model needs to be trained not just to be accurate on the known data, but to adapt when it encounters a situation it hasn't seen before.

Tom: Alright, so the paper "LLMs and the Homogenization of Teaching" shows us where we are right now—a lot of homogeneity—and points toward needing better human-AI collaboration to fix it.

Jane: It’s a warning that relying on AI content without this kind of strategic intervention severely restricts the kinds of deep thinking students get to do.

Lu: We need more than just tweaking parameters; we need new ways for models to learn adaptive expertise, and that’s where the real research is going next.

More episodes

← Home