Have Large Language Models Enhanced the Way Civil & Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over 25 Years

summary

The gist

Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution…

In short

This study analyzed a large corpus of Civil and Environmental Engineering literature from 2000 to 2025 to detect linguistic signals indicative of Large Language Model (LLM) assistance. Using a frequency-shift methodology, the research quantified how LLMs are reshaping scholarly writing style, showing systematic shifts in word choice, complexity, and assertiveness in abstracts.

Key concepts

Vocabulary-based frequency-shift methodology
This is a statistical method used to find unusual patterns in language. The researchers tracked the yearly frequency of specific words (lemmas) across all abstracts. Deviations from expected historical frequencies signal potential changes in writing style, which they hypothesize are caused by LLM use.
Frequency Gap ($\delta$) and Frequency Ratio ($r$)
These are two mathematical metrics used to measure how much the actual word usage in a given year differs from what would be expected based on past trends. These metrics help pinpoint statistically significant shifts in vocabulary that might be linked to how LLMs influence academic writing.
Semantic Properties
These are measurable characteristics of text beyond just word count, such as sentence length, punctuation use, and the diversity of words chosen. By analyzing these properties, the study identified specific stylistic changes—like increased complexity or reduced reliance on passive voice—that occur when LLMs are used.
LLM Adoption Prevalence
This refers to the estimated percentage of abstracts likely written by an LLM in a given year. The study estimates this prevalence (e.g., 15.3% in 2024) using word frequency differences, providing a quantitative measure of how widely LLMs are being used across the field.

Terminology used across episodes

This episode discusses

The paper

Have Large Language Models Enhanced the Way Civil & Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over 25 Years · Read on arXiv

Morgan D. Sanger, Brett W. Maurer

University of Washington

Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution, and communication. However, a gap remains in understanding how prevalent LLM adoption is across CEE scholarship and whether it measurably alters research prose. This study applies a vocabulary-based frequency-shift methodology to 149,452 abstracts published by the American Society of Civil Engineers between 2000 and 2025. Deviations from long-term vocabulary trends suggest that 15.3% and 26.9% of 2024 and 2025 abstracts, respectively, are likely LLM-influenced. Prior to the introduction of LLMs in 2022, CEE publications exhibit long-term trends toward longer abstracts, greater use of segmenting punctuation, higher required reading levels, and use of active, first-person verb constructions. Beginning around 2023, however, the frequencies of many stylistic marker words (e.g., enhance) sharply depart from their historical trajectories, a shift consistent with the emergence of LLMs. Abstracts classified as likely LLM-influenced also depart from historical semantic trends, exhibiting increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident. The artificial intelligence contribution of this study is the use of natural language processing to detect population-level linguistic signals of LLM-generated text, estimating the prevalence of LLM use and its influence on CEE writing. Together, these novel findings provide the first large-scale, data-driven assessment of how CEE scholarly communication is changing alongside LLM adoption.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Have Large Language Models Enhanced the Way Civil & Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over 25 Years".

Jane: Here's the gist of the paper. Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution, and communication <ref:2602.03864#pg2> This study uses a vocabulary-based frequency-shift methodology to detect linguistic signals of LLM-assisted writing in a large corpus of CEE literature <ref:2602.03864#pg6>.

Tom: Pruning redundant reasoning tokens The research aims to quantify the prevalence of LLM use across CEE scholarship and assess how its adoption varies across disciplines and publication types within the field of CEE scholarship <ref:2602.03864#pg2> The objective is to provide the first large-scale, data-driven portrait of how LLMs are reshaping the language of CEE research <ref:2602.03864#pg6>.

Jane: This involves quantifying the frequency of LLM use and investigating how LLM adoption is changing the structure, complexity, voice, and embedded confidence of research prose <ref:2602.03864#pg6>.

Tom: How the training signal is built The study compiles a corpus of 149,452 abstracts published by the American Society of Civil Engineers from 2000 through 2025 <ref:2602.03864#pg2> Preprocessing involves tokenizing each abstract into individual word tokens using the Natural Language Toolkit (NLTK) and converting all text to lowercase for case-insensitive analysis <ref:2602.03864#pg2>.

Paper summary: Tom: So, we're talking about this paper from Morgan Sanger and Brett Maurer called "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years" <ref:2602.03864#pg1,A Quantitative Analysis of Scholarly Communication over 25 Years>. The main idea here is figuring out if these large language models are actually changing how civil and environmental engineers write their research papers.

Jane: That sounds really relevant because we're all seeing AI popping up everywhere, and this paper tries to quantify it by looking at a massive collection of literature from the American Society of Civil Engineers between the years two thousand and twenty-five. It claims there are clear linguistic shifts happening in how people write their work that might be due to using these new tools.

Lu: Exactly. The authors use this vocabulary-based frequency-shift methodology to detect those signals across all those abstracts to see if they deviate from long-term trends in writing style over the last twenty-five years, and they are looking at quantifying how prevalent this LLM adoption is and how it affects the structure, complexity, voice, and even the embedded confidence of that research prose.

Meng: I'm curious about what kind of data they used to track these shifts. When you look at a corpus that big across twenty-five years, what’s actually difficult to manage without a really solid preprocessing step?

Lalam: The study compiles one hundred forty-nine thousand abstracts from ASCE journals and conference proceedings published between two thousand and twenty-five, and they tokenize every abstract into individual word tokens using the Natural Language Toolkit and converting all the text to lowercase for case-insensitive analysis, then they remove common stop words and lemmatize the remaining tokens to reduce inflected word forms to their base or dictionary form <ref:2602.03864#pg1>.

Tom: Okay, so that's a big dataset you're starting with. So what’s the actual core claim they are making about these linguistic signals?

Jane: They are looking for statistically anomalous vocabulary changes by computing an annual document-level frequency of each lemma as the proportion of abstracts each year containing at least one instance of the word normalized by the number of publications that year, and then they compute deviations from expected frequencies using two departure metrics, a frequency gap and a frequency ratio.

Lu: And they aren't just looking at words; they’re quantifying several semantic properties for each abstract, including overall abstract length in terms of words and sentences, mean sentence length, punctuation use, abstract complexity enumerated in terms of word choice diversity and two commonly used readability scores like the Gunning Fog Index and Flesch-Kincaid Reading Level.

Paper summary: Meng: Readability scores are interesting because they give you a concrete idea of how accessible the writing is to someone who isn't deep in the field, but I wonder how those metrics align with what an actual researcher perceives as "better" or "more rigorous" writing.

Lalam: Anecdotally, they use first-person pronoun use to proxy active voice and past participles to proxy passive voice, which is a way of measuring those stylistic shifts.

Tom: So, if we look at the results from this paper on "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years," what’s the big picture they’ve found regarding these changes <ref:2602.03864#pg1,A Quantitative Analysis of Scholarly Communication over 25 Years>?

Jane: The finding is that LLM adoption is measurably impacting the stylistic evolution of scholarly writing in civil and environmental engineering, showing systematic shifts in abstracts that are likely written with an LLM. Specifically, abstracts classified as likely LLM-written show increased word choice diversity, heavier use of commas, reduced readability, decreased reliance on passive constructions and diminished use of qualifying language used to convey uncertainty.

Lu: These features together suggest the prose becomes more segmented and more syntactically complex and more assertive than what we saw in CEE scholarship before the widespread adoption of LLMs.

Meng: The paper also gives some specific numbers, saying the prevalence of LLM adoption is estimated at fifteen point three percent in two thousand twenty-four and twenty-six point two percent in two thousand twenty-five using the mean prevalence differential between the rare and common word sets.

Lalam: A more conservative classification procedure yields corresponding estimates of thirteen point nine percent and twenty point four percent of abstracts likely written by LLMs, which provides that quantitative context for interpreting the stylistic changes driven by LLMs.

Tom: It sounds like they are saying that this wasn't just a random change; it’s a specific pattern emerging around two thousand twenty-three when many excess style words started departing from their historic trajectories and multiple semantic properties also began shifting at that same time.

Jane: And what really stands out is when they isolate those abstracts identified as likely LLM-written, these departures are shown to be largely or entirely attributable to the LLMs themselves, confirming that these specific stylistic changes are linked to the use of these models.

Lu: The study also pointed out that prior to the widespread adoption of LLMs, CEE scholarship had long-term trends toward increasing numbers of authors and longer abstracts and sentences, greater use of segmenting punctuation, higher required reading levels and a shift toward active first-person verb constructions.

Meng: So, before this period starting around two thousand twenty-three the style was already shifting in these directions naturally, but now we see a dramatic departure with the LLMs kicking things into high gear.

Paper summary: Lalam: They also noted that they compiled publicly available abstracts published across ASCE journals and conference proceedings from two thousand and through twenty-five, and they independently repeated this analysis to compare abstracts from two thousand twenty-five to two thousand nineteen as well as from two thousand nineteen to two thousand thirteen.

Tom: So, the paper is doing a lot of work by looking at these long-term trends and then isolating the specific linguistic markers that appear around the time LLMs started gaining traction in this field. What does this mean for someone who only listens to this show?

Jane: It means that when you read a paper from twenty-twenty-five, you might notice it reads differently—maybe it’s more punctuated, or the sentences are longer and more complex than papers from ten years ago—and if it fits the pattern they found, there's a good chance an LLM was involved in drafting or editing that section.

Lu: The core idea is that we need to understand these shifts because we don't fully know how prevalent LLM adoption is across CEE scholarship and at what rate authors disclose their use of these tools.

Meng: And they flag a limitation, which is that the study only looks at abstracts from ASCE journals and conference proceedings, so it might not capture every single type of CEE writing out there.

Lalam: They also state that their classification procedure is intended to capture population-level stylistic changes and no claim is made towards LLM authorship of individual abstracts, which is an important distinction to keep in mind when interpreting the results.

Tom: So, to wrap up this discussion on "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years," what's the final thought on where we go from here <ref:2602.03864#pg1,A Quantitative Analysis of Scholarly Communication over 25 Years>?

Jane: The paper provides the quantitative context necessary for us to interpret these stylistic changes driven by LLMs, showing a clear trend away from earlier writing patterns and toward a more assertive and complex prose.

Lu: It suggests that while LLMs are emerging as tools for project ideation, execution, and communication in CEE research, their adoption is already measurably altering the language of the discipline.

Meng: From my side, it means if we're building models for this field, we need to know how these stylistic shifts manifest so we can build better detection tools or understand the actual impact on scientific rigor.

Lalam: It’s a signal that LLMs are moving beyond just being a helpful assistant and starting to shape the very structure of how CEE science is communicated.

Conclusion: Tom: So we've been looking at how Large Language Models are changing how civil and environmental engineers write their research over twenty-five years, and now we're getting to the actual conclusion of this paper, "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years."

Jane: Basically, the authors found that yes, there are measurable shifts in how people write these papers after LLMs started becoming common tools. They look at the language itself across all those years.

Lu: They pinpoint specific ways this writing changes, like an increase in word choice diversity and a heavier use of commas in abstracts that seem to be written with AI assistance.

Meng: So what does that actually mean for the people doing the engineering work? Is it just about sounding different?

Lalam: The paper shows these shifts are systematic, meaning they aren't random; they happen together across many features like sentence length and how complex the abstract is made to be.

Tom: It suggests that papers flagged as likely LLM-written have a prose that’s more segmented and more assertive than what we saw before the models became widespread.

Jane: That’s the core implication, Tom; it points to a specific style evolving in this field, and it shows those changes are tied to the way these models structure language.

Lalam: They estimate that around two thousand twenty-four and twenty twenty-five, we're seeing about fifteen point three percent and twenty six point two percent of abstracts likely use an LLM in their writing.

Lu: The authors also repeat this analysis over different time slices, looking at the period from two thousand twenty-five back to nineteen or thirteen to make sure these trends are real population-level changes.

Meng: If we're building AI tools for this field, knowing that prose gets more assertive and complex when using LLMs helps us understand what we're actually seeing in the data.

Tom: It’s a signal that LLMs aren't just helping with drafting; they are already altering the very structure of how civil and environmental science communicates its findings.

Jane: And understanding these linguistic signals is important because it gives us a way to look at the output and see if it aligns with those expected stylistic shifts.

Lu: This suggests we need better ways to track not just *if* LLMs are used, but *how* their use is changing the fundamental writing mechanics of this discipline.

Meng: We need to figure out what breaks when we try to run these models on actual engineering data at scale, because if the writing style changes that much, our deployment models might also need a rethink.

Lalam: The paper itself doesn't claim authorship for any single abstract; it’s about the population trend of style change across all of ASCE literature from two thousand to twenty twenty five.

Tom: So, we see a clear pattern emerging where the language of CEE research is shifting in a direction that aligns with how these new models are being used.

Jane: It gives us concrete data on what that shift looks like, which is exactly what we need to understand the impact on the science itself.

Lu: Next up, we're going to look at some of those specific linguistic features—the word choice diversity and readability scores—that they measured in more detail.

More episodes

← Home