Have Large Language Models Enhanced the Way Civil & Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over 25 Years

arXiv:2602.03864 · cs.DL · Submitted 2026-01-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Have Large Language Models Enhanced the Way Civil & Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over 25 Years".

Jane: Here's the gist of the paper. Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution, and communication <ref:2602.03864#pg2> This study uses a vocabulary-based frequency-shift methodology to detect linguistic signals of LLM-assisted writing in a large corpus of CEE literature <ref:2602.03864#pg6>.

Tom: Pruning redundant reasoning tokens The research aims to quantify the prevalence of LLM use across CEE scholarship and assess how its adoption varies across disciplines and publication types within the field of CEE scholarship <ref:2602.03864#pg2> The objective is to provide the first large-scale, data-driven portrait of how LLMs are reshaping the language of CEE research <ref:2602.03864#pg6>.

Jane: This involves quantifying the frequency of LLM use and investigating how LLM adoption is changing the structure, complexity, voice, and embedded confidence of research prose <ref:2602.03864#pg6>.

Tom: How the training signal is built The study compiles a corpus of 149,452 abstracts published by the American Society of Civil Engineers from 2000 through 2025 <ref:2602.03864#pg2> Preprocessing involves tokenizing each abstract into individual word tokens using the Natural Language Toolkit (NLTK) and converting all text to lowercase for case-insensitive analysis <ref:2602.03864#pg2>.

Paper summary: Tom: So, we're talking about this paper from Morgan Sanger and Brett Maurer called "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years" <ref:2602.03864#pg1,A Quantitative Analysis of Scholarly Communication over 25 Years>. The main idea here is figuring out if these large language models are actually changing how civil and environmental engineers write their research papers.

Jane: That sounds really relevant because we're all seeing AI popping up everywhere, and this paper tries to quantify it by looking at a massive collection of literature from the American Society of Civil Engineers between the years two thousand and twenty-five. It claims there are clear linguistic shifts happening in how people write their work that might be due to using these new tools.

Lu: Exactly. The authors use this vocabulary-based frequency-shift methodology to detect those signals across all those abstracts to see if they deviate from long-term trends in writing style over the last twenty-five years, and they are looking at quantifying how prevalent this LLM adoption is and how it affects the structure, complexity, voice, and even the embedded confidence of that research prose.

Meng: I'm curious about what kind of data they used to track these shifts. When you look at a corpus that big across twenty-five years, what’s actually difficult to manage without a really solid preprocessing step?

Lalam: The study compiles one hundred forty-nine thousand abstracts from ASCE journals and conference proceedings published between two thousand and twenty-five, and they tokenize every abstract into individual word tokens using the Natural Language Toolkit and converting all the text to lowercase for case-insensitive analysis, then they remove common stop words and lemmatize the remaining tokens to reduce inflected word forms to their base or dictionary form <ref:2602.03864#pg1>.

Tom: Okay, so that's a big dataset you're starting with. So what’s the actual core claim they are making about these linguistic signals?

Jane: They are looking for statistically anomalous vocabulary changes by computing an annual document-level frequency of each lemma as the proportion of abstracts each year containing at least one instance of the word normalized by the number of publications that year, and then they compute deviations from expected frequencies using two departure metrics, a frequency gap and a frequency ratio.

Lu: And they aren't just looking at words; they’re quantifying several semantic properties for each abstract, including overall abstract length in terms of words and sentences, mean sentence length, punctuation use, abstract complexity enumerated in terms of word choice diversity and two commonly used readability scores like the Gunning Fog Index and Flesch-Kincaid Reading Level.

Paper summary: Meng: Readability scores are interesting because they give you a concrete idea of how accessible the writing is to someone who isn't deep in the field, but I wonder how those metrics align with what an actual researcher perceives as "better" or "more rigorous" writing.

Lalam: Anecdotally, they use first-person pronoun use to proxy active voice and past participles to proxy passive voice, which is a way of measuring those stylistic shifts.

Tom: So, if we look at the results from this paper on "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years," what’s the big picture they’ve found regarding these changes <ref:2602.03864#pg1,A Quantitative Analysis of Scholarly Communication over 25 Years>?

Jane: The finding is that LLM adoption is measurably impacting the stylistic evolution of scholarly writing in civil and environmental engineering, showing systematic shifts in abstracts that are likely written with an LLM. Specifically, abstracts classified as likely LLM-written show increased word choice diversity, heavier use of commas, reduced readability, decreased reliance on passive constructions and diminished use of qualifying language used to convey uncertainty.

Lu: These features together suggest the prose becomes more segmented and more syntactically complex and more assertive than what we saw in CEE scholarship before the widespread adoption of LLMs.

Meng: The paper also gives some specific numbers, saying the prevalence of LLM adoption is estimated at fifteen point three percent in two thousand twenty-four and twenty-six point two percent in two thousand twenty-five using the mean prevalence differential between the rare and common word sets.

Lalam: A more conservative classification procedure yields corresponding estimates of thirteen point nine percent and twenty point four percent of abstracts likely written by LLMs, which provides that quantitative context for interpreting the stylistic changes driven by LLMs.

Tom: It sounds like they are saying that this wasn't just a random change; it’s a specific pattern emerging around two thousand twenty-three when many excess style words started departing from their historic trajectories and multiple semantic properties also began shifting at that same time.

Jane: And what really stands out is when they isolate those abstracts identified as likely LLM-written, these departures are shown to be largely or entirely attributable to the LLMs themselves, confirming that these specific stylistic changes are linked to the use of these models.

Lu: The study also pointed out that prior to the widespread adoption of LLMs, CEE scholarship had long-term trends toward increasing numbers of authors and longer abstracts and sentences, greater use of segmenting punctuation, higher required reading levels and a shift toward active first-person verb constructions.

Meng: So, before this period starting around two thousand twenty-three the style was already shifting in these directions naturally, but now we see a dramatic departure with the LLMs kicking things into high gear.

Paper summary: Lalam: They also noted that they compiled publicly available abstracts published across ASCE journals and conference proceedings from two thousand and through twenty-five, and they independently repeated this analysis to compare abstracts from two thousand twenty-five to two thousand nineteen as well as from two thousand nineteen to two thousand thirteen.

Tom: So, the paper is doing a lot of work by looking at these long-term trends and then isolating the specific linguistic markers that appear around the time LLMs started gaining traction in this field. What does this mean for someone who only listens to this show?

Jane: It means that when you read a paper from twenty-twenty-five, you might notice it reads differently—maybe it’s more punctuated, or the sentences are longer and more complex than papers from ten years ago—and if it fits the pattern they found, there's a good chance an LLM was involved in drafting or editing that section.

Lu: The core idea is that we need to understand these shifts because we don't fully know how prevalent LLM adoption is across CEE scholarship and at what rate authors disclose their use of these tools.

Meng: And they flag a limitation, which is that the study only looks at abstracts from ASCE journals and conference proceedings, so it might not capture every single type of CEE writing out there.

Lalam: They also state that their classification procedure is intended to capture population-level stylistic changes and no claim is made towards LLM authorship of individual abstracts, which is an important distinction to keep in mind when interpreting the results.

Tom: So, to wrap up this discussion on "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years," what's the final thought on where we go from here <ref:2602.03864#pg1,A Quantitative Analysis of Scholarly Communication over 25 Years>?

Jane: The paper provides the quantitative context necessary for us to interpret these stylistic changes driven by LLMs, showing a clear trend away from earlier writing patterns and toward a more assertive and complex prose.

Lu: It suggests that while LLMs are emerging as tools for project ideation, execution, and communication in CEE research, their adoption is already measurably altering the language of the discipline.

Meng: From my side, it means if we're building models for this field, we need to know how these stylistic shifts manifest so we can build better detection tools or understand the actual impact on scientific rigor.

Lalam: It’s a signal that LLMs are moving beyond just being a helpful assistant and starting to shape the very structure of how CEE science is communicated.

Conclusion: Tom: So we've been looking at how Large Language Models are changing how civil and environmental engineers write their research over twenty-five years, and now we're getting to the actual conclusion of this paper, "Have Large Language Models Enhanced the Way Civil and Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over twenty-five Years."

Jane: Basically, the authors found that yes, there are measurable shifts in how people write these papers after LLMs started becoming common tools. They look at the language itself across all those years.

Lu: They pinpoint specific ways this writing changes, like an increase in word choice diversity and a heavier use of commas in abstracts that seem to be written with AI assistance.

Meng: So what does that actually mean for the people doing the engineering work? Is it just about sounding different?

Lalam: The paper shows these shifts are systematic, meaning they aren't random; they happen together across many features like sentence length and how complex the abstract is made to be.

Tom: It suggests that papers flagged as likely LLM-written have a prose that’s more segmented and more assertive than what we saw before the models became widespread.

Jane: That’s the core implication, Tom; it points to a specific style evolving in this field, and it shows those changes are tied to the way these models structure language.

Lalam: They estimate that around two thousand twenty-four and twenty twenty-five, we're seeing about fifteen point three percent and twenty six point two percent of abstracts likely use an LLM in their writing.

Lu: The authors also repeat this analysis over different time slices, looking at the period from two thousand twenty-five back to nineteen or thirteen to make sure these trends are real population-level changes.

Meng: If we're building AI tools for this field, knowing that prose gets more assertive and complex when using LLMs helps us understand what we're actually seeing in the data.

Tom: It’s a signal that LLMs aren't just helping with drafting; they are already altering the very structure of how civil and environmental science communicates its findings.

Jane: And understanding these linguistic signals is important because it gives us a way to look at the output and see if it aligns with those expected stylistic shifts.

Lu: This suggests we need better ways to track not just *if* LLMs are used, but *how* their use is changing the fundamental writing mechanics of this discipline.

Meng: We need to figure out what breaks when we try to run these models on actual engineering data at scale, because if the writing style changes that much, our deployment models might also need a rethink.

Lalam: The paper itself doesn't claim authorship for any single abstract; it’s about the population trend of style change across all of ASCE literature from two thousand to twenty twenty five.

Tom: So, we see a clear pattern emerging where the language of CEE research is shifting in a direction that aligns with how these new models are being used.

Jane: It gives us concrete data on what that shift looks like, which is exactly what we need to understand the impact on the science itself.

Lu: Next up, we're going to look at some of those specific linguistic features—the word choice diversity and readability scores—that they measured in more detail.

Morgan D. Sanger, Brett W. Maurer

University of Washington

cs.DL

Submitted: 2026-01-28

Updated: 2026-11-16

Code: https://github.com/nltk/nltk

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution…

Key concepts

Vocabulary-based frequency-shift methodology
This is a statistical method used to find unusual patterns in language. The researchers tracked the yearly frequency of specific words (lemmas) across all abstracts. Deviations from expected historical frequencies signal potential changes in writing style, which they hypothesize are caused by LLM use.
Frequency Gap ($\delta$) and Frequency Ratio ($r$)
These are two mathematical metrics used to measure how much the actual word usage in a given year differs from what would be expected based on past trends. These metrics help pinpoint statistically significant shifts in vocabulary that might be linked to how LLMs influence academic writing.
Semantic Properties
These are measurable characteristics of text beyond just word count, such as sentence length, punctuation use, and the diversity of words chosen. By analyzing these properties, the study identified specific stylistic changes—like increased complexity or reduced reliance on passive voice—that occur when LLMs are used.
LLM Adoption Prevalence
This refers to the estimated percentage of abstracts likely written by an LLM in a given year. The study estimates this prevalence (e.g., 15.3% in 2024) using word frequency differences, providing a quantitative measure of how widely LLMs are being used across the field.

Terminology

Summary

Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution, and communication <ref:2602. p. 0

Where the paper says it

Sanger1 and Brett W. Maurer1 Abstract Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution, and communication.

03864#pg2> This study uses a vocabulary-based frequency-shift methodology to detect linguistic signals of LLM-assisted writing in a large corpus of CEE literature <ref:2602. p. 0
Where the paper says it

However, it is unknown how prevalent LLM adoption is across CEE scholarship and whether it measurably alters research prose. Inspired by recent analyses of biomedical research, this study uses a vocabulary-based frequency-shift methodology to detect linguistic signals of LLM-assisted writing in a large corpus of CEE literature.

03864#pg6>.

Pruning redundant reasoning tokens

The research aims to quantify the prevalence of LLM use across CEE scholarship and assess how its adoption varies across disciplines and publication types within the field of CEE scholarship <ref:2602.03864#pg2> The objective is to provide the first large-scale, data-driven portrait of how LLMs are reshaping the language of CEE research <ref:2602. p. 4

Where the paper says it

Importantly, the objective of this study is not to prescribe policy regarding LLMs in scholarly writing. Rather, the contribution is empirical: to provide the first large-scale, data-driven portrait of how LLMs are reshaping the language of CEE research.

03864#pg6>. This involves quantifying the frequency of LLM use and investigating how LLM adoption is changing the structure, complexity, voice, and embedded confidence of research prose <ref:2602. p. 4
Where the paper says it

While the results may not generalize perfectly to all publishers, the dataset offers a valuable case study for understanding LLM use within a broad sample of CEE scholarship. In addition to quantifying the frequency of LLM use, we also investigate how LLM adoption is changing the structure, complexity, voice, and embedded confidence of research prose.

03864#pg6>.

How the training signal is built

The study compiles a corpus of 149,452 abstracts published by the American Society of Civil Engineers from 2000 through 2025 <ref:2602.03864#pg2> Preprocessing involves tokenizing each abstract into individual word tokens using the Natural Language Toolkit (NLTK) and converting all text to lowercase for case-insensitive analysis <ref:2602. p. 4

Where the paper says it

Specifically, each abstract is tokenized into individual word tokens using the Natural Language Toolkit (NLTK) (Bird et al. , 2009), with all text converted to lowercase for case-insensitive analysis.

03864#pg2>. Common stop words are removed, and remaining tokens are lemmatized to reduce inflected word forms to their base or dictionary form <ref:2602. p. 4
Where the paper says it

, “the,” “and,” “of”), which carry limited semantic content and can distort frequency-based measures, are removed using NLTK’s curated stop-word lists. The remaining tokens are then lemmatized using NLTK’s WordNet lemmatizer to reduce inflected and derived word forms to their base or dictionary form; for example, “models,” “modeling,” and “modeled” are all mapped to the lemma “model.

03864#pg2>.

Where the gains come from

The methodology employs a frequency-shift approach inspired by Kobak et al. (2025) to identify statistically anomalous vocabulary changes <ref:2602. p. 6

Where the paper says it

Linguistic Signals of LLM Usage To estimate the prevalence of LLM use, we adopt a frequency-shift methodology inspired by Kobak et al. (2025) that identifies statistically anomalous vocabulary changes.

03864#pg2> This involves computing an annual document-level frequency of each lemma as the proportion of abstracts each year containing at least one instance of the word normalized by the number of publications that year <ref:2602. p. 7
Where the paper says it

8 we compute the annual document-level frequency of each lemma (i. , word) as the proportion of abstracts each year containing at least one instance of the word normalized by the number of publications that year.

03864#pg2>. Deviations from expected frequencies are computed according to two complementary departure metrics: frequency gap, δ (Eq. p. 7
Where the paper says it

The expected frequencies for each word are quantified using a counterfactual trendline, where a least-squares linear model is fit to the observed frequencies prior to 2022 and extrapolated for comparison to the observed frequencies in 2023, 2024, and 2025. Deviations from the expected frequencies are computed according to two complementary departure metrics: frequency gap, 𝛿 (Eq.

3), and frequency ratio, r (Eq. p. 7
Where the paper says it

Deviations from the expected frequencies are computed according to two complementary departure metrics: frequency gap, 𝛿 (Eq. 3), and frequency ratio, 𝑟 (Eq.

4) <ref:2602.03864#pg2>.

Semantic Trends and Influence of LLMs

The analysis quantifies several semantic properties for each abstract, including overall abstract length in terms of words and sentences, mean sentence length, punctuation use, abstract complexity enumerated in terms of word choice diversity and two commonly used readability scores (Gunning Fog Index and Flesch-Kincaid Reading Level) <ref:2602.03864#pg2>. Anecdotally, active voice is proxied via first-person pronoun use and passive voice is proxied via past participles <ref:2602. p. 6

Where the paper says it

, “The data were analyzed”). As proxy metrics to observe verbiage trends, we equate active voice to first-person pronoun use and passive voice to instances of "to be" verbs followed by past participles (e.

03864#pg2>.

Conclusions

The findings suggest that LLM adoption is measurably impacting the stylistic evolution of scholarly writing in civil and environmental engineering <ref:2602.03864#pg6>. Specifically, abstracts classified as likely LLM-written exhibit systematic shifts, including increased word choice diversity, heavier use of commas, reduced readability, decreased reliance on passive constructions, and diminished use of qualifying language used to convey uncertainty <ref:2602. p. 0

Where the paper says it

, enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties. Abstracts classified as likely LLM-assisted exhibit increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident.

p. 0
Where the paper says it

, enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties. Abstracts classified as likely LLM-assisted exhibit increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident.

03864#pg6>. These features collectively suggest prose that is more segmented, more syntactically complex, and more assertive <ref:2602. p. 0
Where the paper says it

, enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties. Abstracts classified as likely LLM-assisted exhibit increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident.

p. 0
Where the paper says it

, enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties. Abstracts classified as likely LLM-assisted exhibit increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident.

03864#pg6>.

The prevalence of LLM adoption is estimated at 15.3% in 2024 and 26. p. 1

Where the paper says it

Given their transformative potential, a wealth of general and domain-specific analysis and commentary has rapidly emerged on LLMs, including research on their use in civil and environmental engineering (CEE) education, scholarship, and practice (e. , Sanger and Maurer, 2023; Kumar, 2024; Plevris, 2025; Matzakos and Moundridou, 2025; Li et al.

2% in 2025 using the mean prevalence differential of the rare and common word sets <ref:2602. p. 7
Where the paper says it

These two metrics are considered in tandem, as δ highlights the absolute change in prevalence and 𝑟 emphasizes the relative change in prevalence. Therefore, deviations identified by a large 𝛿 indicate changes in the use of common words with high baseline frequency, whereas departures identified by a large 𝑟 indicate changes in the use of rare words with low baseline frequency.

03864#pg6>. A more conservative multi-marker classification procedure yields corresponding estimates of 13. p. 7
Where the paper says it

, word) as the proportion of abstracts each year containing at least one instance of the word normalized by the number of publications that year. This approach prevents longer abstracts and years with more abstracts from disproportionately affecting frequency estimates and ultimately yields a year-word frequency matrix spanning the temporal and lexical domain of the corpus.

9% and 20.4% of abstracts likely written by LLMs <ref:2602. p. 0
Where the paper says it

, enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties. Abstracts classified as likely LLM-assisted exhibit increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident.

03864#pg6>. These estimates provide the quantitative context necessary for interpreting stylistic changes driven by LLMs <ref:2602. p. 6
Where the paper says it

The Gunning Fog Index and Flesch-Kincaid Reading Level have their own suite of limitations, including but not limited to: (i) the limited empirical samples on which they were fit; (ii) their oversimplification of complexity in terms of syllable quantity and sentence length; and (iii) their lack of consideration for context and reader familiarity with the subject matter. However, rather than classifying a single text excerpt for a given reading level, their primary applications in this study are to observe relative complexity changes over time and to see how adoption of LLMs may affect complexity.

03864#pg6>.

Several clear conclusions emerge regarding how LLM use is reshaping research prose <ref:2602.03864#pg6>. Prior to the widespread adoption of LLMs, CEE scholarship exhibits long-term trends toward increasing numbers of authors, longer abstracts and sentences, greater use of segmenting punctuation, higher required reading levels, and a shift toward active, first-person verb constructions <ref:2602. p. 0

Where the paper says it

A total of 149,452 abstracts published by the American Society of Civil Engineers from 2000 through 2025 are analyzed to quantify deviations from long-term vocabulary trends. Prior to the introduction of LLMs in 2022, CEE publications exhibit long-term trends toward longer abstracts and sentences, greater use of segmenting punctuation, higher required reading levels, and a shift toward active, first-person verb constructions.

03864#pg6>. Beginning around 2023, however, the use of many excess style words (e. p. 0
Where the paper says it

Prior to the introduction of LLMs in 2022, CEE publications exhibit long-term trends toward longer abstracts and sentences, greater use of segmenting punctuation, higher required reading levels, and a shift toward active, first-person verb constructions. Beginning around 2023, however, the frequencies of many stylistic marker words (e.

g., enhance, offer, demonstrate) dramatically depart from their historic trajectories <ref:2602. p. 0
Where the paper says it

Beginning around 2023, however, the frequencies of many stylistic marker words (e. , enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties.

03864#pg6>. Multiple semantic properties also depart from historic trends beginning in 2023 <ref:2602. p. 0
Where the paper says it

Beginning around 2023, however, the frequencies of many stylistic marker words (e. , enhance) sharply depart from historical trajectories, accompanied by deviations in multiple semantic properties.

03864#pg6>.

When abstracts identified as likely LLM-written are isolated, these departures are shown to be largely or entirely attributable to LLMs <ref:2602.03864#pg6>. Specifically, abstracts classified as likely LLM-written exhibit systematic shifts, including increased word choice diversity, heavier use of commas, reduced readability, decreased reliance on passive constructions, and diminished use of qualifying language used to convey uncertainty <ref:2602.03864#pg6>. These features collectively suggest prose that is more segmented, more syntactically complex, and more assertive <ref:2602.03864#pg6>.

The study compiles publicly available abstracts published across ASCE journals and conference proceedings from 2000 through 2025 <ref:2602.03864#pg2>. This analysis is independently repeated to compare abstracts from 2025 to 2019 <ref:2602. p. 1

Where the paper says it

Given their transformative potential, a wealth of general and domain-specific analysis and commentary has rapidly emerged on LLMs, including research on their use in civil and environmental engineering (CEE) education, scholarship, and practice (e. , Sanger and Maurer, 2023; Kumar, 2024; Plevris, 2025; Matzakos and Moundridou, 2025; Li et al.

03864#pg2>. The results of this classification procedure are intended to capture population-level stylistic changes, and no claim is made towards LLM authorship of individual abstracts <ref:2602.03864#pg6>. The study provides the quantitative context necessary for interpreting stylistic changes driven by LLMs <ref:2602. p. 6
Where the paper says it

The Gunning Fog Index and Flesch-Kincaid Reading Level have their own suite of limitations, including but not limited to: (i) the limited empirical samples on which they were fit; (ii) their oversimplification of complexity in terms of syllable quantity and sentence length; and (iii) their lack of consideration for context and reader familiarity with the subject matter. However, rather than classifying a single text excerpt for a given reading level, their primary applications in this study are to observe relative complexity changes over time and to see how adoption of LLMs may affect complexity.

p. 6
Where the paper says it

The Gunning Fog Index and Flesch-Kincaid Reading Level have their own suite of limitations, including but not limited to: (i) the limited empirical samples on which they were fit; (ii) their oversimplification of complexity in terms of syllable quantity and sentence length; and (iii) their lack of consideration for context and reader familiarity with the subject matter. However, rather than classifying a single text excerpt for a given reading level, their primary applications in this study are to observe relative complexity changes over time and to see how adoption of LLMs may affect complexity.

03864#pg6>.

The study compiles publicly available abstracts published across ASCE journals and conference proceedings from 2000 through 2025 <ref:2602.03864#pg2>. This analysis is independently repeated to compare abstracts from 2019 to 2013 <ref:2602. p. 5

Where the paper says it

, Wang et al. , 2013): the Gunning Fog Index (Gunning, 1968) and Flesch-Kincaid Reading Level (Kincaid et al.

03864#pg6>. The results of this classification procedure are intended to capture population-level stylistic changes, and no claim is made towards LLM authorship of individual abstracts <ref:2602.03864#pg6>. The study provides the quantitative context necessary for interpreting stylistic changes driven by LLMs <ref:2602.03864#pg6>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. LLM-powered text analysis tools can be improved to incorporate vocabulary-based frequency-shift methodologies to detect linguistic signals of LLM-assisted writing by comparing observed word frequencies against counterfactual trendlines for pre and post-LLM periods.

  2. The improved system can quantify deviations from long-term vocabulary trends, specifically identifying words with large frequency gaps (delta) and high relative frequency ratios (r), which serve as indicators of LLM influence on CEE scholarship.

  3. The system can utilize a global set of LLM marker words derived from optimized rare- and common-marker sets to classify abstracts as likely LLM-written using a multimarker classification procedure, providing a more conservative estimate than single-marker approaches.

  4. The improved AI can assess how LLM writing changes structural properties such as mean abstract length, sentence length, punctuation use, word choice diversity, and readability scores like the Gunning Fog Index and Flesch-Kincaid Reading Level.

  5. The system can analyze trends in grammatical voice by tracking first-person pronoun use (as a proxy for active voice) and past participles (as a proxy for passive voice), revealing shifts such as reduced reliance on passive constructions and diminished use of qualifying language used to convey uncertainty.

Abstract

Large language models (LLMs) have rapidly emerged in civil and environmental engineering (CEE) research, education, and practice as tools for project ideation, execution, and communication. However, a gap remains in understanding how prevalent LLM adoption is across CEE scholarship and whether it measurably alters research prose. This study applies a vocabulary-based frequency-shift methodology to 149,452 abstracts published by the American Society of Civil Engineers between 2000 and 2025. Deviations from long-term vocabulary trends suggest that 15.3% and 26.9% of 2024 and 2025 abstracts, respectively, are likely LLM-influenced. Prior to the introduction of LLMs in 2022, CEE publications exhibit long-term trends toward longer abstracts, greater use of segmenting punctuation, higher required reading levels, and use of active, first-person verb constructions. Beginning around 2023, however, the frequencies of many stylistic marker words (e.g., enhance) sharply depart from their historical trajectories, a shift consistent with the emergence of LLMs. Abstracts classified as likely LLM-influenced also depart from historical semantic trends, exhibiting increased lexical diversity, comma use, and complexity, with reduced passive voice and hedging language, producing prose that is more segmented, complex, and confident. The artificial intelligence contribution of this study is the use of natural language processing to detect population-level linguistic signals of LLM-generated text, estimating the prevalence of LLM use and its influence on CEE writing. Together, these novel findings provide the first large-scale, data-driven assessment of how CEE scholarly communication is changing alongside LLM adoption.

Sources

Related papers