Beyond the Black Box: Integrating Lexical and Semantic Methods in Quantitative Discourse Analysis with BERTopic

summary

The gist

Quantitative Discourse Analysis (QDA) has seen growing adoption with the rise of Large Language Models (LLMs) and computational tools…

In short

The study developed a transparent framework for Quantitative Discourse Analysis by combining lexical methods (like word counts) with semantic methods (BERTopic using sentence embeddings). This hybrid approach allows researchers to triangulate findings, ensuring that precise term detection is grounded in contextual understanding, leading to more robust and interpretable results than using black-box software alone.

Key concepts

Lexical Analysis
This involves traditional word counting techniques like Bag-of-Words (BOW) and n-grams. It precisely counts how often specific words or sequences of words appear in the text, helping researchers identify key terms and their co-occurrence patterns with high accuracy.
Sentence Embeddings
These are numerical vectors that represent text contextually. Unlike simple word counts, embeddings capture the meaning and nuance of a sentence by placing it in a high-dimensional space where similar sentences are positioned close together, allowing for deeper semantic analysis.
BERTopic
This is a modern topic modeling technique that uses sentence embeddings to group similar sentences into coherent topics. It automatically discovers latent themes in the text and provides context-aware clustering, which is more nuanced than older methods like LDA.

Terminology used across episodes

This episode discusses

The paper

Beyond the Black Box: Integrating Lexical and Semantic Methods in Quantitative Discourse Analysis with BERTopic · Read on arXiv

Thomas Compton

University of York

Quantitative Discourse Analysis has seen growing adoption with the rise of Large Language Models and computational tools. However, reliance on black box software such as MAXQDA and NVivo risks undermining methodological transparency and alignment with research goals. This paper presents a hybrid, transparent framework for QDA that combines lexical and semantic methods to enable triangulation, reproducibility, and interpretability. Drawing from a case study in historical political discourse, we demonstrate how custom Python pipelines using NLTK, spaCy, and Sentence Transformers allow fine-grained control over preprocessing, lemmatisation, and embedding generation. We further detail our iterative BERTopic modelling process, incorporating UMAP dimensionality reduction, HDBSCAN clustering, and c-TF-IDF keyword extraction, optimised through parameter tuning and multiple runs to enhance topic coherence and coverage. By juxtaposing precise lexical searches with context-aware semantic clustering, we argue for a multi-layered approach that mitigates the limitations of either method in isolation. Our workflow underscores the importance of code-level transparency, researcher agency, and methodological triangulation in computational discourse studies. Code and supplementary materials are available via GitHub.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Beyond the Black Box".

Jane: Here's the gist of the paper. Quantitative Discourse Analysis (QDA) has seen growing adoption with the rise of Large Language Models (LLMs) and computational tools, but relying on 'black box' software risks undermining methodological transparency and alignment with research goals The paper presents a hybrid, transparent framework for QDA that combines lexical and semantic methods to enable triangulation, reproducibility, and interpretability.

Tom: Pruning redundant reasoning tokens The authors argue that combining bag-of-words (BOW) frequency analysis with sentence embedding-based topic modelling (via BERTopic) enables methodological triangulation.

Jane: This hybrid approach ensures both precision in term detection and sensitivity to contextual meaning.

Tom: The implementation leverages Python, NLTK, spaCy, and Sentence Transformers, allowing full control over preprocessing and model fine-tuning.

Paper summary: Tom: So wrapping up this look at "Beyond the Black Box: Integrating Lexical and Semantic Methods in Quantitative Discourse Analysis with BERTopic". The authors are really pushing for a transparent, triangulated framework to replace the black box reliance in quantitative discourse analysis.

Jane: They show how mixing lexical methods, like bag-of-words and n-grams, with semantic methods like sentence embeddings and BERTopic allows for triangulation across different types of evidence.

Lu: The key contribution is demonstrating how custom Python pipelines using NLTK, spaCy, and Sentence Transformers give researchers fine-grained control over preprocessing and embedding generation.

Meng: It sounds like the authors are saying that if you want to get accurate results on real data at scale, you need to be able to tune every step of the pipeline yourself instead of just clicking buttons in a commercial tool.

Tom: Right, because they show how they optimized their BERTopic process through UMAP dimensionality reduction and HDBSCAN clustering using multiple runs to ensure the topic coherence was solid.

Jane: And by comparing five different embedding models and selecting all-mpnet-base-v2 because it balanced coherence and topic distribution, they provide a concrete example of making those tough choices.

Lu: It’s a call for interdisciplinary training so that there’s a bridge between the technical investment in these custom methods and the actual need for code-level transparency in AI research.

Tom: So ultimately, the message is that this hybrid approach gives you both precise word counts and context-aware structural insights without relying on any single method alone.

Jane: It’s about moving away from just accepting whatever a black box spits out and demanding that we can see exactly how we got there.

Conclusion: Jane: So, to wrap up this look at "Beyond the Black Box: Integrating Lexical and Semantic Methods in Quantitative Discourse Analysis with BERTopic."

Tom: Basically, they’re saying you can’t just rely on a single black box for discourse analysis; you need to combine counting words with understanding the actual meaning of those words.

Lu: That’s right. They built this hybrid framework using lexical methods like bag-of-words and semantic methods using sentence embeddings and BERTopic clustering.

Meng: So, instead of just getting a list of keywords, you get frequency counts *and* you get themes that actually make sense contextually.

Jane: Exactly. The authors show how they used custom Python pipelines with tools like NLTK and spaCy to have total control over every step, from cleaning the text to making the embeddings themselves.

Tom: I’m interested in how they handled those iterative steps, like optimizing that BERTopic process with UMAP dimensionality reduction and HDBSCAN clustering through multiple runs. That sounds like a real engineering effort.

Lu: It is an effort to keep the model coherent and well-distributed, which is important because the stochastic nature of topic modeling can be messy if you don't tune it right.

Meng: I worry about that tuning part—how much work does it really take? If we’re shipping this, how do we make sure that parameter tuning isn't just a lot of manual guesswork?

Jane: That’s the hurdle. But they argue that this level of transparency is what builds trust in the results, which is crucial when you are analyzing something like historical political discourse.

Tom: It seems like the authors really want to move away from just using commercial software and towards letting researchers have agency over how these models work at a code level.

Lu: Precisely. The core argument is that juxtaposing precise lexical searches with context-aware semantic clustering gives you a more complete picture than either method alone can offer.

Jane: It’s about making sure the quantitative metrics we get from word counts actually align with the qualitative framing we see in the themes.

Tom: So, they are pushing for this multi-layered approach to be standard practice in computational discourse studies instead of just an optional add-on.

Lu: Yeah, they’re advocating for a workflow where you have code-level transparency as a baseline requirement for any serious AI analysis in this field.

Meng: If we were deploying something like this, the initial setup complexity is high, but I see the value in being able to audit why the model chose a specific topic.

Jane: And that level of auditability is what makes this paper so interesting—it shows how you can get both precision and context without sacrificing interpretability.

Tom: So it boils down to demanding transparency from the tools we use, whether it's MAXQDA or a custom Python script.

Lu: It’s about researcher agency in the age of black box models. They’re showing that by combining these methods thoughtfully, you get much better results and a clearer understanding of the process behind them.

More episodes

← Home