Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers
summary
The gist
Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education…
In short
Researchers studied two data visualization tools for analyzing student journal responses in a STEM course. They found that one tool (Dashboard) was rated easier to use overall, but both tools revealed a core disagreement: whether using word frequency as an analysis method actually provides meaningful qualitative insights. The study suggests future tools must offer greater user control and transparency over how data is encoded to serve both quantitative and qualitative needs.
Key concepts
- Dashboard vs. Maker
- These are two different software platforms used to visualize student journal data from a course. The Dashboard was generally rated as easier to use, while the Maker tool was seen as more useful for extending analytical workflows through user control and customization.
- Epistemological Dissensus over Encoding
- This is a fundamental disagreement among researchers about how word frequency encoding works in qualitative research. Some experts see it as a productive starting point, while others argue that simple counts obscure rare but critical student responses, leading to different views on what the encoding should actually represent.
- Thematic Analysis
- This is a qualitative research method used to identify recurring patterns or themes within text data. In this study, researchers used it to explore how experts experienced the visualization tools and what contexts they applied these tools in their actual teaching and assessment practices.
Terminology used across episodes
This episode discusses
- Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers · Paper Radio
- Characterizing Multimodal Interaction in Visualization Authoring Tools
- Characterizing Grounded Theory Approaches in Visualization
- Sycamore: Grounded Synthetic Evaluation for Visualization Systems · Paper Radio
- Visualization Retrieval for Data Literacy: Position Paper
The paper
Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers · Read on arXiv
Huyen N. Nguyen, Kathleen A. Bowe, Minh-Huyen Nguyen, Kit Thompson, Caleb M. Trujillo
Harvard Medical School · University of New Hampshire · Hanoi University of Science and Technology · University of Washington Bothell
Visualizing student reflections collected throughout a course can help instructors track changes in student language and inform teaching decisions. Interpreting these changes, however, requires tools that connect overall patterns with individual responses, and how to design such visualizations for learning analytics remains an open question. To address this gap, we conducted a mixed-methods study with 10 STEM education researchers who offered dual perspectives as both instructors and qualitative researchers. We used two visualization platforms as probes to elicit expert perspectives: the Journal Data Dashboard, for analyzing formative assessments, and WordStream Maker, for authoring custom visualizations. Participants evaluated both platforms by analyzing weekly student journal responses from a data visualization course. Findings revealed shared prioritization of context and control, discipline-rooted differences in perceived usefulness, and, notably, epistemological disagreement about frequency as viable evidence in qualitative work. Some participants regarded frequency-based visualization as a productive entry point to qualitative analysis, while others argued that it conflicts with the values of qualitative data analysis and may obscure rare but critical responses. These findings support the argument that perceived ease of use and usefulness, context of use or role, and epistemic frames are relevant to users of qualitative data visualization and learning analytics tools. We discuss design implications for future tools to better integrate quantitative techniques with qualitative interpretation. Transcripts, tutorials, and all supplementary materials are available at https://osf.io/z2f8d.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Text-Based Visualizations for Learning Analytics".
Jane: Here's the gist of the paper. Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education (Vis4Ed), here we examine the reverse direction (Ed4Vis): what can qualitative education research tell us about building better visualization tools Across two cycles of thematic analysis with confirmatory checking, we report themes spanning tool experience, disciplinary context of use, and, most importantly, a core epistemological dissensus Some instructor-researchers regarded frequency-based visualization as a productive entry point to qualitative analysis; others cautioned it can obscure rare but critical responses We synthesize these findings into design implications for future tools that better integrate quantitative technique with qualitative inquiry Study Design and Methodology We conducted a mixed-methods expert study (n=10) in which STEM education researchers with expertise in qualitative methods and classroom assessment used both platforms to analyze student journal responses from a data visualization course The study design followed a two-phase concurrent mixed-methods design following an initial pilot study The quantitative component compared the two platforms, Dashboard and Maker, on perceived ease of use and usefulness The qualitative component used thematic analysis to explore participants’ experiences and contexts of use beyond what an agreement scale can capture The study procedure overview is shown in Figure 3 Quantitative Findings on Perceived Ease of Use and Usefulness Across all ten participants, the Dashboard was rated at or above the Maker on every Likert item Table 2 reports per-item means derived directly from the study data, and Figure 4 shows detailed agreement ratings on each of the seven Likert items The overall mean was 3.94 for the Dashboard and 3.47 for the Maker; expressed as a share of agree or strongly agree ratings, 77% of Dashboard ratings (54/70) were positive versus 60% for the Maker (42/70) The largest gap is on exploring details (L1, ∆ = 1.30) and almost no difference on broad trends (L2, ∆ = 0.20) and development over time (L6, ∆ = 0.10) Emergent Themes in Tool Experience Three themes characterize how participants experienced the tools and situated them in their own practice Theme T1 focused on "Enhancing navigation and exploration of qualitative patterns across scales of context" where users valued features that enhanced ease of use by making it easier to compare and examine qualitative data in context at different scales of analysis Theme T2 concerned "Extending analytic workflows through user control" where users evaluated the tools’ usefulness largely through the features they wanted added that would extend the visualizations into a fuller analytic pipeline Theme T3 involved "Evaluating features through a deep context-of-use perspective across disciplinary domains" demonstrating how participants evaluated the tools within their specific disciplinary activities Epistemological Dissensus over Encoding Theme T4 emerged only in Cycle 2 and prompted the addition of RQ2, which was a genuine disagreement about whether a frequency-based encoding can carry qualitative meaning At the more skeptical end, Expert 8 elaborated on shifting toward purpose-dependent uses, stating that Word frequency is not a qualitative research tool but can direct researchers to transcripts or videos to help identify patterns Expert 5 made the loss-of-context argument concrete, noting that word count/frequency doesn’t actually tell us a lot, and words in context have a lot more meaning This dissensus is therefore a meaningful disagreement about what the encoding is—and should be—for, instead of a simple deviation from consensus Design Implications for Learning Analytics Tools We translated these findings into five design implications, unifying creation and extraction, making encodings transparent and controllable These include enabling custom data uploads with insight extraction, making encoding choices transparent & controllable, and preserving rare but critical responses Furthermore, we suggest supporting flexible units of analysis and treating frequency as an entry point rather than an endpoint The goal is to integrate quantitative technique with qualitative learning analytics inquiry We set out to learn what qualitative education researchers can teach visualization designers (Ed4Vis), using two WordStreambased platforms: the Journal Data Dashboard and WordStream Maker, as probes into creating and reading visualizations of student journal writing for learning analytics Alongside those ratings, our thematic analysis surfaced a sustained valuing of control and context (Themes 1–3) and, most importantly, an unresolved epistemological dissensus over whether a frequencybased encoding can represent qualitative meaning (Theme 4) We translated these findings into five design implications, unify creation and extraction, make encodings transparent and controllable, preserve rare but critical responses, support flexible units of analysis, and treat frequency as an entry point rather than an endpoint We hope these inform tools that let instructors and researchers move flexibly between the shape of a cohort’s language and the individual voices that make it up Acknowledgments This work was supported in part by the National Aeronautics and Space Administration (NASA) and the Gordon Research Conferences under NASA Award 17-TWSC17-0055 The authors would like to thank Riyansha Goyal (RG), Jaime Serrano (JS), and Jayrylle Jaylo (JJ) for their support of this study References
1: C.
Tom: M.
Jane: Abramson and D.
Tom: Dohan, Beyond text: Using arrays to represent and analyze ethnographic data, Sociological methodology, 45(1):272–319, 2015
3: H.
Title and authors: Tom: So let's talk about the paper, Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers. The title itself tells us they are focusing on how text visualizations can help analyze learning in science, technology, engineering, and math education.
Jane: And the authors are looking at a specific group of people—STEM education researchers—to see what their experiences are with these tools and whether they think they actually work for their research questions.
Lu: The core idea is that when we turn student journal responses into visualizations, we need to think about context, because qualitative research is all about getting a rich understanding of subjects and their specific setting.
Meng: So what's the summary of what they did? Basically, they took two platforms—the Journal Data Dashboard and WordStream Maker—and had these experts test them on things like ease of use and usefulness for analyzing student assessment data.
Lalam: They ran this in two cycles, first with four people and then with six, to make sure their findings were pretty solid.
Tom: And what did they find about the tools themselves? It looks like they found some agreement on how easy the tools were to use overall, but there are some deeper issues surfacing.
Jane: They found that while users liked certain features for navigating and exploring patterns across different scales of context, there's a big debate about how we should be encoding that data.
Lu: That debate is really interesting because it touches on the very nature of what we’re trying to measure when we use these tools.
Meng: I mean, if they can figure out what kind of visual representation makes sense for different research questions, that's a big step toward making these tools more useful in the real world.
The paper's summary: Tom: The summary points out that the study had three main themes emerging from their experience with the tools and how they used them in their own practices.
Jane: They found one theme was about enhancing navigation and exploration of qualitative patterns across different scales of context, which just means making it easier to compare things at different levels of analysis.
Lu: Then there was a theme about extending analytic workflows through user control, meaning users wanted features that would let them build out the visualization pipeline further.
Meng: And the third theme was evaluating features through a deep context-of-use perspective across different academic subjects, showing how specialized those tools need to be.
Lalam: But the big thing they found—and this is key—is this fourth theme about an epistemological dissensus over whether a frequency-based encoding can actually carry qualitative meaning.
Tom: That’s the core conflict, right? Some researchers thought counting how often words appear was a productive way to start analyzing qualitative data, while others were really skeptical that it could capture anything important.
Jane: They pointed out that some experts felt like word count doesn't tell us much and that context around the words is what really gives them meaning.
Lu: It’s like they are disagreeing on whether the encoding itself is supposed to be qualitative or quantitative, which is a deep question for anyone building these systems.
The paper's improvements: Tom: The paper translates those findings into five design implications for future tools, and they focus on making creation and extraction more unified.
Jane: They suggest making the encodings transparent and controllable, which means users should be able to see exactly what's happening with the numbers or words they are seeing.
Lu: They also suggest allowing custom data uploads with insight extraction, so you can bring in your own stuff and have the tool figure out what’s important for you.
Meng: And they want to make encoding choices transparent and controllable, which is important because if a user doesn't control the encoding, they don't trust the output.
Lalam: They also propose preserving rare but critical responses, so even if something doesn't show up often, it’s still there for researchers to see.
Tom: Plus, they suggest supporting flexible units of analysis so you can look at a single student over time or group by question in different ways.
Jane: And treating frequency encoding as an entry point rather than an endpoint, which means using it to find where to go next, not just stopping there with the count.
Conclusion: Tom: So wrapping up the Text-Based Visualizations for Learning Analytics paper, they confirm that control and context are what researchers value most when they use these platforms.
Jane: They also confirmed that there’s a real disagreement about whether frequency based encodings can represent qualitative meaning in this kind of data.
Lu: The implications suggest we need to build tools that let users unify creation and extraction, make choices transparent, and keep those rare but critical responses safe.
Meng: If we take the design implications seriously, it means designing systems where you can import your own data and control how things are encoded rather than just accepting a fixed view.
Lalam: And treating frequency as a starting point for other analysis is a good idea because it lets researchers find where to go next with their deeper qualitative work.
Tom: It’s about moving toward tools that let instructors and researchers move flexibly between the shape of language in a cohort and the individual voices that make up that language.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game