Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Text-Based Visualizations for Learning Analytics".
Jane: Here's the gist of the paper. Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education (Vis4Ed), here we examine the reverse direction (Ed4Vis): what can qualitative education research tell us about building better visualization tools Across two cycles of thematic analysis with confirmatory checking, we report themes spanning tool experience, disciplinary context of use, and, most importantly, a core epistemological dissensus Some instructor-researchers regarded frequency-based visualization as a productive entry point to qualitative analysis; others cautioned it can obscure rare but critical responses We synthesize these findings into design implications for future tools that better integrate quantitative technique with qualitative inquiry Study Design and Methodology We conducted a mixed-methods expert study (n=10) in which STEM education researchers with expertise in qualitative methods and classroom assessment used both platforms to analyze student journal responses from a data visualization course The study design followed a two-phase concurrent mixed-methods design following an initial pilot study The quantitative component compared the two platforms, Dashboard and Maker, on perceived ease of use and usefulness The qualitative component used thematic analysis to explore participants’ experiences and contexts of use beyond what an agreement scale can capture The study procedure overview is shown in Figure 3 Quantitative Findings on Perceived Ease of Use and Usefulness Across all ten participants, the Dashboard was rated at or above the Maker on every Likert item Table 2 reports per-item means derived directly from the study data, and Figure 4 shows detailed agreement ratings on each of the seven Likert items The overall mean was 3.94 for the Dashboard and 3.47 for the Maker; expressed as a share of agree or strongly agree ratings, 77% of Dashboard ratings (54/70) were positive versus 60% for the Maker (42/70) The largest gap is on exploring details (L1, ∆ = 1.30) and almost no difference on broad trends (L2, ∆ = 0.20) and development over time (L6, ∆ = 0.10) Emergent Themes in Tool Experience Three themes characterize how participants experienced the tools and situated them in their own practice Theme T1 focused on "Enhancing navigation and exploration of qualitative patterns across scales of context" where users valued features that enhanced ease of use by making it easier to compare and examine qualitative data in context at different scales of analysis Theme T2 concerned "Extending analytic workflows through user control" where users evaluated the tools’ usefulness largely through the features they wanted added that would extend the visualizations into a fuller analytic pipeline Theme T3 involved "Evaluating features through a deep context-of-use perspective across disciplinary domains" demonstrating how participants evaluated the tools within their specific disciplinary activities Epistemological Dissensus over Encoding Theme T4 emerged only in Cycle 2 and prompted the addition of RQ2, which was a genuine disagreement about whether a frequency-based encoding can carry qualitative meaning At the more skeptical end, Expert 8 elaborated on shifting toward purpose-dependent uses, stating that Word frequency is not a qualitative research tool but can direct researchers to transcripts or videos to help identify patterns Expert 5 made the loss-of-context argument concrete, noting that word count/frequency doesn’t actually tell us a lot, and words in context have a lot more meaning This dissensus is therefore a meaningful disagreement about what the encoding is—and should be—for, instead of a simple deviation from consensus Design Implications for Learning Analytics Tools We translated these findings into five design implications, unifying creation and extraction, making encodings transparent and controllable These include enabling custom data uploads with insight extraction, making encoding choices transparent & controllable, and preserving rare but critical responses Furthermore, we suggest supporting flexible units of analysis and treating frequency as an entry point rather than an endpoint The goal is to integrate quantitative technique with qualitative learning analytics inquiry We set out to learn what qualitative education researchers can teach visualization designers (Ed4Vis), using two WordStreambased platforms: the Journal Data Dashboard and WordStream Maker, as probes into creating and reading visualizations of student journal writing for learning analytics Alongside those ratings, our thematic analysis surfaced a sustained valuing of control and context (Themes 1–3) and, most importantly, an unresolved epistemological dissensus over whether a frequencybased encoding can represent qualitative meaning (Theme 4) We translated these findings into five design implications, unify creation and extraction, make encodings transparent and controllable, preserve rare but critical responses, support flexible units of analysis, and treat frequency as an entry point rather than an endpoint We hope these inform tools that let instructors and researchers move flexibly between the shape of a cohort’s language and the individual voices that make it up Acknowledgments This work was supported in part by the National Aeronautics and Space Administration (NASA) and the Gordon Research Conferences under NASA Award 17-TWSC17-0055 The authors would like to thank Riyansha Goyal (RG), Jaime Serrano (JS), and Jayrylle Jaylo (JJ) for their support of this study References
1: C.
Tom: M.
Jane: Abramson and D.
Tom: Dohan, Beyond text: Using arrays to represent and analyze ethnographic data, Sociological methodology, 45(1):272–319, 2015
3: H.
Title and authors: Tom: So let's talk about the paper, Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers. The title itself tells us they are focusing on how text visualizations can help analyze learning in science, technology, engineering, and math education.
Jane: And the authors are looking at a specific group of people—STEM education researchers—to see what their experiences are with these tools and whether they think they actually work for their research questions.
Lu: The core idea is that when we turn student journal responses into visualizations, we need to think about context, because qualitative research is all about getting a rich understanding of subjects and their specific setting.
Meng: So what's the summary of what they did? Basically, they took two platforms—the Journal Data Dashboard and WordStream Maker—and had these experts test them on things like ease of use and usefulness for analyzing student assessment data.
Lalam: They ran this in two cycles, first with four people and then with six, to make sure their findings were pretty solid.
Tom: And what did they find about the tools themselves? It looks like they found some agreement on how easy the tools were to use overall, but there are some deeper issues surfacing.
Jane: They found that while users liked certain features for navigating and exploring patterns across different scales of context, there's a big debate about how we should be encoding that data.
Lu: That debate is really interesting because it touches on the very nature of what we’re trying to measure when we use these tools.
Meng: I mean, if they can figure out what kind of visual representation makes sense for different research questions, that's a big step toward making these tools more useful in the real world.
The paper's summary: Tom: The summary points out that the study had three main themes emerging from their experience with the tools and how they used them in their own practices.
Jane: They found one theme was about enhancing navigation and exploration of qualitative patterns across different scales of context, which just means making it easier to compare things at different levels of analysis.
Lu: Then there was a theme about extending analytic workflows through user control, meaning users wanted features that would let them build out the visualization pipeline further.
Meng: And the third theme was evaluating features through a deep context-of-use perspective across different academic subjects, showing how specialized those tools need to be.
Lalam: But the big thing they found—and this is key—is this fourth theme about an epistemological dissensus over whether a frequency-based encoding can actually carry qualitative meaning.
Tom: That’s the core conflict, right? Some researchers thought counting how often words appear was a productive way to start analyzing qualitative data, while others were really skeptical that it could capture anything important.
Jane: They pointed out that some experts felt like word count doesn't tell us much and that context around the words is what really gives them meaning.
Lu: It’s like they are disagreeing on whether the encoding itself is supposed to be qualitative or quantitative, which is a deep question for anyone building these systems.
The paper's improvements: Tom: The paper translates those findings into five design implications for future tools, and they focus on making creation and extraction more unified.
Jane: They suggest making the encodings transparent and controllable, which means users should be able to see exactly what's happening with the numbers or words they are seeing.
Lu: They also suggest allowing custom data uploads with insight extraction, so you can bring in your own stuff and have the tool figure out what’s important for you.
Meng: And they want to make encoding choices transparent and controllable, which is important because if a user doesn't control the encoding, they don't trust the output.
Lalam: They also propose preserving rare but critical responses, so even if something doesn't show up often, it’s still there for researchers to see.
Tom: Plus, they suggest supporting flexible units of analysis so you can look at a single student over time or group by question in different ways.
Jane: And treating frequency encoding as an entry point rather than an endpoint, which means using it to find where to go next, not just stopping there with the count.
Conclusion: Tom: So wrapping up the Text-Based Visualizations for Learning Analytics paper, they confirm that control and context are what researchers value most when they use these platforms.
Jane: They also confirmed that there’s a real disagreement about whether frequency based encodings can represent qualitative meaning in this kind of data.
Lu: The implications suggest we need to build tools that let users unify creation and extraction, make choices transparent, and keep those rare but critical responses safe.
Meng: If we take the design implications seriously, it means designing systems where you can import your own data and control how things are encoded rather than just accepting a fixed view.
Lalam: And treating frequency as a starting point for other analysis is a good idea because it lets researchers find where to go next with their deeper qualitative work.
Tom: It’s about moving toward tools that let instructors and researchers move flexibly between the shape of language in a cohort and the individual voices that make up that language.
Huyen N. Nguyen, Kathleen A. Bowe, Minh-Huyen Nguyen, Kit Thompson, Caleb M. Trujillo
Harvard Medical School · University of New Hampshire · Hanoi University of Science and Technology · University of Washington Bothell
cs.CY, cs.HC
Submitted: 2026-06-17
Updated: 2026-11-16
Comments: 11 pages, 6 figures, 2 tables
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 84/100
The gist: Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education…
Key concepts
- Dashboard vs. Maker
- These are two different software platforms used to visualize student journal data from a course. The Dashboard was generally rated as easier to use, while the Maker tool was seen as more useful for extending analytical workflows through user control and customization.
- Epistemological Dissensus over Encoding
- This is a fundamental disagreement among researchers about how word frequency encoding works in qualitative research. Some experts see it as a productive starting point, while others argue that simple counts obscure rare but critical student responses, leading to different views on what the encoding should actually represent.
- Thematic Analysis
- This is a qualitative research method used to identify recurring patterns or themes within text data. In this study, researchers used it to explore how experts experienced the visualization tools and what contexts they applied these tools in their actual teaching and assessment practices.
Terminology
Summary
Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education (Vis4Ed), here we examine the reverse direction (Ed4Vis): what can qualitative education research tell us about building better visualization tools Across two cycles of thematic analysis with confirmatory checking, we report themes spanning tool experience, disciplinary context of use, and, most importantly, a core epistemological dissensus Some instructor-researchers regarded frequency-based visualization as a productive entry point to qualitative analysis; others cautioned it can obscure rare but critical responses We synthesize these findings into design implications for future tools that better integrate quantitative technique with qualitative inquiry
Study Design and Methodology
We conducted a mixed-methods expert study (n=10) in which STEM education researchers with expertise in qualitative methods and classroom assessment used both platforms to analyze student journal responses from a data visualization course The study design followed a two-phase concurrent mixed-methods design following an initial pilot study The quantitative component compared the two platforms, Dashboard and Maker, on perceived ease of use and usefulness The qualitative component used thematic analysis to explore participants’ experiences and contexts of use beyond what an agreement scale can capture The study procedure overview is shown in Figure 3
Quantitative Findings on Perceived Ease of Use and Usefulness
Across all ten participants, the Dashboard was rated at or above the Maker on every Likert item Table 2 reports per-item means derived directly from the study data, and Figure 4 shows detailed agreement ratings on each of the seven Likert items The overall mean was 3.94 for the Dashboard and 3. p. 0 Bowe †
University of New Hampshire
Minh-Huyen Nguyen ‡
Hanoi University of Science and Technology
Kit Thompson§
University of Washington Bothell
Caleb M. Trujillo ¶
University of Washington Bothell
WordStream
Dashboard
Maker
Student
Reflections
STEM instructor-researchers
Figure 1: Ten STEM instructor-researchers evaluate two WordStream-based platforms on student journal reflections: Journal Data
Dashboard for analyzing formative assessments and WordStream Maker for authoring custom visualizations. 47 for the
Maker; expressed as a share of agree or strongly agree ratings,
77% of Dashboard ratings (54/70) were positive versus 60% for
the Maker (42/70). The largest gap is on exploring details (L1,
∆ = 1. The largest gap is on exploring details (L1,
∆ = 1. 30) and almost no difference on broad trends (L2, ∆ = 0. L6 Development. The app gives me insight into the development
of student ideas over time.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Emergent Themes in Tool Experience
Three themes characterize how participants experienced the tools and situated them in their own practice Theme T1 focused on Enhancing navigation and exploration of qualitative patterns across scales of context
where users valued features that enhanced ease of use by making it easier to compare and examine qualitative data in context at different scales of analysis Theme T2 concerned Extending analytic workflows through user control
where users evaluated the tools’ usefulness largely through the features they wanted added that would extend the visualizations into a fuller analytic pipeline Theme T3 involved Evaluating features through a deep context-of-use perspective across disciplinary domains
demonstrating how participants evaluated the tools within their specific disciplinary activities
Epistemological Dissensus over Encoding
Theme T4 emerged only in Cycle 2 and prompted the addition of RQ2, which was a genuine disagreement about whether a frequency-based encoding can carry qualitative meaning At the more skeptical end, Expert 8 elaborated on shifting toward purpose-dependent uses, stating that Word frequency is not a qualitative research tool but can direct researchers to transcripts or videos to help identify patterns Expert 5 made the loss-of-context argument concrete, noting that word count/frequency doesn’t actually tell us a lot, and words in context have a lot more meaning This dissensus is therefore a meaningful disagreement about what the encoding is—and should be—for, instead of a simple deviation from consensus
Design Implications for Learning Analytics Tools
We translated these findings into five design implications, unifying creation and extraction, making encodings transparent and controllable These include enabling custom data uploads with insight extraction, making encoding choices transparent & controllable, and preserving rare but critical responses Furthermore, we suggest supporting flexible units of analysis and treating frequency as an entry point rather than an endpoint The goal is to integrate quantitative technique with qualitative learning analytics inquiry
Conclusion
We set out to learn what qualitative education researchers can teach visualization designers (Ed4Vis), using two WordStreambased platforms: the Journal Data Dashboard and WordStream Maker, as probes into creating and reading visualizations of student journal writing for learning analytics Alongside those ratings, our thematic analysis surfaced a sustained valuing of control and context (Themes 1–3) and, most importantly, an unresolved epistemological dissensus over whether a frequencybased encoding can represent qualitative meaning (Theme 4) We translated these findings into five design implications, unify creation and extraction, make encodings transparent and controllable, preserve rare but critical responses, support flexible units of analysis, and treat frequency as an entry point rather than an endpoint We hope these inform tools that let instructors and researchers move flexibly between the shape of a cohort’s language and the individual voices that make it up
Acknowledgments
This work was supported in part by the National Aeronautics and Space Administration (NASA) and the Gordon Research Conferences under NASA Award 17-TWSC17-0055 The authors would like to thank Riyansha Goyal (RG), Jaime Serrano (JS), and Jayrylle Jaylo (JJ) for their support of this study
References
[1] C. M. Abramson and D. p. 7 REFERENCES
[1] C. Abramson and D.Where the paper says it
[3] H. A. H. Baca et al., Covidstream: Interactive visualization of emotions evolution associated with covid-19, In Annual International Conference on Information Management and Big Data, pp. p. 7 Covidstream: Interactive visualization of emotions evolution associated with covid-19. In Annual International Conference on Information Management and Big Data, pp.Where the paper says it
[37] D. W. Shaffer, Epistemic network analysis: Understanding learning by using big data for thick description, In International handbook of the learning sciences, pp. p. 8 Epistemic network analysis: Understanding learning
by using big data for thick description. In International handbook of
the learning sciences, pp.Where the paper says it
[39] M. Tory, User studies in visualization: A reflection on methods, Handbook of Human Centric Visualization, pp. p. 8 User studies in visualization: A reflection on methods. Handbook of Human Centric Visualization, pp.Where the paper says it
[43] F. B. Viegas et al., Participatory visualization with wordle, IEEE transactions on visualization and computer graphics, 15(6):1137–1144, 2009
[47] S. Zhang et al., Who will support my project? p. 8 153–162, 2010. Who will support
my project? interactive search of potential crowdfunding investors
through insearch. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp.Where the paper says it
Where the paper says it
[5] V. Braun and V. p. 7 08953, 2025. Braun and V.Where the paper says it
[8] S. Chandrasegaran et al. p. 7 1–18, 2022. Chandrasegaran, S.Where the paper says it
[9] L.-T. Chen and L. p. 7 13180 1, 3
[9] L. Chen and L.Where the paper says it
[15] F. D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS quarterly, pp. p. 7 Perceived usefulness, perceived ease of use, and user
acceptance of information technology. MIS quarterly, pp.Where the paper says it
[27] T. Losev et al. p. 7 Our
claims are therefore interpretive, not generalizable. This also resonates with Losev et al.Where the paper says it
[38] Y. Tan et al., Epistemic network analysis and ordered network analysis in learning analytics, In Learning analytics methods and tutorials: A practical guide using R, pp. p. 8 Epistemic network
analysis and ordered network analysis in learning analytics. In Learning analytics methods and tutorials: A practical guide using R, pp.Where the paper says it
[41] A. van den Brandt et al. p. 8 1, 3, 8
[28] S. van den Brandt, E.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified tool development should allow users to
import their own corpus
andadjust the Maker’s display parameters while preserving the Dashboard’s context retrieval path from any stream element back to the verbatim responses.
This enables a single interface that combines data ingestion, preprocessing, and visualization creation for qualitative analysis. -
Encoding transparency must be achieved by providing
clear annotations and explanations of all thresholds and scoring methods in plain-language
and allowing users tocustomize or modify the topic/phrasing list.
This addresses the concern thatExpert 1 did not know the inclusion cutoffs or 'what sudden attention or TF-IDF mean'.
-
The system should implement complementary views, such as a
lowfrequency or outlier panel, or a per-response distribution,
to ensurediversity of voice is not lost to aggregation.
This directly counters the risk thatthe most important factors are those that are not as common
and preserves critical responses. -
The tool should treat frequency encoding as an entry point rather than an endpoint by featuring a
one-click navigation from a stream element to its matching, relevant excerpts, plus export of selected words, phrases, and responses into coding workflows.
This supports the role of frequency as alocator tool to direct me to where I will apply other methodologies.
-
The system should support flexible units of analysis by allowing users
to select the responses of only one student over time
orgrouped by question,
enabling different analytical lenses for the same corpus.
Abstract
Visualizing student reflections collected throughout a course can help instructors track changes in student language and inform teaching decisions. Interpreting these changes, however, requires tools that connect overall patterns with individual responses, and how to design such visualizations for learning analytics remains an open question. To address this gap, we conducted a mixed-methods study with 10 STEM education researchers who offered dual perspectives as both instructors and qualitative researchers. We used two visualization platforms as probes to elicit expert perspectives: the Journal Data Dashboard, for analyzing formative assessments, and WordStream Maker, for authoring custom visualizations. Participants evaluated both platforms by analyzing weekly student journal responses from a data visualization course. Findings revealed shared prioritization of context and control, discipline-rooted differences in perceived usefulness, and, notably, epistemological disagreement about frequency as viable evidence in qualitative work. Some participants regarded frequency-based visualization as a productive entry point to qualitative analysis, while others argued that it conflicts with the values of qualitative data analysis and may obscure rare but critical responses. These findings support the argument that perceived ease of use and usefulness, context of use or role, and epistemic frames are relevant to users of qualitative data visualization and learning analytics tools. We discuss design implications for future tools to better integrate quantitative techniques with qualitative interpretation. Transcripts, tutorials, and all supplementary materials are available at https://osf.io/z2f8d.
Sources
- Characterizing Multimodal Interaction in Visualization Authoring Tools
- Characterizing Grounded Theory Approaches in Visualization
- Sycamore: Grounded Synthetic Evaluation for Visualization Systems
- Visualization Retrieval for Data Literacy: Position Paper
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance