Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers

arXiv:2606.18692 · cs.CY, cs.HC · Submitted 2026-06-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Text-Based Visualizations for Learning Analytics".

Jane: Here's the gist of the paper. Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education (Vis4Ed), here we examine the reverse direction (Ed4Vis): what can qualitative education research tell us about building better visualization tools Across two cycles of thematic analysis with confirmatory checking, we report themes spanning tool experience, disciplinary context of use, and, most importantly, a core epistemological dissensus Some instructor-researchers regarded frequency-based visualization as a productive entry point to qualitative analysis; others cautioned it can obscure rare but critical responses We synthesize these findings into design implications for future tools that better integrate quantitative technique with qualitative inquiry Study Design and Methodology We conducted a mixed-methods expert study (n=10) in which STEM education researchers with expertise in qualitative methods and classroom assessment used both platforms to analyze student journal responses from a data visualization course The study design followed a two-phase concurrent mixed-methods design following an initial pilot study The quantitative component compared the two platforms, Dashboard and Maker, on perceived ease of use and usefulness The qualitative component used thematic analysis to explore participants’ experiences and contexts of use beyond what an agreement scale can capture The study procedure overview is shown in Figure 3 Quantitative Findings on Perceived Ease of Use and Usefulness Across all ten participants, the Dashboard was rated at or above the Maker on every Likert item Table 2 reports per-item means derived directly from the study data, and Figure 4 shows detailed agreement ratings on each of the seven Likert items The overall mean was 3.94 for the Dashboard and 3.47 for the Maker; expressed as a share of agree or strongly agree ratings, 77% of Dashboard ratings (54/70) were positive versus 60% for the Maker (42/70) The largest gap is on exploring details (L1, ∆ = 1.30) and almost no difference on broad trends (L2, ∆ = 0.20) and development over time (L6, ∆ = 0.10) Emergent Themes in Tool Experience Three themes characterize how participants experienced the tools and situated them in their own practice Theme T1 focused on "Enhancing navigation and exploration of qualitative patterns across scales of context" where users valued features that enhanced ease of use by making it easier to compare and examine qualitative data in context at different scales of analysis Theme T2 concerned "Extending analytic workflows through user control" where users evaluated the tools’ usefulness largely through the features they wanted added that would extend the visualizations into a fuller analytic pipeline Theme T3 involved "Evaluating features through a deep context-of-use perspective across disciplinary domains" demonstrating how participants evaluated the tools within their specific disciplinary activities Epistemological Dissensus over Encoding Theme T4 emerged only in Cycle 2 and prompted the addition of RQ2, which was a genuine disagreement about whether a frequency-based encoding can carry qualitative meaning At the more skeptical end, Expert 8 elaborated on shifting toward purpose-dependent uses, stating that Word frequency is not a qualitative research tool but can direct researchers to transcripts or videos to help identify patterns Expert 5 made the loss-of-context argument concrete, noting that word count/frequency doesn’t actually tell us a lot, and words in context have a lot more meaning This dissensus is therefore a meaningful disagreement about what the encoding is—and should be—for, instead of a simple deviation from consensus Design Implications for Learning Analytics Tools We translated these findings into five design implications, unifying creation and extraction, making encodings transparent and controllable These include enabling custom data uploads with insight extraction, making encoding choices transparent & controllable, and preserving rare but critical responses Furthermore, we suggest supporting flexible units of analysis and treating frequency as an entry point rather than an endpoint The goal is to integrate quantitative technique with qualitative learning analytics inquiry We set out to learn what qualitative education researchers can teach visualization designers (Ed4Vis), using two WordStreambased platforms: the Journal Data Dashboard and WordStream Maker, as probes into creating and reading visualizations of student journal writing for learning analytics Alongside those ratings, our thematic analysis surfaced a sustained valuing of control and context (Themes 1–3) and, most importantly, an unresolved epistemological dissensus over whether a frequencybased encoding can represent qualitative meaning (Theme 4) We translated these findings into five design implications, unify creation and extraction, make encodings transparent and controllable, preserve rare but critical responses, support flexible units of analysis, and treat frequency as an entry point rather than an endpoint We hope these inform tools that let instructors and researchers move flexibly between the shape of a cohort’s language and the individual voices that make it up Acknowledgments This work was supported in part by the National Aeronautics and Space Administration (NASA) and the Gordon Research Conferences under NASA Award 17-TWSC17-0055 The authors would like to thank Riyansha Goyal (RG), Jaime Serrano (JS), and Jayrylle Jaylo (JJ) for their support of this study References

1: C.

Tom: M.

Jane: Abramson and D.

Tom: Dohan, Beyond text: Using arrays to represent and analyze ethnographic data, Sociological methodology, 45(1):272–319, 2015

3: H.

Title and authors: Tom: So let's talk about the paper, Text-Based Visualizations for Learning Analytics: Perspectives from STEM Education Researchers. The title itself tells us they are focusing on how text visualizations can help analyze learning in science, technology, engineering, and math education.

Jane: And the authors are looking at a specific group of people—STEM education researchers—to see what their experiences are with these tools and whether they think they actually work for their research questions.

Lu: The core idea is that when we turn student journal responses into visualizations, we need to think about context, because qualitative research is all about getting a rich understanding of subjects and their specific setting.

Meng: So what's the summary of what they did? Basically, they took two platforms—the Journal Data Dashboard and WordStream Maker—and had these experts test them on things like ease of use and usefulness for analyzing student assessment data.

Lalam: They ran this in two cycles, first with four people and then with six, to make sure their findings were pretty solid.

Tom: And what did they find about the tools themselves? It looks like they found some agreement on how easy the tools were to use overall, but there are some deeper issues surfacing.

Jane: They found that while users liked certain features for navigating and exploring patterns across different scales of context, there's a big debate about how we should be encoding that data.

Lu: That debate is really interesting because it touches on the very nature of what we’re trying to measure when we use these tools.

Meng: I mean, if they can figure out what kind of visual representation makes sense for different research questions, that's a big step toward making these tools more useful in the real world.

The paper's summary: Tom: The summary points out that the study had three main themes emerging from their experience with the tools and how they used them in their own practices.

Jane: They found one theme was about enhancing navigation and exploration of qualitative patterns across different scales of context, which just means making it easier to compare things at different levels of analysis.

Lu: Then there was a theme about extending analytic workflows through user control, meaning users wanted features that would let them build out the visualization pipeline further.

Meng: And the third theme was evaluating features through a deep context-of-use perspective across different academic subjects, showing how specialized those tools need to be.

Lalam: But the big thing they found—and this is key—is this fourth theme about an epistemological dissensus over whether a frequency-based encoding can actually carry qualitative meaning.

Tom: That’s the core conflict, right? Some researchers thought counting how often words appear was a productive way to start analyzing qualitative data, while others were really skeptical that it could capture anything important.

Jane: They pointed out that some experts felt like word count doesn't tell us much and that context around the words is what really gives them meaning.

Lu: It’s like they are disagreeing on whether the encoding itself is supposed to be qualitative or quantitative, which is a deep question for anyone building these systems.

The paper's improvements: Tom: The paper translates those findings into five design implications for future tools, and they focus on making creation and extraction more unified.

Jane: They suggest making the encodings transparent and controllable, which means users should be able to see exactly what's happening with the numbers or words they are seeing.

Lu: They also suggest allowing custom data uploads with insight extraction, so you can bring in your own stuff and have the tool figure out what’s important for you.

Meng: And they want to make encoding choices transparent and controllable, which is important because if a user doesn't control the encoding, they don't trust the output.

Lalam: They also propose preserving rare but critical responses, so even if something doesn't show up often, it’s still there for researchers to see.

Tom: Plus, they suggest supporting flexible units of analysis so you can look at a single student over time or group by question in different ways.

Jane: And treating frequency encoding as an entry point rather than an endpoint, which means using it to find where to go next, not just stopping there with the count.

Conclusion: Tom: So wrapping up the Text-Based Visualizations for Learning Analytics paper, they confirm that control and context are what researchers value most when they use these platforms.

Jane: They also confirmed that there’s a real disagreement about whether frequency based encodings can represent qualitative meaning in this kind of data.

Lu: The implications suggest we need to build tools that let users unify creation and extraction, make choices transparent, and keep those rare but critical responses safe.

Meng: If we take the design implications seriously, it means designing systems where you can import your own data and control how things are encoded rather than just accepting a fixed view.

Lalam: And treating frequency as a starting point for other analysis is a good idea because it lets researchers find where to go next with their deeper qualitative work.

Tom: It’s about moving toward tools that let instructors and researchers move flexibly between the shape of language in a cohort and the individual voices that make up that language.

Huyen N. Nguyen, Kathleen A. Bowe, Minh-Huyen Nguyen, Kit Thompson, Caleb M. Trujillo

Harvard Medical School · University of New Hampshire · Hanoi University of Science and Technology · University of Washington Bothell

cs.CY, cs.HC

Submitted: 2026-06-17

Updated: 2026-11-16

Comments: 11 pages, 6 figures, 2 tables

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 84/100

The gist: Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education…

Key concepts

Dashboard vs. Maker
These are two different software platforms used to visualize student journal data from a course. The Dashboard was generally rated as easier to use, while the Maker tool was seen as more useful for extending analytical workflows through user control and customization.
Epistemological Dissensus over Encoding
This is a fundamental disagreement among researchers about how word frequency encoding works in qualitative research. Some experts see it as a productive starting point, while others argue that simple counts obscure rare but critical student responses, leading to different views on what the encoding should actually represent.
Thematic Analysis
This is a qualitative research method used to identify recurring patterns or themes within text data. In this study, researchers used it to explore how experts experienced the visualization tools and what contexts they applied these tools in their actual teaching and assessment practices.

Terminology

Summary

Data-driven learning analytics can surface trends across a student cohort over time, helping instructors improve the learning environment Where the prior work built these platforms for education (Vis4Ed), here we examine the reverse direction (Ed4Vis): what can qualitative education research tell us about building better visualization tools Across two cycles of thematic analysis with confirmatory checking, we report themes spanning tool experience, disciplinary context of use, and, most importantly, a core epistemological dissensus Some instructor-researchers regarded frequency-based visualization as a productive entry point to qualitative analysis; others cautioned it can obscure rare but critical responses We synthesize these findings into design implications for future tools that better integrate quantitative technique with qualitative inquiry

Study Design and Methodology

We conducted a mixed-methods expert study (n=10) in which STEM education researchers with expertise in qualitative methods and classroom assessment used both platforms to analyze student journal responses from a data visualization course The study design followed a two-phase concurrent mixed-methods design following an initial pilot study The quantitative component compared the two platforms, Dashboard and Maker, on perceived ease of use and usefulness The qualitative component used thematic analysis to explore participants’ experiences and contexts of use beyond what an agreement scale can capture The study procedure overview is shown in Figure 3

Quantitative Findings on Perceived Ease of Use and Usefulness

Across all ten participants, the Dashboard was rated at or above the Maker on every Likert item Table 2 reports per-item means derived directly from the study data, and Figure 4 shows detailed agreement ratings on each of the seven Likert items The overall mean was 3.94 for the Dashboard and 3. p. 0

Where the paper says it

Bowe † University of New Hampshire Minh-Huyen Nguyen ‡ Hanoi University of Science and Technology Kit Thompson§ University of Washington Bothell Caleb M. Trujillo ¶ University of Washington Bothell WordStream Dashboard Maker Student Reflections STEM instructor-researchers Figure 1: Ten STEM instructor-researchers evaluate two WordStream-based platforms on student journal reflections: Journal Data Dashboard for analyzing formative assessments and WordStream Maker for authoring custom visualizations.

47 for the Maker; expressed as a share of agree or strongly agree ratings, 77% of Dashboard ratings (54/70) were positive versus 60% for the Maker (42/70) The largest gap is on exploring details (L1, ∆ = 1. p. 4
Where the paper says it

47 for the Maker; expressed as a share of agree or strongly agree ratings, 77% of Dashboard ratings (54/70) were positive versus 60% for the Maker (42/70). The largest gap is on exploring details (L1, ∆ = 1.

30) and almost no difference on broad trends (L2, ∆ = 0. p. 4
Where the paper says it

The largest gap is on exploring details (L1, ∆ = 1. 30) and almost no difference on broad trends (L2, ∆ = 0.

20) and development over time (L6, ∆ = 0. p. 3
Where the paper says it

L6 Development. The app gives me insight into the development of student ideas over time.

10)

Emergent Themes in Tool Experience

Three themes characterize how participants experienced the tools and situated them in their own practice Theme T1 focused on Enhancing navigation and exploration of qualitative patterns across scales of context where users valued features that enhanced ease of use by making it easier to compare and examine qualitative data in context at different scales of analysis Theme T2 concerned Extending analytic workflows through user control where users evaluated the tools’ usefulness largely through the features they wanted added that would extend the visualizations into a fuller analytic pipeline Theme T3 involved Evaluating features through a deep context-of-use perspective across disciplinary domains demonstrating how participants evaluated the tools within their specific disciplinary activities

Epistemological Dissensus over Encoding

Theme T4 emerged only in Cycle 2 and prompted the addition of RQ2, which was a genuine disagreement about whether a frequency-based encoding can carry qualitative meaning At the more skeptical end, Expert 8 elaborated on shifting toward purpose-dependent uses, stating that Word frequency is not a qualitative research tool but can direct researchers to transcripts or videos to help identify patterns Expert 5 made the loss-of-context argument concrete, noting that word count/frequency doesn’t actually tell us a lot, and words in context have a lot more meaning This dissensus is therefore a meaningful disagreement about what the encoding is—and should be—for, instead of a simple deviation from consensus

Design Implications for Learning Analytics Tools

We translated these findings into five design implications, unifying creation and extraction, making encodings transparent and controllable These include enabling custom data uploads with insight extraction, making encoding choices transparent & controllable, and preserving rare but critical responses Furthermore, we suggest supporting flexible units of analysis and treating frequency as an entry point rather than an endpoint The goal is to integrate quantitative technique with qualitative learning analytics inquiry

Conclusion

We set out to learn what qualitative education researchers can teach visualization designers (Ed4Vis), using two WordStreambased platforms: the Journal Data Dashboard and WordStream Maker, as probes into creating and reading visualizations of student journal writing for learning analytics Alongside those ratings, our thematic analysis surfaced a sustained valuing of control and context (Themes 1–3) and, most importantly, an unresolved epistemological dissensus over whether a frequencybased encoding can represent qualitative meaning (Theme 4) We translated these findings into five design implications, unify creation and extraction, make encodings transparent and controllable, preserve rare but critical responses, support flexible units of analysis, and treat frequency as an entry point rather than an endpoint We hope these inform tools that let instructors and researchers move flexibly between the shape of a cohort’s language and the individual voices that make it up

Acknowledgments

This work was supported in part by the National Aeronautics and Space Administration (NASA) and the Gordon Research Conferences under NASA Award 17-TWSC17-0055 The authors would like to thank Riyansha Goyal (RG), Jaime Serrano (JS), and Jayrylle Jaylo (JJ) for their support of this study

References

[1] C. M. Abramson and D. p. 7

Where the paper says it

REFERENCES [1] C. Abramson and D.

Dohan, Beyond text: Using arrays to represent and analyze ethnographic data, Sociological methodology, 45(1):272–319, 2015

[3] H. A. H. Baca et al., Covidstream: Interactive visualization of emotions evolution associated with covid-19, In Annual International Conference on Information Management and Big Data, pp. p. 7

Where the paper says it

Covidstream: Interactive visualization of emotions evolution associated with covid-19. In Annual International Conference on Information Management and Big Data, pp.

540–551, Springer, 2020

[37] D. W. Shaffer, Epistemic network analysis: Understanding learning by using big data for thick description, In International handbook of the learning sciences, pp. p. 8

Where the paper says it

Epistemic network analysis: Understanding learning by using big data for thick description. In International handbook of the learning sciences, pp.

520–531, Routledge, 2018

[39] M. Tory, User studies in visualization: A reflection on methods, Handbook of Human Centric Visualization, pp. p. 8

Where the paper says it

User studies in visualization: A reflection on methods. Handbook of Human Centric Visualization, pp.

411–426, 2014

[43] F. B. Viegas et al., Participatory visualization with wordle, IEEE transactions on visualization and computer graphics, 15(6):1137–1144, 2009

[47] S. Zhang et al., Who will support my project? p. 8

Where the paper says it

153–162, 2010. Who will support my project?

, interactive search of potential crowdfunding investors through insearch, In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. p. 8
Where the paper says it

interactive search of potential crowdfunding investors through insearch. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp.

1–6, 2022

[5] V. Braun and V. p. 7

Where the paper says it

08953, 2025. Braun and V.

Clarke, Using thematic analysis in psychology, Qualitative research in psychology, 3(2):77–101, 2006

[8] S. Chandrasegaran et al. p. 7

Where the paper says it

1–18, 2022. Chandrasegaran, S.

, Integrating visual analytics support for grounded theory practice in qualitative text analysis, Computer Graphics Forum, 36(3):201–212, 2017

[9] L.-T. Chen and L. p. 7

Where the paper says it

13180 1, 3 [9] L. Chen and L.

Liu, Methods to analyze likert-type data in educational technology research, Journal of Educational Technology Development and Exchange (JETDE), 13(2):3, 2021

[15] F. D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS quarterly, pp. p. 7

Where the paper says it

Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, pp.

319–340, 1989

[27] T. Losev et al. p. 7

Where the paper says it

Our claims are therefore interpretive, not generalizable. This also resonates with Losev et al.

, Embracing disciplinary diversity in visualization, IEEE Computer Graphics and Applications, 42(6):64–71, 2022

[38] Y. Tan et al., Epistemic network analysis and ordered network analysis in learning analytics, In Learning analytics methods and tutorials: A practical guide using R, pp. p. 8

Where the paper says it

Epistemic network analysis and ordered network analysis in learning analytics. In Learning analytics methods and tutorials: A practical guide using R, pp.

569–636, Springer, 2024

[41] A. van den Brandt et al. p. 8

Where the paper says it

1, 3, 8 [28] S. van den Brandt, E.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified tool development should allow users to import their own corpus and adjust the Maker’s display parameters while preserving the Dashboard’s context retrieval path from any stream element back to the verbatim responses. This enables a single interface that combines data ingestion, preprocessing, and visualization creation for qualitative analysis.

  2. Encoding transparency must be achieved by providing clear annotations and explanations of all thresholds and scoring methods in plain-language and allowing users to customize or modify the topic/phrasing list. This addresses the concern that Expert 1 did not know the inclusion cutoffs or 'what sudden attention or TF-IDF mean'.

  3. The system should implement complementary views, such as a lowfrequency or outlier panel, or a per-response distribution, to ensure diversity of voice is not lost to aggregation. This directly counters the risk that the most important factors are those that are not as common and preserves critical responses.

  4. The tool should treat frequency encoding as an entry point rather than an endpoint by featuring a one-click navigation from a stream element to its matching, relevant excerpts, plus export of selected words, phrases, and responses into coding workflows. This supports the role of frequency as a locator tool to direct me to where I will apply other methodologies.

  5. The system should support flexible units of analysis by allowing users to select the responses of only one student over time or grouped by question, enabling different analytical lenses for the same corpus.

Abstract

Visualizing student reflections collected throughout a course can help instructors track changes in student language and inform teaching decisions. Interpreting these changes, however, requires tools that connect overall patterns with individual responses, and how to design such visualizations for learning analytics remains an open question. To address this gap, we conducted a mixed-methods study with 10 STEM education researchers who offered dual perspectives as both instructors and qualitative researchers. We used two visualization platforms as probes to elicit expert perspectives: the Journal Data Dashboard, for analyzing formative assessments, and WordStream Maker, for authoring custom visualizations. Participants evaluated both platforms by analyzing weekly student journal responses from a data visualization course. Findings revealed shared prioritization of context and control, discipline-rooted differences in perceived usefulness, and, notably, epistemological disagreement about frequency as viable evidence in qualitative work. Some participants regarded frequency-based visualization as a productive entry point to qualitative analysis, while others argued that it conflicts with the values of qualitative data analysis and may obscure rare but critical responses. These findings support the argument that perceived ease of use and usefulness, context of use or role, and epistemic frames are relevant to users of qualitative data visualization and learning analytics tools. We discuss design implications for future tools to better integrate quantitative techniques with qualitative interpretation. Transcripts, tutorials, and all supplementary materials are available at https://osf.io/z2f8d.

Sources

Related papers