TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education

arXiv:2608.26184 · cs.AI, cs.HC · Submitted 2026-08-22 · Read on arXiv

cs.AI, cs.HC

Submitted: 2026-08-22

Updated: 2026-08-22

Comments: Accepted to ACM UIST 2026, Detroit, MI, USA, 14 pages, 5 figures Dataset: https://vizpi.org/dataset

Journal ref: Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26), Detroit, MI, USA, 2026

DOI: 10.1145/3830398.3830712

License: http://creativecommons.org/licenses/by/4.0/

The gist: AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners' needs.

Abstract

AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners' needs. We present TutorTrace, a dataset and behavioral abstraction pipeline that makes learners' behavioral context visible and computable in real time from low-level IDE telemetry. Across four deployments in two introductory Python courses (N=480), TutorTrace captures approximately 180K telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics. From this foundation, we derive a taxonomy of learner activity before the first AI query, between consecutive queries, and across the full session, enabling systems to respond not just to what learners say, but to what they have done leading up to the help-seeking moment. In a preliminary classroom evaluation, behavior-aware prompts were associated with a decrease in intervals between queries with no independent work from 50.0% to 20.7%. As an additional demonstration of downstream utility, we evaluate TutorTrace on two held-out prediction tasks: whether a learner will query within the next 60 seconds (AUROC=.726) and whether an upcoming query reflects guided or dependent help-seeking (AUROC=.717). Together, these findings show how behavioral context can enable adaptive AI tutoring at scale.

Related papers