ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation
cs.CY, cs.AI, cs.SE
Submitted: 2025-07-16
Updated: 2026-09-01
Comments: Accepted to the Main Conference of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)
Code: https://github.com/mmiroyan/ParaStudent
License: http://creativecommons.org/licenses/by/4.0/
The gist: Evaluating Artificial Intelligence (AI) tutor feedback before deployment requires anticipating student engagement, typically assessed through real interaction data.
Terminology
Abstract
Evaluating Artificial Intelligence (AI) tutor feedback before deployment requires anticipating student engagement, typically assessed through real interaction data. We introduce ParaStudent, a fine-tuning framework for simulating novice programming revisions to support AI tutor evaluation. Compared with prompted baselines, ParaStudent's revisions more closely match real student code distributions across functional, stylistic, and semantic metrics. Our best variant achieves AUCs of 0.80 for both feedback relevance and successful uptake when distinguishing streams with real engagement above versus at or below the median, while prompted baselines remain near chance on successful uptake. These findings demonstrate the promise of simulated engagement for pre-deployment feedback triage.
Sources
- Evaluating Large Language Models Trained on Code
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
- Qwen2.5-Coder Technical Report
- A Survey on Large Language Models for Code Generation
- StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
- CodeMonkeys: Scaling Test-Time Compute for Software Engineering
- The Llama 3 Herd of Models
- An Exploratory Study on Fine-Tuning Large Language Models for Secure Code Generation
- CoIR: A Comprehensive Benchmark for Code Information Retrieval Models
- Is Self-Repair a Silver Bullet for Code Generation?
- CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval
- StarCoder 2 and The Stack v2: The Next Generation
- Training Software Engineering Agents and Verifiers with SWE-Gym
- MemGPT: Towards LLMs as Operating Systems
- An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation
- Code Llama: Open Foundation Models for Code
- LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation
- Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models
Related papers
- Reasoning Enhances Robustness to Prompt Injection in LLM-Based Consensus
- Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot
- PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks
- What is an intelligent system?
- AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study
- Generative AI Use in Entrepreneurship: An Integrative Review and an Empowerment-Entrapment Framework