Large Language Models (GPT) for automating feedback on programming assignments
summary
The gist
Large Language Models (GPT) can be used to generate personalized hints for students solving programming assignments on an automated assessment platform…
In short
The study tested using OpenAI's GPT-3.5 model to provide personalized hints for programming assignments on an automated platform. Students found these GPT-generated hints useful and performed better in terms of successful submissions across consecutive attempts compared to those without the help. This suggests that integrating AI feedback can improve student learning outcomes in programming.
Key concepts
- GPT-3.5 Model
- This is a large language model developed by OpenAI used in the study to automatically generate personalized hints for students solving programming problems. It was employed to provide suggestions for code improvement, explain compiler errors, and offer debugging tips based on the assignment's context.
- Experimental Group
- This group of computer science students received additional features on the assessment platform that enabled them to receive feedback generated by GPT-3.5 for 38 out of 46 tasks. They were compared against a control group that did not have this GPT feedback feature enabled.
- Consecutive Attempts Performance
- This metric measures how well students performed on a task across multiple tries, specifically looking at the percentage of successful submissions after each attempt. The research found that for tasks with GPT hints, the experimental group showed a higher success rate after subsequent attempts compared to when those hints were unavailable.
Terminology used across episodes
This episode discusses
- Large Language Models (GPT) for automating feedback on programming assignments · Paper Radio
- Learning gain differences between ChatGPT and human tutor generated algebra hints
The paper
Large Language Models (GPT) for automating feedback on programming assignments · Read on arXiv
Maciej PANKIEWICZ, Ryan S. BAKER
Warsaw University of Life Sciences · Penn Center for Learning Analytics, University of Pennsylvania
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Large Language Models (GPT) for automating feedback on programming assignments".
Jane: Here's the gist of the paper. Large Language Models (GPT) can be used to generate personalized hints for students solving programming assignments on an automated assessment platform, and experimental results suggest that enabling GPT-generated hints led to better performance in terms of successful submissions across consecutive attempts Study Setup and Methodology The study involved second-semester computer science students at the Warsaw University of Life Sciences (Poland) enrolled in an Object-oriented programming course using the C hash programming language A total of 132 students consented to participate and were randomly assigned to either the control or experimental group The experiment utilized the RunCode online application for automated execution and testing of programming code Forty-six programming assignments covering basic object-oriented programming concepts were made available on the platform for practicing course material Each assignment was composed of a collection of more specific subtasks, such as the creation of a class and the addition of fields with appropriate access modifiers Both groups had access to the same set of tasks within the application For 38 out of 46 tasks, students in the experimental group had an additional feature enabled providing them with feedback generated by GPT-3.5 API This GPT-generated feedback provided suggestions for code improvement, explanations of compiler errors, and hints for debugging Perceived Usefulness and Feedback Usage The examination of the perceived usefulness of the GPT generated hints showed that the majority of users rated the GPT hints they received as useful A total of 1,442 responses were collected from N=46 participants within the experimental group throughout the duration of the study A statistically significant difference was found between the experimental and control groups for tasks where GPT feedback was enabled The experimental group used the regular feedback significantly less than the control group for these tasks However, no significant difference was found between the experimental and control groups for the set of tasks where GPT feedback was not available Performance Metrics Across Conditions The analysis of consecutive task attempts for tasks where GPT hints were enabled revealed that performance worsened across attempts for both groups, but the experimental group appeared to have a slightly higher percentage of successful submissions after the second attempt A linear mixed-effects model revealed a non-significant interaction effect between group and attempt concerning slopes Conversely, for tasks where GPT feedback was not available, there was a significant interaction effect between group and attempt, indicating that the slopes of the control and experimental groups were significantly different Temporal Efficiency and Affective State The analysis of time needed to successfully complete tasks showed that for tasks where GPT feedback was available, the group was not a statistically significant predictor of the time needed to solve tasks However, for tasks where GPT feedback was not available, the experimental group needed less time than the control group Students in the experimental group spent an additional average of 40 seconds after each attempt compared to the control group Regarding affective states, both conditions predominantly reported the focused state, and no significant differences were observed between conditions for any of the reported affective states The availability of GPT hints did not appear to have a substantial impact on student affect The findings indicate that integrating GPT-generated feedback into computer programming education may positively impact student performance, ultimately contributing to enhanced learning outcomes The study suggests that GPT hints played a pivotal role in enhancing the students' learning process by rectifying misconceptions and providing supplementary information Limitations noted include the study duration spanning only 9 weeks of the 15-week semester term and focusing solely on GPT-generated hints in Polish Future research should explore longer-term effects, different languages, and larger sample sizes to establish generalizability The paper concludes that integrating GPT-generated feedback into computer programming education may positively impact student performance The authors also suggest that fine-tuning the model could further improve the quality of generated hints for effective programming education on a larger scale Acknowledgements This paper was written with the assistance of ChatGPT, which was used to improve the writing clarity and grammar of first drafts written by humans All outputs were reviewed and modified by two human authors prior to submission References Benjamini, Y., & Hochberg, Y.
Tom: (1995).
Jane: Controlling the false discovery rate: A practical and powerful approach to multiple testing Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.
Tom: D., Dhariwal, P., - & Amodei, D.
Paper summary: Tom: So we've seen how this study on "Large Language Models (GPT) for automating feedback on programming assignments" tested the idea of using GPT-three point five to give students personalized hints during programming tasks <ref:2307.00150#pg1,Large Language Models (GPT) for automating feedback on programming assignments>. The core finding is that enabling these GPT hints led to better performance in terms of successful submissions across consecutive attempts for those specific tasks>
Jane: That means that having that AI guidance, even if they relied on it less than the regular platform feedback, helped them get more right over time on the problems where GPT was available>
Lu: The study suggests GPT hints played a pivotal role in enhancing the students' learning process by rectifying misconceptions and providing supplementary information>
Meng: It seems like the implication for deployment is that you might be able to support complex tasks better if you can provide that layer of personalized, immediate clarification without needing constant human intervention>
Lalam: The authors also suggest that fine-tuning the model could further improve the quality of generated hints for effective programming education on a larger scale>
Tom: Yeah, so the paper points toward GPT being a tool that helps bridge those gaps in learning when coding is complicated, and they aren't just saying it's good; they're showing better submission percentages>
Conclusion: Tom: So, this paper is about using GPT three point five to give students personalized hints for programming homework on an automated testing platform>
Jane: Right, so we're talking about taking those tricky programming assignments and letting an AI generate tailored help for every student>
Tom: The authors are looking at how much students actually used these hints and if it made a difference in their success rate across multiple tries>
Jane: They found that the majority of users thought the GPT hints were useful, which is a good start, but then they looked at performance metrics across different conditions>
Tom: Yeah, they saw that for tasks where the GPT feedback was enabled, students actually performed better in terms of their percentage of successful submissions after several attempts>
Jane: That's interesting because it shows that this personalized guidance helped them get more right over time on those specific problems>
Tom: And when the GPT feedback wasn't available, they needed less time to solve things, but they were initially less likely to get it right on the first try>
Jane: So what's the bigger picture here for how we teach coding? It suggests that having that immediate AI clarification can really help students get past those sticking points>
Tom: Yeah, it moves beyond just giving them a generic answer; this is about providing feedback tailored to their specific errors on a specific task>
Jane: And the authors are hinting at something important, they're suggesting that we might be able to use this kind of personalized feedback system more widely in computer science education>
Tom: Exactly, and they even mention that fine-tuning the underlying model could really boost the quality of those hints for a lot more assignments>
Jane: It makes you wonder what happens when you start integrating these kinds of smart assistants into everyday learning tools, not just programming courses>
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game