Large Language Models (GPT) for automating feedback on programming assignments

arXiv:2307.00150 · cs.HC, cs.AI · Submitted 2023-06-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Large Language Models (GPT) for automating feedback on programming assignments".

Jane: Here's the gist of the paper. Large Language Models (GPT) can be used to generate personalized hints for students solving programming assignments on an automated assessment platform, and experimental results suggest that enabling GPT-generated hints led to better performance in terms of successful submissions across consecutive attempts Study Setup and Methodology The study involved second-semester computer science students at the Warsaw University of Life Sciences (Poland) enrolled in an Object-oriented programming course using the C hash programming language A total of 132 students consented to participate and were randomly assigned to either the control or experimental group The experiment utilized the RunCode online application for automated execution and testing of programming code Forty-six programming assignments covering basic object-oriented programming concepts were made available on the platform for practicing course material Each assignment was composed of a collection of more specific subtasks, such as the creation of a class and the addition of fields with appropriate access modifiers Both groups had access to the same set of tasks within the application For 38 out of 46 tasks, students in the experimental group had an additional feature enabled providing them with feedback generated by GPT-3.5 API This GPT-generated feedback provided suggestions for code improvement, explanations of compiler errors, and hints for debugging Perceived Usefulness and Feedback Usage The examination of the perceived usefulness of the GPT generated hints showed that the majority of users rated the GPT hints they received as useful A total of 1,442 responses were collected from N=46 participants within the experimental group throughout the duration of the study A statistically significant difference was found between the experimental and control groups for tasks where GPT feedback was enabled The experimental group used the regular feedback significantly less than the control group for these tasks However, no significant difference was found between the experimental and control groups for the set of tasks where GPT feedback was not available Performance Metrics Across Conditions The analysis of consecutive task attempts for tasks where GPT hints were enabled revealed that performance worsened across attempts for both groups, but the experimental group appeared to have a slightly higher percentage of successful submissions after the second attempt A linear mixed-effects model revealed a non-significant interaction effect between group and attempt concerning slopes Conversely, for tasks where GPT feedback was not available, there was a significant interaction effect between group and attempt, indicating that the slopes of the control and experimental groups were significantly different Temporal Efficiency and Affective State The analysis of time needed to successfully complete tasks showed that for tasks where GPT feedback was available, the group was not a statistically significant predictor of the time needed to solve tasks However, for tasks where GPT feedback was not available, the experimental group needed less time than the control group Students in the experimental group spent an additional average of 40 seconds after each attempt compared to the control group Regarding affective states, both conditions predominantly reported the focused state, and no significant differences were observed between conditions for any of the reported affective states The availability of GPT hints did not appear to have a substantial impact on student affect The findings indicate that integrating GPT-generated feedback into computer programming education may positively impact student performance, ultimately contributing to enhanced learning outcomes The study suggests that GPT hints played a pivotal role in enhancing the students' learning process by rectifying misconceptions and providing supplementary information Limitations noted include the study duration spanning only 9 weeks of the 15-week semester term and focusing solely on GPT-generated hints in Polish Future research should explore longer-term effects, different languages, and larger sample sizes to establish generalizability The paper concludes that integrating GPT-generated feedback into computer programming education may positively impact student performance The authors also suggest that fine-tuning the model could further improve the quality of generated hints for effective programming education on a larger scale Acknowledgements This paper was written with the assistance of ChatGPT, which was used to improve the writing clarity and grammar of first drafts written by humans All outputs were reviewed and modified by two human authors prior to submission References Benjamini, Y., & Hochberg, Y.

Tom: (1995).

Jane: Controlling the false discovery rate: A practical and powerful approach to multiple testing Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.

Tom: D., Dhariwal, P., - & Amodei, D.

Paper summary: Tom: So we've seen how this study on "Large Language Models (GPT) for automating feedback on programming assignments" tested the idea of using GPT-three point five to give students personalized hints during programming tasks <ref:2307.00150#pg1,Large Language Models (GPT) for automating feedback on programming assignments>. The core finding is that enabling these GPT hints led to better performance in terms of successful submissions across consecutive attempts for those specific tasks>

Jane: That means that having that AI guidance, even if they relied on it less than the regular platform feedback, helped them get more right over time on the problems where GPT was available>

Lu: The study suggests GPT hints played a pivotal role in enhancing the students' learning process by rectifying misconceptions and providing supplementary information>

Meng: It seems like the implication for deployment is that you might be able to support complex tasks better if you can provide that layer of personalized, immediate clarification without needing constant human intervention>

Lalam: The authors also suggest that fine-tuning the model could further improve the quality of generated hints for effective programming education on a larger scale>

Tom: Yeah, so the paper points toward GPT being a tool that helps bridge those gaps in learning when coding is complicated, and they aren't just saying it's good; they're showing better submission percentages>

Conclusion: Tom: So, this paper is about using GPT three point five to give students personalized hints for programming homework on an automated testing platform>

Jane: Right, so we're talking about taking those tricky programming assignments and letting an AI generate tailored help for every student>

Tom: The authors are looking at how much students actually used these hints and if it made a difference in their success rate across multiple tries>

Jane: They found that the majority of users thought the GPT hints were useful, which is a good start, but then they looked at performance metrics across different conditions>

Tom: Yeah, they saw that for tasks where the GPT feedback was enabled, students actually performed better in terms of their percentage of successful submissions after several attempts>

Jane: That's interesting because it shows that this personalized guidance helped them get more right over time on those specific problems>

Tom: And when the GPT feedback wasn't available, they needed less time to solve things, but they were initially less likely to get it right on the first try>

Jane: So what's the bigger picture here for how we teach coding? It suggests that having that immediate AI clarification can really help students get past those sticking points>

Tom: Yeah, it moves beyond just giving them a generic answer; this is about providing feedback tailored to their specific errors on a specific task>

Jane: And the authors are hinting at something important, they're suggesting that we might be able to use this kind of personalized feedback system more widely in computer science education>

Tom: Exactly, and they even mention that fine-tuning the underlying model could really boost the quality of those hints for a lot more assignments>

Jane: It makes you wonder what happens when you start integrating these kinds of smart assistants into everyday learning tools, not just programming courses>

Maciej PANKIEWICZ, Ryan S. BAKER

Warsaw University of Life Sciences · Penn Center for Learning Analytics, University of Pennsylvania

cs.HC, cs.AI

Submitted: 2023-06-30

Updated: 2023-06-30

Comments: Submitted to the ICCE 2023 (31st International Conference on Computers in Education)

Journal ref: Proceedings of the 31st International Conference on Computers in Education ICCE2023

DOI: 10.58459/icce.2023.950

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 76/100

The gist: Large Language Models (GPT) can be used to generate personalized hints for students solving programming assignments on an automated assessment platform…

Key concepts

GPT-3.5 Model
This is a large language model developed by OpenAI used in the study to automatically generate personalized hints for students solving programming problems. It was employed to provide suggestions for code improvement, explain compiler errors, and offer debugging tips based on the assignment's context.
Experimental Group
This group of computer science students received additional features on the assessment platform that enabled them to receive feedback generated by GPT-3.5 for 38 out of 46 tasks. They were compared against a control group that did not have this GPT feedback feature enabled.
Consecutive Attempts Performance
This metric measures how well students performed on a task across multiple tries, specifically looking at the percentage of successful submissions after each attempt. The research found that for tasks with GPT hints, the experimental group showed a higher success rate after subsequent attempts compared to when those hints were unavailable.

Summary

Large Language Models (GPT) can be used to generate personalized hints for students solving programming assignments on an automated assessment platform, and experimental results suggest that enabling GPT-generated hints led to better performance in terms of successful submissions across consecutive attempts

Study Setup and Methodology

The study involved second-semester computer science students at the Warsaw University of Life Sciences (Poland) enrolled in an Object-oriented programming course using the C hash programming language A total of 132 students consented to participate and were randomly assigned to either the control or experimental group The experiment utilized the RunCode online application for automated execution and testing of programming code Forty-six programming assignments covering basic object-oriented programming concepts were made available on the platform for practicing course material Each assignment was composed of a collection of more specific subtasks, such as the creation of a class and the addition of fields with appropriate access modifiers Both groups had access to the same set of tasks within the application For 38 out of 46 tasks, students in the experimental group had an additional feature enabled providing them with feedback generated by GPT-3.5 API This GPT-generated feedback provided suggestions for code improvement, explanations of compiler errors, and hints for debugging

Perceived Usefulness and Feedback Usage

The examination of the perceived usefulness of the GPT generated hints showed that the majority of users rated the GPT hints they received as useful A total of 1,442 responses were collected from N=46 participants within the experimental group throughout the duration of the study A statistically significant difference was found between the experimental and control groups for tasks where GPT feedback was enabled The experimental group used the regular feedback significantly less than the control group for these tasks However, no significant difference was found between the experimental and control groups for the set of tasks where GPT feedback was not available

Performance Metrics Across Conditions

The analysis of consecutive task attempts for tasks where GPT hints were enabled revealed that performance worsened across attempts for both groups, but the experimental group appeared to have a slightly higher percentage of successful submissions after the second attempt A linear mixed-effects model revealed a non-significant interaction effect between group and attempt concerning slopes Conversely, for tasks where GPT feedback was not available, there was a significant interaction effect between group and attempt, indicating that the slopes of the control and experimental groups were significantly different

Temporal Efficiency and Affective State

The analysis of time needed to successfully complete tasks showed that for tasks where GPT feedback was available, the group was not a statistically significant predictor of the time needed to solve tasks However, for tasks where GPT feedback was not available, the experimental group needed less time than the control group Students in the experimental group spent an additional average of 40 seconds after each attempt compared to the control group Regarding affective states, both conditions predominantly reported the focused state, and no significant differences were observed between conditions for any of the reported affective states The availability of GPT hints did not appear to have a substantial impact on student affect

Conclusion and Future Directions

The findings indicate that integrating GPT-generated feedback into computer programming education may positively impact student performance, ultimately contributing to enhanced learning outcomes The study suggests that GPT hints played a pivotal role in enhancing the students' learning process by rectifying misconceptions and providing supplementary information Limitations noted include the study duration spanning only 9 weeks of the 15-week semester term and focusing solely on GPT-generated hints in Polish Future research should explore longer-term effects, different languages, and larger sample sizes to establish generalizability The paper concludes that integrating GPT-generated feedback into computer programming education may positively impact student performance The authors also suggest that fine-tuning the model could further improve the quality of generated hints for effective programming education on a larger scale

Acknowledgements

This paper was written with the assistance of ChatGPT, which was used to improve the writing clarity and grammar of first drafts written by humans All outputs were reviewed and modified by two human authors prior to submission

References

Benjamini, Y., & Hochberg, Y. p. 3 (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing Brown, T. p. 9, Mann, B., Ryder, N. p. 9, Subbiah, M. p. 9, Kaplan, J. p. 9 D., Dhariwal, P. p. 9, - & Amodei, D. p. 9 (2020). Language models are few-shot learners Brusilovsky, P. p. 9, Malmi, L. p. 9, Hosseini, R. p. 9, Guerra, J. p. 9, Sirkiä, T. p. 9, - & Pollari-Malmi, K. p. 9 (2018). An integrated practice system for learning programming in Python: design and evaluation Carlini, N. p. 9, Tramer, F. p. 9, Wallace, E. p. 9, Jagielski, M. p. 9, Herbert-Voss, A. p. 9, Lee, K., - & Raffel, C. p. 9 (2021). Extracting Training Data from Large Language Models Dai, W. p. 9, Lin, J., Jin, F., Li, T., Tsai, Y., Gasevic, D. p. 9, & Chen, G. p. 9 (2023). Can Large Language Models Provide Feedback to Students? p. 9 A Case Study on ChatGPT Deeva, G. p. 9, Bogdanova, D. p. 9, Serral, E. p. 9, Snoeck, M. p. 9, & De Weerdt, J. p. 9 (2021). A review of automated feedback systems for learners: Classification framework, challenges and opportunities Edwards, S. p. 9 H., & Murali, K. p. 1 P. (2017). CodeWorkout: short programming exercises with built-in data collection Finnie-Ansley, J. p. 9, Denny, P. p. 9, Becker, B. p. 9 A., Luxton-Reilly, A. p. 9, and Prather, J. p. 9 (2022). The robots are coming: Exploring the implications of openai codex on introductory programming Hull, A. p. 9, & du Boulay, B. p. 0 (2015). Motivational and metacognitive feedback in SQL-Tutor Jackson, G. p. 9 T., & Graesser, A. p. 0 C. (2007). Content matters: An investigation of feedback categories within an ITS Karumbaiah, S. p. 9, Baker, R. p. 9S., Tao, Y., Liu, Z. (2022). How does Students' Affect in Virtual Learning Relate to Their Outcomes? A Systematic Review Challenging the Positive-Negative Dichotomy Kloke, J. p. 9 D., & McKean, J. p. 6 W. (2012). Rfit: Rank-based estimation for linear models Maier, U. p. 9, & Klotz, C. p. 1 (2022). Personalized feedback in digital learning environments: Classification framework and literature review Paiva, J. p. 9 C., Leal, J. P., & Figueira, Á. p. 9 (2022). Automated assessment in computer science education: A state-of-the-art review Pankiewicz, M. p. 9 (2020). Move in the Right Direction: Impacting Students’ Engagement With Gamification in a Programming Course Pardos, Z. p. 9 A., & Bhandari, S. p. 0 (2023). Learning gain differences between ChatGPT and human tutor generated algebra hints Pinheiro, J. p. 9, Bates, D. p. 9, DebRoy, S. p. 9, Sarkar, D. p. 9, Heisterkamp, S. p. 9, Van Willigen, B. p. 9, & Maintainer, R. p. 9 (2017). Package ‘nlme’. p. 5 Linear and nonlinear mixed effects models Radford, A. p. 9, Wu, J., Child, R. p. 9, Luan, D., Amodei, D. p. 9, & Sutskever, I. p. 9 (2019). Language models are unsupervised multitask learners Stanger, N. p. 9 (2018). Semi-automated assessment of SQL schemas via database unit testing

Pruning redundant reasoning tokens

The study utilized the GPT-3.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated feedback generation can be improved by fine-tuning GPT models using examples on which they can improve, as suggested in section 4, to enhance hint quality and increase the proportion of useful hints. This could lead to more effective programming education on a larger scale.

  2. The system can be optimized by employing prompt engineering to further optimize the hint generation process, aiming to increase the efficacy of GPT-based automated feedback and improve student performance.

  3. Future research should explore longer-term effects of GPT-generated hints on student performance and learning outcomes, moving beyond the nine-week study duration to provide a more comprehensive view.

  4. The system's generalizability can be expanded by conducting additional experiments in different languages, as the current findings are limited to Polish. This would establish broader applicability across various educational contexts.

  5. To increase robustness and external validity, future work should examine this type of intervention with a larger sample size than the current 132 students.

Sources

Related papers