From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations
The episode discusses a paper using Large Language Models to calibrate programming exam difficulty. The study showed AI can rank problem difficulty, correlating with student pass rates, using both solving-based and cheaper review-based methods. Key findings include the need for context like exposure adjustment and identifying course boundaries where the AI tool is most effective.