Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation
A researcher used multiple LLM agents to automate generating and evaluating 1,071 SAT Math MCQs from textbooks. While surface qualities like grammar were high, generated questions lacked deep cognitive alignment and difficulty calibration compared to expert questions. The study shows a labor shift where human work moved from writing content to system specification and validation.