Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

The gist: an empirical account of an end-to-end research workflow executed by a single researcher orchestrating multiple LLMs and agentic tools, making labor shifts from artifact production to specification, coordination, and validation.

Research Objective and Workflow

The study aimed to answer the central empirical question: To what extent are SAT questions generated by LLMs comparable to human-authored questions along pedagogical quality, cognitive demand, and assessment validity dimensions? (Page 2). This was addressed through a research application defined as the large-scale generation and evaluation of SAT Math multiple-choice questions (MCQs) derived from open-access textbooks (Page 2). The workflow proceeded through six structured phases: Phase 1: Data Identification and Acquisition, Phase 2: Data Extraction and Corpus Construction, Phase 3: Content Alignment and Structuring, Phase 4: Artifact Generation, Phase 5: Artifact Evaluation, and finally, Phase 6: Human Interpretation and Insight Development (Page 3).

Data Sources and Generation Process

The baseline dataset consisted of 1,071 SAT Math MCQs collected from the College Board educator question bank (Page 3). To create the generated sets, the process involved several steps:

** Phase 2 involved converting PDF content into structured JSON files using Google AI Studio Gemini 3 Pro (Page 5). **

** The researcher coordinated AI to search open-source textbooks like OpenStax and convert PDFs into markdown format, chunking the text into retrievable segments (Page 5). **

** Each baseline MCQ was mapped to its most semantically relevant textbook chunk(s) using a combined relevance score: score = 0.3 × simmetadata + 0.7 × simcontent (Page 6). **

** New MCQs were generated by prompting GPT-5-nano and Gemini-2.5-Flash with the baseline MCQ's metadata and retrieved grounding text (Page 6). **

Evaluation Metrics and Findings

The quality of the generated MCQs was assessed using a 24-criterion quality framework (Page 3). This rubric covered six thematic categories, including "Accuracy & Grounding," "Clarity & Presentation, and Difficulty & Cognitive Calibration" (Page 6). The evaluation utilized a dual-judge design, employing two representative LLMs as independent evaluators to ensure methodological triangulation (Page 6).

** Aggregated quality showed that mean scores were high across all conditions, with average scores clustering near the top of the 1-5 scale (Page 7). **

** Universally strong criteria included grammar fluency, no duplicates, and factual accuracy, which achieved near-ceiling scores regardless of the generator or judge (Page 9). **

However, criterion-level analysis revealed significant gaps: criterion-level analysis and equivalence testing show that generated MCQs are not fully comparable to expert-vetted baseline questions (Page 1). Specifically, strict similarity (24/24 criteria equivalent) was never achieved, and only 9-12/24 criteria met an equivalence threshold depending on LLM-based judge and analysis track (Page 1). The weakest dimensions consistently included skill depth, difficulty balance, and cognitive engagement (Page 8).

Labor Shift and Operational Implications

The study documented a labor shift where the researcher’s work moved from “authoring items” toward “specification, orchestration, verification, and governance” (Page 1). The primary activities became Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance (Page 1).

** Task classification showed that the largest share of researcher time (52.5%) went to Pipeline Engineering and Operations, rather than content creation (Page 12). **

** The efficiency comparison illustrated a striking compression of elapsed time, reducing a task estimated at six months for a full-time doctoral student to approximately ten days for the AI-orchestrated workflow (Page 13). **

The conclusion suggests that the primary value of LLMs is enabling new forms of high-throughput research by shifting human effort to system-level control, including specifying constraints, orchestrating heterogeneous tools, verifying correctness and alignment, and maintaining auditable records (Page 14).

Limitations and Future Directions

The study acknowledged limitations regarding the single researcher/domain scope and the use of LLMs as judges which may share systematic blind spots (Page 14). Future work is recommended to include independent human expert raters scoring a representative subset of items to anchor LLM judge outputs and to conduct multi-researcher replication to assess whether the observed labor shift and efficiency gains hold across different skill levels and working styles (Page 14). Furthermore, extending the study to include item-response theory (IRT) modeling for empirical difficulty estimates is suggested as a follow-up direction (Page 14).

1. Bold Header: Implement rigorous evaluation governance for LLM judges. The paper states, Responsible adoption, therefore, requires explicit governance, which includes documenting how items are evaluated, testing sensitivity to different judges, defining failure thresholds, and logging model and prompt versions. This ensures that conclusions about “quality” are not arbitrary but tied to specific evaluator provenance.

2. Bold Header: Develop a mechanism to explicitly address extraction artifacts in PDF processing. The study found that A subset of SAT items contain references to figures or tables (e.g., “according to the graph above”) that were not recoverable from plain-text extraction, leading to penalties under the Gemini judge, suggesting a need for visual grounding checks in data ingestion pipelines.

3. Bold Header: Integrate targeted refinement strategies for low-scoring criteria. Since skill depth was identified as the single weakest criterion in all six configurations, future AI systems must incorporate specific prompting or self-correction loops aimed at ensuring questions probe the designated skill with sufficient rigor rather than superficially invoking it.

4. Bold Header: Establish automated equivalence threshold enforcement for LLM output. To move beyond the finding that strict similarity (24/24 criteria equivalent) was never achieved, systems should be designed to enforce a higher practical similarity threshold, such as ensuring ≥19/24 criteria equivalent through iterative refinement loops during the generation phase.

5. Bold Header: Shift agent focus from artifact production to orchestration and verification. The paper notes that the researcher’s work moved from 'authoring items' toward specification, orchestration, verification, and governance, implying AI systems should be designed as orchestrators capable of defining constraints and building validation loops.

6. Bold Header: Incorporate human expert raters for final quality anchoring. To mitigate the concern that LLMs are used to both generate and evaluate artifacts based on a set of criteria, future systems should include a step where human verification with the LLM-evaluation results is conducted, specifically using independent human expert raters scoring a representative subset of items to anchor LLM judge outputs.

Advances in large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape research activities remains limited. We report a mixed-methods pilot evaluation of an AI-orchestrated research workflow in which a human researcher coordinated multiple LLM-based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation. Using the generation and assessment of multiple-choice questions (MCQs) as a testbed, we collected 1,071 SAT Math MCQs and employed LLM agents to extract questions from PDFs, retrieve and convert open textbooks into structured representations, align each MCQ with relevant textbook content, generate new MCQs under specified difficulty and cognitive levels, and evaluate both original and generated MCQs using a 24-criterion quality framework. Across all evaluations, average MCQ quality was high. However, criterion-level analysis and equivalence testing show that generated MCQs are not fully comparable to expert-vetted baseline questions. Strict similarity (24/24 criteria equivalent) was never achieved. Persistent gaps concentrated in skill depth, cognitive engagement, difficulty calibration, and metadata alignment, while surface-level qualities, such as grammar fluency, clarity options, no duplicates, were consistently strong. Beyond MCQ outcomes, the study documents a labor shift. The researcher's work moved from ``authoring items'' toward specification, orchestration, verification, and governance. Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance constituted the primary activities. We discuss implications for the future of scientific work, including emerging ``AI research operations'' skills required for AI-empowered research pipelines.