LLM-as-a-judge validity is strongly task-dependent across physics assessment formats

arXiv:2603.14732 · physics.ed-ph, cs.CL · Submitted 2026-03-16 · Read on arXiv

physics.ed-ph, cs.CL

Submitted: 2026-03-16

Updated: 2026-09-26

Comments: 29 pages, 28 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 31× in this paper · explained in 1 episode
• LLM-as-a-Judge — 21× in this paper · explained in 3 episodes
• Calibration — 18× in this paper · explained in 7 episodes
• Benchmark — 16× in this paper · explained in 2 episodes
• Alignment — 13× in this paper · explained in 1 episode
• Aggregation — 11× in this paper · explained in 1 episode
• Prompt Engineering — 6× in this paper · explained in 4 episodes
• Consistency — 4× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 4× in this paper · explained in 6 episodes
• Mean Absolute Error — 4× in this paper
• Core — 3× in this paper · explained in 2 episodes
• Embeddings — 2× in this paper · explained in 2 episodes
• Empirical Methods in Natural Language Processing — 2× in this paper
• Human-in-the-Loop — 2× in this paper · explained in 3 episodes
• Self-consistency — 2× in this paper · explained in 2 episodes
• Structured Prompting — 2× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 3 episodes
• Data Augmentation — 1× in this paper · explained in 7 episodes
• Distribution Matching — 1× in this paper · explained in 2 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Positional Bias — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Structural Constraints — 1× in this paper · explained in 2 episodes
• Synthetic Data Generation — 1× in this paper · explained in 5 episodes
• Systematic Bias — 1× in this paper · explained in 3 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• Assessing Confidence in AI-Assisted Grading of Physics Exams through Psychometrics: An Exploratory Study
• Using Large Language Models to Assign Partial Credit to Students' Explanations of Problem-Solving Process: Grade at Human Level Accuracy with Grading Confidence Index and Personalized Student-facing Feedback
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
• LLM Evaluators Recognize and Favor Their Own Generations
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• Report on the Scoping Workshop on AI in Science Education Research 2025
• AI-assisted Automated Short Answer Grading of Handwritten University Level Mathematics Exams

Related papers

• Developing an LLM-Based Feedback System Grounded in Evidence-Centered Design to Support Physics Problem Solving
• Using Learning Analytics to Study Novice Learners' Work on Quantum Circuit Simulator Tasks
• Artificial Neural Network Assisted Modelling of Tangent Galvanometer Measurements for the Determination of Horizontal Component of Earth's Magnetic Field