Choices Speak Louder than Questions
This research introduces Normalized Probability Shift by the Question (NPSQ) to evaluate Large Language Models better than traditional methods. The study found that model decisions are often biased by answer choices rather than question comprehension. NPSQ isolates this influence, showing that models' preferences are truly driven by the question itself, leading to a more reliable assessment of understanding.