API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
cs.AI, cs.SE
Submitted: 2026-09-08
Updated: 2026-10-08
Code: https://github.com/jennjwang/interface-audit
License: http://creativecommons.org/licenses/by/4.0/
The gist: Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy.
Terminology
Abstract
Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access exceeds the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.
Sources
- Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
- Auditing Meta and TikTok Research API Data Access under Article 40(12) of the Digital Services Act
- Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models
- A Survey on Evaluation of Large Language Models
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Towards Ecologically Valid Research on Language User Interfaces
- Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
- More than Marketing? On the Information Value of AI Benchmarks for Practitioners
- A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
- Measuring Massive Multitask Language Understanding
- Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy
- AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models
- metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
- LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces
- Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
- RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems