OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
cs.AI, cs.CL, stat.ML
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: Accepted at NeurIPS 2026
Code: https://github.com/LinerXiang/OTROPE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reliable evaluation of large language models (LLMs) is essential for their development and deployment, yet is often costly, risky, and difficult to perform safely online.
Terminology
Abstract
Reliable evaluation of large language models (LLMs) is essential for their development and deployment, yet is often costly, risky, and difficult to perform safely online. We study off-policy evaluation for LLMs, where limited human-labeled data from a behavior model are used to evaluate a newer target LLM. This setting is challenging because labels are scarce, behavior--target distribution shift is common, and response likelihoods are often unavailable for black-box LLMs. We propose the Optimal Transport-based Robust Off-Policy Evaluation (OTROPE), a likelihood-free evaluation that performs distributional correction in a semantic space via optimal transport to align labeled behavior-policy samples with unlabeled target-policy samples. OTROPE combines corrected human-labeled residuals with proxy predictors, yielding a doubly robust-style evaluation without behavior-policy modeling or density-ratio estimation. We theoretically characterize why baseline evaluators fail under LLM distribution shift, and establish consistency and convergence rates for OTROPE when either the reweighted behavior distribution or the proxy predictor converges. Experiments on synthetic and real LLM evaluation tasks show that OTROPE consistently outperforms baselines while enabling ensembles of weaker LLM evaluators to approach and sometimes surpass stronger evaluators. Code is available at https://github.com/LinerXiang/OTROPE.
Sources
- PPI++: Efficient Prediction-Powered Inference
- Off-Policy Evaluation from Logged Human Feedback
- AutoEval Done Right: Using Synthetic Data for Model Evaluation
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Multiple-Prediction-Powered Inference
- Prediction-Powered Inference with Inverse Probability Weighting
- Pluralistic Off-policy Evaluation and Alignment
- DeepSeek-V3 Technical Report
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Bandits
- OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems