Amortized Off-Policy Evaluation for LLMs

arXiv:2610.10848 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Amortized Off-Policy Evaluation for LLMs".

Jane: The gist The proposed PFN-OPE network amortizes off-policy evaluation across a distribution of contextual-bandit tasks,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're diving into this paper, "Amortized Off-Policy Evaluation for LLMs," which sounds pretty technical, but it’s tackling a big problem in using AI models in the real world.

Jane: Exactly. The core issue is that when you test a new AI model on live traffic data, you run into two kinds of shifts: the model itself might have changed after training, and the way we define what success means—the reward—can change too.

Lu: It’s a problem because traditional ways of doing off-policy evaluation, or OPE, have to fit a new model from scratch every single time you get new logged data or a new reward definition.

Meng: That sounds incredibly inefficient for deployment cycles; retraining on every small change just doesn't scale well for real-time updates.

Tom: That’s the problem they are addressing. The paper introduces PFN-OPE, which is a prior-data fitted network designed to amortize that off-policy evaluation across many different tasks.

Jane: They propose this network gets pretraining on a broad set of tasks from which it can learn how to predict the correction value needed for evaluation in just one forward pass.

Lu: The input representation they use is pretty specific, with two tokens: one representing the logged response and its reward, and another representing the target response without a reward.

Meng: So instead of needing a separate fit for every new dataset, this single network uses the context to estimate that correction value C.

Tom: Right. And they train this network using quantile regression to predict that correction value C given those inputs, which they call O.

Jane: The key insight here is that the label needed—the true target policy value—is unobservable on real logs, but it becomes computable if you build tasks from a pool of stored responses and rewards.

Lu: They construct this task prior using a pool of LLM responses with rewards from several different reward functions, where combinations of those reward functions act as new ways to define the reward.

Meng: That means they can sample pretraining tasks without needing extra LLM generation or having to evaluate every single reward model separately just to build the training set.

Tom: And when you test it, that pretrained network uses the observation O from a new logged dataset and sampled target responses to produce quantile estimates, which they then aggregate into a correction value Cb theta of O.

Jane: Then they estimate the target policy value V b theta of pi; O by adding that correction Cb theta of O to the logged mean reward r bar.

Title and authors: Lu: The paper highlights that PFNs remove the per-dataset fit because they are transformers pretrained on tasks sampled from a prior over tasks, so given a new dataset and a query in context, it returns a prediction in just one forward pass.

Meng: That sounds like it could speed up evaluation significantly if you're testing many candidates quickly against live traffic data.

Tom: It does. The paper shows that PFN-OPE achieves the lowest mean squared error under all five reward definitions, which is two point zero to nine point three times lower than the best baselines in reward-shifted settings.

Jane: That improvement comes from a few clever diagnostic measures they use, like B2 delta and L delta, which help show how well the prior covers the necessary task space.

Lu: The prior coverage analysis shows that their two priors cover almost all the same region except for a specific left corner and upper region where they differ.

Meng: So even in those tricky areas, where things are hard to predict, they still get low error rates because of how the network reads the shift between logged and target responses.

Tom: But there are some limitations mentioned too. They point out that their evaluation targets are other LLMs, and the policy shifts they test are changes within LLMs themselves.

Jane: And another limitation is that the prior is built from three LLMs, which suggests that generalizing to completely unseen families of models might still be an open question right now.

Lu: The paper also notes that the transfer to an unseen checkpoint of a seen family and to one entirely new family proves the estimator generalizes beyond just the initial pool of LLMs.

Meng: So, while it's much better than previous methods, we still need more testing with totally new model architectures to be sure it works everywhere.

Tom: Overall, PFN-OPE is presented as a step toward evaluation that keeps pace with deployment by reusing one pretraining run across the stream of target policies that get produced in the field.

Jane: It suggests a way to handle those continual changes in post-training objectives and feedback signals by building on prior knowledge instead of starting over every time.

Lu: The architecture itself is quite clever; it uses a directional attention mask where logged tokens only attend to logged tokens, while target tokens attend to all logged and target tokens.

Meng: That setup allows it to process both the policy shift and the reward shift simultaneously in that single forward pass, which is really powerful for keeping up with live traffic.

Title and authors: Tom: It’s about making a single pretraining run serve as a foundation for evaluating many different future model behaviors without needing new training runs every time.

Jane: And they showed that even when the correction is quite large, above a standard deviation of zero point two, PFN-OPE keeps an error between zero point zero four and zero point zero seven, whereas other methods have their error growing with the correction size.

Lu: That handling of large corrections is important because it means it doesn't break down when the shift is significant, unlike some direct method estimators that struggle when the distribution shifts a lot.

Meng: So if you’re worried about a massive divergence between your deployed model and what you trained on, this approach seems quite robust to those large changes.

Tom: And the experimental results show that above a correction standard deviation of zero point two, every estimator achieves an accuracy at or below zero point zero one two for direct methods because the logged mean is already close to the target value in those cases.

Jane: So even when things are close, it still provides this consistent floor of error compared to other approaches like IPS or DR estimation that you see in Jiang and Li, two thousand sixteen <ref:2610.10848#pg2>.

Lu: The paper also mentions that the combination of reward-tilted reweightings of an LLM’s responses playing the role of post-trained policies helps it learn how far the target value differs from logged rewards even when the target policy has diverged significantly.

Meng: That ability to handle those policy shifts through these reweightings is what makes it seem robust when dealing with different business requirements for reward definitions.

Tom: So, to sum up PFN-OPE, we have a network that learns to read the shift off the observation itself by being pretrained on tasks built from a response pool where every target value is known by construction.

Jane: It’s about amortizing OPE across a distribution of contextual bandit tasks so you don't have to fit everything individually.

Lu: The implication is that for evaluating LLMs in continuous deployment settings, we can reuse one pretraining run to evaluate the stream of target policies that get produced in the field.

Meng: I think this means the engineering side gets a huge win because it simplifies the evaluation pipeline significantly.

Tom: It does. We'll leave you there for now, but keep an eye on these kinds of amortized methods because they are definitely moving toward making model evaluation much more practical for real-world systems.

The paper's summary: Tom: So we’re looking at the summary of this paper, "Amortized Off-Policy Evaluation for LLMs." Basically, they figured out a way to do off-policy evaluation—which is figuring out how good a new AI model is using old data—in one single pass.

Jane: That’s right. Instead of having to retrain or do a whole new fitting process for every single task or every change in the reward system, this PFN-OPE network just uses its pretraining to predict the necessary correction value.

Lu: It takes two inputs, one from the logged data and one from a target response without a reward, and it outputs an estimate of that correction value. They call this prediction C.

Meng: So instead of fitting a new model for every dataset, you just feed it the context in one go to get that correction estimate. That sounds much faster for deployment cycles.

Tom: Exactly. The key is how they build the task prior and then use that network at test time to produce those quantile estimates directly from the observation O.

Jane: They pretrain this network on a massive pool of responses with known rewards, and that training lets it learn how to read the difference between what you logged and what you’re targeting.

Lu: The idea is they learn to see how far the target policy value departs from the logged rewards just by looking at the input observation itself. That's a huge shift from fitting a whole new model every time.

Meng: So, if you’re testing five different versions of an AI model on live traffic data with slightly different reward definitions, this single network can handle all five in one go?

Tom: That’s the core promise. The results they show are pretty strong—they see error reductions between two point zero and nine point three times better than the best methods they compared it to.

Jane: And they're even hitting the lowest mean squared error when both the target AI model and the reward model aren't included in that initial pretraining pool.

Lu: The coverage diagnostics show that their prior covers almost everything, except for a specific region, but even there, things are accurate because of how they’ve defined those priors.

Meng: It’s about making evaluation more practical for continuous deployment because it reuses one training run to evaluate a stream of different target behaviors.

Tom: Right. It moves the goal from fitting and retraining constantly to simply querying a pre-learned network with new data points and getting an estimate in one forward pass.

Jane: So, this means we can keep pace with the constant changes in how AI is deployed by using this prior knowledge to adjust our expectations about performance.

Lu: And they also show that it handles those large shifts well—when the correction goes above a standard deviation of two point zero, their error stays between zero point zero four and zero point zero seven.

Meng: That’s important because other methods get worse as the shift gets bigger; this PFN-OPE keeps its accuracy much more stable.

Tom: It shows that even when things are changing a lot, we don't lose precision as fast if we use this kind of amortized approach.

Jane: So the next thing to look at is what they say about how these models generalize when it comes to entirely new AI systems or completely new reward functions.

The paper's improvements: Tom: So we’re talking about how this PFN-OPE method actually improves things over what we had before in terms of evaluation speed and accuracy.

Jane: Well, one major improvement is that it’s all done in one forward pass now without needing per-task fitting for every single change.

Lu: That means you can test a new AI candidate on live traffic data really quickly just by feeding it the context. No need to retrain anything for every reward definition switch.

Meng: That’s huge for the engineering side, because it simplifies the whole pipeline and cuts down on massive retraining cycles during deployment updates.

Tom: And they handle those large correction settings much better too, which is a big deal for robustness.

Jane: They maintain an error between zero point zero four and zero point zero seven when the correction goes above a standard deviation of two point zero.

Lu: That’s because it doesn't just grow with the correction like other direct methods do; it stays relatively stable even when the shift is big.

Meng: So for someone who only listens to this, it means if your AI drifts a lot from what you trained on, this method keeps giving you a reliable error estimate instead of breaking down.

Tom: Exactly. It’s about making the evaluation process more resilient to those significant distribution shifts that happen in real-world testing.

Jane: Then there's the generalization aspect; they showed it can actually handle unseen target AI models and reward models too, which is pretty impressive for a prior-based system.

Lu: They proved this by showing transfer to an unseen checkpoint of a known family, and even to one completely new family, proving it learns beyond just the initial training pool.

Meng: So this isn't just a hack for the current setup; it suggests we can build evaluators that work across a wider range of future AI architectures.

Tom: It really points toward building evaluation tools that can keep up with the continuous flow of new target policies in production environments.

Jane: And they mention how they use reward-tilted reweightings to model policy shifts, which helps it learn how far the target value is from logged rewards even when the AI has changed a lot.

Lu: So this architecture is designed to be quite flexible, using that directional attention mask so that the target tokens look at both what’s logged and what’s targeting simultaneously.

Meng: I see a way this could improve how we test safety; if we can evaluate against policy shifts effectively, we can catch those changes sooner in the pipeline.

Tom: That's a good point. This work is definitely moving toward making AI evaluation less of a bottleneck and more of an integrated part of the deployment process.

Conclusion: Tom: So we’re wrapping up on "Amortized Off-Policy Evaluation for LLMs." The big picture here is that they’ve figured out how to make off-policy evaluation practical for real deployment streams.

Jane: Basically, this paper shows a method where you can reuse one pretraining run across many different target AI policies without needing to retrain every single time.

Lu: It really shifts the focus from fitting per-task models to having one powerful network learn how to read the shift between logged data and target responses.

Meng: From an engineering standpoint, this means we can build evaluation systems that keep up with continuous deployment because they don't require massive retraining infrastructure for every little update.

Lalam: If this works as well as they claim, it gives us a way to measure the real-world performance of AI in a production setting without having to rebuild the entire foundation constantly.

Tom: Right. It’s about keeping pace with how fast things change in the deployment pipeline. The results show it’s significantly better than baselines, two point zero to nine point three times lower error in reward-shifted settings.

Jane: And they showed that this approach is robust even when those shifts are large, maintaining a low error between zero point zero four and zero point zero seven when the correction is above two point zero standard deviations.

Lu: The implication for research is that we don't have to build entirely new estimation methods from scratch every time; we can build on these prior-data fitted networks.

Meng: It means the engineering cost of evaluation drops a lot because you are amortizing the training effort across all those different tasks.

Tom: So, this whole paper, "Amortized Off-Policy Evaluation for LLMs," is a step toward evaluation that keeps pace with deployment by reusing pretraining runs.

Jane: It’s about making AI evaluation much more practical for real-world systems by using context to predict the necessary correction values in one forward pass.

Lu: And the fact that it generalizes to unseen target AI models and reward models is really interesting because it means the prior knowledge is quite strong.

Meng: That generalization capability suggests we can apply this idea to a wider variety of AI systems than just the ones they tested on initially.

Lalam: For me, this feels important because if we can evaluate these behaviors more efficiently, it helps us build a better culture around responsible and predictable AI deployment.

Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee

RBC Borealis 2 University of Toronto Vector Institute

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 83/100

The gist: The gist The proposed PFN-OPE network amortizes off-policy evaluation across a distribution of contextual-bandit tasks, achieving 2.0 to 9.3 times lower error than best baselines in reward-shifted

Key concepts

PFN-OPE: A prior-data fitted network that amortizes off-policy evaluation for LLMs. It is pretrained on a pool of tasks to learn to predict the correction value needed to estimate target policy values from logged data and target responses in a single forward pass.
In-context Prediction Problem: PFN-OPE frames OPE as an in-context prediction task. The input observation consists of two tokens: one representing the logged data (behavior policy) and another representing the target policy's responses. The network learns to predict a correction value based on this context.
Prior-data Fitted Networks: These networks are trained on datasets sampled from a prior over tasks, rather than fitting per dataset. This allows the network to generalize; given new data and a query, it returns an estimate in one pass by reading the shift between logged and target responses.
Task Prior for LLM Off-Policy Evaluation: The task prior is built from a pool of LLM responses with rewards derived from several reward functions. This allows the system to sample pretraining tasks without needing additional LLM generation or reward model evaluation, enabling the estimation of target policy values.

Summary

The gist The proposed PFN-OPE network amortizes off-policy evaluation across a distribution of contextual-bandit tasks, achieving 2.0 to 9.3 times lower error than best baselines in reward-shifted settings

How it works

PFN-OPE formulates Off-Policy Evaluation (OPE) as an in-context prediction problem where the target policy value is estimated from a behavior policy sampled logged data and target policy sampled target responses The input observation O is represented by two tokens: Li = fµϕx(xi), ϕa(aµi), rµi, Qi = fπϕx(xi), ϕa(aπi), MASK The network is trained via quantile regression to predict the correction value C in Equation (2) given O rather than predicting the target-policy value directly At test-time, the pretrained network uses O to produce quantile estimates Qbκθ(O)κ∈K, which are aggregated as Cbθ(O) and then Vbθ(π; O) is estimated as ¯r + Cbθ(O)

Prior-data fitted networks

PFNs remove the per-dataset fit by being a transformer pretrained on datasets sampled from a prior over tasks, where given a new dataset and a query in context, it returns a prediction in one forward pass The key insight is that the label a PFN needs, the target policy’s value, is unobservable on real logs but computable on tasks built from a pool of stored responses with rewards The network learns to read the shift off the observation itself, how the target responses differ from the logged ones and what the logged rewards say under the current definition

Task prior for LLM off-policy evaluation

The task prior is built from a pool of LLM responses with rewards from several reward functions, where affine combinations of reward functions play the role of new reward definitions, and reward-tilted and routed policies as policy shifts The target-policy value Vτ (π) can then be evaluated as a finite sum over the pool, allowing us to sample pretraining tasks without additional LLM generation or reward-model evaluation

Evaluation with LLM policies

PFN-OPE has the lowest MSE under all five reward definitions, four of them absent from pretraining, by 2.0 to 9.3 times over the best of the baselines It has the lowest MSE in 11 of 12 behavior–target pairs when both the target LLM and reward model are absent from pretraining

Prior coverage diagnostics

The prior-coverage analysis shows that the two priors cover the same region except the left corner and upper region, and that corner is where the two PFNs differ The full prior has tasks there, and its PFN’s error there is low

Limitations

The limitations indicate that evaluation targets are other LLMs, and the policy shifts tested are changes of LLM The prior is built from three LLMs, and transfer to an unseen checkpoint of a seen family and to one unseen family is the evidence that the estimator generalizes beyond the pool’s LLMs The prior coverage analysis shows that tilted policies reach the kind of task a post-trained checkpoint would pose

The paper introduces PFN-OPE, a prior-data fitted network that amortizes off-policy evaluation of LLMs This is a new problem setting for PFNs where the input is a set of paired responses, one with a reward and one without, and the label is a functional of the whole set rather than the label of one query The core mechanism involves pretraining on tasks constructed from a response pool where every target value is known by construction At test-time, it maps a logged dataset and sampled target responses to a value estimate in one forward pass, with no per-task fitting On HelpSteer2 and UltraFeedback, PFN-OPE has 2.0 to 9.3 times lower MSE than the best baseline As post-training objectives and feedback signals continue to shift, we see PFN-OPE as a step toward evaluation that keeps pace by reusing one pretraining run across the stream of target policies that deployment produces

How it works

The input representation for PFN-OPE is represented by two tokens: Li = fµϕx(xi), ϕa(aµi), rµi, Qi = fπϕx(xi), ϕa(aπi), MASK The attention mask uses a directional attention mask where each logged token Li attends only to the logged tokens, whereas each target token Qi attends to all logged and target tokens The readout includes a learned summary token whose final representation is passed to an output head that predicts the quantiles Cbθ(O) of the correction C

Prior coverage diagnostics

The diagnostic B2∆ is defined as (∆b − ∆)2/σ2µ, which is zero when the probe misjudges both values by the same amount and large only when its error differs between the two response sets The diagnostic L∆ measures the ridgeregularized leverage of the shift, averaged over folds

E Limitations

Evaluation targets are other LLMs, and every evaluation target is the pool policy of one LLM, so the policy shifts we test are changes of LLM Rewards come from reward models, and every reward in pretraining and evaluation is the output of ArmoRM or GRM The prior is built from three LLMs, and transfer to an unseen checkpoint of a seen family and to one unseen family is the evidence that the estimator generalizes beyond the pool’s LLMs Widening the pool to more families and to actual post-training checkpoints is the natural extension, at the one-time cost of generating and scoring their responses

B Algorithms

Algorithm 1 summarizes pretraining, which involves sampling a task τ, forming the observation O = (Dµ, Dπ), setting C = Vτ (π) − r¯, and updating θ on the loss of Equation (3) at (O, C) Algorithm 2 gives every step of the procedure summarized in Algorithm 1

C Experimental details

The experiments ask whether PFN-OPE replaces per-task fitting under the reward shift and policy shift The metric is the MSE between Vb(π) and Vτ (π) Table 1 shows PFN-OPE achieves the lowest MSE in 11 of 12 behavior–target pairs when both the target LLM and reward model are absent from pretraining The gain shrinks with distance from the prior but does not vanish Above Cstd = 0.2 every estimator is accurate, with MSE at or below 0.012 for every direct method, because the logged mean is already close to the target value The ratio to the best baseline widens from 1.6 to 3.

Improvements for AI systems

A single PFN-OPE network can estimate target policy values in one forward pass, with no per-task fitting, enabling rapid evaluation of new LLM candidates on live traffic data without requiring retraining for every reward definition change.
The system can generalize to unseen target LLMs under an unseen reward model, achieving a 3.1 to 19 times improvement in MSE compared to baselines by leveraging the prior-data fitted network's ability to learn how far the target value departs from the logged rewards.
The PFN-OPE architecture uses a directional attention mask where Target tokens attend to all logged and target tokens because the reward of a target response must be inferred from the logged responses and their observed rewards, allowing it to process both policy shift and reward shift simultaneously.
The system can handle large correction settings effectively, as PFN-OPE maintains an MSE between 0.04 and 0.07 when the correction is above a standard deviation of 0.2, whereas baselines' MSE grows with the correction.
The system is robust to policy shifts by incorporating reward-tilted reweightings of an LLM’s responses play the role of post-trained policies, allowing it to learn how the target value differs from logged rewards even when the target policy has diverged significantly.

Abstract

Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed models. This is off-policy evaluation (OPE), and it faces two distribution shifts: as a model is updated in post-training, its responses diverge from the logged ones (policy shift), and the reward definition under which it is judged changes with business requirements (reward shift). Classical OPE methods are ill-suited to this continual-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition. To address this, we propose PFN-OPE, a prior-data fitted network that amortizes OPE across a distribution of contextual-bandit tasks. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur. At test-time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per-task fitting. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN-OPE achieves 2.0 to 9.3 times lower error than the best baselines across all tested configurations in the reward-shifted settings.