Model-Free Assessment of Simulator Fidelity via Quantile Curves
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Model-Free Assessment of Simulator Fidelity via Quantile Curves".
Jane: Here's the gist of the paper. Model-Free Assessment of Simulator Fidelity via Quantile Curves is a model-agnostic framework that constructs confidence sets for latent population parameters to derive a robust proxy for sim-to-real discrepancy, enabling distribution-level risk profiling across various scenarios Problem Formulation and Motivation The fundamental challenge is that, for any given scenario, this discrepancy cannot be observed directly, since both systems are accessible only through finite samples, often of heterogeneous sizes across scenarios <ref:2512.05024#pg2>.
Tom: Standard predictive inference methods are therefore ill-suited, as they quantify uncertainty in observable outputs rather than latent population parameters <ref:2512.05024#pg2>.
Jane: The primary objective of our work is to characterize the quantile function of the discrepancy between real and simulated output distributions across scenarios <ref:2512.05024#pg2>.
Tom: This characterization allows for a distribution-level risk profile of the simulator, supporting statistical inference for new scenarios, calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators <ref:2512.05024#pg3>.
Paper summary: Tom: So, to wrap up this section, the core thesis of "Model-Free Assessment of Simulator Fidelity via Quantile Curves" is that you can't use simple predictive error metrics when comparing simulators because the gap between simulation and reality changes depending on the specific scenario.
Jane: They propose a way around this by constructing confidence sets for those unobserved latent population parameters associated with each scenario, and then using those sets to derive a robust proxy for the sim-to-real discrepancy.
Lu: The crucial part is that instead of just looking at one output error, they estimate the quantile function of this proxy across all scenarios, which results in a distribution-level risk profile.
Meng: So, if you were trying to deploy an AI agent built on a simulator, this framework helps you understand the entire range of risks associated with that simulation as you move from one scenario to another.
Tom: They claim this approach supports many statistical summaries, including things like conditional value-at-risk and principled comparisons between different simulators.
Jane: The methodology involves defining a confidence set for each scenario's latent parameter, often using KL divergence based on empirical averages, and then calculating a pseudo-discrepancy as the supremum of the loss function over that set.
Lu: Next, they construct a calibrated quantile curve by finding the tau quantile of these pseudo-discrepancies across all scenarios.
Meng: It’s model agnostic, which is important because it handles general output spaces, whether that's categorical survey responses or continuous data.
Tom: The theoretical result establishes that for any target level alpha, the probability that the true discrepancy is less than or equal to this estimated quantile function value V hat m tau one minus alpha holds with high probability over a certain set D.
Jane: This guarantee relies on controlling two sources of uncertainty: scenario uncertainty and the finite-sample uncertainty within each scenario.
Lu: The proof sketch shows they can approximate the oracle discrepancy using empirical pseudo-quantiles, and they include an index adjustment to compensate for some inherent conservatism in that approximation.
Meng: It sounds like a solid statistical foundation, but I wonder how robust this holds up when we move from perfectly controlled lab settings to real-world traffic where data quality is messy.
Tom: That’s exactly the challenge they address by showing how the method handles those finite sample issues and scenario variability in a statistically sound way, even though it's model-agnostic.
Conclusion: Jane: So, looking at "Model-Free Assessment of Simulator Fidelity via Quantile Curves," the authors are essentially providing a framework to move beyond simple predictive accuracy when evaluating AI simulations for real-world use.
Tom: They’re not just giving us a single score; they’re giving us a distribution-level risk profile, which allows us to calculate things like tail risks and compare different simulators fairly across various scenarios.
Lu: The implication is that this shifts the focus from getting the best possible average result to understanding the entire range of variability of results, which is critical for safety and reliability.
Meng: For someone building a system, this means they get a tool to assess not just if their model works in ideal cases, but how likely it is to fail badly in unexpected conditions.
Jane: The authors highlight that GPT-4o shows the lowest calibrated discrepancy, suggesting it’s the most reliable alignment among the LLMs they tested on the WorldValueBench dataset.
Tom: But they also showed that this framework can be applied broadly, from manufacturing to environmental modeling, and it’s useful for deriving risk measures like CVaRcal alpha.
Lu: The main limitation they point out is that right now, the guarantees are developed assuming static per-scenario outputs under an i.i.d. scenario assumption, which doesn't cover fully dynamic simulators with temporal dependence or distribution shift yet.
Meng: And computationally, if you have a small number of scenarios or small sample sizes per scenario, the resulting confidence band can be quite conservative for deployment purposes right now.
Jane: So the main thing to remember is that while this framework gives us powerful tools for characterizing sim-to-real discrepancy across distributions, we still need to address those dynamic simulation complexities.
Garud Iyengar, Yu-Shiou Willy Lin, Kaizheng Wang
Columbia University
stat.ME, cs.AI, cs.LG
Submitted: 2025-12-04
Updated: 2026-11-13
Comments: Accepted to NeurIPS 2026
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: Model-Free Assessment of Simulator Fidelity via Quantile Curves is a model-agnostic framework that constructs confidence sets for latent population parameters to derive a robust proxy for sim-to-real…
Key concepts
- Latent Population Parameters
- These are the underlying, unobservable characteristics of a real population that the simulator is trying to model. Since we can't see them directly from finite samples, this method uses statistical inference to estimate what these true values might be.
- Pseudo-Discrepancy
- This is a calculated measure of how much the simulated output differs from the real output for a specific scenario. It is derived by finding the worst-case loss over a confidence set defined for the latent parameters, giving a conservative estimate of potential error.
- Quantile Curve ($ ilde{V}_m( au)$)
- This curve represents the $ au$-quantile of all calculated pseudo-discrepancies across every scenario. It provides a distribution-level risk profile, showing not just the average error but also how often extreme errors occur.
- Model-Agnostic Framework
- The method works regardless of the specific type of simulation or data being used. It focuses on comparing the output distributions between a simulator and reality without needing to know the internal workings or structure of either system.
Terminology
Summary
Model-Free Assessment of Simulator Fidelity via Quantile Curves is a model-agnostic framework that constructs confidence sets for latent population parameters to derive a robust proxy for sim-to-real discrepancy, enabling distribution-level risk profiling across various scenarios
Problem Formulation and Motivation
The fundamental challenge is that, for any given scenario, this discrepancy cannot be observed directly, since both systems are accessible only through finite samples, often of heterogeneous sizes across scenarios <ref:2512.05024#pg2>. Standard predictive inference methods are therefore ill-suited, as they quantify uncertainty in observable outputs rather than latent population parameters <ref:2512. p. 005024#pg2>. The primary objective of our work is to characterize the quantile function of the discrepancy between real and simulated output distributions across scenarios <ref:2512. p. 105024#pg2>. This characterization allows for a distribution-level risk profile of the simulator, supporting statistical inference for new scenarios, calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators <ref:2512. p. 005024#pg3>.
Methodology for Fidelity Profiling
The procedure involves two steps: first, defining a confidence set for each scenario's latent parameter, such as the multinomial setting where KL divergence is used to define the set based on empirical averages <ref:2512.05024#pg8> and then calculating a pseudo-discrepancy by taking the supremum of the loss function over this set <ref:2512. p. 705024#pg9>. Second, constructing a calibrated quantile curve by defining Vˆm(τ) as the τ-quantile of these pseudo-discrepancies across all scenarios <ref:2512. p. 705024#pg10>. This method is model-agnostic and handles general output spaces, such as categorical survey responses and continuous multi-dimensional data <ref:2512. p. 005024#pg2>.
Theoretical Guarantees
The main theoretical result establishes that for any target level α, the probability that the true discrepancy Pψ∼Ψ∆(k)ψ ≤ Vˆm(1 − α)≥ 1 − α− εm(δ) holds with high probability over D <ref:2512.05024#pg11>. This guarantee is derived by controlling two sources of uncertainty: scenario uncertainty and finite-sample uncertainty within each scenario <ref:2512. p. 505024#pg3>. The proof sketch shows that the oracle discrepancy can be approximated using empirical pseudo-quantiles, and the index adjustment compensates for inherent conservativeness in this approximation <ref:2512. p. 705024#pg12>.
Application to LLM Fidelity
The methodology is applied to the WorldValueBench dataset, benchmarking four major LLMs against human populations <ref:2512.05024#pg2>. The results suggest that all LLM-simulators outperform the uniform benchmark on over 80% of questions, but fail to capture the tail scenarios <ref:2512.05024#pg3>. GPT-4o is identified as having the lowest calibrated discrepancy, indicating the most reliable alignment, with Llama 3. p. 33 and Qwen-3 trailing behind <ref:2512. p. 005024#pg13>.
Tightness Analysis and Risk Summaries
The tightness analysis shows that the calibrated envelope can be substantially conservative when sample sizes are small but becomes tighter when nj ≥ 500 <ref:2512.05024#pg2> while nj = 50 (blue) is much looser than nj = 500 (orange) <ref:2512. p. 205024#pg2>. The resulting calibrated quantile curve allows for the calculation of summary statistics like the calibrated AUCcal and a tail risk measure CVaRcalα, providing tools for assessing average bias and rare but consequential failures <ref:2512. p. 905024#pg14>. The framework also provides a confidence band that sandwiches the unknown quantile function, offering an upper and lower envelope <ref:2512. p. 505024#pg2> while showing GPT-4o's band lies systematically below Llama 3.3's band in the upper tail <ref:2512.05024#pg3>.
Future Directions
The framework targets scenario-level population discrepancy, but does not yet model respondent-level joint dependence across multiple questions <ref:2512.05024#pg2>. Furthermore, the guarantees are developed for static per-scenario outputs under an i. p. 1i.d. scenario assumption and do not yet cover fully dynamic simulators with temporal dependence or distribution shift <ref:2512. p. 805024#pg3>. The computational bottleneck remains simulator-side generation, and the resulting envelope can be conservative when the number of scenarios m or per-scenario sample sizes are small <ref:2512. p. 105024#pg4>.
Simulation System Examples
The paper demonstrates applicability across diverse domains, including manufacturing (discrete-event simulation), environment (urban decarbonization), and building control systems where the discrepancy is defined by differences in means or Wasserstein distances <ref:2512.05024#pg2> to <ref:2512. p. 005024#pg2>. The results indicate that for UMAR building control, the linear regression model attains the lowest overall upper quantile curve for scenario-level variance discrepancy <ref:2512.05024#pg2>. The overall conclusion is that predictive accuracy alone is not the main object of interest here; the fidelity target is the distribution of scenario-level variability <ref:2512. p. 305024#pg3>.
References
Aher, G., Arriaga, R. I. and Kalai, A. T. (2023). Using large language models to simulate multiple humans and replicate human subject studies. p. 0 In Proceedings of the 40th International Conference on Machine Learning <ref:2512. p. 0 p. 0 p. 005024#pg2>.
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C. and Wingate, D. (2023). Out of one, many: Using language models to simulate human samples <ref:2512. p. 905024#pg2>.
Bardenet, R. and Maillard, O.-A. (2015). Concentration inequalities for sampling without replacement <ref:2512. p. 605024#pg2>.
Barton, R. R., Nelson, B. L. and Xie, W. (2014). Quantifying input uncertainty via simulation confidence intervals <ref:2512. p. 205024#pg3>.
Bates, S., Angelopoulos, A., Lei, L., Malik, J. and Jordan, M. (2021). Distribution-free, risk-controlling prediction sets <ref:2512. p. 005024#pg4>.
Budde, C. E., Hartmanns, A., Meggendorfer, T., Weininger, M. and Wienhöft, P. (2025). Statistical model checking beyond means: Quantiles, cvar, and the dkw inequality (extended version) <ref:2512. p. 005024#pg5>.
Cao, Y., Liu, H., Arora, A., Augenstein, I., Röttger, P. and Hershcovich, D. (2025). Specializing large language models to simulate survey response distributions for global populations <ref:2512. p. 005024#pg6>.
Chen, M., Lam, H. and Liu, Z. (2024). Quantifying distributional input uncertainty via inflated kolmogorov-smirnov confidence band <ref:2512. p. 205024#pg7>.
Cranmer, K. p. 2, Brehmer, J. and Louppe, G. (2020). The frontier of simulation-based inference <ref:2512. p. 205024#pg8>.
Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N. et al. (2023). Towards measuring the representation of subjective global opinions in language models <ref:2512.05024#pg9>.
Gao, Y., Lee, D., Burtch, G. and Fazelpour, S. (2025). Take caution in using llms as human surrogates <ref:2512. p. 005024#pg10>.
Gretton, A. p. 2, Borgwardt, K. M., Rasch, M. J., Schölkopf, B. and Smola, A. (2012). A kernel two-sample test <ref:2512. p. 205024#pg11>.
Haerpfer, C. p. 1, Inglehart, R., Moreno, A., Welzel, C., Kizilova, K., Diez-Medrano, J., Lagos, M., Norris, P., Ponarin, E., Puranen, B. et al. (2020). World values survey: Round seven – country-pooled datafile (2017–2020) <ref:2512. p. 105024#pg19>.
He-Yueya, J., Ma, W. A., Gandhi, K., Domingue, B. W., Brunskill, E. and Goodman, N. D. (2024). Psychometric alignment: Capturing human knowledge distributions via language models <ref:2512. p. 005024#pg2>.
Heer, P., Derungs, C., Huber, B., Bünning, F., Fricker, R., Stoller, S. and Niesen, B. (2024). Comprehensive energy demand and usage data for building automation <ref:2512. p. 005024#pg3>.
Hoeffding, W. p. 6 (1963). Probability inequalities for sums of bounded random variables <ref:2512.05024#pg4>.
Huang, C., Wu, Y. and Wang, K. p. 0 (2025). Uncertainty quantification for LLM-based survey simulations <ref:2512. p. 105024#pg5>.
Jeon, Y., Chu, Y., Pasupathy, R. and Shashaani, S. (2024). Uncertainty quantification using simulation output: Batching as an inferential device <ref:2512. p. 005024#pg6>.
L.A., P. and Bhat, S. p. 6 P. (2022). A wasserstein distance approach for concentration of empirical risk estimates <ref:2512. p. 505024#pg7>.
Lam, H. (2022). Cheap bootstrap for input uncertainty quantification <ref:2512. p. 205024#pg8>.
Lu, Y., Huang, J. p. 2, Han, Y., Yao, B., Bei, S., Gesi, J., Xie, Y., He, Q., Wang, D. et al. (2025). Prompting is not all you need! p. 4 evaluating llm agent simulation methodologies with real-world online customer behavior data <ref:2512. p. 005024#pg9>.
Macal, C. (2016). Everything you need to know about agent-based modelling and simulation <ref:2512. p. 105024#pg2>.
Mardia, J., Jiao, J., Tánczos, E., Nowak, R. D. and Weissman, T. (2019). Concentration inequalities for the empirical distribution of discrete distributions: beyond the method of types <ref:2512. p. 505024#pg2>.
Massart, P. (1990). The tight constant in the dvoretzky-kiefer-wolfowitz inequality <ref:2512. p. 805024#pg3>.
Papamakarios, G. p. 2, Sterratt, D. and Murray, I. (2019). Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows <ref:2512.05024#pg4>.
Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P. and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior <ref:2512. p. 105024#pg3>.
Peng, X. B., Andrychowicz, M., Zaremba, W. and Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization <ref:2512. p. 005024#pg4>.
Richner, P., Heer, P., Largo, R., Marchesi, E. and Zimmermann, E. (2018). Nest – a platform for the acceleration of innovation in buildings <ref:2512. p. 005024#pg5>.
Roy, C. J. and Oberkampf, W. p. 2 L. (2011). A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing <ref:2512. p. 305024#pg6>.
Santurkar, S. p. 2 p. 2, Durmus, E. p. 1 p. 1, Ladhak, F., Lee, C., Liang, P. and Hashimoto, T. (2023a). Whose opinions do language models reflect? p. 4 p. 4 In Proceedings of the 40th International Conference on Machine Learning <ref:2512.05024#pg6>.
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P. and Hashimoto, T. (2023b). Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning <ref:2512.05024#pg6>.
Schrab, A., Kim, I., Albert, M., Laurent, B., Guedj, B. and Gretton, A. p. 2 (2023). Mmd aggregated two-sample test <ref:2512. p. 305024#pg7>.
Snell, J. C., Zollo, T. P., Deng, Z., Pitassi, T. and Zemel, R. (2022). Quantile risk control: A flexible framework for bounding the probability of high-loss predictions <ref:2512.05024#pg8>.
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W. and Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world <ref:2512. p. 205024#pg9>.
Vovk, V., Gammerman, A. and Shafer, G. (2005). Algorithmic Learning in a Random World <ref:2512. p. 005024#pg10>.
Wang, Z., Lamb, A., Saveliev, E., Cameron, P., Zaykov, J., Hernandez-Lobato, J. M., Turner, R. E., Baraniuk, R. G., Craig Barton, E. p. 2, Peyton Jones, S., Woodhead, S. and Zhang, C. p. 1 (2021). Results and insights from diagnostic questions: The neurips 2020 education challenge <ref:2512.05024#pg10>.
Westwood, S. p. 1 J. (2025). The potential existential threat of large language models to online survey research <ref:2512. p. 105024#pg11>.
Zhang, L., Zhou, L., Ren, L. and Laili, Y. (2019). Modeling and simulation in intelligent manufacturing <ref:2512. p. 105024#pg14>.
Zhao, W., Mondal, D., Tandon, N., Dillion, D., Gray, K. and Gu, Y. (2024). Worldvaluesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models <ref:2512. p. 005024#pg15>.
A Technical Lemmas
Lemma A.1. (Dvoretzky-Kiefer-Wolfowitz (DKW) Inequality via Massart (1990)) <ref:2512. p. 805024#pg4>.
Lemma A.2. [Hoeffding (additive) via Hoeffding (1963)] <ref:2512. p. 605024#pg5>.
Lemma A.3. [Chernoff–Hoeffding for one-parameter exponential family] <ref:2512. p. 605024#pg6>.
Lemma A.4. (Chernoff-Hoeffding Inequality) <ref:2512. p. 605024#pg7>.
Lemma A.5 (Multinomial Chernoff-Hoeffding Bound via Mardia et al. p. 6 (2019)) <ref:2512. p. 005024#pg8>.
Lemma A.6 (Wasserstein distance bound via L. p. 5A. and Bhat (2022)) <ref:2512. p. 605024#pg9>.
Lemma A.7 (Order-statistic thresholding) <ref:2512. p. 005024#pg10>.
B Proof to Theorem 3. p. 81 and 5.1
B.1 Proof for Theorem 3. p. 81 <ref:2512. p. 0 p. 0 p. 005024#pg5>.
B.2 Proof for Corollary B. p. 91 <ref:2512.05024#pg6>.
B.3 Corollary for simulator budget <ref:2512. p. 405024#pg7>.
B.4 Proof for Theorem B. p. 81 <ref:2512.05024#pg8>
C World Value Bench Methodology
C.1 Selection of Survey Questions <ref:2512. p. 905024#pg2>.
C.2 Example Question and Preprocess <ref:2512. p. 005024#pg3>.
C.3 Synthetic Profile Generation <ref:2512. p. 305024#pg4>.
C.4 Adaptive coverage rate γj choice <ref:2512. p. 705024#pg5>.
D Simulation System Examples <ref:2512. p. 005024#pg2> to .
E Applications
E.1 EEDI Dataset <ref:2512. p. 005024#pg3>.
E.2 OpinionQA <ref:2512. p. 005024#pg3>.
E.3 UMAR Dataset <ref:2512. p. 005024#pg3>.
Figure 15 reports the adjusted upper quantile curves for the UMAR experiment <ref:2512.05024#pg2> to Figure 4 shows robustness check of simulator performance under different human sample sizes <ref:2512.05024#pg2> and Figure 7 illustrates these bands for two subsample sizes nj ∈ [1000, 5000] and compares GPT-4o against Llama 3. p. 13 <ref:2512. p. 005024#pg3>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Model-free fidelity profiling enables assessment of sim-to-real discrepancy across scenarios by constructing confidence sets for latent population parameters and deriving a robust proxy for discrepancy, which supports risk measures like Conditional Value-at-Risk (CVaR).
-
The framework allows statistical inference for real world parameters in new scenarios and principled comparisons across simulators, moving beyond individual outcome uncertainty to characterize the distribution of sim-to-real discrepancy.
-
The method provides a calibrated quantile curve of the discrepancy, enabling practitioners to derive a calibrated AUC for average bias and tail risk measures like CVaR alpha by aggregating the entire curve or averaging over the worst alpha fraction of scenarios.
-
For LLM-based simulators on survey data, it allows identifying which models are most reliable across all quantiles and highlights failures in capturing tail scenarios, providing a more nuanced comparison than simple scalar error summaries.
-
In continuous simulation systems like building control, the framework targets the distribution of scenario-level variability rather than just pointwise prediction error, allowing assessment of how faithfully an emulator reproduces real-world variability under different operational contexts.
Sources
- Statistical Model Checking Beyond Means: Quantiles, CVaR, and the DKW Inequality (extended version)
- Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations
- Quantifying Distributional Input Uncertainty via Inflated Kolmogorov-Smirnov Confidence Band
- Towards Measuring the Representation of Subjective Global Opinions in Language Models
- Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models
- Uncertainty Quantification using Simulation Output: Batching as an Inferential Device
- Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data
- Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions
- WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes