Model Criticism via Stochastic Optimization

arXiv:2610.37837 · stat.ME, stat.CO · Submitted 2026-10-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Model Criticism via Stochastic Optimization".

Jane: Here's the gist of the paper. Many generative models are specified by how data are simulated The procedure splits the data in two and is formulated as a two-stage stochastic program This simulation-based procedure tests whether such a model fits the data and, when it does not, locates where it fails without choosing a test statistic Model Criticism Framework The paper proposes a simulation-based procedure that tests whether a generative model fits the data and identifies where it fails by splitting the data into two blocks and formulating the problem as a two-stage stochastic program The estimation stage fits the structural parameter θ on one block while holding the noise law fixed, minimizing a Wasserstein distance The recourse stage freezes θ and uses representative noise values to optimize weights on the held-out block This structure allows the resulting optimal weights to be compared with theoretical weights under correct specification, yielding a calibrated Pearson test Two-Stage Stochastic Program Details The two stages of the program are defined as follows: Stage 0 (here-and-now) involves choosing the structural parameter θ to minimize the Wasserstein distance between data and simulations from the model, while keeping the noise law exactly as written Stage 1 (recourse) involves optimizing a vector w of weights on model atoms, where these atoms are generated by pushing a finite set of representative noise values through the fitted mechanism G(˜θ, ·) Under correct specification, the optimal reweighting returns the mass that the fitted model itself assigns to its cells The departure between optimized w and theoretical w points to the part of the noise law that is wrong Calibration and Diagnostics The split makes the test calibrated because θ is determined from the estimation block (F0-measurable) while w is a wait-and-see decision dependent on ξ1 The resulting counts are conditionally multinomial with known cell probabilities under the null, yielding a calibrated Pearson test The diagnostic is the comparison between the optimized weights w˜ and theoretical weights w vor(˜θ), which indicates where the model fails Consistency and Robustness The paper proves strong consistency of both stages under checkable assumptions The estimation stage is robust because it guards ˜θ against mild misspecification of the structure G or the noise law ν, leaving misfit for the recourse stage to expose The consistency of ˜θn0 near θ0 is established by showing that it concentrates near θ0 for large n0 under appropriate conditions Experimental Validation The procedure is illustrated on four models, including a Gaussian location–scale family, the number of segregating sites under Kingman’s coalescent, a Yule process, and an Ornstein–Uhlenbeck diffusion The experiments show that the diagnostic separates symmetric tail defects from one-sided gross errors in the Gaussian example The recourse stage successfully locates a state-dependent birth rate in the Yule process by identifying excess atoms corresponding to fast growth trajectories The results demonstrate that the procedure can recover that a driving noise is more concentrated than Brownian motion in the Ornstein–Uhlenbeck diffusion example Limitations and Extensions The theory requires i.i.d.

Tom: replicates, meaning a single long trajectory of a process is not covered by the current framework The recourse identifies the noise law only on the sigma-algebra generated by the fitted mechanism, reporting departures jointly for all noise values in a fiber The resolution of the diagnostic is bounded by the mesh, and refining it trades resolution for noisier cell weights through the ratio n1/N1 Finally, since structure, noise law, and prior are entangled, a misspecified structure can be absorbed into the noise law Code Availability The code is publicly available on GitHub at https://github.

Title and authors: Tom: So, to summarize what they're doing in "Model Criticism via Stochastic Optimization," they split the data into two blocks and formulate this as a two-stage stochastic program. The estimation stage fits the structural parameter theta on the first block by minimizing a Wasserstein distance between that data and simulations from the model, while holding the noise law exactly as written.

Jane: And then in this second stage, they freeze that fitted theta at its best estimate and use representative noise values to push through it to get model atoms. Then they optimize weights on the held-out block of data to best match its own distribution. This structure allows them to compare those final optimized weights with what we'd expect theoretically if the model was actually correct, which gives them a calibrated Pearson test for when it fails.

Lu: What I find interesting is that under correct specification, the optimal reweighting they get in that second stage will be exactly the mass that their fitted model itself assigns to its cells. If you see a departure between those optimized weights and what's theoretical under the true noise law, it points directly to where the noise law is wrong.

Meng: That implies if we are using an Ornstein-Uhlenbeck diffusion model, for instance, and we see this mismatch, it tells us something specific about that driving noise—maybe it’s more concentrated than standard Brownian motion. I need to know if this procedure is practical enough for production systems with millions of data points.

Lalam: The paper shows that the estimation stage is robust because it guards theta against mild misspecification of the structure G or the noise law nu, which means any misfit that remains is then exposed in the recourse stage. Furthermore, they proved strong consistency for both stages under checkable assumptions, meaning as you get more data, theta n0 converges near the true parameter theta0.

Tom: So it's a two-part test: first, fit the structure robustly on one set of data and second, use that structure to see if the model can correctly predict what happens in a separate held-out block. It sounds like they’re replacing some ad hoc test statistic with a structured optimization approach.

Jane: Exactly. The key is that this split makes the test calibrated because the structural parameter theta comes from an estimation block, while the weights w are a wait-and-see decision that depends on noise values from another part of the data. It yields counts that follow a known conditional multinomial distribution under no assumptions about what's wrong, giving us a calibrated Pearson test.

The paper's summary: Tom: Now let's talk about the improvements they propose, because this isn't just a baseline method; they suggest ways to make it even better. First up is the idea of using a split predictive check by splitting data into an estimation block and a held-out block to determine if theta is typical under the prior.

Jane: That sounds like tightening up the initial fitting process, making sure we are estimating a parameter that truly reflects what's happening across the whole dataset, not just one slice. Then there’s this idea for closed-form recourse decision making using Voronoi recourse, which they say is a closed form solution.

Lu: Closed-form means you don't have to run another optimization loop in the second stage; you can solve it directly using Equation twenty-three which eliminates the need to choose any test statistic for model criticism. That removes one of those subjective choices we usually have to make when we want to diagnose a failure.

Meng: If it's a closed form solution, that’s good for scaling. Running another complex optimization step on every new batch of data is not ideal if you're trying to deploy this at scale with real traffic. But does that closed form solution handle the complexity of the noise law identification well?

Lalam: The paper also points out that the diagnostic is calibrated because disagreement between the optimal weights and theoretical weights points to specific parts of the noise law that are wrong, enabling precise localization of model failure right down to the fibers of those fitted mechanisms. They can even recover things like a driving noise being more concentrated than Brownian motion in diffusion examples by looking at which mesh paths are flagged as excess or deficit.

Tom: So, we have robust estimation, closed-form recourse, and a diagnostic that points to specific noise law errors. It seems they've built a much more surgical tool for model criticism than just looking at a single error rate or loss function.

Jane: It’s about moving from general suspicion to pinpointing the exact mechanism of failure in the generative process itself, which is what I find really compelling for science communication. This moves beyond just saying "this model is wrong" to saying "the noise term here is behaving like X instead of Y."

The paper's improvements: Tom: So to wrap up, the paper "Model Criticism via Stochastic Optimization" proposes a two-stage stochastic program that uses data splitting to test generative models. They estimate a structural parameter theta robustly on one block and then use that structure in the second stage to find weights that match the held-out block.

Jane: The main implication is that this procedure yields a calibrated Pearson test, allowing us to identify exactly where a model fails by comparing optimized weights against theoretical weights derived from the fitted model. This moves past vague error detection toward precise localization of noise law errors.

Lu: For researchers, the improvement with closed-form recourse decision making is significant because it removes the need to arbitrarily choose a test statistic, providing a more objective way to diagnose structural failures in complex models.

Meng: From an engineering standpoint, if this procedure can reliably flag specific noise characteristics—like identifying when driving noise is more concentrated than standard Brownian motion—then we might actually be able to build better checks into our pipelines that alert us when the underlying assumptions about the data generation process are violated.

Lalam: The paper also has limitations they mention; for example, it requires i.i.d. replicates, meaning a single long trajectory of a process isn't covered by this framework right now. And they state that the resolution of the diagnostic is bounded by the mesh, and refining it trades resolution for noisier cell weights through the ratio n1 over N1.

Tom: So, while it’s a very powerful tool for calibration and localization, we need to keep in mind those constraints—the need for i.i.d. data and the trade-off between diagnostic precision and weight noise. That’s where the next set of work will have to focus if we want to push this into production environments where things are messy.

Jane: It's a solid piece of work that takes a complex problem—how do we know if a model is right?—and solves it by making the test itself structurally sound. We’ll be looking forward to seeing how these calibrated tests get applied in real-world AI systems.

Conclusion: Tom: So, to wrap up, "Model Criticism via Stochastic Optimization" shows how you can use data splitting and two-stage programs to test if a generative model actually fits what we see in the data, and it gives us a calibrated way to find out exactly where it’s going wrong.

Jane: It’s about taking that initial structural parameter theta, fitting it robustly on one piece of data, and then using that fit to see how well the model predicts what happens in a separate block of data.

Lu: The core idea is that if the model fails, the optimized weights from the second stage won't match what we’d expect theoretically under the true noise law, and that difference points us directly to which part of that noise law is incorrect.

Meng: For deployment at scale, it means we can use this optimization approach instead of guessing a test statistic to figure out if our model assumptions are breaking down. It’s about finding the specific failure point without needing a massive benchmark nobody has run on real traffic.

Lalam: I'm pretty sure I can say with high confidence that this procedure gives us a calibrated Pearson test, meaning when the weights don't match up, we know it’s not just random noise; it’s a specific flaw in the model’s understanding of the underlying process.

Tom: Exactly. It moves us from guessing to knowing where to look for the error. It sounds like a very practical way to debug generative models that is grounded in how data actually looks, not just some abstract loss function.

Jane: The implication for us is that when we build these kinds of models, we aren't just checking if they’re accurate overall; we’re checking if the *structure* of the noise term is correct too. That’s a much deeper level of understanding for how AI actually learns things.

Lu: It suggests that structural parameters are not always perfectly separable from the noise law, and this framework handles that entanglement by letting both stages work together sequentially.

Meng: I just worry about the overhead; if we have to do two separate optimization steps for every model iteration, it adds significant computational cost compared to just running one standard training pass.

Lalam: But the paper shows consistency under checkable assumptions, so if you can prove your assumptions hold up, this method should give you a reliable answer about where the model is failing.

Tom: Well, that’s the gist of "Model Criticism via Stochastic Optimization." It’s a powerful way to pinpoint generative failures by using a two-stage program to compare fitted structure against held-out predictions.

Jane: It gives us a much clearer picture of where the model breaks down, moving beyond just seeing that it's wrong somewhere.

Lu: And it lays out the groundwork for how we can use these calibrated diagnostics to guide better choices in our representation learning methods moving forward.

Meng: We’ll see if we can adapt this structure to a faster estimation stage for production models soon.

Lalam: Keep an eye on this work; it really formalizes how we diagnose generative model failures now.

Irving Institute for Cancer Dynamics, Columbia University

stat.ME, stat.CO

Submitted: 2026-10-27

Updated: 2026-11-11

Code: https://github.com/RenYiW/Model-Criticism-via-Stochastic-Optimization

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Many generative models are specified by how data are simulated The procedure splits the data in two and is formulated as a two-stage stochastic program This simulation-based procedure tests whether…

Key concepts

Two-Stage Stochastic Program
This is a mathematical framework used to solve complex decision problems in two sequential steps. Stage 0 involves making an initial choice (like setting structural parameters), and Stage 1 involves making a subsequent decision based on the outcome of the first stage. It allows researchers to test model fit by comparing decisions made under different assumptions.
Estimation vs. Recourse Stages
The estimation stage fits the model's structural parameter ($ heta$) using one data block while keeping noise fixed, aiming to minimize distance from simulations. The recourse stage then uses this fitted structure and representative noise values to optimize weights on the remaining data block. This separation allows for testing both structural assumptions and noise law effects.
Calibrated Pearson Test
This is the diagnostic tool used to measure model failure. Under correct assumptions, the optimal weights found in the recourse stage should match theoretical weights derived from the fitted model. The difference between these two sets of weights points precisely to which parts of the noise law are incorrect.
Robust Estimation
The method uses a robust minimum Wasserstein estimator during estimation. This technique helps ensure that the estimated structural parameter ($ ilde{ heta}$) remains reliable even if there are minor errors or misspecifications in the assumed structure ($G$) or noise law ($ u$). This makes the initial parameter fitting more stable.

Summary

Many generative models are specified by how data are simulated The procedure splits the data in two and is formulated as a two-stage stochastic program This simulation-based procedure tests whether such a model fits the data and, when it does not, locates where it fails without choosing a test statistic

Model Criticism Framework

The paper proposes a simulation-based procedure that tests whether a generative model fits the data and identifies where it fails by splitting the data into two blocks and formulating the problem as a two-stage stochastic program The estimation stage fits the structural parameter θ on one block while holding the noise law fixed, minimizing a Wasserstein distance The recourse stage freezes θ and uses representative noise values to optimize weights on the held-out block This structure allows the resulting optimal weights to be compared with theoretical weights under correct specification, yielding a calibrated Pearson test

Two-Stage Stochastic Program Details

The two stages of the program are defined as follows: Stage 0 (here-and-now) involves choosing the structural parameter θ to minimize the Wasserstein distance between data and simulations from the model, while keeping the noise law exactly as written Stage 1 (recourse) involves optimizing a vector w of weights on model atoms, where these atoms are generated by pushing a finite set of representative noise values through the fitted mechanism G(˜θ, ·) Under correct specification, the optimal reweighting returns the mass that the fitted model itself assigns to its cells The departure between optimized w and theoretical w points to the part of the noise law that is wrong

Calibration and Diagnostics

The split makes the test calibrated because θ is determined from the estimation block (F0-measurable) while w is a wait-and-see decision dependent on ξ1 The resulting counts are conditionally multinomial with known cell probabilities under the null, yielding a calibrated Pearson test The diagnostic is the comparison between the optimized weights w˜ and theoretical weights w vor(˜θ), which indicates where the model fails

Consistency and Robustness

The paper proves strong consistency of both stages under checkable assumptions The estimation stage is robust because it guards ˜θ against mild misspecification of the structure G or the noise law ν, leaving misfit for the recourse stage to expose The consistency of ˜θn0 near θ0 is established by showing that it concentrates near θ0 for large n0 under appropriate conditions

Experimental Validation

The procedure is illustrated on four models, including a Gaussian location–scale family, the number of segregating sites under Kingman’s coalescent, a Yule process, and an Ornstein–Uhlenbeck diffusion The experiments show that the diagnostic separates symmetric tail defects from one-sided gross errors in the Gaussian example The recourse stage successfully locates a state-dependent birth rate in the Yule process by identifying excess atoms corresponding to fast growth trajectories The results demonstrate that the procedure can recover that a driving noise is more concentrated than Brownian motion in the Ornstein–Uhlenbeck diffusion example

Limitations and Extensions

The theory requires i.i.d. replicates, meaning a single long trajectory of a process is not covered by the current framework The recourse identifies the noise law only on the sigma-algebra generated by the fitted mechanism, reporting departures jointly for all noise values in a fiber The resolution of the diagnostic is bounded by the mesh, and refining it trades resolution for noisier cell weights through the ratio n1/N1 Finally, since structure, noise law, and prior are entangled, a misspecified structure can be absorbed into the noise law

Code Availability

The code is publicly available on GitHub at https://github.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Calibrated Model Criticism using Split Predictive Checks: The improved system can perform model criticism by splitting data into an estimation block and a held-out block to determine if a structural parameter theta is typical under the prior, and then determining how the held-out data reweights the fitted model atoms.

  2. Closed-form Recourse Decision Making: The system can solve the recourse stage using Voronoi recourse (Equation 23), which is a closed form solution, eliminating any need to choose a test statistic for model criticism.

  3. Robust Estimation of Structural Parameters: By using a robust minimum Wasserstein estimator (Algorithm 1), the system can estimate the structural parameter theta by minimizing the W1 distance between data and simulations from the model, providing qualitative robustness against mild misspecification of the noise law or structure G.

  4. Calibrated Diagnostic for Model Failure: The system provides a calibrated Pearson test where disagreement between optimal and theoretical weights points to specific parts of the noise law that are wrong, enabling precise localization of model failure at the level of the fibers of the fitted mechanism.

  5. Automated Noise Law Identification: In cases where the noise law is misspecified (e.g., Ornstein-Uhlenbeck diffusion), the recourse stage can recover that a driving noise is more concentrated than standard Brownian motion by examining which mesh paths are flagged as excess or deficit.

Abstract

Many generative models are specified by how data are simulated. An observation is a deterministic transform Y=G(θ,X) of a draw X from an assumed noise law, indexed by a structural parameter θ. Such models cover discrete, continuous and functional data, as well as i.i.d. replicates of stochastic processes, but their likelihoods are often unavailable. We propose a simulation-based procedure that tests whether such a model fits the data and, when it does not, locates where it fails, without choosing a test statistic. The procedure splits the data in two and is formulated as a two-stage stochastic program. In the estimation stage, θ is fitted on the first block by minimizing a Wasserstein distance between the data and simulations from the model. In the recourse stage, θ is frozen at the fitted θ, and a finite set of representative noise values x j is pushed through the fitted mechanism to give model atoms G(θ, x j). The held-out block optimizes weights w on the model atoms to best match its own distribution. The optimized weights are then compared with theoretical weights under correct specification. Since each model atom is the image of known noise values, the disagreement between optimal and theoretical weights points to the part of the noise law that is wrong. The split makes the scaled weights conditionally multinomial, yielding a calibrated Pearson test. We prove strong consistency of both stages under checkable assumptions and illustrate the procedure on a Gaussian location--scale family, the number of segregating sites under Kingman's coalescent, a Yule process, and an Ornstein--Uhlenbeck diffusion.

Sources

Related papers