A regularization-independent Information Criterion for correlated data
summary
The gist
A regularization-independent Information Criterion for correlated data allows for model and data subset selection when regularized fits are performed on correlated data The main result is represented…
In short
The RIAIC is a new Information Criterion designed to select optimal models and data subsets when performing regularized fits on correlated data. It aims to be regularization-independent by defining it as an estimator of the model-dependent contribution to the Kullback–Leibler divergence, allowing for reliable comparison across different regularization methods.
Key concepts
- Information-Theoretic Criteria (ICs)
- These are methods used to choose the best model from several options. They are theoretically sound because they are based on information theory, specifically the Kullback–Leibler divergence. They balance how well a model fits the data against how complex that model is.
- Regularization Dependence
- Standard Information Criteria, like AIC, become unreliable when regularized fits (using techniques like a covariance matrix C) are performed on correlated data because they explicitly depend on the regularization parameter. This dependence makes comparing models difficult.
- RIAIC (Regularization-Independent Information Criterion)
- The RIAIC is the proposed solution. It is constructed to be 'on average, regularization-independent' by defining it as an estimator of the model's contribution to the correlated divergence. This allows users to compare models fitted with different regularizations without systematic bias.
- RIBAIC (Regularization-Independent Bayesian Information Criterion)
- This is the Bayesian extension of RIAIC, used for MAP fits. It provides a unified framework that allows for comparing frequentist and Bayesian fits, even when they use different regularization techniques, simplifying model selection in complex analyses.
Terminology used across episodes
This episode discusses
- A regularization-independent Information Criterion for correlated data · Paper Radio
- On fits to correlated and auto-correlated data
- Monte Carlo errors with less errors
- Improved information criteria for Bayesian model averaging in lattice field theory
- Model averaging approaches to data subset selection
- Leading hadronic contribution to the muon magnetic moment from lattice QCD
- Bayesian model averaging for analysis of lattice field theory results
- Constrained Curve Fitting
- Preliminary study of the H dibaryon in N f=2+1 lattice QCD
- Two-nucleon scattering amplitudes at light quark masses from lattice QCD · Paper Radio
- Quark-mass dependence of two-nucleon scattering · Paper Radio
- D and D s decay constants in N f=2+1 QCD with Wilson fermions
The paper
A regularization-independent Information Criterion for correlated data · Read on arXiv
Davide Laudicina
Ruhr-Universitຜ Bochum
We generalize the Akaike Information Criterion (AIC) to allow model and data subset selection when regularized fits are performed on correlated data. While the standard generalization to regularized fits is inherently dependent on the type of regularization applied to the data covariance matrix, our definition, the RIAIC, is, on average, regularization-independent. The regularization-independence is achieved by defining the new Information Criterion as an estimator of the model-dependent contribution to the correlated Kullback--Leibler divergence. The Bayesian extension is derived as well. We test the new criterion on lattice QCD data by performing regularized fits and show that it reproduces the selection properties of the standard correlated AIC, with fluctuations which are compatible with the expected statistical uncertainties. These results enable the application of subset and model selection techniques to cases where regularizations are applied to the covariance matrix of the fitted dataset.
Transcript
Introduction to the show: ident: High Energy Physics: Lattice. The day in lattice field theory: what the simulations put on the table.
Kai: We're looking at the paper: "A regularization-independent Information Criterion for correlated data".
Mira: Here's the gist of the paper. A regularization-independent Information Criterion for correlated data allows for model and data subset selection when regularized fits are performed on correlated data The main result is represented by the RIAIC and is reported in eq.
Kai: (3.4) for standard frequentist fits Information-Theoretic Criteria Background Information-Theoretic Criteria are a widely used approach to model selection problems because they are theoretically sound, since their definition is rooted in information theory through the Kullback–Leibler divergence, and by construction, they represent a trade-off between quality of the fit and complexity of the description One commonly used IC is the Akaike Information Criterion (AIC)
1, 2: The standard definition of the AIC explicitly assumes that correlations in the dataset under study are properly taken into account Regularization Dependence and the RIAIC Definition The standard generalization of the AIC to regularized fits induces an explicit dependence on the regularization which makes it unreliable for systematic comparison of models and subsets The authors propose a new Information Criterion which is explicitly built to be, on average, regularization-independent This is achieved by defining the new Information Criterion as an estimator of the model-dependent contribution to the correlated Kullback–Leibler divergence The resulting criterion is defined as RIAICη = −2 log L(y, C θ C MLE, η) + B (y, C, η) Derivation of the Regularization-Independent Criterion The crucial step to absorb any regularization-dependent term is taking the regularized log-likelihood in the equation above as an estimator of eq.
Mira: (2.3) The bias to be computed reads B(y, C, η) ≡ E h 2 log L(y, C θ C MLE, η) + ∆(Σ θ Σ MLE, η) i Under the assumption of normally-distributed data and correct model specification, the bias reads B(y, C, η) = nd − n eff d + nη + n eff η + log det Σ − log det C This leads to the expression of the regularization-independent AIC RIAICη = χ 2 C + nd − n eff d + nη + n eff η + nd log 2π + log det Σ Application in Model and Dataset Selection The RIAIC allows the comparison between models and subsets when regularizations are applied to the covariance matrix of the fitted dataset In dataset selection, this leads to the final expression RIAICfull = χ 2 C + nd − n eff d + nη + n eff η + 2nC This expression inherits all the properties of the RIAIC: it reproduces, in the correlated limit, the standard result in eq.
Kai: (3.6) and its expectation value is regularization-independent Bayesian Extension and Statistical Properties The Bayesian extension, RIBAICη = −2 log L(y, C θ C MAP, η) + B (y, C, η) In the limit of non-informative priors, both P0 and P1 reduce to the projector in eq.
Title and authors: Kai: So we’re looking at this paper called A regularization-independent Information Criterion for correlated data. It looks like they're tackling a real problem when you try to pick the right model or the right subset of data if you've used some kind of regularization on your covariance matrix.
Mira: Right, it sounds like they are trying to fix a known weakness in how we use standard criteria when the data has correlations and we are using regularized fits. It seems they want a criterion that doesn't depend so much on *how* you regularized things.
Lev: If I’m being honest, when you try to run something on real hardware with noise, the choice of how you handle those covariance matrices really messes with your error estimates, and this paper seems to be aiming at making that part more robust.
Kai: Exactly. The authors propose a new one called the Regularization-Independent Information Criterion or RIAIC, which is built to be on average regularization-independent. They achieve this by defining it as an estimator of the model dependent contribution to the correlated Kullback Leibler divergence, which is basically a way to measure how well your model describes the data while accounting for those correlations in a more stable way.
Mira: That sounds like they're taking something theoretically sound, like information theory, and modifying it so that when you apply a regularization technique—say SVD cuts or some other method to tame that noisy covariance matrix—the resulting criterion gives you a fair comparison between different model choices.
Lev: I wonder how much of this actually saves us on the ground. Because if the criterion is regularized independent, it means when we compare two models fitted with different regularization strengths, the comparison isn't skewed just because one fit used a heavier regularization than the other.
Kai: That's what they are testing. They test it on lattice QCD data by performing those regularized fits and show that the RIAIC reproduces the selection properties of the standard correlated Akaike Information Criterion, but with fluctuations that match what we expect from statistical uncertainties.
Mira: So, in simple terms, they're saying this new criterion gives you the same optimal model and data subset choices as the standard one, even when you’ve tamed your covariance matrix with regularization. But it does it without being overly sensitive to the specific regularization method you choose.
Lev: That makes sense from an engineering standpoint. If I have to pick a fit for my simulation, I don't want to have to re-run everything just because I slightly changed the cutoff or the covariance matrix structure for a more stable result.
Kai: The paper also extends this idea into the Bayesian framework, creating something called the Regularization-Independent Bayesian Information Criterion or RIBAIC. This allows you to compare frequentist and Bayesian fits that might use different regularization techniques in a unified way for model averaging or subset selection.
Title and authors: Mira: That’s a big step for consistency across different statistical approaches. If you're doing model selection, whether you're using maximum likelihood or maximum a posteriori estimation, this framework should give you comparable results when dealing with correlated data and regularized fits.
Lev: For hardware running on real systems, that unified comparison is helpful because it means the overhead of the regularization choice doesn't dictate which physics we choose to study. It just becomes part of the noise we account for in a standard way.
Kai: They also looked at how much the criterion actually changes when you introduce regularization, calculating something called delta IC equals AIC minus RIAIC. They found that the RIAIC exhibits deviations of a few units from the AIC over the whole range of tmin in dataset and model selection scenarios, which is expected because they have similar average values.
Mira: So while they show it's close to the standard one, it’s not identical. That small difference is what they think reflects the actual impact of that regularization choice on your final result, which is a kind of insight you don't get from just looking at the average value alone.
Lev: I like that specificity. Knowing the deviation size gives us a concrete idea of how much we are potentially losing or gaining by choosing one path over another in our analysis pipeline.
Kai: So, to wrap up on this paper, they've developed the RIAIC and RIBAIC which lets you select models and data subsets even when using regularized fits on correlated data, making the criterion more robust to regularization choices while still matching the standard results in the correlated limit.
Mira: It’s a solid methodological improvement because it addresses that dependency head-on by redefining the information criterion as an estimator of the model dependent contribution to the Kullback Leibler divergence.
Lev: For us, it means we can trust our selection process more when we are dealing with complex covariance matrices, even if we have to apply some kind of regularization to keep things numerically stable on our finite samples.
Kai: So that’s what they did with this paper titled A regularization-independent Information Criterion for correlated data. It shows how you can do model and dataset selection when regularized fits are performed on correlated data, and it gives us the RIAIC and the RIBAIC which are reported in equations three point four and three point eleven respectively.
Mira: It’s a useful tool that lets us compare models or subsets when we apply regularization to the covariance matrix of our fitted dataset, because it's on average regularization independent.
Lev: It gives us a way to quantify the impact of regularization through delta IC, showing how much it deviates from the AIC, which is useful for understanding those trade-offs in our analysis pipeline.
The paper's summary: Kai: So, to quickly recap, this paper introduces the Regularization-Independent Information Criterion or RIAIC to handle model and data subset selection when you’re using regularized fits on correlated data.
Mira: Right, it takes a standard tool from information theory, like the Akaike Information Criterion, and fixes its problem where it gets too tied up in the specific way you regularize your covariance matrix.
Lev: It's about making sure that if I use a different regularization technique—say I change my cutoff or my truncation scale—the criterion still gives me a fair comparison between two models.
Kai: Exactly, it’s supposed to be on average regularization-independent, which is a big deal because in real experiments, you have to choose some kind of regularization anyway.
Mira: The paper shows this by defining the RIAIC as an estimator of that model dependent contribution to the Kullback Leibler divergence. It’s built so that the bias term absorbs whatever regularization dependency there is.
Lev: From an engineering standpoint, that means if I run a fit with one kind of regularization and another kind, I don't have to re-derive the entire criterion just because I tweaked my method slightly.
Kai: And they test it on lattice QCD data using frequentist fits, showing that the RIAIC reproduces the standard result in the correlated limit, but with expected statistical fluctuations around its average value.
Mira: It’s also extended to Bayesian fits with the RIBAIC, which lets you compare different estimation methods—frequentist and Bayesian—when they both use different regularization techniques for model averaging.
Lev: That unified framework is useful because it means you can combine results from both approaches if your hardware setup favors one over the other in terms of noise handling.
Kai: They also calculate a delta IC, which is the difference between the RIAIC and the standard AIC, showing that while they are close on average, they do show deviations that reflect how much regularization actually matters for a specific set of data.
Mira: So while it’s designed to be robust against your choice of regularization method, the deviation itself tells you something about the actual impact of that choice on your final selection.
Lev: It gives us a way to quantify that impact without having to run multiple full simulations just to see how sensitive our selection is.
Kai: This sets up a nice contrast with other papers we've seen, like those focusing on variance reduction techniques for lattice QCD, which aim to improve the quality of the data estimation itself.
Mira: So this paper is more about refining the selection process *after* you have made your fit and regularization choices, rather than just making the initial data estimation cleaner.
Lev: It’s about making sure that when we pick our best subset or model, we aren't just picking a model that happens to work well for the specific regularization method we chose for that one run.
The paper's improvements: Kai: So, we’re looking at how this paper suggests improving their original idea by actually quantifying the impact of that regularization choice on the selection decisions.
Mira: Right, they're not just giving you a criterion that works; they’re giving you a way to measure how much it deviates from the standard Akaike Information Criterion.
Lev: That delta IC, which is AIC minus RIAIC, actually shows deviations of a few units across the whole range of tmin in both dataset and model selection scenarios.
Kai: So, that means even though they are on average regularized independent, you can still see a measurable difference depending on what you’re looking at in your simulation.
Mira: It confirms what we suspected—that the regularization choice doesn't just vanish; it actually leaves a footprint on the selection process.
Lev: That kind of insight is useful because it tells me exactly how sensitive my final result is to whether I use a more aggressive or a more conservative regularization scheme.
Kai: And they point out that this deviation happens over the entire range of the simulation parameter, which means it’s not just a fluke happening at one specific point.
Mira: It also shows that the RIAIC identifies the same optimal window for selecting models and data subsets as you would with the standard AIC, which is good because it validates their approach.
Lev: So, if I'm trying to decide where to cut my data or which model parameters are most important, this new measure tells me that even with regularization noise, I'm still looking at the same physics.
Kai: It’s a way to reconcile the theoretical hope of independence with the practical reality of using finite, regularized datasets.
Mira: And they also mention how this framework extends to Bayesian fits, RIBAIC, which helps unify frequentist and Bayesian comparisons when regularization is involved.
Lev: That unification is key for real hardware testing because it lets you compare your frequentist analysis against a Bayesian one without having to invent a whole new statistical language for every case.
Kai: So this paper moves beyond just saying "it works" to showing precisely how much the choice of regularization actually alters the final selection outcome, which is what we need for rigorous analysis.
Mira: It’s an improvement because it gives you a concrete number—the delta IC—that tells you exactly how much information loss or gain you are seeing due to your finite covariance matrix.
Lev: That quantitative measure is what I need when I'm trying to design error-correction schemes; knowing the sensitivity of the selection criterion is vital for setting those parameters correctly.
Kai: It’s a nice piece of work because it bridges the gap between theoretical information criteria and the messy reality of applying regularization to complex, correlated data sets.
Conclusion: Kai: So we’re wrapping up on this paper, "A regularization-independent Information Criterion for correlated data." It boils down to them creating the RIAIC and RIBAIC to handle model selection when you’ve used regularized fits on correlated data.
Mira: Right, they managed to make a criterion that doesn't depend so much on the specific way you regularize things, which is a major win for consistent model comparison.
Lev: It means that when I’m trying to compare two different regularization methods for my error correction, the selection process itself shouldn't be biased by which method I picked.
Kai: They showed that this new criterion reproduces the standard result in the correlated limit, but with expected statistical fluctuations around its average value.
Mira: And they quantified that deviation through a delta IC, showing exactly how much your regularization choice actually shifts the selection results away from the ideal case.
Lev: That's useful because it gives us a number on how sensitive our decision is to that regularization noise in our simulation setup.
Kai: The RIBAIC extension is also important for linking frequentist and Bayesian analysis when dealing with those same regularized fits, which is pretty big for model averaging procedures.
Mira: It’s a solid methodological step because it addresses that dependency head-on by defining the criterion as an estimator of the model dependent contribution to the Kullback Leibler divergence.
Lev: For building real hardware, having a unified framework like RIBAIC means you can compare different theoretical models without needing to re-derive everything for every new regularization scheme.
Kai: So, this paper gives us a more robust tool for making decisions about model and data subset selection even when the underlying covariance matrix is being tamed with regularization.
Mira: It’s a useful tool because it lets us compare models or subsets when we apply regularization to the covariance matrix of our fitted dataset, because it’s on average regularized independent.
Lev: So yeah, it makes the selection process more reliable when you’re dealing with complex correlations and finite sampling.
Kai: We'll keep an eye on how this RIAIC is used in future lattice QCD studies to see if those theoretical expectations hold up under real experimental conditions.
Mira: Definitely, because understanding these trade-offs between fit quality and regularization complexity is something we need to keep exploring in the condensed matter side.
Lev: For me, it's good to see methods that are grounded in information theory but also have a clear path on how they behave when you introduce practical constraints like those regularization schemes.
More episodes
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2606.20029-Finite-Volume Step Scaling at Fixed Truncation in (2+1)D Hamiltonian U(1) Lattice Gauge Theory
- 2603.16266-Lattice QCD study of the K*(892) resonance at the physical point
- 2607.15742-Path optimization method for the sign problem: Insights from random matrix models
- The day in conversation for 2026-11-18
- Daily Summary for 2026-11-18
- Daily Summary for 2026-11-17
- 2608.18239-Searching for symmetric mass generation with staggered fermions in four dimensions
- The day in conversation for 2026-11-17