Estimating Implicit Regularization in Deep Learning

arXiv:2605.05436 · stat.ML, cs.LG · Submitted 2026-05-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Estimating Implicit Regularization in Deep Learning".

Jane: Here's the gist of the paper. Deep learning systems exhibit implicit regularization that favors simple solutions instead of merely minimizing the loss function.

Tom: The authors provide gradient matching methods to empirically estimate this implicit regularization, which is crucial for understanding complex, modern deep learning systems.

Jane: Estimating Implicit Bias The core task is to find a mathematically expressible regularizer R such that optimizing the empirical loss augmented by R is functionally equivalent to an implicitly biased procedure optimizing L alone.

Tom: This involves matching the gradients of the loss to the gradients of a proposed regularizer based on a decomposition of the empirical loss.

Title and authors: Tom: So we're looking at a paper called "Estimating Implicit Regularization in Deep Learning". This whole topic is about how deep learning systems naturally favor simpler solutions instead of just finding the absolute lowest loss value.

Jane: Right, so it’s about this implicit bias, this tendency to pick simpler models when there are many possible solutions. The authors are looking at how to actually measure or estimate that hidden bias in these complicated deep learning systems.

Lu: Basically, they're trying to find a mathematical way—a regularizer R—so that if you optimize the loss plus this regularizer R, it behaves exactly the same way as just optimizing the loss alone.

Meng: But the authors say that for modern deep learning systems with things like early stopping or dropout, we can’t always figure out what that implicit bias actually is because those techniques make the training procedure different.

Tom: That’s where this paper steps in by proposing gradient matching methods to empirically estimate this implicit regularization. They want to find a way to see the signal that tells us about the bias without having a perfect analytical formula for every single architecture.

Jane: So, the core idea is they set up a system of equations where they match the gradients of what we actually measure—the loss gradient—to the gradients of some proposed regularizer.

Lu: They assume that the learned weights are an exact solution to a regularized empirical risk minimization problem, meaning the sum of their loss gradient and this proposed regularizer's gradient is zero at the solution.

Tom: That’s a heavy assumption, but it gives them something concrete to work with. They then talk about two ways they can succeed in this estimation: recovering the ground truth regularization or recovering a regularizer that just mimics what the implicit regularization actually does.

Jane: So, one way is if they can recover the actual penalty that was used during training, and another way is if they can find a new regularizer that produces the same effect on the weights.

Meng: I’m interested in how much of a headache this actually is to set up these gradient matching equations. Does it require us to know the exact architecture beforehand?

Tom: Not necessarily, but they say the gains in estimation accuracy depend on assumptions about how correlated these gradients are and how many parameters there are effective. It scales as big O of p over peff.

Title and authors: Jane: That scaling factor, that big O of p over peff, is important because it tells us how much better our estimate gets as we get more data or more complex models involved. They suggest using multiple independent endpoints or bootstrap resampling can increase that effective parameter count and improve coverage in the estimate.

Lu: They also look at stability by testing two modes: a constant regularization term Lambda, and a time-varying one Lambda of t, which lets them track how the implicit regularization evolves as training progresses.

Tom: Tracking it over time is really useful because it shows us if that bias is something that changes during the training process, not just some static feature. They show they can recover Barrett and Dherin's implicit gradient regularization in OLS by fitting a scalar least-squares fit to the trajectory deviations, which gives them an analytic coefficient lambda equals eta p over four when compared against known continuous gradient flow.

Jane: So they are using that known coefficient as a calibration check for their trajectory estimator. And they show that the error scales with terms involving log NR of rho and c of R and R prime.

Meng: It sounds like they’re proving it works on something we actually see, like OLS, which is a good starting point because it’s mathematically simpler than a full deep network. But what about when we move to these huge nonlinear models?

Tom: That’s the next part of the paper. They extend this framework to complex networks that are too complicated for pure math analysis, like estimating the implicit l2 penalty induced by dropout in deep nonlinear networks.

Lu: They show that this estimated effective l2 penalty increases monotonically with the dropout rate, which matches what theory predicted about how dropout works. This gives practitioners a way to use gradient matching to understand regularization in those complicated networks that are otherwise hard to derive analytically.

Jane: So it’s not just about fitting a simple line; it’s about applying this concept across different training techniques like early stopping, dropout, and even discrete gradient steps. They show they can recover the quadratic weight penalty from early stopping in OLS when using an underparametrized diagonal estimator.

Tom: That's a solid result because it connects theoretical concepts like regularization penalties to concrete empirical observations we see in practice. It shows that these methods are actually useful tools for interpreting hyperparameter choices.

Jane: They also discuss what the method can do for validation, showing that if you retrain the model using an estimated explicit regularizer, you recover the original weights under local convexity assumptions. That’s a big deal because it connects back to validating theoretical results.

Title and authors: Lu: The authors are quite careful about their limitations here. They note that fitting near zero can hinder estimation because if many weights are small, you can’t actually tell what the true regularization parameters are from the noise.

Meng: And how does that affect deployment? If we have to rely on a method where small weights make estimation unstable, that adds risk when we try to deploy these models at scale.

Tom: That’s a fair question, Meng. The paper suggests normalizing gradients by rescaling each candidate gradient to unit norm before fitting coefficients in that normalized basis, which helps when candidate regularizers produce gradients with wildly different norms. It also points out that regularization parameters associated with collinear gradients can yield estimates with high variance.

Jane: So, if we're using this for production work, we have to be careful about the noise introduced by those collinearities and the scaling issues from different gradient norms.

Lu: And they mention that bootstrap recovery beats reporting zero only when the OLS sampling variance is non-negligible relative to the coefficient magnitude. This is a practical caveat about when you can trust those empirical estimates over a null result.

Meng: So, for me, what I’m hearing is that this isn't some magic tool that gives us perfect answers; it's more of a way to gain insight into training configurations early on in our deep learning systems.

Tom: Exactly. The authors emphasize that estimating implicit regularization provides a cheap and practical way for practitioners to study training configurations early on, which is something we really need when we are trying to generalize beyond our training data.

Jane: So, to wrap up the paper "Estimating Implicit Regularization in Deep Learning", the main implication is that we can move beyond just observing loss curves and start empirically characterizing the hidden regularizing forces at work in complex deep learning models.

Lu: It opens exciting avenues for future research into how these implicit regularization effects interact with many of the engineering tricks used in state of the art deep learning systems.

Meng: I think it’s a good starting point for understanding why our models train the way they do, even if we can't fully predict every single weight update.

Tom: Agreed. This work gives us a tool to empirically characterize that implicit regularization and helps us understand how these complex systems generalize beyond their training data.

The paper's summary: Jane: So, to recap where we are, these authors are showing us how to actually measure that hidden bias in deep learning systems, this implicit regularization thing we talked about earlier.

Tom: Right. They’re proposing gradient matching methods so you can solve for those unknown regularization parameters, Lambda, by matching the loss gradients against the proposed regularizer's gradients.

Lu: It’s not just a fancy trick; they set up a system of equations where the only unknowns are those regularization parameters Lambda, and they assume that the learned weights are an exact solution to a regularized minimization problem.

Jane: That means they can distinguish between two things: first, if they can recover the true ground truth regularization penalty from training data, and second, if they can find a completely different regularizer that produces the same effect on those weights.

Tom: That’s where it gets practical; we’re not just guessing what the bias is; we’re trying to characterize successful estimation by seeing which of those two things matches up with reality.

Lu: They show this framework can recover explicit penalties like l1 or l2, and even known implicit effects, like that quadratic weight penalty you get from early stopping in gradient descent.

Jane: That’s huge because it lets us test theories about implicit regularization by showing the method works on networks with known regularization; it validates popular explicit penalties.

Tom: And they extend this to things that are way too complicated for pure math, like characterizing dropout effects in deep nonlinear networks, showing those implicit l2 effects.

Lu: They also look at tracking how the implicit regularization evolves over time along the training trajectory, which is better than just looking at a model once it has converged.

Jane: So they can see if that hidden bias is something that changes as the model learns, not just some fixed setting from the start.

Tom: And they validate their results by showing you can recover the original weights if you retrain using an estimated explicit regularizer, which connects back to theoretical validation.

Lu: They are very careful about what this method doesn't do; for instance, fitting near zero can hinder estimation because if many weights are small, you can’t reliably identify the true regularization parameters.

Jane: So it’s a tool with clear limits; it won't work perfectly when things get too noisy or when the weights are very tiny.

Tom: But they suggest normalizing the gradients by rescaling them to unit norm before fitting, which helps when candidate regularizers produce gradients with wildly different scales.

Lu: Plus, they mention that if you use multiple independent data points or bootstrap resampling, you can increase the effective parameter count, which improves coverage in the estimate.

Jane: So what this means for us is that we can start empirically characterizing these hidden forces during training rather than just looking at the final loss number.

Tom: It’s a cheap and practical way to study training configurations early on, which is essential when we need to generalize beyond our own training data.

Lu: This paper opens up exciting avenues for researching how these implicit regularization effects interact with all those complex engineering tricks used in state of the art deep learning systems.

The paper's improvements: Tom: So, we’ve seen how they figured out that gradient matching can reveal implicit bias, and now they’re talking about how to make this method even better for real-world use.

Jane: Right. They're suggesting a few ways to improve the estimation process itself, moving beyond just getting a basic idea of the regularization parameter Lambda.

Lu: They propose normalizing the gradients by rescaling each candidate gradient to unit norm before fitting those coefficients in that normalized basis, which helps when those candidate regularizers have wildly different gradient norms.

Tom: That’s smart; it addresses that problem we talked about earlier where if one regularizer has a huge gradient and another is tiny, you get skewed results.

Jane: And they point out something else, and this is important for understanding the estimation stability; they say that smoothing near zero hinders the estimation because when many weights are small, you can’t actually tell what those true regularization parameters are from the noise.

Lu: They also look at how we track these regularization parameters along training trajectories, not just at one final point, which lets us see if the implicit bias is changing during training.

Tom: That time-varying Lambda is really powerful because it tells you if that bias evolves as the model learns over time.

Jane: And they show that for recovering Barrett and Dherin's implicit gradient regularization in OLS, they fit a single parameter ridge bias to those trajectory deviations against a known analytic coefficient of eta p over four.

Lu: That gives them a calibration check against the known continuous gradient flow, which is a solid way to validate their estimator.

Tom: And they also showed that retraining on the estimated regularizer can actually recover the original weights under local convexity assumptions, which validates that connection we mentioned earlier.

Jane: But there are still caveats; they say the recovered weight matching only implies stationarity for the regularized objective, not necessarily that retraining with those specific Lambda estimates will perfectly recover you without more curvature assumptions.

Lu: Plus, when they look at bootstrap recovery, it’s only qualitatively better than reporting zero if the OLS sampling variance is actually non-negligible compared to the coefficient magnitude.

Tom: So what this means for deployment is that we can use this to study training configurations early on, but we still have to watch out for noise from collinear gradients and scaling issues.

Jane: It’s a starting point for characterizing that implicit regularization, and it opens up exciting avenues for future research into how these effects interact with the engineering tricks in state of the art deep learning systems.

Conclusion: Tom: So, to wrap up, we’ve looked at how the paper "Estimating Implicit Regularization in Deep Learning" uses gradient matching to empirically estimate that hidden bias in systems like dropout and early stopping.

Jane: Exactly, they show us a practical way to characterize what those implicit forces actually are without needing a perfect analytical formula for every single architecture.

Lu: This work provides a starting point for empirically characterizing the implicit regularization in deep learning systems, and it opens exciting avenues for future research into how these effects interact with the many engineering tricks used in state of the art deep learning systems.

Tom: It’s a cheap and practical way to study training configurations early on, which is really important when we need to generalize beyond our training data.

Jane: I think it confirms that we can move beyond just observing loss curves and start empirically characterizing the hidden regularizing forces at work in complex deep learning models.

Lu: The direct estimation of regularization parameters instead of just a scalar quantity is what makes this approach so useful for empirical analysis and contrasting theoretical predictions about implicit bias in realistic models.

Meng: I just wonder how much of this actually translates to running a model on real traffic, because we saw some scaling issues with those effective parameter counts.

Tom: That’s fair; the paper flags that regularization parameters associated with collinear gradients can give estimates with high variance, so we have to be careful there.

Jane: So it’s a starting point for understanding why our models train the way they do, even if we can't fully predict every single weight update.

Lu: I think the method is solid because it connects theoretical concepts like regularization penalties to concrete empirical observations we see in practice, like recovering l1 or l2.

Meng: It’s good that you can validate those theoretical results by showing that retraining with an estimated explicit regularizer recovers the original weights under local convexity assumptions.

Tom: That’s a big win for validating analytical results, I agree.

Jane: Overall, "Estimating Implicit Regularization in Deep Learning" gives us a tool to empirically characterize that implicit regularization and helps us understand how these complex systems generalize beyond their training data.

Joseph H. Rudoler, Kevin Tan, Giles Hooker, Konrad P. Kording

University of Pennsylvania

stat.ML, cs.LG

Submitted: 2026-05-06

Updated: 2026-11-04

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: Deep learning systems exhibit implicit regularization that favors simple solutions instead of merely minimizing the loss function.

Key concepts

Implicit Regularization
This refers to the inherent bias in deep learning models that favors simpler solutions rather than just finding the absolute lowest loss. It's an unstated preference built into the optimization process, which this method seeks to measure and quantify.
Gradient Matching
This is a core technique where researchers equate the gradients of the actual loss function with the gradients of a proposed regularizer. By solving this system, they can empirically find regularization parameters that reproduce or approximate the model's implicit bias.
Trajectory Estimation
Instead of just looking at a model at its final state, this framework estimates how regularization changes over time during training. This allows researchers to track the evolution of implicit bias as the model learns, providing a richer understanding than static analysis.

Terminology

Summary

Deep learning systems exhibit implicit regularization that favors simple solutions instead of merely minimizing the loss function. p. 0

Where the paper says it

Kording University of Pennsylvania Abstract Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizing the loss function.

The authors provide gradient matching methods to empirically estimate this implicit regularization, which is crucial for understanding complex, modern deep learning systems. p. 0
Where the paper says it

It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization.

Estimating Implicit Bias

The core task is to find a mathematically expressible regularizer R such that optimizing the empirical loss augmented by R is functionally equivalent to an implicitly biased procedure optimizing L alone. This involves matching the gradients of the loss to the gradients of a proposed regularizer based on a decomposition of the empirical loss. p. 1

Where the paper says it

A novel empirical framework for estimating parametrized models of implicit regularization. We do so by matching the gradients of the loss to the gradients of a proposed regularizer, based on a decomposition of the empirical loss.

The framework assumes that learned weights are an exact solution to a regularized empirical risk minimization (RERM) problem, meaning ∇θL(θ, D)θˆ(D) + ∇θR(θ, D)θˆ(D) = 0. p. 2
Where the paper says it

Our key assumption is to view the residual loss as a quantity associated with an additional objective, namely the regularization objective. We understand implicit effects by assuming the learned ˆθ is an exact solution to a regularized empirical risk minimization (RERM) problem: ˆθ(D) = arg min θ∈Θ L(θ, D) | {z } empirical loss + R(θ, D) | {z } regularization term for some regularizer R.

Characterizing Successful Estimation

The method distinguishes between two modes of success: recovering ground truth regularization and recovering a regularizer that mimics the effect of the implicit regularization. Recovery of hypothesized Λ is achieved in simple settings where results are predicted by theory, while recovery of original model weights is validated by explicitly regularizing the training feature and comparing it to the original solution. p. 1

Where the paper says it

We show that indeed we achieve (1) in simple settings where results are predicted by theory. We can evaluate (2) by estimating a regularizer and using it to retrain the model, then comparing the retrained model to the original.

From Theory to Practice

The approach bridges theory and practice by allowing theoreticians to empirically validate analytical results, demonstrated by recovering the implicit regularization of gradient descent in OLS. This tool also extends the study of implicit bias to complex networks that evade mathematical analysis, such as estimating the implicit regularization induced by dropout in deep nonlinear networks. p. 1

Where the paper says it

• An interpretability tool for training complex networks. Our method also extends the study of implicit bias to complex networks that evade mathematical analysis.

Where the Gains Come From

The estimation framework relies on matching gradients, which leads to a system of equations involving computable gradients of the loss and candidate regularizer, where the only unknowns are the regularization parameters Λ. The gains in estimation accuracy depend on assumptions regarding collinearity of gradients and the effective parameter count, scaling as O(p/peff). p. 4

Where the paper says it

We elaborate and provide theoretical guarantees for both this parametric and the associated nonparametric case in Appendix B. There we show that estimation error scales in O( p r/peff), where in the linear parametric case r is the number of regularizers and peff is the effective parameter count (this is analogous to effective sample size, since parameters are samples in our regression of ∇θL onto ∇θR).

Furthermore, using multiple independent endpoints or bootstrap resampling can increase the effective parameter count and improve coverage. p. 4
Where the paper says it

In realistic data settings, many datasets (especially those from the same domain, like images) are correlated and therefore will lead to more similar ˆθ – but even sampling variability in ˆθ will improve our coverage. Another natural way to obtain additional estimating equations, when constrained to a single training dataset, is through bootstrap resampling: each bootstrap replicate yields a new fitted parameter vector and thus a new approximate stationarity condition, expanding the gradient equations available for estimating Λ.

Stability under Approximate Gradient Matching

The study examines two modes of trajectory estimation: constant Λ and time-varying Λ(t), allowing for tracking how implicit regularization evolves over training. The recovery of Barrett and Dherin's implicit gradient regularization in OLS is achieved by fitting a scalar least-squares fit to trajectory deviations, which yields the analytic coefficient λ = ηp/4 when compared against the known continuous gradient flow. The generalization bound derived shows that error scales with terms involving log NR(ρ) and c(R, R′).

Computing Loss and Regularizer Gradients

Computational complexity is managed by computing the vector-Jacobian product instead of materializing the full Jacobian matrix. This avoids explicitly forming the full Jacobian ∇θYˆ(θ) ∈ Rk×p, which is computationally expensive. p. 2

Where the paper says it

We discuss this and other challenges, along with suggested remedies, in Section 2. Appendix D discusses computing the gradients in more detail: the loss gradient decomposes into a vector-Jacobian product that can be computed efficiently without materializing the full Jacobian.

The gradient matching condition implies a system of p equations where p is the dimensionality of θ. p. 2
Where the paper says it

So, if we assume the observed parameters ˆθ are an optimum, then the regularizer gradients must negate the loss gradient! Note that our stationarity condition implies a system of p equations where p is the dimensionality of θ.

Related Work on Estimating Inductive Bias

Other approaches quantify inductive bias as a scalar quantity rather than estimating regularization parameters, which contrasts with this work that directly estimates the parameters of hypothesized regularizers. This direct estimation enables empirical analysis and contrasting of theoretical predictions about implicit bias in realistic models. p. 1

Where the paper says it

Estimating implicit bias should allow theoreticians to empirically validate their analytical results and extend the study of implicit bias to complex networks that evade mathematical analysis, thereby providing a stronger bridge between theory and practice. For empirical researchers training deep neural networks, even heuristic estimates of implicit regularization can be a useful diagnostic.

Bootstrapping

Bootstrap resampling is used to replicate the OLS early-stopping recovery experiment, showing that while multi-dataset experiments provide exact consistency, bootstrap recovery is qualitatively weaker. The spread in ˆθ∗(b) scales as σ/√n, and bootstrap recovery beats reporting zero only when the OLS sampling variance is non-negligible relative to the coefficient magnitude.

Additional Considerations for Stable Estimation of Implicit Regularization

The authors suggest normalizing gradients by rescaling each candidate gradient to unit norm before fitting coefficients in that normalized basis, mapping back to the original regularization parameters. This normalization helps when candidate regularizers induce gradients with very different norms. p. 4

Where the paper says it

Collinearity of gradients. A natural application of our method is to simultaneously fit a number of candidate regularizers with scale coefficients – essentially turning the problem into linear regression over a set of regularizer gradients.

The analysis of elastic net recovery highlights that smoothing near zero hinders estimation because the true parameters are not identifiable when many θ are small. p. 5
Where the paper says it

Figure 3 shows the recovered penalties as a function of the true penalties. Note this is a difficult task because smoothing ℓ1 near zero makes the two norms indistinguishable for very small weights (≤ 10−3 ).

Further Experiment Details

The experiments include recovering explicit regularization like elastic net and theorized implicit regularization from early stopping, dropout, and discrete gradient steps. The recovery of the quadratic weight penalty induced by early stopping in OLS is shown to be exact when using an underparametrized diagonal estimator. p. 5

Where the paper says it

Section 3 studies a time-varying quadratic weight penalty induced by gradient descent for OLS. 3 Empirical Studies The following experiments validate we can recover both known explicit regularization (elastic net) and theorized implicit regularization (from early stopping, dropout, and discrete GD steps).

The estimated ridge penalty from dropout in deep networks increases monotonically with the dropout rate, consistent with theoretical predictions. p. 7
Where the paper says it

Figure 5 shows the estimated ridge penalty λˆ as a function of dropout rate, faceted by depth and width. When all other hyperparameters are held equal, the estimated ℓ2 penalty increases monotonically with dropout rate across all architectures, consistent with the theoretical prediction that dropout acts as adaptive ℓ2 regularization.

The recovery of Barrett and Dherin's implicit gradient regularization is shown by fitting a single-parameter ridge bias to trajectory deviations against the known analytic coefficient λ = ηp/4. The stability analysis shows that retraining on the estimated regularizer recovers the original weights under local convexity assumptions. p. 4
Where the paper says it

Appendix A provides a detailed discussion of theoretical guarantees for recovering the original model weights. Intuitively, if the loss converges to a relatively flat minimum, local convexity guarantees that the solution is unique and we will recover the model.

The generalization bound derived from within-trajectory analysis provides a rigorous measure of error. p. 3
Where the paper says it

∆θ = θt+1 − θt = −η∇θ(L + R) (4) Rearranging this expression implies that regressing −η −1∆θ − ∇θL on ∇θR, where each point in the trajectory yields one datapoint, offers an alternative method for learning a regularizer. We further describe this procedure and provide theoretical guarantees for the above in the form of a martingale generalization bound within Appendix C.

Caveats on the Per-step Estimator

The identity T = (η/2) Hg + O(η 2) is a local Taylor/backward-error prediction under smoothness of L, meaning the recovery panel in Figure 6 should be read as a calibration check for the trajectory estimator against a known analytic coefficient. The recovered weight matching only implies that the parameter ˆθ is stationary for the regularized objective, not necessarily that retraining with Λˆ recovers ˆθ without additional curvature assumptions. p. 2

Where the paper says it

Our key assumption is to view the residual loss as a quantity associated with an additional objective, namely the regularization objective. We understand implicit effects by assuming the learned ˆθ is an exact solution to a regularized empirical risk minimization (RERM) problem: ˆθ(D) = arg min θ∈Θ L(θ, D) | {z } empirical loss + R(θ, D) | {z } regularization term for some regularizer R.

The PAC bounds show that error scales in O(p/peff), illustrating how increasing independent endpoints doubles the effective parameter count. p. 4
Where the paper says it

We elaborate and provide theoretical guarantees for both this parametric and the associated nonparametric case in Appendix B. There we show that estimation error scales in O( p r/peff), where in the linear parametric case r is the number of regularizers and peff is the effective parameter count (this is analogous to effective sample size, since parameters are samples in our regression of ∇θL onto ∇θR).

The empirical evidence suggests that for clean data, bootstrap recovery requires sufficient OLS sampling variance for the system to be identifiable. p. 4
Where the paper says it

In realistic data settings, many datasets (especially those from the same domain, like images) are correlated and therefore will lead to more similar ˆθ – but even sampling variability in ˆθ will improve our coverage. Another natural way to obtain additional estimating equations, when constrained to a single training dataset, is through bootstrap resampling: each bootstrap replicate yields a new fitted parameter vector and thus a new approximate stationarity condition, expanding the gradient equations available for estimating Λ.

The analysis of dropout shows the estimated penalty decays over time, matching theory when tested with a scalar l2 regularization parameter. p. 7
Where the paper says it

Figure 4E shows that our heuristic estimate decays over time, with an extremely tight match to theory. We take this as an encouraging sign that we can use underparameterized models of implicit regularization to test our theory and intuitions.

The final summary indicates that estimating implicit regularization provides a cheap and practical way for practitioners to study training configurations early on in training. p. 8
Where the paper says it

However, as model size and compute budgets scale, training runs become massively expensive. Estimating implicit regularization could provide a relatively cheap and practical way for practitioners to study the effects of various training configurations early on in training, by studying how the dynamics deviate from strict loss minimization.

The work is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

The paper concludes by noting that regularization parameters associated with collinear gradients yield estimates with high variance. p. 8
Where the paper says it

2 we discussed ways to increase the sample size, but ultimately they may be limited by the data at hand. We also face familiar identifiability problems – regularization parameters associated with collinear gradients yield estimates with high variance.

The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

p. 8
Where the paper says it

However, many deep learning techniques can be entirely ablated, making them amenable to study. This paradigm opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems.

The authors consider their work to be essential to their capacity to generalize beyond their training data. p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep learning systems. The work opens exciting avenues for future research into the implicit regularization effects of the many complex innovations and engineering tricks used in state of the art deep learning systems. The authors consider their work to be essential to their capacity to generalize beyond their training data. The paper is presented as a starting point for empirically characterizing the implicit regularization in deep… p. 8
Where the paper says it

This is another axis along which to evaluate models, which would complement existing heuristics like validation loss, weight and gradient norms, or perplexity. We consider our work here to be just a starting point for empirically characterizing the implicit regularization in deep learning systems, which we believe to be essential to their capacity to generalize beyond their training data.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Gradient matching methods can be used to empirically estimate implicit regularization by solving for unknown regularization parameters Lambda in equations derived from matching loss and regularizer gradients. This allows practitioners to characterize successful estimation by distinguishing between recovering ground truth regularization and 2) recovering a regularizer that mimics the effect of the implicit regularization.

  2. The framework can recover explicit penalties like l1 and l2, as well as known implicit effects such as the quadratic weight penalty induced by early stopping in gradient descent. This enables testing theories of implicit regularization by demonstrating that our method works on networks with known regularization, recovering popular explicit penalties like l1 and l2.

  3. The system can be extended to characterize dropout effects in deep nonlinear networks, showing implicit l2 effects in this popular method, allowing practitioners to use gradient matching to understand regularization in complex networks that are too complicated to derive analytically.

  4. The method allows for the estimation of implicit regularization along training trajectories, enabling the characterization of time-varying Lambda (t), which is more general than studying a model at convergence. This provides a way for researchers to track how the implicit regularization evolves over the course of training.

  5. The technique can be used to validate theoretical results by recovering the original model weights when retraining with an estimated explicit regularizer, establishing that the weights learned with implicit regularization are equivalent to the weights learned when early stopping is replaced with the estimated regularizer.

Abstract

Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization -- connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like 1 and 2. It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing implicit 2 effects of dropout in deep networks. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically.

Sources

Related papers