Symbolic Density Estimators for Unnormalized Distributions

Estimating symbolic or analytical forms of probability density functions from observed samples using deep generative models and symbolic regression allows for the automated discovery of interpretable relationships characterizing complex physical phenomena like Hamiltonian functions. The gist Estimating the symbolic or analytical form of probability density functions from observed samples is a fundamental challenge in statistical and computational modelling>

Framework Overview

The proposed framework integrates deep generative modeling with Symbolic Regression (SR) to estimate density functions as symbolic expressions using domain-specific prior knowledge, such as the range of interactions and a predefined set of primitive functions> The framework consists of training a deep generative model using samples, fitting an SR model using likelihood information from the generative model and density factorization, and incorporating noisy perturbations for low-density regions> The central idea is to convert an unsupervised density estimation problem into a supervised symbolic regression problem by leveraging quantities obtained from learned flow-based and score-based models, such as log-likelihoods and score fields>

Generative Models

The paper examines flow-based and score-based models for density estimation, which are used to generate samples from the target distribution> Flow models apply a sequence of simple and learnable bijective transformations on samples from a known standard distribution to obtain the density of the transformed variable using Eq. (4)> Score matching involves training a deep model sθ(x) to estimate the target score s(x) by minimizing an expected square distance, J(θ), which can be simplified under certain regularity conditions> Denoising score matching addresses high-dimensional issues by perturbing x with a pre-defined noise distribution qσ(x˜x), and training is carried out across multiple noise levels to generate samples from low-density regions of p(x)>

Symbolic Regression (SR)

SR is a supervised learning approach that aims to estimate an explicit mathematical expression describing the relationship between input and output variables, seeking an optimal expression f∗ = arg min f∈F L(f) where L is a loss function like the mean squared error (MSE)> Two approaches are used: genetic programming based algorithm implemented using PySR library, which uses evolutionary algorithms to optimize a tree data structure representing a mathematical expression in terms of mean squared error, and deep neural network based equation learner (EQL) approach, which minimizes the MSE between y and yˆ> The estimated expression is related to the energy function as f(x) = Hˆ (x) + log Zˆ (5), where Hˆ(x) is obtained by dropping terms independent of x>

Key Improvements and Strategies

The framework incorporates three key components to improve symbolic estimation in practice:

1. Local factorization of the Hamiltonian, exploiting spatial locality in many-body systems to reduce the effective dimensionality of the symbolic search space and enable scalable estimation in high-dimensional settings>

2. Incorporation of task-specific inductive biases through the choice of SR primitives and structural constraints, which encode prior knowledge about the underlying physical system>

3. A multi-scale perturbation strategy that augments samples from high-density regions produced by generative models with controlled noise, thereby improving coverage of low-density regions that are critical for stable symbolic estimation>

Experimental Validation

Experiments were conducted on several benchmarks, including multivariate Gaussian and many-well distributions, as well as problems in computational physics like the XY model and scalar ϕ4 theory> The framework was applied to estimate energy functions H(x) or effective Hamiltonians arising from renormalization in non-perturbative settings where analytical derivations are challenging> Results show effectiveness for estimating the Hamiltonian of the XY model and density functions for several popular toy distributions, with MSE close to zero and R2 values near one on independent test sets> For long-range interaction systems, the framework successfully estimates the correct interaction structure when patch size is chosen to encompass the full interaction neighborhood, particularly with PySR>

Performance Metrics

The quality of symbolic estimation is assessed using several metrics:

1. Mean Squared Error (MSE) and Coefficient of Determination (R2) compare the estimated Hamiltonian Hˆ (x) with the ground-truth Hamiltonian H(x)> The MSE calculation involves mean normalization to remove additive constants>

2. Relative Coefficient Error (RCE), Precision (P), and Recall (R) quantify symbolic expression quality by measuring coefficient accuracy, fraction of true terms, and fraction of ground-truth terms estimated respectively>

3. F1 Score summarizes precision and recall through their harmonic mean>

Limitations

The scalability of the SR stage is constrained by the rapid growth of the search space with the number of variables, especially in systems with long-range interactions where larger patches are required, which increases complexity and likelihood of spurious terms> The framework relies on domain knowledge for specifying symbolic libraries and patch structures, meaning expression quality depends on these choices> Furthermore, mode-covering behavior in deep generative models can lead to biased density estimates and consequently larger coefficient errors and spurious symbolic terms in the estimated Hamiltonian>

Impact

The potential of this work extends to many areas in computational sciences, including Physics, Chemistry and Climate science by enabling scientists to harness machine learning for analyzing large datasets while simultaneously estimating interpretable symbolic equations for locally interacting systems using domain-specific prior knowledge> The paper demonstrates the ability of the proposed framework to estimate compact symbolic approximations of effective Hamiltonians from discrete samples of locally interacting systems in non-perturbative regimes> The work can be taken up for follow up studies in Condensed matter Physics, Nuclear Physics and Climate sciences>

Conclusion

The proposed framework successfully combines deep generative models and symbolic regression to estimate the energy function or Hamiltonian of a system from observed samples using domain-specific prior knowledge> It demonstrates the ability to estimate compact symbolic approximations of effective Hamiltonians from discrete samples of locally interacting systems in non-perturbative regimes where exact analytical derivations are challenging> The framework is applicable to systems with long-range interactions provided that the patch size is large enough to capture the relevant dependencies>

--- Page 17

Published in Transactions on Machine Learning Research (09/2026)

1. The framework can estimate compact symbolic approximations of effective Hamiltonians from discrete samples of locally interacting systems, which allows AI systems to derive interpretable physical laws from raw data in non-perturbative regimes where analytical derivations are challenging.

2. AI can perform model discovery by converting an unsupervised density estimation problem into a supervised symbolic regression problem by leveraging quantities obtained from learned flow-based and score-based models, such as log-likelihoods and score fields.

3. The system can handle complex physical modeling, such as estimating the energy functions of the XY model and scalar phi4 theory, even in non-perturbative settings where traditional perturbative or analytic approaches fail.

4. For long-range interactions, the framework can estimate the correct symbolic structure with perfect precision and recall and no spurious terms when using PySR, demonstrating its ability to identify interpretable long-range interaction terms from samples alone.

5. The AI system can be robust against model inaccuracies; for instance, in strong misspecification settings, it shows that PySR substantially reduces the error relative to the underlying flow model, suggesting that SR acts as a denoising or compression step by reducing the approximation error present in the learned generative model.

Estimating the symbolic or analytical form of probability density functions (PDFs) from observed samples is a fundamental challenge in statistical and computational modelling. This process is critical for deriving interpretable and generalizable relationships characterizing the underlying phenomenon. Traditionally, this estimation depends strongly on domain expertise and prior field-specific knowledge, with experts selecting appropriate functional forms or parametric families based on empirical evidence and theoretical understanding. The coefficients of these forms are then typically determined through parameter estimation. In this paper, we develop a framework for estimating symbolic expressions of unnormalized distributions from observed samples using domain-specific prior knowledge, such as the range of interactions and a predefined set of primitive functions. We integrate deep generative models with symbolic regression (SR), incorporating inductive biases, such as factorizing large distributions, to keep the problem tractable. The deep generative models we examine include likelihood-based models, viz., flow models, and score-based models. Experiments show the effectiveness of the proposed framework for estimating density functions for multivariate toy distributions as well as lattices from computational physics, namely, XY model and ϕ 4 theory. When applied to the renormalization problem in ϕ 4 theory, the proposed framework estimates compact symbolic approximations of the hamiltonian function at different scales directly from samples, yielding expressions that may be challenging to derive using traditional perturbative or analytic approaches in nonperturbative settings.