Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features

arXiv:2505.21247 · physics.chem-ph, physics.comp-ph · Submitted 2025-05-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features".

Mira: Here's the gist of the paper. A computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input is proposed; the method is based on rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, yielding physics-motivated trigonometric neural networks The core idea involves rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, which yields numerically efficient neural networks characterized by using trigonometric activation functions and fast Walsh-Hadamard transform The proposed method is referred to as Multilevel SORF (MSORF) because it involves several SORF layers corresponding to different levels of molecular structure, such as conformer ensemble, individual conformer, and atoms This approach is designed to be computationally efficient for machine learning in chemical space using Boltzmann ensembles of conformers as input Kernel Representation Strategies The paper outlines three strategies for representing molecular conformers and defining the kernel function k in terms of those representations 1.

Kai: The first strategy represents each molecule A with a vector X(A) and calculates the kernel function k(A,B) using the (global) Gaussian kernel function kglobal(A,B):= exp − X⃗ (A) − X⃗ (B) 2 / 2Ã 2!

Mira: Examples of such molecular representations considered in this work are Coulomb Matrix (CM)

37, 38: and Spectrum of London and Axilrod-Teller-Muto potential (SLATM)

4: 2.

Kai: The second strategy represents each atom in molecule A with a vector, resulting in the molecular representation being a matrix X(A) Two popular choices for defining the corresponding kernel function are the local Gaussian kernel function klocal(A,B):= N (A) Xa j′=1 N (B) Xa j′′=1 kglobal(X⃗ (A) j′, X⃗ (B) j′′), and a modified version kdlocal(A,B):= N (A) Xa j′=1 N (B) Xa j′′=1 ¶(n (A) j ′ − n (B) j ′′)kglobal(X⃗ (A) j′, X⃗ (B) j′′) 3.

Title and authors: Kai: So, to summarize what they did in "Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features," they took a standard Kernel Ridge Regression protocol, which usually involves calculating a kernel function between two molecules, and they reformulate that whole process using these Structured Orthogonal Random Features.

Mira: They are essentially mapping the problem into a different mathematical space where the regression coefficients solve an equation related to the kernel function as if it were just a dot product between vectors in this new high-dimensional space.

Lev: That's like taking a massive, complicated calculation and finding a shortcut by changing how you look at the input data, which is something we do constantly when trying to simplify quantum simulations.

Kai: The paper shows three different ways they can represent those molecular conformers and define that kernel function, starting with a global Gaussian kernel using representations like the Coulomb Matrix or the Spectrum of London and Axilrod-Teller-Muto potential.

Mira: Then there’s a local strategy where they represent each atom in a molecule as a vector, leading to two options for defining the local kernel function based on those atomic vectors.

Lev: And then they have this third strategy which is specific to FJK representation, using results from cheap Hartree Fock calculations to assign weights and vectors to different contributions. It shows how flexible their method is across different types of molecular descriptors.

The paper's summary: Kai: The paper points out a major improvement in handling ensembles by introducing a mixed extensive layer called the Ensemble-applied SORF, which switches between local kernels and other kernels in a way that mimics how something like kGauss FJK is related to klin FJK.

Mira: That switching mechanism is designed specifically to manage the transition between different structural levels—from the whole ensemble down to individual atoms—which they call several SORF layers.

Lev: From an engineering standpoint, managing that switching dynamically on hardware would be a huge overhead if not done right, so how they make this computationally efficient is key here.

Kai: They also introduced a normalization layer called FNWF for intensive properties to remove unphysical dependence on how the importance weights are normalized, which helps when you're dealing with those kinds of measurements.

Mira: And for properties that are intensive—things that don't depend on the number of molecules—they use feature combinations like FNWFNWFNWF or FN Sz¶F Wz*Sz¶F Wz*, which aims to make the prediction invariant with respect to molecule multiplication.

The paper's improvements: Kai: So, wrapping up on "Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features," the main result is that they have a computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input.

Mira: The core finding is that rewriting the Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features yields numerically efficient neural networks characterized by using trigonometric activation functions and a fast Walsh-Hadamard transform.

Lev: That means the theoretical efficiency translates into a practical speedup, which is what we need when scaling up these models for real tasks like predicting oxidation potentials or hydration energies.

Kai: They tested this on experimental datasets of two quantities related to battery electrolyte design, specifically oxidation potentials in acetonitrile and hydration energies using several popular molecular representations.

Mira: And they observed a systematic decrease of machine learning error with increased training set size in all cases, reaching experimental accuracy after training on hundreds of molecules.

Lev: That systematic decrease is reassuring, but we have to remember their limitation here: they only used computationally cheap forcefield calculations for conformer generation when generating those inputs.

Conclusion: Kai: So, to wrap up the "Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features" paper, they’ve shown a way to make machine learning in chemical space efficient by using structured orthogonal random features to create physics motivated trigonometric neural networks.

Mira: It moves the focus from just fitting data points to building architectures that inherently respect molecular structure through those several SORF layers that cover the ensemble down to the atoms.

Lev: The improvement they suggest, which is generating conformer representations with another neural network and training a combined model on computational data before re-optimizing on experimental data, sounds like a practical path for improving those errors.

Kai: And they also suggested that MSORF could analogously combine representations of molecular ensembles into representations of mixtures of molecules. That’s a pretty big thought for how we model complex chemical systems.

Mira: It suggests that the way we represent an ensemble isn't just about averaging them; it’s about finding the right mathematical combination, which is something I think will be very useful for our theoretical work on condensed matter.

Lev: If this method holds up when you move it to real hardware, it could significantly reduce the training time needed for these complex property predictions.

Kai: That’s what we’ll keep an eye on as we look at how this actually performs when we get these representations running on our cryogenic systems.

Konstantin Karandashev

University of Vienna

physics.chem-ph, physics.comp-ph

Submitted: 2025-05-27

Updated: 2026-10-30

Code: https://github.com/qml2code/qml2

Project page: https://kjelljorner.github.io/morfeus

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: A computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input is proposed…

Key concepts

Multilevel SORF (MSORF)
This method builds deep neural networks with multiple layers, each corresponding to a different level of molecular structure. It considers inputs at the conformer ensemble level, individual conformers, and finally the atoms within those conformers simultaneously to capture hierarchical structural information.
Structured Orthogonal Random Features
These are special random features used in the kernel representation that allow for numerically efficient neural networks. They enable the use of trigonometric activation functions and a fast Walsh-Hadamard transform, leading to better modeling of physical phenomena.
Kernel Ridge Regression Rewriting
The core technique involves rewriting standard Kernel Ridge Regression expressions into terms involving Structured Orthogonal Random Features. This mathematical transformation is key to achieving the computational efficiency and the resulting physics-motivated trigonometric neural network structure.
Boltzmann Ensembles
Instead of using just one optimal molecular structure, this approach uses an ensemble of conformers weighted by Boltzmann probabilities. This allows the machine learning model to account for the statistical distribution of possible shapes a molecule can take.

Terminology

Summary

A computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input is proposed; the method is based on rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, yielding physics-motivated trigonometric neural networks

Methodology Overview

The core idea involves rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, which yields numerically efficient neural networks characterized by using trigonometric activation functions and fast Walsh-Hadamard transform The proposed method is referred to as Multilevel SORF (MSORF) because it involves several SORF layers corresponding to different levels of molecular structure, such as conformer ensemble, individual conformer, and atoms This approach is designed to be computationally efficient for machine learning in chemical space using Boltzmann ensembles of conformers as input

Kernel Representation Strategies

The paper outlines three strategies for representing molecular conformers and defining the kernel function k in terms of those representations 1. The first strategy represents each molecule A with a vector X(A) and calculates the kernel function k(A,B) using the (global) Gaussian kernel function kglobal(A,B):= exp − X⃗ (A) − X⃗ (B) 2 / 2Ã 2! p. 2

Where the paper says it

We will now outline three strategies for representing molecular conformers and defining the kernel function k in terms of those representations that were used in this work. The first strategy is to represent each molecule A with a vector X⃗ (A) and then calculate the kernel function k(A,B) using the (global) Gaussian kernel function kglobal(A,B) := exp − |X⃗ (A) − X⃗ (B)| 2 2Ã 2 !

Examples of such molecular representations considered in this work are Coulomb Matrix (CM) [37, 38] and Spectrum of London and Axilrod-Teller-Muto potential (SLATM) [4] 2. p. 2
Where the paper says it

| is a vector’s l 2 norm and à is a hyperparameter. Examples of such molecular representations considered in this work are Coulomb Matrix (CM) [37, 38] and Spectrum of London and Axilrod-Teller-Muto potential (SLATM) [4].

The second strategy represents each atom in molecule A with a vector, resulting in the molecular representation being a matrix X(A) Two popular choices for defining the corresponding kernel function are the local Gaussian kernel function klocal(A,B):= N (A) Xa j′=1 N (B) Xa j′′=1 kglobal(X⃗ (A) j′, X⃗ (B) j′′), and a modified version kdlocal(A,B):= N (A) Xa j′=1 N (B) Xa j′′=1 ¶(n (A) j ′ − n (B) j ′′)kglobal(X⃗ (A) j′, X⃗ (B) j′′) 3. p. 2
Where the paper says it

The second strategy is to represent each atom in molecule A with a vector, the resulting molecular representation being a matrix X(A). We consider two popular choices for defining the corresponding kernel function, the first one being the local Gaussian kernel function klocal(A,B) := N (A) Xa j ′=1 N (B) Xa j ′′=1 kglobal(X⃗ (A) j ′ , X⃗ (B) j ′′ ), (4) where X⃗ (A) j ′ and X⃗ (B) j ′′ are the atomic representation vectors of atoms with indices j ′ and j ′′ in molecules A and B, while N (A) a and N (B) a are numbers of atoms in molecules A and B.

The third strategy is specific to the FJK representation [44], which takes results of cheap Hartree Fock calculations and assigns a weight wj (È(A) o) and vector representation ⃗vj (È(A) o) to each contribution

Ensemble Kernel Extensions

To extend the local kernel klocal to ensembles, a mixed-extensive layer EWESF is introduced where the mixed-extensive layer E is defined as E⃗(X, ⃗ D, φ, Ãmix):= 1 p Nstack(1 + Ã−1 mix) n X⃗ oNstack s=1 + X⃗ √ 1 + Ãmix F⃗ XX, D, ϕ, Ã! This layer controls switching between kens local (21) and a different kernel related to kens local in a way similar to how kGauss FJK (7) is related to klin FJK (8) For intensive properties, the authors use FNWF, normalization layer added to remove unphysical dependence of the result on how the importance weights are normalized

Performance and Optimization

The method demonstrates a systematic decrease of machine learning error with increased training set size in all cases, with experimental accuracy reached after training on hundreds of molecules and prediction errors being comparable to state-of-the-art machine learning approaches The protocol relies on a novel version of the LogCosh [34] loss function that is documented in Appendix B along with its counterpart for the Huber [35] loss function Hyperparameter optimization is performed by optimizing leave-one-out crossvalidation errors, i.e., prediction errors for a point given a model had been trained on all other points in the dataset The optimization procedure involves using Huber or LogCosh loss functions, both of which can be thought of as “smoothed” versions of MAE with continuous derivatives

Conclusion and Outlook

The MSORF algorithm has been made part of the QML2 code [57] with the implementation being based on Numpy [91], Numba [92], and SciPy [93] The method both demonstrates systematic decrease of prediction error with increased training set size and reaches experimental accuracy with hundreds of molecules as training set It is noted that the difference between representing a molecule with an ensemble and one optimal conformer is of the order of statistical error, and is significantly less pronounced than differences observed in other studies The authors suggest a possible avenue to improve on errors by generating conformer representations with another NN, training the combined model on computational data, and then re-optimizing the MSORF part of the model on experimental data Lastly, MSORF could analogously combine representations of molecular ensembles into representations of mixtures of molecules

Tables and Figures

Table 1 summarizes the combinations of layers used in this work for different representations and quantities Table 3 summarizes the results for machine learning Eox from Ref. [1] while representing molecules with minimum energy conformers labeled analogously to Table 1 Figure 3 compares the MAEs obtained by representing molecules either with a conformer ensemble or the lowest energy conformer (“min. p. 0

Where the paper says it

This has led to creation of a large number of representation functions encoding geometrical [3, 4, 5] or orbital [6, 7, 8] information about conformers, which can then be combined with standard regressors for accurate ML of computational [9] or experimental [10] data. However, it makes more physical sense to think of molecules in terms of not single conformers, but their Botlzmann ensembles, which was the motivation behind a family of algorithms that perform Kernel Ridge Regression (KRR) [11] while using either representations of averaged conformers [12] or representations averaged over molecular configurations [13, 14].

conf.”) for different representations and training set sizes Ntrain Table 4 summarizes the results for machine learning Eox from Ref. p. 0
Where the paper says it

To evaluate the method’s utility for materials discovery, we test it on experimental datasets of two quantities related to battery electrolyte design, namely oxidation potentials in acetonitrile and hydration energies, using several popular molecular representations to demonstrate the method’s flexibility. Despite only using computationally cheap forcefield calculations for conformer generation, we observe systematic decrease of machine learning error with increased training set size in all cases, with experimental accuracy reached after training on hundreds of molecules and prediction errors being comparable to state-of-the-art machine learning approaches.

[1] while representing molecules with minimum energy conformers labeled analogously to Table 1

Code Availability

The MSORF algorithm has been made part of the QML2 code [57] with the implementation being based on Numpy [91], Numba [92], and SciPy [93] The version used in this work together with the ML scripts has been uploaded as the 0.1.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Optimization of Kernel Ridge Regression for Boltzmann ensembles allows AI systems to model complex molecular properties using physics-motivated trigonometric neural networks, which are characterized by using trigonometric activation functions and fast Walsh-Hadamard transform.

  2. The Multilevel SORF (MSORF) method enables the construction of deep neural networks with several layers corresponding to different levels of molecular structure, allowing the model to consider conformer ensemble, individual conformer, conformer’s atoms simultaneously.

  3. The system can perform machine learning on experimental datasets for battery electrolyte design by predicting oxidation potentials in acetonitrile and hydration energies using various molecular representations like Coulomb Matrix (CM), SOAP descriptors, and FJK representations.

  4. The AI system can be made more robust to statistical noise by using cut versions of Boltzmann weights, defined as w cb j:= max(w b j − Ä−, 0), to avoid processing conformers whose contribution is irrelevant.

  5. The model can handle intensive properties by employing feature combinations like FNWFNWFNWF for global representations or FN Sz¶F Wz∗Sz¶F Wz∗S, allowing it to create models whose prediction is invariant w.r.t. “molecule multiplication” for intensive quantities.

  6. The system can be optimized to find hyperparameters using novel loss functions like LSCLC, which is a self-consistent version of LogCosh loss function whose smoothness is controlled by a smoothness parameter µ, ensuring the resulting model's MAE does not differ from the global minimum by more than µ in relative terms.

  7. The AI system can be fine-tuned using transfer learning techniques where the MSORF part is re-optimized on experimental data, bridging the gap between MSORF and transfer learning techniques.

Abstract

A computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input is proposed; the method is based on rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, yielding physics-motivated trigonometric neural networks. To evaluate the method's utility for materials discovery, we test it on experimental datasets of two quantities related to battery electrolyte design, namely oxidation potentials in acetonitrile and hydration energies, using several geometry- and orbital-based molecular representations to demonstrate the method's flexibility. Despite only using computationally cheap force field calculations for conformer generation and generating orbital representations with GFN2-xTB calculations, we observe systematic decrease of prediction errors with increased training set size in all cases, with experimental accuracy reached after training on hundreds of molecules.

Sources

Related papers