Diffusion Maps Kernel Ridge Regression

This paper provides a theoretical foundation for kernel ridge regression based on the data-driven diffusion maps (DM) kernel. We show that, under suitable assumptions, the appropriately scaled DM kernel converges to the heat kernel on the underlying manifold and that its associated RKHS inherits desirable approximation and generalization properties through its connection to Gaussian, Matérn, and Sobolev spaces. These results justify the use of the DM kernel within the established framework of kernel methods and are supported by numerical experiments demonstrating its advantages over the Gaussian kernel, particularly for learning functions on manifolds with boundary and across a range of frequencies and co-dimensions for large enough data.

Improvements to AI systems:

1. Adaptive kernel selection for manifold learning: Replace fixed Gaussian kernels in kernel ridge regression (and by extension, Gaussian processes and SVMs) with the data-driven diffusion maps (DM) kernel. The DM kernel automatically adapts to the underlying data manifold's geometry, including boundaries and varying curvature, without manual hyperparameter tuning of the kernel width.

2. Enhanced generalization on bounded domains: Use the DM kernel for regression tasks where data lies on manifolds with boundaries (e.g., physical simulations, robotics state spaces, or image manifolds with edges). The theoretical guarantee of convergence to the heat kernel with boundary conditions means the AI system will avoid the boundary bias that plagues standard Gaussian kernels, improving prediction accuracy near edges.

3. Multi-frequency learning capability: The DM kernel's connection to Sobolev and Matérn spaces allows the AI system to learn target functions spanning a wide range of frequencies (from smooth to oscillatory) more uniformly. This is critical for surrogate modeling of PDEs, signal reconstruction, or any task where the target function has mixed smoothness.

4. Data-efficient high-dimensional regression: For high-dimensional input spaces where the data lies on a low-dimensional manifold (high co-dimension), the DM kernel leverages the intrinsic geometry rather than the ambient dimension. This reduces the sample complexity—the AI system can achieve the same accuracy with far fewer training points than with a Gaussian kernel, which wastes capacity on off-manifold directions.

5. Principled kernel scaling: The paper's theoretical result on the appropriate scaling of the DM kernel provides a rule for setting the kernel bandwidth based on the local data density and manifold dimension. An AI system can automatically compute this scaling from the data, eliminating a common source of instability in kernel methods.

What the improved AI system can do:

• Accurately predict physical quantities on irregular domains (e.g., stress on a cracked material, heat flow in a non-convex container) where standard kernels fail near boundaries.

• Learn control policies or dynamics from sparse demonstrations on high-dimensional robot state manifolds (e.g., a robotic arm with joint limits) with fewer samples and better extrapolation near constraint boundaries.

• Perform regression on high-dimensional data (e.g., images, sensor arrays) that intrinsically lies on a low-dimensional manifold, achieving state-of-the-art accuracy with orders of magnitude less training data than Gaussian-kernel-based models.

• Provide uncertainty estimates (via the RKHS norm) that are geometrically meaningful, reflecting the true difficulty of prediction near boundaries or in sparsely sampled regions of the manifold, rather than isotropic distance in ambient space.

• Seamlessly switch between smooth and non-smooth target functions (e.g., in multi-fidelity modeling or meta-learning across tasks with different frequency content) without re-tuning the kernel, because the DM kernel's RKHS spans a rich function space that includes both Sobolev and Matérn-like smoothness classes.