Connecting Neural Models Latent Geometries with Relative Geodesic Representations
cs.LG
Submitted: 2025-06-02
Updated: 2026-10-31
Code: https://github.com/marc0git/RelativeGeodesics
License: http://creativecommons.org/licenses/by/4.0/
The gist: Neural models learn representations of high-dimensional data on low-dimensional manifolds.
Terminology
Abstract
Neural models learn representations of high-dimensional data on low-dimensional manifolds. Multiple factors, including stochasticities in the training process, model architectures, and additional inductive biases, may induce different representations, even when learning the same task on the same data. However, it has recently been shown that when a latent structure is shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions. Building on this idea, we demonstrate that exploiting the differential-geometric structure of latent spaces of neural models, it is possible to capture precisely the transformations between representational spaces trained on similar data distributions. Specifically, we assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the pullback metric that captures the intrinsic structure of the latent space, while scaling efficiently to large models. We validate experimentally our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models, across diverse architectures, datasets, and pretraining schemes.
Sources
- Representation Learning: A Review and New Perspectives
- How do Variational Autoencoders Learn? Insights from Representational Similarity
- A Geometric Perspective on Variational Autoencoders
- The Geometry of Multilingual Language Model Representations
- Word Translation Without Parallel Data
- Similarity and Matching of Neural Network Representations
- A Rainbow in Deep Network Black Boxes
- What Representational Similarity Measures Imply about Decodable Information
- Only Bayes should learn a manifold (on the estimation of differential geometric structure from data)
- The Platonic Representation Hypothesis
- Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?
- Auto-Encoding Variational Bayes
- Similarity of Neural Network Models: A Survey of Functional and Representational Measures
- Understanding image representations by measuring their equivariance and equivalence
- Convergent Learning: Do different neural networks learn the same representations?
- Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections
- Linearly Mapping from Image to Text Space
- Insights on representational similarity in neural networks with canonical correlation
- Comparing noisy neural population dynamics using optimal transport distances
- When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis