Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
cs.LG
Submitted: 2026-08-31
Updated: 2026-08-31
Code: https://github.com/amaralibey/nanoCLIP
License: http://creativecommons.org/licenses/by/4.0/
The gist: The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties.
Terminology
Abstract
The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mode connectivity, demonstrating that independently trained network modes can be connected via a continuous low-loss path. However, existing mode connectivity research is predominantly confined to classifier-based models, leaving it an open question whether similar geometric properties exist in modern complex models. In this paper, we extend the boundaries of mode connectivity to generative and contrastive domains (specifically DDPM and NanoCLIP). Addressing the unique architecture of DDPM and CLIP, we propose an architecture-aware connection building algorithm. Extensive empirical results demonstrate for the first time that we successfully discover mode connectivity between independently trained DDPM and NanoCLIP modes. Our work provides a novel perspective for understanding the geometric properties of the loss landscapes in modern generative and contrastive models.
Sources
- Escaping the Big Data Paradigm with Compact Transformers
- Denoising Diffusion Probabilistic Models
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Vanishing Variance Problem in Fully Decentralized Neural-Network Systems
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks