RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities
cs.LG
Submitted: 2026-06-03
Updated: 2026-11-02
Comments: Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)
License: http://creativecommons.org/licenses/by/4.0/
The gist: To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without…
Terminology
Abstract
To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information. Learning disentangled representations is a principled way to identify the underlying shared and unique factors that are hidden in observational data. However, while multimodal disentanglement is a compelling paradigm, existing methods are largely confined to the two-modality regime due to its inherent scalability bottleneck. To address this, we propose RePercENT, a self-supervised framework designed to surpass these limitations and unlocks scalable pairwise disentanglement beyond two modalities. Through a multimodal `plug-and-play' architecture, our approach operates directly on pre-extracted embeddings, eliminating the need for extensive joint pre-training while making no assumptions regarding the underlying modalities or foundation model backbones. Moreover, we introduce a joint optimization objective for simultaneously deriving the shared and unique components, and provide formal theoretical guarantees that characterize the optimality of our solution. Spanning modalities from images and videos to clinical reports and omics, and tasks from sentiment analysis to tumor classification, RePercENT successfully recovers disentangled components while maintaining competitive performance and significantly reducing computational complexity.
Sources
- Representation Learning with Contrastive Predictive Coding
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Gemini: A Family of Highly Capable Multimodal Models
- Self-Normalizing Foundation Model for Enhanced Multi-Omics Data Analysis in Oncology
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis