CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations
cs.SD, cs.AI, cs.LG, eess.AS, eess.SP
Submitted: 2026-08-31
Updated: 2026-09-06
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Joint-Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space.
Terminology
Abstract
Joint-Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space. However, it typically relies on teacher--student architecture with an EMA to stabilise training, and can tend to yield uninformative representations. Contrastive learning is stable to train and produces strong global representations, but remains limited on local tasks by the global nature of its objective. In this work, we combine both into CoJEPA: a single shared backbone jointly trained with a JEPA objective on masked sequence tokens and a contrastive objective on the class token. The contrastive gradient provides stability, removing the need for an EMA teacher entirely, while JEPA enriches the sequence tokens via local predictions that contrastive learning alone cannot provide. Crucially, no extra parameters are added to the backbone: the same model is guided towards richer representations purely through the design of its training signal. CoJEPA takes the best of both worlds, outperforming or matching both individual methods across global and local MIR tasks, with a particularly strong advantage on tonal and harmonic understanding, and without any task-specific architectural changes. CoJEPA shows that combining objectives with complementary inductive biases can substitute for scale, encouraging future work to invest in smarter training objectives over ever-larger models.
Sources
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- Contrastive Learning of Musical Representations
- Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation
- MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning
- Understanding self-supervised Learning Dynamics without Contrastive Pairs
- Toward Fully Self-Supervised Multi-Pitch Estimation
- W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
- Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- The Llama 3 Herd of Models
- Parameter-Efficient Transfer Learning for Music Foundation Models
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
- Harmonic Representations in Symbolic Music Transformers are Localized and Controllable
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving