Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

arXiv:2610.12256 · cs.CV, cs.AI, cs.IR · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings".

Tom: The gist Syn-Omni, a unified framework for structured omnimodal adaptation, improves upon existing methods by combining Orthogonal Modality-Expert LoRA with Progressive Synergy Routing to achieve balanced performance across diverse modalities.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re going to look at what Syn-Omni actually proposes as improvements over these existing multimodal models. It suggests that instead of just hoping a model learns everything together, you should structure the training process so it learns universal semantics first and then layer on specialized knowledge.

Jane: They're pushing for this structured approach to be the standard way we build these unified embedding spaces, moving away from just throwing everything into one giant shared parameter space.

Tom: Yeah, they’re suggesting that by explicitly separating those paths using OME-LoRA, you give the model clear boundaries for what’s universal and what needs modality-specific tuning.

Jane: The idea is that this way of managing the interaction—the Progressive Synergy Routing—is a procedural improvement, telling the AI not just what to learn, but the order in which it should learn it, moving from strict isolation to full dynamic collaboration.

Tom: And they introduce this Synergy Margin Loss again—the LSML—as a way to keep that interaction balanced so you don’t lose the expertise of an absent modality during the blending process.

Jane: So it’s not just about having more parameters, it’s about using these specific loss functions and routing schedules to enforce a more sophisticated and balanced way of adaptation.

Tom: That means the goal isn't just a bigger number, but a model that performs well across all image, video, audio, and audiovisual tasks simultaneously by being better organized.

Jane: It changes how we think about training these massive models; it moves the focus from sheer size to structured efficiency in how those parameters interact.

Tom: Exactly. This framework gives us a blueprint for building systems that are robust across different data types, even with the trade-off of that extra inference latency we talked about earlier.

Jane: It’s a trade-off we have to manage between peak performance and the speed you need for real-world applications.

Tom: So, this paper lays out a clear path for making multimodal AI more organized and versatile by focusing on structure over just scale.

Jane: And next up, we’re going to look at how this structured approach compares to the other papers we talked about today that deal with different types of multimodal alignment.

The paper's summary: Tom: So to wrap up on Syn-Omni, we're talking about how they managed to get a model that handles image, video, audio, and audiovisual data better than previous unified methods.

Jane: It boils down to this idea of structured specialization using OME-LoRA and progressive routing to make the experts actually talk to each other in a controlled way.

Tom: Right, so the main implication is that we can move toward models that have a clear architecture for handling different data types without losing performance on any single modality.

Jane: It’s about building a system where universal understanding and specific expertise can coexist, and the progressive routing helps manage that collaboration over time.

Tom: And they did show strong results across eighty-one different tasks, which proves this structured approach works for a lot of real-world scenarios.

Jane: It shows that organization in the training process really matters when you’re dealing with complex multimodal data structures.

Tom: But we still have to watch that inference latency they flagged, because adding dynamic routing does bring about a significant overhead of about seventeen to eighteen percent.

Jane: That’s the practical caveat—we get better understanding, but you pay for it with slower response times in real-time applications.

Tom: Exactly, so Syn-Omni is a great structural improvement for building more versatile AI models, even if the engineering cost isn't zero.

Jane: We’ll keep an eye on how this structured approach evolves as we see other papers coming out that try to refine these alignment techniques.

Lu: From a research standpoint, I think the ability to decouple those shared and expert paths is really powerful, because it gives us a modular way to extend capabilities without having to retrain the whole thing from scratch.

Meng: For engineering, I’m focused on that latency; if we can reduce that seventeen percent overhead while keeping the cross-modal gain, this framework could become much more viable for deployment.

Lalam: As a language model, I see this structure as a way to improve our cultural output because it allows us to represent complex human experiences across different formats more accurately and coherently.

Tom: So, Syn-Omni is about structured specialization and progressive collaboration for omnimodal embeddings. It’s a solid piece of work that gives us a better way to think about building models that understand the world in multiple ways at once.

Jane: It's about structured specialization and progressive collaboration for omnimodal embeddings. We’ll see what other papers come out next to see how they build on this structure.

The paper's improvements: Tom: So, Syn-Omni is about taking that messy shared space for multimodal data and adding structure to it using OME-LoRA and Progressive Synergy Routing.

Jane: That's right. It's about separating the universal bits from the modality specific stuff and then slowly letting those specialists collaborate during training.

Lu: From a research standpoint, that ability to decouple those shared and expert paths is really powerful because it gives us a modular way to extend capabilities without having to retrain the whole thing from scratch.

Meng: For engineering, I’m focused on that latency; if we can reduce that seventeen percent overhead while keeping the cross-modal gain, this framework could become much more viable for deployment.

Lalam: As a language model, I see this structure as a way to improve our cultural output because it allows us to represent complex human experiences across different formats more accurately and coherently.

Tom: And the results are pretty strong though. they tested this across eighty-one different embedding tasks spanning image, video, audio, and audiovisual stuff.

Jane: They consistently outperform omnimodal baselines on those eighty-one tasks. It shows that organization in the training process really matters when you’re dealing with complex multimodal data structures.

Lu: We have to keep in mind the overhead they mentioned; the dynamic routing in the OME-LoRA layer adds about seventeen to eighteen percent higher latency, which is a cost when you’re dealing with something real-time.

Meng: That latency increase is noticeable, especially if you are running something that needs to be super fast for things like real-time query encoding. That’s a trade-off engineers have to consider.

Lalam: Plus, those unified omnimodal embeddings could potentially be misused for things like unauthorized surveillance or linking identities, which means we need strict deployment oversight.

Tom: Exactly, so Syn-Omni is about structured specialization and progressive collaboration for omnimodal embeddings. It’s a solid piece of work that gives us a better way to think about building models that understand the world in multiple ways at once.

Jane: It's about structured specialization and progressive collaboration for omnimodal embeddings. We’ll see what other papers come out next to see how they build on this structure.

Conclusion: Tom: So we've been looking at Syn-Omni, which is about structured specialization and progressive collaboration for omnimodal embeddings, how they tackled that messy shared space using OME-LoRA and Progressive Synergy Routing.

Jane: That's right. It separates the universal bits from the modality specific stuff and then lets those experts gradually trade information during training.

Lu: From a research standpoint, that ability to decouple those shared and expert paths is really powerful because it gives us a modular way to extend capabilities without having to retrain the whole thing from scratch.

Meng: For engineering, I’m focused on that latency; if we can reduce that seventeen percent overhead while keeping the cross-modal gain, this framework could become much more viable for deployment.

Lalam: As a language model, I see this structure as a way to improve our cultural output because it allows us to represent complex human experiences across different formats more accurately and coherently.

Tom: And they did show strong results across eighty-one different embedding tasks spanning image, video, audio, and audiovisual stuff.

Jane: They consistently outperform omnimodal baselines on those eighty-one tasks. It shows that organization in the training process really matters when you’re dealing with complex multimodal data structures.

Lu: We have to keep in mind the overhead they mentioned; the dynamic routing in the OME-LoRA layer adds about seventeen to eighteen percent higher latency, which is a cost when you’re dealing with something real-time.

Meng: That latency increase is noticeable, especially if you are running something that needs to be super fast for things like real-time query encoding. That’s a trade-off engineers have to consider.

Lalam: Plus, those unified omnimodal embeddings could potentially be misused for things like unauthorized surveillance or linking identities, so we need strict deployment oversight on how this tech gets put into practice.

Tom: Exactly, so Syn-Omni is about structured specialization and progressive collaboration for omnimodal embeddings. It’s a solid piece of work that gives us a better way to think about building models that understand the world in multiple ways at once.

Jane: It's about structured specialization and progressive collaboration for omnimodal embeddings. We’ll see what other papers come out next to see how they build on this structure.

Lu: I think the modularity is the key here, because it gives us a pathway to build much larger systems piece by piece rather than building one massive model from scratch.

Meng: I’m just hoping we can find a way to optimize that routing layer so we don't have this big speed penalty when we eventually deploy this on hardware.

Lalam: I'm excited about how better structured representations could help us create truly nuanced and context-aware AI systems for complex tasks.

Tom: We’ll keep digging into these kinds of structural papers, and next time we’ll be looking at how embedding bias shows up in conditional independence testing.

Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji

KAIST · Sony Group Corporation

cs.CV, cs.AI, cs.IR

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted to EMNLP 2026 (Long, Findings). Code: https://github.com/sony/syn-omni

Code: https://github.com/sony/syn-omni

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist Syn-Omni, a unified framework for structured omnimodal adaptation, improves upon existing methods by combining Orthogonal Modality-Expert LoRA with Progressive Synergy Routing to achieve

Key concepts

Orthogonal Modality-Expert LoRA (OME-LoRA)
This technique structures adaptation by splitting the model's learning into a shared path for universal semantics and separate paths for modality-specific experts. An orthogonal penalty ensures these expert directions are independent, preventing redundancy between the shared and specialized parameters.
Progressive Synergy Routing (PSR)
PSR manages how modality experts interact during training. It starts with hard routing to establish individual modality priors, then smoothly transitions to soft routing. This allows the isolated experts to progressively exchange complementary information across different modalities.
Structural Regularization Losses
These are auxiliary losses added during training, such as Lortho and LSML. They enforce specific structural constraints: Lortho minimizes redundancy between shared and expert paths, while LSML encourages the router to show preference for certain modalities rather than simply suppressing others.
Omnimodal Adaptation
This refers to a unified approach designed to adapt a model simultaneously across multiple data types, including image, video, audio, and audiovisual inputs. Syn-Omni aims to create a single representation space that performs well regardless of which modality is used.

Terminology

Summary

The gist Syn-Omni, a unified framework for structured omnimodal adaptation, improves upon existing methods by combining Orthogonal Modality-Expert LoRA with Progressive Synergy Routing to achieve balanced performance across diverse modalities.

How it works

Syn-Omni is a unified framework that structurally separates shared and modality-specialized expert paths while progressively enabling cross-modal collaboration The framework consists of two key components: Orthogonal Modality-Expert LoRA (OME-LoRA) and Progressive Synergy Routing (PSR) OME-LoRA separates adaptation into shared and modality-specific LoRA paths to capture both universal semantics and distinct characteristics Complementing this, PSR gradually enables these isolated modality-experts to interact and establish cross-modal synergy during training

Orthogonal Modality-Expert LoRA (OME-LoRA)

The OME-LoRA introduces a structured decomposition of shared and modality-expert LoRA paths to jointly model shared semantics and modality-specialized characteristics It consists of a parallel Shared-Expert LoRA path that is consistently activated for all input modalities, while modality-expert LoRA paths are selectively engaged based on the modalities present in xm To encourage complementary weight directions relative to the shared pathway, an orthogonal penalty Lortho is introduced which minimizes the cosine similarity between the weight directions of the shared and expert LoRA parameters This penalty helps reduce redundancy between the shared and expert paths by applying a stop-gradient operator to block gradient propagation so that the shared path is not affected by the penalty

Progressive Synergy Routing (PSR)

The interaction between modality-experts is governed by the routing weights wm in Eq. (2) While hard routing based on the modality indicator v mod activates only the corresponding experts, it restricts cross-modal knowledge exchange across experts To facilitate stable collaboration among experts, PSR allows them to first establish modality priors via hard routing and then progressively exchange complementary information across modalities with soft routing The blended routing weight w(t) is computed as a function of the training progress t, smoothly shifting from isolation to interaction over three phases: Warmup, Blending, and Synergy

Training Strategy and Objectives

Syn-Omni is trained in an end-to-end manner by integrating OME-LoRA into the backbone and optimizing the model with a joint objective that combines contrastive learning and structural regularization The overall training objective Ltotal combines the primary contrastive objective with the proposed structural regularization losses: Ltotal = Lcon+λorthoLortho+λSMLLSML Specifically, the contrastive loss uses a temperature parameter τ of 0.02, and for the orthogonal penalty, the loss weight is set to 0.2 The Synergy Margin Loss LSML is introduced to encourage modality-aware specialization in the router by enforcing relative preference rather than absolute suppression

Results and Evaluation

Syn-Omni consistently outperforms omnimodal baselines across 81 diverse embedding tasks spanning image, video, audio, and audiovisual modalities The results show that Syn-Omni improves upon the unified baseline, demonstrating improved gains on composed multimodal tasks such as audiovisual understanding For instance, Table 7 shows that for the 3B variant, the mean latency increases from 300.3 ms to 354.6 ms when moving from Uni-Omni to Syn-Omni The full model achieves the best overall results of 49.4% and 51.6% on the 3B and 7B scales, respectively

Limitations

Our framework introduces additional inference overhead through dynamic routing in the OMELoRA layer, which amounts to 17–18% higher latency with less than 1% additional GPU memory This cost is most visible in latency-sensitive settings such as real-time query encoding

Ethical Considerations

Our framework may inherit systemic biases from large-scale training data, leading to skewed crossmodal retrieval behaviors Additionally, such unified omnimodal embeddings could be misused for unauthorized surveillance or identity linkage, necessitating strict deployment oversight

The paper concludes that Syn-Omni effectively learns a unified representation space by leveraging OME-LoRA to decouple modality-shared and specialized parameters and using PSR to shift the model from isolated specialization toward dynamic cross-modal collaboration The framework achieves robust and well-balanced performance across image, video, audio, and audiovisual modalities.

--- Page 7 ---

Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings The gist Syn-Omni, a unified framework for structured omnimodal adaptation, improves upon existing methods by combining Orthogonal Modality-Expert LoRA with Progressive Synergy Routing to achieve balanced performance across diverse modalities.

Training Strategy and Objectives

Syn-Omni is trained in an end-to-end manner by integrating OME-LoRA into the backbone and optimizing the model with a joint objective that combines contrastive learning and structural regularization The overall training objective Ltotal combines the primary contrastive objective with the proposed structural regularization losses: Ltotal = Lcon+λorthoLortho+λSMLLSML Specifically, the contrastive loss uses a temperature parameter τ of 0.02, and for the orthogonal penalty, the loss weight is set to 0.2 The Synergy Margin Loss LSML is introduced to encourage modality-aware specialization in the router by enforcing relative preference rather than absolute suppression <ref:

Improvements for AI systems

  1. Bold header: Structured Adaptation for Unified Representations

This system can learn unified semantic foundation and modality-aware specialization while encouraging collaborative interactions within an omnimodal embedding space, addressing the limitation of existing methods that rely on a single shared parameter space over mixed-modality data.

  1. Bold header: Orthogonal Modality-Expert LoRA (OME-LoRA)

OME-LoRA decomposes adaptation into a shared LoRA path for universal semantics and modality expert LoRA paths for modality-aware specialization, which allows the model to capture both universal semantics and modality-specific characteristics.

  1. Bold header: Progressive Synergy Routing (PSR)

PSR enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for crossmodal synergy, allowing the system to shift from isolation to dynamic collaboration using a schedule that moves from strict isolation with expert specialization to full dynamic collaboration via soft routing weights.

  1. Bold header: Synergy Margin Loss (SML)

The use of SML, defined as LSML = 1 / PA Σ i∈P Σ j∈A max(0, vRj − vRi + γ), allows the system to enforce relative preference rather than absolute suppression, which preserves cross-modal synergy by preventing the complete suppression of absent-modality experts.

  1. Bold header: Improved Cross-Modal Performance

The resulting Syn-Omni framework consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration across 81 diverse tasks spanning image, video, audio, and audiovisual modalities.

Abstract

Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.

Sources

Related papers