DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generation

arXiv:2606.26058 · cs.CV · Submitted 2026-06-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generation".

Jane: Here's the gist of the paper. Open domain subject-driven text-to-video (S2V) generation has drawn significant interest in academia and industry because existing methods primarily focus on maximizing subject fidelity in in-domain scenarios, which limits their editability and adaptability in cross-domain scenarios.

Tom: Proposed Framework Overview The ideal S2V method should flexibly shuttle between different domains, achieving strong performance in both in-domain and cross-domain scenarios.

Jane: To achieve this, the authors propose DomainShuttle, which could achieve high fidelity and generative flexibility for open domain video personalization.

Tom: Specifically, DomainShuttle introduces three key components: (1) Domain-MoT, which decouples videos and reference features to facilitate domain-aware reference modeling, (2) Video-Reference DualRoPE, which places reference image tokens and video tokens in separate RoPE spaces to enable precise subject-level spatial modeling, and (3) Cross-Pair Consistent Loss, which aims to extract intrinsic subject features unaffected by irrelevant features.

Paper summary: Tom: So we're summarizing this paper again. The main thesis is that an ideal S2V method needs to flexibly shuttle between different domains, getting strong performance in both in-domain and cross-domain scenarios >

Jane: They address the problem where most methods only focus on keeping the subject perfect in a specific domain, which limits how adaptable they are when you need to make it look like something completely new with a text prompt >

Lu: DomainShuttle tackles this by decoupling video and reference features through DomainMoT and adding domain aware modeling to the reference image branch >

Meng: Decoupling sounds like it's about separating the representations for video from the representation for your reference image, but what's the actual mechanism behind that separation?

Lu: DomainMoT introduces an independent attention mapping pathway for reference image features within the in context self attention to effectively disentangle them from video features, and they also introduce a domain aware AdaLN to distinguish different domains in the reference feature space >

Jane: So they aren't just mixing the information together; they're creating separate processing streams for those two different types of input >

Tom: And then on top of that, they have the Video Reference Dual RoPE mechanism which assigns reference image tokens to a separate RoPE space from video tokens >

Jane: That separation is key because it allows for precise subject level control, meaning you can explicitly separate different subjects in the latent space >

Meng: So if we want to change one person in a video while keeping the background consistent, this mechanism helps us isolate that person's representation?

Lu: Exactly. They achieve this by having different temporal indices for the reference and video spaces, which keeps those RoPE spaces independent >

Tom: And they finish it off with the Cross Pair Consistent Loss to pull out those intrinsic subject features that are unaffected by irrelevant features like lighting or style >

Jane: So the whole point is that this combination aims to extract what makes a subject *it*, regardless of how it's being rendered in different contexts across domains >

Meng: It’s ambitious—designing these independent paths and losses—but if the results hold up, it means we can move beyond just mimicking reference images to truly generating new things based on text prompts >

Lu: The ablation study showed that removing Ditto 1M didn't hurt performance much, and removing the Cross Pair Consistent Loss just made the model copy from the reference image instead of learning better representations > <ref:2606.26058#pg1>

Tom: So, basically, they built a framework where you have separate processing streams for video and reference input, precise spatial control over subjects via dual RoPE spaces, and a loss function to clean up those features >

Jane: It sounds like they've engineered a system that prioritizes subject consistency while maximizing the freedom to change external attributes based on text instructions >

Conclusion: Tom: So, wrapping up this discussion on "DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generation". The authors propose this framework as a way to achieve high fidelity and generative flexibility in open domain video personalization >

Jane: They've done it by decoupling the video and reference inputs with DomainMoT, adding dual RoPE for precise subject level control, and using Cross Pair Consistent Loss to pull out those intrinsic features >

Lu: The results are solid; they show that this approach significantly outperforms existing methods in terms of subject consistency and text controllability across various complex scenarios >

Meng: For a production team, it means we have a system that can generate high quality videos with competitive text controllability and subject consistency when you need to use it in open domain applications >

Tom: The main implication is that we have a method for S2V that handles the challenges of shifting between domains much more effectively than previous approaches >

Jane: It moves the focus from just maximizing fidelity in one domain to building systems that are inherently flexible enough to handle real world text-driven changes >

Nan Chen, Yiyang Cai, Rongchang Xie, Junwen Pan, Cheng Chen, Weinan Jia, Zhuowei Chen, Wen Zhou, Zhenbang Sun, Wenhan Luo

Hong Kong University of Science and Technology

cs.CV

Submitted: 2026-06-24

Updated: 2026-11-04

Comments: 19 pages, 9 figures

Code: https://github.com/HKUST-C4G/DomainShuttle

Project page: https://cn-makers.github.io/DomainShuttle

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Open domain subject-driven text-to-video (S2V) generation has drawn significant interest in academia and industry because existing methods primarily focus on maximizing subject fidelity in in-domain…

Key concepts

DomainShuttle
A proposed S2V framework that flexibly handles video generation across different domains by decoupling video and reference images. It uses Domain-MoT to separate domain information and DualRoPE for fine subject control, enabling high fidelity in open-domain personalization.
Domain-MoT
This module decouples the processing of videos and reference images using an independent attention mapping pathway for the reference features. It also incorporates a domain-aware AdaLN to explicitly distinguish between different domains within the feature space, which is crucial for accurate cross-domain modeling.
Video-Reference DualRoPE
This mechanism places tokens from the reference image and video in separate RoPE spaces. This separation allows for precise subject-level spatial control, ensuring that different subjects are distinctly separated while representations of the same subject are pulled closer together in the latent space.
Cross-Pair Consistent Loss (CCL)
This loss function aligns pairs of reference images corresponding to the same video. It is specifically designed to capture intrinsic subject features—like appearance—that remain consistent even when irrelevant factors such as lighting or style change between the two reference images.

Terminology

Summary

Open domain subject-driven text-to-video (S2V) generation has drawn significant interest in academia and industry because existing methods primarily focus on maximizing subject fidelity in in-domain scenarios, which limits their editability and adaptability in cross-domain scenarios. p. 0

Proposed Framework Overview

The ideal S2V method should flexibly shuttle between different domains, achieving strong performance in both in-domain and cross-domain scenarios. To achieve this, the authors propose DomainShuttle, which could achieve high fidelity and generative flexibility for open domain video personalization. p. 0 Specifically, DomainShuttle introduces three key components: (1) Domain-MoT, which decouples videos and reference features to facilitate domain-aware reference modeling, (2) Video-Reference DualRoPE, which places reference image tokens and video tokens in separate RoPE spaces to enable precise subject-level spatial modeling, and (3) Cross-Pair Consistent Loss, which aims to extract intrinsic subject features unaffected by irrelevant features. p. 0

Decoupling and Domain Modeling

Domain-MoT introduces an independent attention mapping pathway for reference image features within the in-context self-attention to effectively disentangle them from video features, and further introduces a domain-aware AdaLN to distinguish different domains in the reference feature space. The authors describe DomainMoT as decoupling video latents and reference image features into two independent processing paths, and explicitly injects domain attributes into the reference image branch via Domain-aware AdaLN to distinguish different domains in the feature space, achieving more precise cross-domain S2V. p. 4 The modulation coefficients are given by the scale gamma ∈ Rd, shift beta ∈ Rd, and residual gate g ∈ Rd conditioned on t and a. p. 4

Subject-Level Spatial Control

Video-Reference DualRoPE assigns reference image tokens to a RoPE space independent of video tokens, enabling precise subject-level control: different subjects are explicitly separated, while representations of the same subject are pulled closer in the latent space. The model achieves joint optimization of subject consistency and text controllability in open-domain scenarios through this design. p. 5 For reference images, the temporal index is set to 0 while the temporal index for video starts from 1, explicitly separating the reference RoPE space from the video RoPE space. p. 5

Feature Extraction and Consistency

Cross-Pair Consistent Loss (CCL) aligns two sets of reference images corresponding to the same video to better capture the intrinsic subject features, which are unaffected by irrelevant features such as lighting, composition, or style. This loss is defined as LC = ∥Gtheta (zt, t, ct, cr) − G∗theta (zt, t, ct, c∗ r) ∥22. p. 3 The authors state that combining VR-DualRoPE and CCL achieves the best performance. p. 8

Experimental Results and Contributions

Extensive experiments demonstrate that DomainShuttle achieves significant performance improvements over existing methods, exhibiting high subject fidelity and generative flexibility across diverse open domain application scenarios. Specifically, the experimental results show that DomainShuttle achieves competitive performance, with a significant 18. p. 27% improvement in Cross-Domain Score over the SOTA methods. p. 2 p. 2 The main contributions are summarized as follows: (1) We propose a novel S2V framework termed DomainShuttle, which decouples video and reference images into independent branches via the DomainMoT module. p. 2 (2) Building on DomainMoT, we further design the Video-Reference DualRoPE mechanism to accurately distinguish subject-level spatial distance, and Cross-Pair Consistent Loss to precisely extract the intrinsic subject features. p. 2 (3) Extensive experiments show that DomainShuttle comprehensively outperforms existing methods in terms of subject consistency and text controllability across various complex scenarios, especially achieving an 18. p. 27% improvement in Cross-Domain Score over the SOTA methods.

Human Evaluation Validation

In a human preference evaluation, our method achieves superior performance in all metrics, showing significant advantages in open-domain subject consistency. The authors invite 40 volunteers to conduct a human preference evaluation comparing DomainShuttle with well-performed methods, and the results validate the effectiveness of DomainShuttle. p. 8 The qualitative comparisons demonstrate that DomainShuttle can achieve flexible text controllability and precisely preserve the intrinsic features of the reference subject. p. 7

Ablation Study Insights

The ablation study on whether to use Ditto-1M shows that without Ditto-1M, our method remains effective and still achieves cross-domain SOTA performance, improving the CD-Score by 13.5% over baselines. p. 6 Furthermore, removing CCL causes the model to directly copy the object from the reference image, indicating that CCL facilitates learning more precise representations of the reference subject. p. 9 Combining VR-DualRoPE and CCL, DomainShuttle (ID-4) achieves the best performance. p. 8 The authors also note that using both yields the best results. p. 9

Data Construction Details

The training data comprises two components: image and video personalization datasets. The authors build a 200K image dataset containing multiple subjects based on open-source datasets to endow the model with basic personalization capability. p. 5 For video personalization, they use three datasets: Phantom-Data [36], OpenS2V [37], and Ditto-1M [38]. p. 5 In total, we obtain a 750K high-quality, open-domain video personalization dataset covering various scenarios (e. p. 6g., humans, objects, fantasy subjects, and backgrounds) and supporting cross-pair configurations of both “multiple reference set → single video” and “single reference set → multiple videos”. p. 6

Evaluation Metrics Summary

We evaluate the effectiveness of DomainShuttle from three aspects: (1) Video Quality, using the normalized Aesthetic Score (AES) and Motion Smoothness (MS) [37] to evaluate the quality of videos generated by different methods. (2) Text Controllability, where GMEScore[44] is used to evaluate text controllability. p. 6 (3) Subject Consistency, conducted under both In-Domain and Cross-Domain settings using the standard DINO-I [45] and CLIP-I [46] metrics. p. 6 For the cross-domain scenario, four metrics are designed to evaluate different methods: NANO-CLIP, Qwen-CLIP, Cross-Domain (CD) Score, and Qwen-Score. p. 7

Conclusion on Performance

Compared with baselines, DomainShuttle achieves the best performance in motion smoothness and text controllability. Our method significantly outperforms baselines in most subject consistency metrics, particularly with a significant 18. p. 87% improvement in cross-domain (CD) score. p. 2 These quantitative experiments demonstrate that DomainShuttle can generate high-quality videos with competitive text controllability and subject consistency in open-domain scenarios. p. 8 The authors conclude that DomainShuttle achieves high fidelity and generative flexibility for open domain video personalization. p. 0

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decoupling video latents and reference image features via Domain-MoT allows for domain-aware reference modeling by injecting domain attributes into the reference image branch through Domain-aware AdaLN, which enables high subject fidelity and flexible controllability in open-domain scenarios.

  2. Implementing Video-Reference DualRoPE assigns reference image tokens to a RoPE space independent of video tokens, enabling precise subject-level spatial distance relationships and explicitly separating different subjects while keeping representations of the same subject closer together.

  3. Utilizing Cross-Pair Consistent Loss aligns two sets of reference images corresponding to the same video to precisely capture subject-specific features, which suppresses overfitting to redundant features in a single frame by forcing the model to extract intrinsic subject features unaffected by irrelevant features such as lighting, composition, or style.

  4. The improved system can generate videos that maintain high subject consistency across both in-domain and cross-domain scenarios, achieving a significant 18.7% improvement in Cross-Domain Score over the SOTA methods.

  5. The system can perform flexible transformations of subjects across domains, such as real-world subjects to fantasy domain or fantasy subjects to real-world domain, while preserving intrinsic subject features like hairstyle, skin color, and clothing regardless of the domain attributes.

Sources

Related papers