Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis
The study explored how unified decoder-only transformers can synchronize different data types—video, text, and speech—which have different sampling rates. Researchers tested various indexing methods like global sequential and co-temporal ordering to see which best aligned the modalities. Findings show that co-temporal ordering works well without needing explicit timestamps, and using text first for ordering helps with generalization across new domains.