VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "VidEoMT: Your ViT is Secretly Also a Video Segmentation Model".
Jane: Here's the gist of the paper. VidEoMT proposes an encoder-only video segmentation model that unifies segmentation and temporal association within a single ViT encoder, achieving significant speedups while maintaining competitive accuracy.
Tom: Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules, which introduce significant architectural complexity and computational overhead.
Jane: The authors hypothesize that a simpler approach can match the accuracy of more complex models by making use of powerful vision foundation models (VFMs).
Tom: They suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules.
Paper summary: Tom: So looking at VidEoMT: Your ViT is Secretly Also a Video Segmentation Model, the authors are essentially arguing that you can get high accuracy in video segmentation by trusting a large Vision Transformer enough to handle both the segmentation and tracking without needing those extra specialized components.
Jane: It boils down to using a single encoder-only model and introducing clever mechanisms, like query propagation and fusion, to manage the temporal aspect while leaning on the pre-trained knowledge of foundation models.
Lu: The implication is that we might be able to simplify complex video processing pipelines by relying on massive pre-training rather than building intricate task-specific modules from scratch for every new video segmentation challenge.
Meng: For deployment, it means we can potentially ship much faster models because the architecture is simpler, but we still have to manage the computational cost of running a large ViT at this speed.
Lalam: The title itself says it clearly: your ViT is secretly also a video segmentation model, which highlights that this capability isn't something you have to bolt on later with extra code.
Tom: It seems like the big shift here is moving the complexity from complex, hand-crafted modules into the training process of a single powerful transformer.
Jane: It gives us a path forward where scaling up those base models directly translates to better video segmentation performance across multiple metrics at once.
Conclusion: Tom: So we’re wrapping up VidEoMT, and that title really captures what they did here: your Vision Transformer is actually doing video segmentation too.
Jane: Right, so this paper takes a huge vision model, like a ViT, which is usually just for images, and makes it segment video frames while also tracking things across time.
Lu: The core idea they push is that you don't need those heavy extra modules for tracking or context awareness anymore if the base ViT is already trained well enough.
Meng: But what about the speed? If you’re using a giant ViT, it could still be slow when you’re processing live video at scale.
Tom: That’s where they show the numbers; they claim this thing is five to ten times faster than some of the old systems, hitting one hundred sixty frames per second with a big model.
Jane: And they did that by using a clever way to pass information between frames—something called query propagation—instead of having separate tracking code for every single frame.
Lu: That propagation is balanced by this query fusion step that lets the model adapt to new things it sees in the next scene, which keeps the accuracy high despite being streamlined.
Meng: If I’m shipping this to production, I’m looking at how robust those learned queries are when we introduce completely new types of action or objects.
Tom: The authors show that by just dropping out the heavy tracking components and replacing them with this unified ViT structure, they can still keep most of the accuracy from their more complex video segmentation models.
Jane: So what does this mean for someone just watching a video online? It suggests we could see much smoother, faster results in real-time processing on our devices.
Lu: The big picture is that you might be able to get really strong video understanding by focusing your training on making the base transformer powerful enough to handle the whole task at once.
Meng: My concern is that if you scale up those pre-trained models, we need to make sure they don't start breaking down when the data gets messy or unexpected.
Tom: That’s exactly what they test—they show how performance gaps shrink as you use bigger and better base models.
Jane: It really makes you think about the future of video processing; maybe we don't need so many specialized pieces, just one really good foundation model doing the heavy lifting.
Eindhoven University of Technology · RWTH Aachen University
cs.CV
Submitted: 2026-02-19
Updated: 2026-11-12
Comments: CVPR 2026. Code: https://github.com/tue-mps/videomt
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: VidEoMT proposes an encoder-only video segmentation model that unifies segmentation and temporal association within a single ViT encoder…
Key concepts
- Encoder-only Video Segmentation Model
- This is a model that uses a standard Vision Transformer (ViT) structure without any additional specialized components for segmentation or tracking. The core idea is that a large, well-trained ViT can learn to perform both tasks by itself, simplifying the architecture significantly compared to models that use separate trackers.
- Query Propagation Mechanism
- This is a technique where information about objects or features from one video frame is passed forward to the next frame by reusing and propagating queries. It helps the model maintain temporal consistency, allowing it to understand how segments evolve over time without needing dedicated, complex tracking modules for every frame.
- Query Fusion Strategy
- This strategy combines the temporally propagated queries with a set of learned, temporally-agnostic queries. This combination allows the model to adapt effectively to new content in each frame while still benefiting from the temporal context carried by the propagated information, ensuring robust performance.
- ViT Backbone Scaling
- The performance of VidEoMT is highly dependent on scaling up the Vision Transformer backbone (e.g., ViT-L). The paper shows that using a larger model size leads to better accuracy and a smaller gap between different model architectures, suggesting that more capacity helps the ViT learn the necessary complex functionalities.
Terminology
Summary
VidEoMT proposes an encoder-only video segmentation model that unifies segmentation and temporal association within a single ViT encoder, achieving significant speedups while maintaining competitive accuracy. p. 1
Motivation and Problem Statement
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules, which introduce significant architectural complexity and computational overhead. The authors hypothesize that a simpler approach can match the accuracy of more complex models by making use of powerful vision foundation models (VFMs). p. 0 They suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. p. 0 The core motivation is to eliminate the need for dedicated tracking modules by making the strong pre-trained ViT learn to take over many of the functionalities of specialized components. p. 0
Proposed Architecture and Temporal Modeling
VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. This is balanced with a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries to enable adaptability to new content. p. 0 The architecture unifies segmentation and temporal association within the ViT, as illustrated in Figure 3. p. 1
Simplification Strategy
The authors verify their hypothesis by taking state-of-the-art video segmentation models and evaluating the effect of removing their specialized modules. They systematically replace the heavy segmenter with EoMT [19] and progressively remove video-specific components like context-aware features and re-identification layers. p. 3 This process demonstrates that strong pre-trained ViT encoders can learn to take over the functionalities of these specialized components without requiring additional complex components. p. 1
Performance and Efficiency Gains
VidEoMT attains competitive accuracy while being 5×–10× faster, running at up to 160 FPS with a ViT-L backbone. Stepwise removal of CAVIS modules toward EoMT, and modifications extending it to VidEoMT, shows that the elimination of the tracker yields a speedup of more than 10× to 162 FPS compared to CAVIS’s 15 FPS, but also leads to a substantial 7. p. 56 AP drop. Applying query propagation in step (5) is necessary to introduce temporal modeling in EoMT, improving the AP by +2. p. 56 without increasing the computational cost. p. 5 The final query fusion step recovers nearly all of the original accuracy while achieving a speedup of more than 10× compared to CAVIS. p. 5
Comparative Analysis and Impact
VidEoMT consistently outperforms state-of-the-art methods across six major benchmarks for video segmentation, including YouTube-VIS 2019, VIPSeg, and VSPW. The results show that VidEoMT is over 14× faster while achieving higher accuracy on all benchmarks except OVIS. p. 6 The model performs better with larger-scale pre-training, as evidenced by the performance gap between CAVIS and VidEoMT decreasing as model size increases. p. 7 The speedup factors can be a veritable game changer for applications, enabling online video processing across a wide range of use cases. p. 1
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Video segmentation models can be simplified by removing complex specialized modules, such as a segmenter and a tracker, replacing them with an encoder-only design that
unifies segmentation and temporal association within a single ViT encoder.
This results in systems that are5×–10× faster
while maintaining competitive accuracy. -
The proposed VidEoMT model can perform online video segmentation by introducing a lightweight query propagation mechanism to carry information across frames and a query fusion strategy to combine propagated queries with learned ones, achieving
temporal association without specialized tracking components.
-
The improved AI system can achieve processing speeds of up to 160 FPS using a ViT-L backbone, which is
over 10× faster than existing state-of-the-art methods on the YouTube-VIS [38] benchmarks
while maintaining comparable accuracy. -
By leveraging large-scale pre-training with Vision Foundation Models (VFMs) like DINOv2 or DINOv3, the ViT encoder can
learn to take over many of the functionalities of the specialized components that are typically added on top,
making these components redundant for video segmentation tasks. -
The system can be optimized for various model sizes, as
the gap between the CAVIS baseline and VidEoMT decreases as model size increases,
indicating that using a larger ViT backbone (ViT-L) provides better performance even with smaller models compared to state-of-the-art trackers. -
The architecture allows for the elimination of computationally expensive auxiliary features, such as
context-aware features
andre-identification layers,
which are hypothesized to be unnecessary when leveraging a strong pre-trained ViT encoder.
Sources
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs