EVIDENT: Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding

arXiv:2605.26104 · cs.CV · Submitted 2026-05-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "EVIDENT: Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding".

Title and authors: Tom: So we’re looking at this paper called "EVIDENT: Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding." It sounds like they are tackling a problem where fine-tuning models for video understanding works great on familiar videos, but then it totally breaks when you show it something new.

Jane: That’s exactly right. The paper is looking at how to make these models better at figuring out *when* things happen in a video, especially when the videos are from different sources. It suggests that the problem isn't just about seeing new concepts, but more about seeing a completely different visual style or domain.

Lu: Essentially, they find that standard fine-tuning creates these tangled visual representations where the model relies too much on specific scene context from the training set instead of actually looking at what’s in the video itself. It causes this kind of attention decoupling between what it knows about concepts and what it sees visually.

Meng: So, if I'm deploying a model, that sounds like a huge risk because those brittle shortcuts you mentioned could cause weird failures on real traffic or real footage from a new camera setup. What’s the core idea behind their solution?

Tom: EVIDENT introduces a parameter-efficient framework to fix that. They anchor the temporal grounding directly into the model's built-in entity attention, routing the adaptation through explicit visual evidence of those entities instead of relying on dataset shortcuts.

Jane: That sounds like they are trying to use the inherent strengths of the pre-trained model while forcing it to look at actual visual evidence when adapting for a new domain. They have three main parts for this: an Entity Bottleneck Adapter, an Entity-Binding Distillation loss, and an Entity-to-eVidence gating mechanism.

Lu: The first part is the Entity Bottleneck Adapter which takes those dense visual tokens and compresses them into compact slots that represent entities, trying to preserve the pre-trained model's entity attention while getting rid of those domain-fragile representations.

Tom: Okay, so they are squeezing the visuals down to focus on these key objects or concepts first. And then there’s this Entity-Binding Distillation loss which is designed to instill some kind of objectness prior into that unstructured visual space, guiding each slot to stick with a coherent entity across the video frames.

Jane: That sounds like they are teaching the model what an entity *is* in a general sense, not just how it looks in one specific training video. But how do you actually use those entities to ground the time?

Lu: The third component is the Entity-to-eVidence gating mechanism. This part takes all those captured entities and uses them as evidence to steer the model, telling it exactly which moments in a sequence contain the query-relevant entities. That's where they inject this idea that the target moment is where those query entities actually co-occur.

Title and authors: Meng: So, instead of just looking at general visual patterns from the old data, the model is now explicitly being told to look for evidence of specific things—like a person and a book—and only focus on frames where that evidence appears together. That sounds much more controllable for deployment.

Tom: Exactly. The training objective is defined as Ltotal equals LCE plus lambda times the EB Distillation loss, where that lambda controls how strong this objectness prior is. They are basically tuning the strength of that entity-grounded guidance.

Jane: The results on cross-domain benchmarks show that EVIDENT consistently improves out-of-domain robustness while keeping performance competitive in the same training domain, all with a modest parameter overhead. That's a pretty strong signal for transferability.

Lu: They found that this entity-grounded visual evidence acts as an effective inductive bias for temporal localization, which is what we need when we move to new domains. And they also managed to decompose the attention decoupling into a concept domain gap and a visual domain gap, finding that the latter is the main driver of out-of-domain degradation.

Meng: If I'm thinking about scaling this up for production, how much overhead are we talking about? This sounds like it adds complexity, and complexity means more compute time.

Tom: They are very focused on parameter efficiency here. Only the Entity Bottleneck Adapter and the LoRA modules need to be trainable, which amounts to roughly two hundred one million parameters for a seven billion model. That’s pretty lean compared to what you’d expect for a system like this.

Jane: And they showed that even when they look at how the attention weights behave, training with EB Distillation drives both the entropy and the cosine similarity between slots down substantially, which means each slot is attending to a sharper region.

Lu: That sharpening of attention across frames is key; qualitative analysis showed that each slot consistently attends to the same entity across consecutive frames, forming those temporally coherent trajectories even on out-of-domain samples.

Meng: So if we look at what they found about the domain gap, it suggests that splitting the out-of-domain samples by visual similarity to the source domain yields a much bigger gap than splitting by how much the query concept is different. That tells us where to focus our data collection efforts for better robustness.

Tom: The paper concludes that evidence-grounded entity-attentive adaptation, rather than just fitting to the source context, is what makes this robust cross-domain video temporal grounding work. It successfully anchors localization in the inherent entity attention of the pre-trained model.

Jane: For those of us listening who aren't reading code, it means that for these large models to be reliable across different video sources, we need to force them to base their time predictions on actual visual evidence of objects and events rather than just memorizing patterns from the training videos.

Title and authors: Lu: I think the implication here is that we can use these entity-centric representations as a much more stable way to handle domain shift in video tasks. It moves away from those brittle dataset shortcuts we see in earlier models, like how naïve fine-tuning causes attention-localization decoupling.

Meng: It sounds like this gives us a path to better generalization, which is what I need when I'm trying to deploy something that has to handle real-world variability. But they also noted a limitation: the authors found that the visual feature source used for the Entity-to-eVidence gating after down projection yields the best overall performance on out-of-domain R1 at zero point five, while remaining competitive on in-domain performance.

Tom: That’s a practical piece of information for deployment—knowing which visual stream to use matters when you're optimizing for speed and accuracy. And they also pointed out that the concept and visual axes carry independent information, as shown by their Pearson correlation between the binary concept indicator and the continuous similarity score being only zero point one hundred six <ref:2605.26104#pg1>.

Jane: So, what does that independence mean for how we think about these models? It suggests that knowing a query concept doesn't automatically dictate where in the visual space the model should look.

Lu: It promotes entity-grounded reasoning instead of just relying on those dataset-specific shortcuts, which is what they were trying to avoid initially. The framework is built upon Qwen2, and it’s designed to be parameter efficient with around two hundred one million trainable parameters for a seven billion model.

Meng: So in short, this framework offers a way to make video understanding more robust by using entity evidence as the anchor for adaptation, and it seems like it does that without requiring massive retraining.

Tom: That’s the mechanic: bottlenecking tokens to entities, distilling objectness priors, and then gating with entity evidence. It's a very structured way to guide the model toward grounded predictions in OOD settings.

Jane: So, if you're working on video search or content accessibility applications right now, this framework offers a robust pathway by leveraging the pre-trained model’s attention capabilities while making the grounding more explicit and evidence-based.

Lu: I think the biggest change is shifting the focus from just matching concepts to explicitly locating where those concepts actually appear in the video stream under different visual conditions.

Meng: It seems like a solid step toward diagnosing and mitigating distribution shifts in video understanding, but we still have to see how stable this performance holds when we move to completely unrelated modalities beyond just visual domain shift.

Tom: Yeah, that’s the next big question. But for now, EVIDENT is showing that entity-grounded evidence is a powerful tool for making these models more reliable in the real world. That’s what we have today.

The paper's summary: Tom: So, to recap, EVIDENT is tackling that problem where fine-tuning models for video understanding works great on familiar videos but then totally breaks when you show them something new, especially across different sources.

Jane: Right. And the big takeaway is that this failure isn't just about the model not knowing a new concept; it’s mostly because of a visual domain shift. The paper finds that models learn these tangled visual patterns based on what they saw in the training set, instead of actually looking at what’s in the video itself.

Lu: They call this an attention-localization decoupling, and they break it down into two parts: a concept domain gap and a visual domain gap. The paper shows that the visual domain gap is actually the main reason why out-of-domain performance drops so fast.

Meng: So, what’s the fix here? Is it just more data from the new domain? Because that sounds like a lot of work for production pipelines right now.

Tom: It’s not just more data; it’s about changing *how* we teach the model to adapt. EVIDENT proposes this parameter-efficient framework that forces adaptation to rely on entity-grounded visual evidence instead of those brittle dataset shortcuts you mentioned.

Jane: They build three main parts for this: first, an Entity Bottleneck Adapter which squeezes the dense visuals down into compact slots representing entities while keeping the model's existing entity attention intact.

Lu: Then there’s an Entity-Binding Distillation loss, which instills these objectness priors into the model’s visual space to make sure each slot binds tightly to a specific entity across all frames.

Meng: And finally, they have this Entity-to-eVidence gating mechanism that uses those captured entities as actual evidence to steer the model toward moments in the video where those specific things actually co-occur. That sounds like a lot of fine-tuning setup.

Tom: It is structured, though. The training objective is just a simple combination of a standard loss plus this entity distillation loss, and they show that this method consistently improves out-of-domain robustness while keeping the performance on familiar videos competitive with only modest parameter overhead.

Jane: That’s the crucial part for anyone using these models; it means you can leverage the powerful entity attention from the pre-trained model without having to retrain everything from scratch every time you get a new video source. It creates a robust pathway where entity-grounded localization is an effective inductive bias for generalizable temporal localization.

Lu: They showed that by splitting out-of-domain samples based on visual similarity to the source domain, the gap they find is much larger than if they split based on concept overlap, confirming that the visual difference is definitely the dominant factor driving degradation.

Meng: If this works as well as they claim, it means we can stop worrying so much about just matching a text query and start focusing on where those specific objects actually appear in the video stream under different lighting or camera setups.

Tom: Exactly. The paper concludes that evidence-grounded entity-attentive adaptation is the key to robust cross-domain video temporal grounding, anchoring localization in what the model inherently knows about entities.

Jane: It’s a really practical idea for applications like video search, because it gives you a more reliable way to find things even when the visual style changes drastically. But they did point out that they found the best performance on out-of-domain results came from using one specific visual feature source for that gating mechanism, which is something you need to keep in mind when you’re deploying this.

Lu: And they also confirmed that the concept and visual axes are actually independent pieces of information, which suggests we don't have to assume that knowing a query concept automatically tells us where to look visually.

Meng: It sounds like a solid step toward making these models more reliable in the real world by focusing on visual evidence rather than just memorizing training shortcuts. We’ll see how this scales when we start dealing with completely different types of shifts, though.

The paper's improvements: Tom: So, we’ve seen how EVIDENT works mechanically—it bottlenecks tokens to entities, distills objectness priors, and gates on entity evidence—and now we’re looking at what this actually means for performance gains.

Jane: The main improvement is that this entity-centric representation learning creates a visual representation that regularizes the fine-tuning process, meaning it can consistently capture things like a person or a book across frames, no matter what domain those videos come from.

Lu: It’s about getting past those brittle representations you saw in previous work; instead of relying on dataset-specific scene context, this method grounds the predictions in the actual visual evidence present in the video.

Meng: So, if I’m shipping a model to production and it needs to handle real-world variability, does this mean we can be more confident that our temporal grounding is stable across different types of footage?

Tom: It does. The experiments show that this entity-grounded localization provides a robust pathway to fully leverage the pre-trained model’s inherent entity attention capabilities under domain shift, and they keep the in-domain performance competitive with only modest parameter overhead.

Jane: What’s interesting is how they diagnose the problem too; they found that the visual domain gap—the difference between training visuals and test visuals—is actually much bigger than the difference in query concepts. That tells us we should focus our efforts on making sure our data covers more visual diversity, not just more concept variety.

Lu: They also showed that training with this entity distillation loss significantly lowers both the entropy and the cosine similarity between different slots, which means each slot is attending to a much sharper region of the video record.

Meng: Sharper attention sounds good for efficiency, but how does that translate to real-time performance when you’re running on a server? Does it add too much computational load?

Tom: The framework is quite parameter-efficient; only the Entity Bottleneck Adapter and LoRA modules need training, which totals about two hundred one million parameters for a seven billion model. That keeps the overhead manageable.

Jane: And they confirmed that this approach is robust to how you set the loss scale for that distillation process, with differences staying within one or two points across various loss weights.

Lu: The method’s strength is anchoring temporal localization in the pre-trained model’s attention structure, which solves that attention-localization decoupling problem head-on. It shifts the focus from just matching concepts to explicitly locating where those concepts appear in the video stream under different visual conditions.

Meng: That explicit steering based on entity evidence sounds like it could be very useful for things like video search, because you can ask for a person and it will look for frames where that specific person is actually visible, even if the background looks totally different.

Tom: Exactly. This framework is a first step toward systematically diagnosing and mitigating distribution shifts in these video tasks by forcing the model to use evidence instead of just relying on dataset-specific shortcuts.

Jane: So, while they show great improvements across various benchmarks like Charades-STA and DiDeMo, the paper does flag one limitation: they found that using a specific visual feature source for the Entity-to-eVidence gating yields the best overall performance on out-of-domain results.

Lu: And they also noted that there's an independence between the concept information and the visual similarity score, which means you don’t have to assume those two things are directly tied together in a simple way.

Meng: So, if we want to deploy this for real-world video understanding, what’s the next hurdle? What doesn't this work on yet?

Tom: They didn't cover everything. The authors pointed out that they only focused on visual domain shift; they haven’t really shown how robust it is when you introduce completely different modalities beyond just visual changes.

Conclusion: Tom: So we’re wrapping up on EVIDENT, which is all about routing MLLM adaptation through entity-grounded visual evidence for cross-domain video temporal grounding.

Jane: The big picture here is that we’ve moved away from brittle dataset shortcuts and toward a system where the model actually looks at what it sees when adapting to new environments.

Lu: It really shows that using entity-grounded localization as an inductive bias is a very solid way to get generalizable temporal localization in video tasks.

Meng: It’s impressive that they managed this with only about two hundred one million trainable parameters for a seven billion model; I like the efficiency of it.

Lalam: I’m confident in the answer here; because they proved that visual domain gap is the primary cause of degradation, and their proposed three-part mechanism directly targets that visual misalignment.

Jane: And what this changes for us on the ground is that we can expect much more stable performance when deploying these models to handle real-world video data from sources we haven't seen before.

Tom: It’s a clear message: if you want cross-domain robustness in video understanding, stop relying on simple fine-tuning and start anchoring your adaptation in explicit entity evidence.

Lu: The idea that the concept and visual axes are independent is also important; it means the model isn't forced to assume those two things are perfectly aligned just because they appear together often.

Meng: I guess for us, it means we have a better tool now to diagnose *why* a model breaks when we move its training data outside of our control.

Jane: Exactly; it gives us a way to debug the distribution shift by looking at the visual evidence rather than just guessing what concept might be missing.

Tom: We’ll keep an eye on this research as they explore how this entity-centric view interacts with even more complex shifts, like audio or text domain changes.

Lu: It opens up a lot of avenues for integrating these entity representations into other multimodal tasks that need to maintain temporal coherence across different visual scenes.

Meng: That sounds promising for building more resilient AI systems that don't just work on pretty training data but actually perform in varied real-world conditions.

Geo Ahn, Jiwook Han, Youngrae Kim, Joonseok Lee, Jinwoo Choi

Kyung Hee University · University of Southern California · Seoul National University

cs.CV

Submitted: 2026-05-25

Updated: 2026-11-19

Comments: NeurIPS 2026. Project page: https://mreraser.github.io/EVIDENT_project_page/

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Fine-tuning MLLMs for Video Temporal Grounding (VTG) often improves indomain performance but degrades sharply under domain shift This work finds that this failure is primarily driven not just by…

Key concepts

Entity Bottleneck Adapter (EB Adapter)
This component transforms dense visual tokens into compact slots representing specific entities. It preserves the original MLLM's entity-attention while replacing domain-fragile representations with a more structured, entity-level view.
Entity-Binding Distillation Loss (EB Distillation)
This loss instills objectness priors into the visual space. It guides each learned slot to reliably bind itself to a coherent, recognizable entity, helping the model learn transferable object representations.
Entity-to-eVidence (E2V) Gating Mechanism
This mechanism uses captured entities as evidence to steer localization. It directs the model to focus on specific moments in the video where the query entities actually co-occur, grounding predictions in visual evidence.

Terminology

Summary

Fine-tuning MLLMs for Video Temporal Grounding (VTG) often improves indomain performance but degrades sharply under domain shift This work finds that this failure is primarily driven not just by unseen query concepts, but by visual domain shift EVIDENT introduces a parameter-efficient adaptation framework that anchors temporal grounding in the inherent entity-attention of pre-trained MLLMs by routing VTG adaptation through explicit visual entity evidence The method consists of three components: an Entity Bottleneck Adapter, an Entity-Binding Distillation loss, and an Entity-to-eVidence gating mechanism These components enable VTG fine-tuning to rely on entity-grounded evidence rather than brittle dataset shortcuts

Pruning redundant reasoning tokens

The paper identifies a critical adaptation paradox where naïve fine-tuned MLLMs learn entangled visual representations that rely on dataset-specific scene context rather than the actual visual content, indiscriminately responding to memorized patterns across frames in OOD videos In contrast, the proposed method learns an entity-centric visual representation that regularizes VTG fine-tuning to ground its predictions in the visual evidence of the video, yielding transferability to OOD domains The authors observe that naïve fine-tuning causes attention-localization decoupling in OOD VTG, which is primarily driven by visual domain shift This decoupling is decomposed into a concept domain gap and a visual domain gap, with the latter being the primary cause of OOD degradation

How the training signal is built

EVIDENT utilizes three key components to achieve evidence-grounded entity-attentive adaptation 1) The Entity Bottleneck Adapter (EB Adapter) transforms dense visual tokens into compact entity-level slots, preserving the pre-trained MLLM's entity-attention while replacing domain-fragile representations 2) The Entity-Binding Distillation (EB Distillation) loss instills objectness priors into the semantically unstructured MLLM visual space, guiding each slot to bind to a coherent entity 3) The Entity-to-eVidence (E2V) gating mechanism leverages captured entities as evidence, steering the model to localize moments containing query-relevant entities The training objective is defined as Ltotal = LCE + λLEBD, where λ controls the strength of the objectness prior

Where the gains come from

Experiments on cross-domain VTG benchmarks show that EVIDENT consistently improves out-of-domain robustness while preserving competitive in-domain performance with modest parameter overhead The results demonstrate that entity-grounded visual evidence provides a robust pathway to fully leverage the entity-attention capabilities of pre-trained MLLMs under domain shift Ablation studies confirm that EB Distillation is crucial for learning transferable representations, and E2V gating provides an additional improvement by exploiting the MLLM’s inherent entity-attention Qualitative analysis shows that each slot consistently attends to the same entity across consecutive frames, forming temporally coherent trajectories along the time axis even on out-of-domain samples

The authors systematically analyzed the domain gap and found that splitting OOD samples by visual similarity to the source domain yields a substantially larger gap than splitting by query concept overlap, identifying the visual domain gap as the primary cause of degradation Furthermore, they verified that DINOv2 cluster maps serve as a semantically grounded training signal for EB Distillation by mirroring the structure of entity-level decomposition in both ID and OOD domains The EVIDENT framework is parameter-efficient, with only the EB Adapter and LoRA modules being trainable, amounting to approximately 201M parameters for a 7B model

The EVIDENT framework consistently improves OOD performance across all source-target configurations while maintaining competitive ID performance For instance, on the Cha.→QVH setting, EVIDENT delivers consistent OOD gains over the Qwen2. p. 7

Where the paper says it

6 points on QVH→Cha. EVIDENT bridges these two regimes, delivering consistent OOD gains over the Qwen2.

5-VL baseline (e. p. 3
Where the paper says it

Acquired grounding capability fails to transfer. To probe the source of this collapse, we measure how strongly the fine-tuned model’s visual-to-query cross-attention shifts toward ground-truth-interval frames, relative to the zero-shot baseline (Figure 2 (c)).

g., +5.5 R1@0.5 and +7.2 R1@0.7) The analysis of slot attention weights shows that training with EB Distillation drives both entropy and inter-slot cosine similarity down substantially, indicating that each slot attends to a sharper, more localized region The EVIDENT framework is shown to be robust to the choice of loss scale for EB Distillation, with differences remaining within roughly 1–2 points across various loss weights

The paper concludes that evidence-grounded entity-attentive adaptation, rather than direct fitting to source-domain context, is the key to robust cross-domain VTG EVIDENT successfully addresses attention-localization decoupling by anchoring temporal localization in the inherent entity-attention of pre-trained MLLMs The framework has potential positive impacts on applications such as video search and content accessibility Standard considerations applicable to general video understanding research should be observed by downstream users

The EVIDENT framework is shown to be effective across diverse benchmarks, including Charades-STA, QVHighlights, and DiDeMo The visual feature source used in E2V gating after down projection yields the best overall performance on OOD R1@0.5 (59.8) while remaining competitive on ID This indicates that E2V gating should be computed from the LLM’s visual stream rather than the bottlenecked slot output The framework is a first step toward systematically diagnosing and mitigating distribution shifts in MLLM-based video temporal grounding

The authors propose that the concept and visual axes carry independent information, as evidenced by the Pearson correlation between the binary concept indicator and the continuous similarity score being only ρ= +0.106 The framework is designed to be parameter-efficient, with a total of approximately 201M trainable parameters for the 7B model This approach promotes entity-grounded reasoning rather than reliance on dataset-specific shortcuts The framework is built upon Qwen2. p. 1

Where the paper says it

These findings point to a different design principle for cross-domain MLLM-based VTG: robust adaptation should anchor temporal localization in the inherent entity-attention of the pre-trained MLLM, rather than merely fitting dense visual tokens to source-domain scene context. To this end, we propose EVIDENT, a parameter-efficient framework for EVIDdence-grounded ENTity-attentive adaptation for cross-domain VTG.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Entity-centric Representation Learning: The system can learn a entity-centric visual representation that regularizes VTG fine-tuning to ground its predictions in the visual evidence of the video, yielding transferability to OOD domains, which enables consistent capture of both person and book across frames regardless of domain.

  2. Parameter Efficient Adaptation: EVIDENT uses a parameter-efficient framework consisting of an Entity Bottleneck Adapter, Entity-Binding Distillation loss, and an Entity-to-eVidence gating mechanism to anchor temporal grounding in the inherent entity-attention of pre-trained MLLMs by routing adaptation through explicit visual entity evidence.

  3. Robust Cross-Domain Generalization: The improved system consistently improves out-of-domain robustness while maintaining competitive in-domain performance, suggesting that entity-grounded localization is an effective inductive bias for generalizable temporal localization.

  4. Diagnosis of Domain Shift: The framework can diagnose the primary cause of OOD degradation by showing that visual domain gap dominates, indicating that adaptation should anchor localization in entity-attention rather than brittle dataset shortcuts.

  5. Entity-Level Grounding Mechanism: The E2V gating mechanism allows the model to steer localization by injecting an inductive bias where the target moment is where the query entities co-occur, enabling evidence-grounded localization under OOD shifts.

Abstract

Fine-tuning MLLMs for Video Temporal Grounding (VTG) often improves in-domain performance but degrades sharply under domain shift. In this work, we find that this failure is primarily driven not just by unseen query concepts, but by visual domain shift, which prevents the model from coupling its learned temporal localization knowledge with its inherent entity-attention capability. To address this, we introduce EVIDENT, a parameter-efficient adaptation framework that anchors temporal grounding in the inherent entity-attention of pre-trained MLLMs by routing VTG adaptation through explicit visual entity evidence. EVIDENT consists of three components: (i) an Entity Bottleneck Adapter that transforms dense visual tokens into compact entity-level slots, (ii) an Entity-Binding Distillation loss that instills objectness priors into the semantically unstructured MLLM visual space, guiding each slot to bind to a coherent entity, and (iii) an Entity-to-eVidence gating mechanism that leverages the captured entities as evidence, steering the model to localize moments containing query-relevant entities. Together, these components enable VTG fine-tuning to rely on entity-grounded evidence rather than brittle dataset shortcuts. Experiments on cross-domain VTG benchmarks show that EVIDENT consistently improves out-of-domain robustness while preserving competitive in-domain performance with modest parameter overhead. These results suggest that entity-level grounding is an effective inductive bias for generalizable temporal localization.

Sources

Related papers