Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG
summary
The gist
Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs) Sparse frame sampling fails to capture finegrained visual details…
In short
Standard scene segmentation fails for movie understanding because current methods focus too much on visual changes rather than narrative events. The authors introduced NarraScene, a dataset using Event Segmentation Theory to define scenes based on physical, character, and narrative shifts. These narrative-centric segments significantly improve performance in movie retrieval tasks compared to uniform chunking.
Key concepts
- Sparse Frame Sampling
- This technique attempts to select only a few representative frames from a long video. However, it fails because it misses fine-grained visual details crucial for understanding complex movie plots and events.
- Retrieval-Augmented Generation (RAG)
- RAG helps large models answer questions by searching through relevant video segments. Its success depends entirely on how well the retrieved segments are defined; poor boundaries lead to irrelevant or incomplete context.
- Event Segmentation Theory (EST)
- This theory defines a scene as a continuous experience that changes when the underlying situation model shifts. The authors used this to create scene boundaries based on physical, character, and narrative changes, ensuring retrieved units capture meaningful story progression.
Terminology used across episodes
This episode discusses
- Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG · Paper Radio
- Qwen3-VL Technical Report
- Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models
- StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification
- Qwen3-ASR Technical Report
- RAG-Adapter: A Plug-and-Play RAG-enhanced Framework for Long Video Understanding
- Vidi: Large Multimodal Models for Video Understanding and Editing
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- How Well Do Vision-Language Models Understand Movies? · Paper Radio
- SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding
- TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs
The paper
Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG · Read on arXiv
Korea University · KAIST
Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs). Sparse frame sampling fails to capture fine-grained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units. In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content. Scene segmentation, a long-studied problem that partitions movies into semantically coherent units, is a natural candidate for defining such retrieval units. We reexamine whether existing methods actually serve this role through comprehensive evaluation on downstream movie understanding tasks, and find that they do not provide a consistent or dominant advantage over naive uniform temporal chunking. Our audit of the most standard scene segmentation benchmarks reveals why: current annotations prioritize visually salient transitions over narrative event structure. Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level shift. When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks, suggesting that the central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding. The dataset is available at https://huggingface.co/datasets/queez0405/NarraScene.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Beyond Visual Boundaries".
Jane: Here's the gist of the paper. Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs) Sparse frame sampling fails to capture finegrained visual details, while dense sampling quickly exceeds context length limits Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content Scene segmentation is a natural candidate for defining such retrieval units in this pipeline We find that existing methods consistently fail to outperform naive uniform temporal chunking when used as retrieval units on downstream movie understanding tasks Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks Pruning redundant reasoning tokens Existing scene segmentation methods have been evaluated across three downstream tasks—temporal grounding, multi-choice question answering, and claim verification—and consistently fail to outperform naive temporal uniform splitting This suggests that standard scene segmentation targets may not align with the context structure needed by downstream movie understanding tasks when utilizing movie RAG The audit of the most standard scene segmentation benchmarks reveals why this failure occurs: current annotations prioritize visually salient transitions over narrative event structure This mismatch means that existing segmentation methods are not dominant retrieval units for movie RAG across temporal grounding, question answering, and fact verification How the training signal is built To understand the root cause of this failure, the authors examined existing scene boundary annotations and hypothesized that they capture more superficial visual transitions rather than narrative continuity They tested this hypothesis by re-annotating a stratified sample of ten films from MovieNet using a definition grounded in Event Segmentation Theory (EST) This definition characterizes event perception as the segmentation of continuous experience into meaningful units when the underlying situation model changes The authors operationalized this by annotating three levels of scene transition: Level 1 (physical change), Level 2 (character change), and Level 3 (narrative change) A valid narrative scene boundary must include a Level 3 transition, which prevents the task from collapsing into visual chunking Where the gains come from The authors introduced NarraScene, a narrative-centric scene segmentation dataset annotated with this EST-grounded three-level definition This dataset contains 53 full-length open-licensed movies, yielding 2,371 narrative scenes When used as retrieval units in the movie RAG pipeline, hand-labeled narrative scenes consistently outperform uniform chunking on MF2 claim verification This establishes that narrative-valid segmentation is useful for movie RAG in principle, suggesting the central challenge is identifying narrative event units that matter for movie understanding Establishing the benchmark The paper shows that hand-labeled semantic boundaries can outperform other chunking methods on downstream movie claim verification, establishing a practical upper bound for retrieval quality The results indicate that retrieval quality in movie RAG is bottlenecked not only by the embedding model but also by whether chunk boundaries preserve coherent narrative units NarraScene serves as an oracle evaluation target for testing whether retrieval units that preserve narrative events improve movie RAG The findings motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream movie understanding Qualitative analysis Qualitative analysis illustrates the main failure mode, showing that evidence needed to verify a claim is concentrated in narratively coherent exchanges rather than being distributed arbitrarily across time When segmentation fragments this exchange into shorter or visually defined chunks, the retriever can return only partial evidence, causing downstream verifiers to miss key statements The narrative scene groups the full conversational event into a single retrieval unit, providing both motivation and supporting quotes in one coherent context This highlights the transfer gap between legacy scene segmentation targets and downstream movie RAG, where decisive evidence is often goal-relevant rather than visually salient In this work, existing scene segmentation methods do not consistently outperform simple uniform splitting on downstream movie understanding tasks The gap originates from existing segmentation practice, where annotations are more closely aligned with visually salient physical transitions than with narrative event change NarraScene defines scene boundaries through physical, character, and narrative change, requiring every valid boundary to include a narrative-level shift Narrative-centric scene boundaries can outperform uniform chunking on MF2 claim verification, suggesting they are useful for movie RAG in principle Scene segmentation for movie RAG should be evaluated by its ability to recover narrative evidence units that improve downstream movie understanding The results suggest that scene segmentation should be evaluated by the narrative evidence they preserve, not only by boundary detection accuracy under legacy benchmarks References
Title and authors: Tom: So we're looking at this paper called "Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG". It sounds like they're tackling a big problem in how large language models understand long-form video.
Jane: Yeah, it's about movie understanding, which is really hard because you need to grasp the whole story—characters, events, and the overall narrative arc that can span hours of content.
Lu: The authors are basically looking at scene segmentation as a way to chunk movies up for retrieval-augmented generation. They want to see if existing ways of cutting movies into scenes actually work well for this task.
Meng: I'm curious about what they suggest is the main issue with how current scene segmentation methods work in this context.
Tom: Well, the paper points out that most standard scene segmentation methods are failing when used as retrieval units because they focus too much on visually obvious transitions instead of the actual story structure.
Jane: So it's not just a technical detail about cutting up a video; it's about whether those cuts actually help an AI understand what happened in the movie.
Lu: Exactly. The core idea is that these existing methods often create "spurious cuts," meaning they miss real narrative boundaries or create artificial ones, which fragments the continuous story into pieces that don't make sense for reasoning over hours of content.
Meng: So if we use these standard scene boundaries, the retrieval system might pull in chunks that are visually interesting but narratively irrelevant, which would hurt the final answer.
Tom: Right. And they are suggesting a way to fix that by using a different kind of segmentation that focuses on narrative goals instead of just visual changes.
Jane: That sounds promising for grounding movie understanding because it means the system retrieves evidence that actually connects to the plot, not just what looks visually dramatic at a certain moment.
The paper's summary: Tom: So, this paper introduces NarraScene, which is a new dataset built specifically for this purpose. They've annotated it using a three-level cognitive taxonomy that goes from physical change to character change to narrative change.
Jane: That sounds like they are trying to define what a true narrative scene boundary actually is—it has to include that highest level of shift, the narrative one.
Lu: They explicitly state that a valid boundary must include a level three transition, which is defined as a narrative change, and this makes NarraScene explicitly target those narrative event boundaries rather than just visual ones.
Meng: So they're not just looking at what happens visually in one shot; they're trying to capture when the situation model itself changes in a meaningful way for the story.
Tom: That's it. They found that when these narrative-grounded scenes are used as retrieval units, they actually outperform uniform chunking on downstream movie understanding tasks like claim verification.
Jane: That's the key finding here: using these narrative-centric segments gives the movie RAG pipeline better evidence to work with than just splitting the video up evenly in time.
Lu: It really highlights that for long-form video, you need to segment based on what matters for reasoning—the story structure—not just how the pixels change from one frame to the next.
Meng: If this is true, it changes how we think about training models on movie data; instead of just chunking by time, we need a way to label those chunks according to narrative events.
The paper's improvements: Tom: The paper argues that existing scene segmentation methods fail because their annotations prioritize visually salient transitions over the actual structure of narrative events.
Jane: They showed this by looking at how current methods perform on things like temporal grounding, question answering, and claim verification and found they consistently underperform naive uniform temporal splitting.
Lu: To show why this happens, they re-annotated a sample of films using an Event Segmentation Theory definition that breaks the experience down into meaningful units when the underlying situation model changes.
Meng: So they tested their hypothesis by defining boundaries based on physical change, character change, and narrative change to see if it fixed the problem.
Tom: They created NarraScene with fifty-three full-length open-licensed movies, resulting in two thousand three hundred seventy one narrative scenes annotated this way.
Jane: The main improvement they show is that hand-labeled narrative scenes outperform uniform chunking on MF2 claim verification, which is a specific test for movie understanding.
Lu: They established a practical upper bound for retrieval quality by showing that semantic scene boundaries, defined narratively, can beat other chunking methods when it comes to verifying claims in movies.
Meng: This suggests that the real bottleneck isn't just the embedding model; it's whether the chunk boundaries themselves preserve coherent narrative units for accurate evidence retrieval.
Conclusion: Tom: So, to wrap up, they found that existing scene segmentation methods don't consistently beat simple uniform splitting on movie understanding tasks because their annotations are too focused on physical cuts rather than narrative event shifts.
Jane: The solution is NarraScene, which defines boundaries using physical, character, and narrative change levels, requiring every boundary to have a narrative shift. This makes those scenes useful for movie RAG in principle because they preserve coherent events.
Lu: The implication is that when we evaluate scene segmentation for movie RAG, we should judge it by its ability to recover the narrative evidence units that actually improve downstream understanding, not just how accurate the boundary detection is on old benchmarks.
Meng: From an engineering standpoint, this gives us a new oracle target. NarraScene becomes the standard way to test if future segmentation models can actually deliver measurable gains for movie RAG pipelines.
Lalam: I'm confident in this direction because the qualitative analysis showed that evidence needed for verification is concentrated in narratively coherent exchanges rather than being scattered randomly across time.
Tom: So, by focusing on narrative continuity, we get better retrieval results, which means more reliable answers from our movie understanding systems.
Jane: It's a practical shift: move away from purely visual segmentation towards segmenting based on the story structure itself. That's what this paper has shown us with Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI