Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Beyond Visual Boundaries".
Jane: Here's the gist of the paper. Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs) Sparse frame sampling fails to capture finegrained visual details, while dense sampling quickly exceeds context length limits Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content Scene segmentation is a natural candidate for defining such retrieval units in this pipeline We find that existing methods consistently fail to outperform naive uniform temporal chunking when used as retrieval units on downstream movie understanding tasks Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks Pruning redundant reasoning tokens Existing scene segmentation methods have been evaluated across three downstream tasks—temporal grounding, multi-choice question answering, and claim verification—and consistently fail to outperform naive temporal uniform splitting This suggests that standard scene segmentation targets may not align with the context structure needed by downstream movie understanding tasks when utilizing movie RAG The audit of the most standard scene segmentation benchmarks reveals why this failure occurs: current annotations prioritize visually salient transitions over narrative event structure This mismatch means that existing segmentation methods are not dominant retrieval units for movie RAG across temporal grounding, question answering, and fact verification How the training signal is built To understand the root cause of this failure, the authors examined existing scene boundary annotations and hypothesized that they capture more superficial visual transitions rather than narrative continuity They tested this hypothesis by re-annotating a stratified sample of ten films from MovieNet using a definition grounded in Event Segmentation Theory (EST) This definition characterizes event perception as the segmentation of continuous experience into meaningful units when the underlying situation model changes The authors operationalized this by annotating three levels of scene transition: Level 1 (physical change), Level 2 (character change), and Level 3 (narrative change) A valid narrative scene boundary must include a Level 3 transition, which prevents the task from collapsing into visual chunking Where the gains come from The authors introduced NarraScene, a narrative-centric scene segmentation dataset annotated with this EST-grounded three-level definition This dataset contains 53 full-length open-licensed movies, yielding 2,371 narrative scenes When used as retrieval units in the movie RAG pipeline, hand-labeled narrative scenes consistently outperform uniform chunking on MF2 claim verification This establishes that narrative-valid segmentation is useful for movie RAG in principle, suggesting the central challenge is identifying narrative event units that matter for movie understanding Establishing the benchmark The paper shows that hand-labeled semantic boundaries can outperform other chunking methods on downstream movie claim verification, establishing a practical upper bound for retrieval quality The results indicate that retrieval quality in movie RAG is bottlenecked not only by the embedding model but also by whether chunk boundaries preserve coherent narrative units NarraScene serves as an oracle evaluation target for testing whether retrieval units that preserve narrative events improve movie RAG The findings motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream movie understanding Qualitative analysis Qualitative analysis illustrates the main failure mode, showing that evidence needed to verify a claim is concentrated in narratively coherent exchanges rather than being distributed arbitrarily across time When segmentation fragments this exchange into shorter or visually defined chunks, the retriever can return only partial evidence, causing downstream verifiers to miss key statements The narrative scene groups the full conversational event into a single retrieval unit, providing both motivation and supporting quotes in one coherent context This highlights the transfer gap between legacy scene segmentation targets and downstream movie RAG, where decisive evidence is often goal-relevant rather than visually salient In this work, existing scene segmentation methods do not consistently outperform simple uniform splitting on downstream movie understanding tasks The gap originates from existing segmentation practice, where annotations are more closely aligned with visually salient physical transitions than with narrative event change NarraScene defines scene boundaries through physical, character, and narrative change, requiring every valid boundary to include a narrative-level shift Narrative-centric scene boundaries can outperform uniform chunking on MF2 claim verification, suggesting they are useful for movie RAG in principle Scene segmentation for movie RAG should be evaluated by its ability to recover narrative evidence units that improve downstream movie understanding The results suggest that scene segmentation should be evaluated by the narrative evidence they preserve, not only by boundary detection accuracy under legacy benchmarks References
Title and authors: Tom: So we're looking at this paper called "Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG". It sounds like they're tackling a big problem in how large language models understand long-form video.
Jane: Yeah, it's about movie understanding, which is really hard because you need to grasp the whole story—characters, events, and the overall narrative arc that can span hours of content.
Lu: The authors are basically looking at scene segmentation as a way to chunk movies up for retrieval-augmented generation. They want to see if existing ways of cutting movies into scenes actually work well for this task.
Meng: I'm curious about what they suggest is the main issue with how current scene segmentation methods work in this context.
Tom: Well, the paper points out that most standard scene segmentation methods are failing when used as retrieval units because they focus too much on visually obvious transitions instead of the actual story structure.
Jane: So it's not just a technical detail about cutting up a video; it's about whether those cuts actually help an AI understand what happened in the movie.
Lu: Exactly. The core idea is that these existing methods often create "spurious cuts," meaning they miss real narrative boundaries or create artificial ones, which fragments the continuous story into pieces that don't make sense for reasoning over hours of content.
Meng: So if we use these standard scene boundaries, the retrieval system might pull in chunks that are visually interesting but narratively irrelevant, which would hurt the final answer.
Tom: Right. And they are suggesting a way to fix that by using a different kind of segmentation that focuses on narrative goals instead of just visual changes.
Jane: That sounds promising for grounding movie understanding because it means the system retrieves evidence that actually connects to the plot, not just what looks visually dramatic at a certain moment.
The paper's summary: Tom: So, this paper introduces NarraScene, which is a new dataset built specifically for this purpose. They've annotated it using a three-level cognitive taxonomy that goes from physical change to character change to narrative change.
Jane: That sounds like they are trying to define what a true narrative scene boundary actually is—it has to include that highest level of shift, the narrative one.
Lu: They explicitly state that a valid boundary must include a level three transition, which is defined as a narrative change, and this makes NarraScene explicitly target those narrative event boundaries rather than just visual ones.
Meng: So they're not just looking at what happens visually in one shot; they're trying to capture when the situation model itself changes in a meaningful way for the story.
Tom: That's it. They found that when these narrative-grounded scenes are used as retrieval units, they actually outperform uniform chunking on downstream movie understanding tasks like claim verification.
Jane: That's the key finding here: using these narrative-centric segments gives the movie RAG pipeline better evidence to work with than just splitting the video up evenly in time.
Lu: It really highlights that for long-form video, you need to segment based on what matters for reasoning—the story structure—not just how the pixels change from one frame to the next.
Meng: If this is true, it changes how we think about training models on movie data; instead of just chunking by time, we need a way to label those chunks according to narrative events.
The paper's improvements: Tom: The paper argues that existing scene segmentation methods fail because their annotations prioritize visually salient transitions over the actual structure of narrative events.
Jane: They showed this by looking at how current methods perform on things like temporal grounding, question answering, and claim verification and found they consistently underperform naive uniform temporal splitting.
Lu: To show why this happens, they re-annotated a sample of films using an Event Segmentation Theory definition that breaks the experience down into meaningful units when the underlying situation model changes.
Meng: So they tested their hypothesis by defining boundaries based on physical change, character change, and narrative change to see if it fixed the problem.
Tom: They created NarraScene with fifty-three full-length open-licensed movies, resulting in two thousand three hundred seventy one narrative scenes annotated this way.
Jane: The main improvement they show is that hand-labeled narrative scenes outperform uniform chunking on MF2 claim verification, which is a specific test for movie understanding.
Lu: They established a practical upper bound for retrieval quality by showing that semantic scene boundaries, defined narratively, can beat other chunking methods when it comes to verifying claims in movies.
Meng: This suggests that the real bottleneck isn't just the embedding model; it's whether the chunk boundaries themselves preserve coherent narrative units for accurate evidence retrieval.
Conclusion: Tom: So, to wrap up, they found that existing scene segmentation methods don't consistently beat simple uniform splitting on movie understanding tasks because their annotations are too focused on physical cuts rather than narrative event shifts.
Jane: The solution is NarraScene, which defines boundaries using physical, character, and narrative change levels, requiring every boundary to have a narrative shift. This makes those scenes useful for movie RAG in principle because they preserve coherent events.
Lu: The implication is that when we evaluate scene segmentation for movie RAG, we should judge it by its ability to recover the narrative evidence units that actually improve downstream understanding, not just how accurate the boundary detection is on old benchmarks.
Meng: From an engineering standpoint, this gives us a new oracle target. NarraScene becomes the standard way to test if future segmentation models can actually deliver measurable gains for movie RAG pipelines.
Lalam: I'm confident in this direction because the qualitative analysis showed that evidence needed for verification is concentrated in narratively coherent exchanges rather than being scattered randomly across time.
Tom: So, by focusing on narrative continuity, we get better retrieval results, which means more reliable answers from our movie understanding systems.
Jane: It's a practical shift: move away from purely visual segmentation towards segmenting based on the story structure itself. That's what this paper has shown us with Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG.
Korea University · KAIST
cs.CV
Submitted: 2026-08-27
Updated: 2026-11-05
Comments: Accepted to NeurIPS 2026 (Evaluations & Datasets Track)
Code: https://github.com/Breakthrough/PySceneDetect
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 91/100
The gist: Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs) Sparse frame sampling fails to capture finegrained visual details…
Key concepts
- Sparse Frame Sampling
- This technique attempts to select only a few representative frames from a long video. However, it fails because it misses fine-grained visual details crucial for understanding complex movie plots and events.
- Retrieval-Augmented Generation (RAG)
- RAG helps large models answer questions by searching through relevant video segments. Its success depends entirely on how well the retrieved segments are defined; poor boundaries lead to irrelevant or incomplete context.
- Event Segmentation Theory (EST)
- This theory defines a scene as a continuous experience that changes when the underlying situation model shifts. The authors used this to create scene boundaries based on physical, character, and narrative changes, ensuring retrieved units capture meaningful story progression.
Terminology
Summary
Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs) Sparse frame sampling fails to capture finegrained visual details, while dense sampling quickly exceeds context length limits. p. 0 Kim3 Donghyun Kim1
1Korea University 2KT 3KAIST
Abstract
Understanding long-form video remains a fundamental challenge for multimodal
large language models (MLLMs). Sparse frame sampling fails to capture finegrained visual details, while dense sampling quickly exceeds context length limits. Sparse frame sampling fails to capture finegrained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval
units. In this paper, we investigate RAG for movie understanding, which demands
story-level reasoning over characters, events, and narrative arcs spanning hours
of content. Scene segmentation, a long-studied problem that partitions movies
into semantically coherent units, is a natural candidate for defining such retrieval
units. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval
units. In this paper, we investigate RAG for movie understanding, which demands
story-level reasoning over characters, events, and narrative arcs spanning hours
of content. Scene segmentation, a long-studied problem that partitions movies
into semantically coherent units, is a natural candidate for defining such retrieval
units. We reexamine whether existing methods actually serve this role through
comprehensive evaluation on downstream movie understanding tasks, and find that
they consistently fail to outperform naive uniform temporal chunking.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The Mismatch in Existing Segmentation Targets
Existing scene segmentation methods often prioritize visually salient transitions over narrative event structure. An audit of the most standard scene segmentation benchmarks reveals why this mismatch exists. p. 0 Our audit of
the most standard scene segmentation benchmarks reveals why: current annotations
prioritize visually salient transitions over narrative event structure. Motivated by
this mismatch, we introduce NarraScene, a narrative-centric scene segmentation
dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level
shift. To test this hypothesis, we re-annotate a stratified sample of ten films
from the MovieNet test set using a scene boundary definition grounded in Event Segmentation
Theory [37, 22], which characterizes event perception as the segmentation of continuous experience
into meaningful units when the underlying situation model changes. Comparing these re-annotations
against the original MovieNet labels, we find that the majority of existing boundaries are driven
primarily by physical transitions, with only a small fraction aligning with narrative event change
under our definition. Comparing these re-annotations
against the original MovieNet labels, we find that the majority of existing boundaries are driven
primarily by physical transitions, with only a small fraction aligning with narrative event change
under our definition. This suggests that current scene segmentation practice is more strongly aligned
with visually salient cues than with narrative event structure.Where the paper says it
Where the paper says it
Where the paper says it
The Proposed Narrative-Centric Solution
Motivated by this mismatch, the authors introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change. Every valid boundary in NarraScene requires a narrative-level shift. p. 0 Our audit of
the most standard scene segmentation benchmarks reveals why: current annotations
prioritize visually salient transitions over narrative event structure. Motivated by
this mismatch, we introduce NarraScene, a narrative-centric scene segmentation
dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level
shift. Concretely, we annotate scene
transitions along three levels, namely physical change, character change, and narrative change, and
require every valid boundary to include a narrative shift. This makes NarraScene explicitly target
narrative event boundaries rather than visual discontinuities alone.Where the paper says it
Where the paper says it
Evaluation Through Downstream Tasks
To directly assess whether existing scene segmentation methods provide effective retrieval units for movie RAG, the authors conduct controlled experiments across three downstream tasks: temporal grounding [29], multi-choice question answering [12], and claim verification [41]. They find that models trained on the most widely used scene segmentation dataset consistently fail to outperform naive temporal uniform splitting. p. 1 To directly assess whether existing scene segmentation methods provide effective retrieval units for
movie RAG, we conduct controlled experiments across three downstream tasks for movie understanding: temporal grounding [29], multi-choice question answering [12], and claim verification [41]. Fixing the retrieval backbone and downstream models across all tasks, we vary only the segmentation method and find that models trained on the most widely used scene segmentation dataset [14]
consistently fail to outperform naive temporal uniform splitting.Where the paper says it
Performance of Narrative Boundaries
When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks. Specifically, hand-labeled narrative scene boundaries can outperform other chunking methods on MF2 claim verification. p. 2 Under the same retrieval and downstream verification pipeline, hand-labeled narrative scenes consistently outperform uniform chunking
on MF2
claim verification. These results suggest that the central challenge for scene segmentation
in movie RAG is not simply to detect more accurate boundaries under legacy benchmarks, but to
recover the narrative evidence units that downstream tasks need. • We introduce NarraScene, a narrative-centric scene segmentation dataset for full-length movies, in
which every valid boundary includes a narrative transition under a three-level boundary definition. • We show that hand-labeled narrative scene boundaries can outperform other chunking methods
on downstream movie claim verification, establishing that narrative-valid segmentation is useful
for movie RAG in principle.Where the paper says it
Where the paper says it
Conclusion and Future Direction
The central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding. The work suggests that scene segmentation should be evaluated by its ability to recover narrative evidence units rather than just boundary detection accuracy under legacy benchmarks. p. 8 Finally,
we show that narrative-centric scene boundaries can outperform uniform chunking on MF2
claim
verification, suggesting that these boundaries are useful for movie RAG in principle. Taken together,
our results suggest that scene segmentation for movie RAG should be evaluated not only by boundary
detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that
improve downstream movie understanding. At the same time, these results should be interpreted as an oracle analysis rather than as evidence that
current automatic scene segmentation methods already solve the problem. Instead, they establish a
practical upper bound and motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream
movie understanding.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper reexamines scene segmentation as a chunking interface for modern movie RAG and shows that existing scene segmentation methods do not provide a reliable advantage over simple uniform chunking across temporal grounding, question answering, and fact verification. The results indicate that standard scene segmentation models do not automatically yield better retrieval units for downstream movie temporal grounding. p. 4 best uniform chunking on average. Overall, standard scene segmentation models do not automatically
yield better retrieval units for downstream movie temporal grounding.Where the paper says it
How the training signal is built
The authors define scene boundaries based on Event Segmentation Theory (EST), which characterizes event perception as the segmentation of continuous experience into meaningful units when the underlying situation model changes. This involves annotating three levels of scene transition: Level 1 (physical change), Level 2 (character change), and Level 3 (narrative change). p. 2 • We audit MovieNet scene segmentation annotations under an Event Segmentation Theory based
definition and show that existing scene segmentation targets are more strongly aligned with
physical boundary cues than with narrative event change. • We introduce NarraScene, a narrative-centric scene segmentation dataset for full-length movies, in
which every valid boundary includes a narrative transition under a three-level boundary definition. These levels are not mutually exclusive and may co-occur at the same timestamp. Our key design
choice is that a valid narrative scene boundary must include a Level 3 transition.Where the paper says it
Where the paper says it
Where the gains come from
The superiority of narrative boundaries is demonstrated in claim verification on MF2 [41]. Hand-labeled semantic boundaries consistently outperform all fixed-length baselines and automatic GenreDur segmentation under the same retrieval pipeline. p. 7 We report pairwise accuracy, which counts a prediction
as correct when the model ranks the true claim above its paired false claim for the same movie. Table 6 shows that hand-labeled semantic boundaries consistently outperform all fixed-length baselines and the automatic GenreDur segmentation under the same retrieval pipeline. The gains are not marginal. NarraScene consistently outperforms the strongest non-oracle
baseline in every setting, showing that retrieval quality depends not just on chunk size, but on whether
boundaries align with coherent narrative units.Where the paper says it
Where the paper says it
The qualitative analysis shows that evidence needed to verify a claim is concentrated in a narratively coherent exchange rather than being distributed arbitrarily across time. The narrative scene groups the full conversational event into a single retrieval unit, providing both motivation and supporting quote in one context. p. 8 A fixed window may overlap part of the relevant exchange and
therefore remain competitive in aggregate, but it does not explicitly preserve the event-level structure
that makes the claim easy to verify. In contrast, the narrative scene groups the full conversational
event into a single retrieval unit, so the model receives both the motivation and the supporting quote
in one coherent context.Where the paper says it
The key finding is that useful scene boundaries for movie RAG should be evaluated by the narrative evidence they preserve, not only by how well they match legacy boundary annotations. NarraScene provides an oracle evaluation target for testing this hypothesis. p. 2 The resulting benchmark contains 2,371 narrative boundaries and 2,424 annotated scenes. Beyond providing new labels, NarraScene serves as an oracle evaluation target for testing whether
retrieval units that preserve narrative events improve movie RAG.Where the paper says it
The paper concludes that existing scene segmentation methods do not consistently outperform simple uniform splitting on downstream movie understanding tasks across temporal grounding, question answering, and fact verification. The findings motivate the creation of NarraScene to test whether future segmentation models can recover narrative boundaries. p. 7 At the same time, these results should be interpreted as an oracle analysis rather than as evidence that
current automatic scene segmentation methods already solve the problem. Instead, they establish a
practical upper bound and motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream
movie understanding.Where the paper says it
The results suggest that scene segmentation for movie RAG should be evaluated not only by boundary detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that improve downstream movie understanding. p. 8 Finally,
we show that narrative-centric scene boundaries can outperform uniform chunking on MF2
claim
verification, suggesting that these boundaries are useful for movie RAG in principle. Taken together,
our results suggest that scene segmentation for movie RAG should be evaluated not only by boundary
detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that
improve downstream movie understanding. Finally,
we show that narrative-centric scene boundaries can outperform uniform chunking on MF2
claim
verification, suggesting that these boundaries are useful for movie RAG in principle. Taken together,
our results suggest that scene segmentation for movie RAG should be evaluated not only by boundary
detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that
improve downstream movie understanding. Finally,
we show that narrative-centric scene boundaries can outperform uniform chunking on MF2
claim
verification, suggesting that these boundaries are useful for movie RAG in principle. Taken together,
our results suggest that scene segmentation for movie RAG should be evaluated not only by boundary
detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that
improve downstream movie understanding. NarraScene consistently outperforms the strongest non-oracle
baseline in every setting, showing that retrieval quality depends not just on chunk size, but on whether
boundaries align with coherent narrative units. These results provide direct evidence that retrieval
quality in movie RAG is bottlenecked not only by the embedding model, but also by whether chunk
boundaries preserve coherent narrative units.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The authors highlight two main findings from the downstream experiments: (i) Existing scene segmentation methods do not yield effective retrieval units for movie RAG. (ii) On MovieStory101, oracle 3-minute clips substantially outperform retrieved top-1 chunks. p. 5 (i) Existing scene
segmentation methods, including both trained models and classical computer-vision (PySceneDetect) baselines, are not dominant retrieval units for movie RAG; across tasks, they fail to show a
consistent advantage over simple uniform splitting. (ii) On MovieStory101, the oracle 3-minute
clips substantially outperform retrieved top-1 chunks. To better understand this gap, we manually
examined 63 randomly sampled oracle clips out of 632 test clips (10%) and found that all of them
contained coherent narrative scenes, whereas segments predicted by existing scene segmentation
methods often failed to maintain narrative coherence. This observation motivates the next section,
where we more directly examine whether standard scene annotations align with narrative structure.Where the paper says it
Where the paper says it
The study provides a practical upper bound by showing that hand-labeled semantic boundaries substantially improve downstream movie RAG when the boundaries are correct. NarraScene is established as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream movie understanding. p. 7 At the same time, these results should be interpreted as an oracle analysis rather than as evidence that
current automatic scene segmentation methods already solve the problem. Instead, they establish a
practical upper bound and motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream
movie understanding.Where the paper says it
The paper concludes that scene segmentation for movie RAG should be evaluated not only by boundary detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that improve downstream movie understanding. The results suggest that scene segmentation for movie RAG should be evaluated not only by boundary detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that improve downstream movie understanding.
The authors argue that cognitively grounded narrative scene boundaries provide a stronger substrate for downstream retrieval, localization, and reasoning than fixed temporal chunks or visually defined segments. This suggests that scene segmentation should be evaluated through both retrieval quality and chunk quality rather than final QA accuracy alone. p. 8 Lower downstream QA accuracy can arise because the
retriever fails to select the best chunk, because the chunk itself does not preserve the right evidence,
or both. Scene segmentation for movie RAG should therefore be evaluated through both retrieval
quality and chunk quality rather than final QA accuracy alone.Where the paper says it
The paper is motivated by the gap between semantic validity and practical retrieval strength, which is addressed by the benchmark audit in the next section. The work establishes a practical upper bound and motivates NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream movie understanding. p. 7 At the same time, these results should be interpreted as an oracle analysis rather than as evidence that
current automatic scene segmentation methods already solve the problem. Instead, they establish a
practical upper bound and motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream
movie understanding.Where the paper says it
The paper provides context units that better reflect narrative events, potentially benefiting long-form video retrieval, accessibility, and media analysis. Potential risks include misuse for large-scale indexing or monitoring of audiovisual content. The work presents NarraScene as a research benchmark rather than a definitive account of how movies should be segmented. p. 7 At the same time, these results should be interpreted as an oracle analysis rather than as evidence that
current automatic scene segmentation methods already solve the problem. Instead, they establish a
practical upper bound and motivate NarraScene as a benchmark for testing whether future segmentation models can recover narrative boundaries that translate into measurable gains in downstream
movie understanding.Where the paper says it
The paper's findings suggest that scene segmentation for movie RAG should be evaluated not only by boundary detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that improve downstream movie understanding. The results suggest that scene segmentation for movie RAG should be evaluated not only by boundary detection accuracy under legacy benchmarks, but by its ability to recover narrative evidence units that improve downstream movie understanding.
The authors show that NarraScene defines scene boundaries through physical, character, and narrative change while requiring every valid boundary to include a narrative-level shift. This design targets the kind of coherent event units needed by movie RAG. p. 6 We adopt the
Event Segmentation Theory (EST)-grounded three-level scene boundary definition introduced in
Figure 3, where valid scene boundaries must include Level 3 narrative change [37, 39]. This design
targets the kind of coherent event units needed by movie RAG, where retrieval quality depends on
preserving interactions, goals, and story progression rather than only local visual continuity.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved retrieval units for movie understanding by using narrative-centric scene segmentation, which
outperform uniform chunking on downstream movie understanding tasks.
This allows retrieval to selectthe narrative event units that matter for movie understanding
instead of just visually salient transitions. -
Enhanced performance on downstream tasks such as temporal grounding, question answering, and claim verification by utilizing retrieval units aligned with Event Segmentation Theory (EST). This is achieved because
hand-labeled narrative scene boundaries can outperform other chunking methods on downstream movie claim verification.
-
More robust evaluation of existing scene segmentation methods by auditing them against the required narrative structure. This audit shows that
existing scene segmentation targets are more strongly aligned with physical boundary cues than with narrative event change,
highlighting a mismatch for RAG applications. -
Creation of the NarraScene dataset to serve as an
oracle evaluation target for testing whether retrieval units that preserve narrative events improve movie RAG.
This provides a benchmark wheresemantic scene boundaries are inherently more ambiguous than physical cuts,
allowing researchers to test if retrieval quality depends on preserving coherent narrative units.
Abstract
Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs). Sparse frame sampling fails to capture fine-grained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units. In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content. Scene segmentation, a long-studied problem that partitions movies into semantically coherent units, is a natural candidate for defining such retrieval units. We reexamine whether existing methods actually serve this role through comprehensive evaluation on downstream movie understanding tasks, and find that they do not provide a consistent or dominant advantage over naive uniform temporal chunking. Our audit of the most standard scene segmentation benchmarks reveals why: current annotations prioritize visually salient transitions over narrative event structure. Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level shift. When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks, suggesting that the central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding. The dataset is available at https://huggingface.co/datasets/queez0405/NarraScene.
Sources
- Qwen3-VL Technical Report
- Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models
- StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification
- Qwen3-ASR Technical Report
- RAG-Adapter: A Plug-and-Play RAG-enhanced Framework for Long Video Understanding
- Vidi: Large Multimodal Models for Video Understanding and Editing
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- How Well Do Vision-Language Models Understand Movies?
- SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding
- TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs