Long Story Short: Story-level Video Understanding from 20K Short Films

arXiv:2406.10221 · cs.CV, cs.AI, cs.CL · Submitted 2024-06-14 · Read on arXiv

cs.CV, cs.AI, cs.CL

Submitted: 2024-06-14

Updated: 2026-08-28

Comments: International Journal of Computer Vision (IJCV)

Journal ref: International Journal of Computer Vision - Volume 134, article number 396 (2026)

DOI: 10.1007/s11263-026-02940-x

Code: https://github.com/Breakthrough/PySceneDetect

Project page: https://ridouaneg.github.io/sf20k.html

License: http://creativecommons.org/licenses/by/4.0/

The gist: Recent developments in vision-language models have significantly advanced video understanding.

Terminology

Accuracy — 31× in this paper · explained in 1 episode
Benchmark — 26× in this paper · explained in 2 episodes
Instruction Tuning — 15× in this paper · explained in 2 episodes
Open-Ended Question Answering — 7× in this paper
Transformer — 7× in this paper · explained in 2 episodes
Multiple-Choice Question Answering — 6× in this paper
Ablation — 4× in this paper · explained in 1 episode
Actor — 3× in this paper · explained in 3 episodes
Alignment — 2× in this paper · explained in 1 episode
Foundation Model — 2× in this paper · explained in 3 episodes
Grounding — 2× in this paper · explained in 3 episodes
Temporal Reasoning — 2× in this paper · explained in 2 episodes
Abundance — 1× in this paper · explained in 1 episode
Causal Reasoning — 1× in this paper · explained in 2 episodes
Edge AI — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Feature Extraction — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Multimodal Reasoning — 1× in this paper · explained in 5 episodes
Regret — 1× in this paper · explained in 3 episodes
Tokenization — 1× in this paper · explained in 2 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
alpha — 1× in this paper · explained in 1 episode

Abstract

Recent developments in vision-language models have significantly advanced video understanding. Existing datasets and tasks, however, have notable limitations. Most datasets are confined to short videos with limited events and narrow narratives. For example, datasets with instructional and egocentric videos often depict the activities of one person in a single scene. Although existing movie datasets offer richer content, they are often limited to short-term tasks, lack publicly available videos, and frequently encounter data leakage issues given the use of subtitles and other information about commercial movies during LLM pretraining. To address the above limitations, we propose Short-Films 20K (SF20K), the largest publicly available movie dataset. SF20K consists of 20,143 amateur films, amounting to 3,582 hours of video, with an average of 12 minutes per movie. We accompany this dataset with SF20K-Test, a manual, open-ended question answering benchmark. SF20K-Test consists of 95 movies and 979 question-answer pairs. Our extensive analysis of SF20K-Test reveals limited data leakage, emphasizes the need for long-term reasoning, and demonstrates the strong performance of recent VLMs. Finally, we show that instruction tuning on the large-scale dataset substantially improves model performance, paving the way for future progress in long-term video understanding.

Terminology

Abstract

Related papers