FOLIO: Focused Semantic Memory for Streaming Video Understanding
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "FOLIO: Focused Semantic Memory for Streaming Video Understanding".
Jane: Here's the gist of the paper. In online streaming video understanding, FOLIO introduces a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact because this system addresses the core challenge of deciding what information to retain and how to organize the maintained history as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes Pruning redundant reasoning tokens The core idea of FOLIO is focused memory construction where important entities, actions, and events are recorded with richer details while surrounding context is kept compact.
Tom: As the stream arrives, FOLIO updates a hybrid memory at the segment level guided by a dynamic focus state.
Jane: This hybrid memory combines a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache.
Tom: How the training signal is built The focus state update uses two groups of features where the positive features include visibility, first appearance, reappearance, persistence, event participation, movement, state change, manipulable-target status and interaction-conditioned relevance.
Paper summary: Tom: So we've talked about the core idea of this paper, which is the focused memory construction. They are addressing the fundamental challenge of deciding what information to retain from a continuously arriving video stream and how to organize that history as it grows.
Jane: The central claim is that you can record important parts of the stream in higher detail while keeping everything else compact, which solves the problem where memory cost increases with stream growth and redundant visual details get retained.
Lu: They propose a hybrid memory at the segment level guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual evidence cache.
Meng: The way they determine what to keep is through two groups of features: positive features like visibility, first appearance, reappearance, persistence, event participation, movement, state change and manipulable target status and interaction conditioned relevance.
Tom: And negative features include absence and long disappearance as well as static background behavior which signals that we can safely discard those details.
Jane: The score update is defined by pi of o equals clip of zero one gamma pi minus one of o plus delta i of o, where delta i is the difference between the positive and negative feature scores.
Lu: They show their results on StreamingBench SQA with a five hundred sixty nine second video, showing focus-guided writing with adaptive keyframe selection at specific time points like t equals one hundred ninety-six seconds or t equals two hundred twenty-eight seconds <ref:2607.13298#pg2>.
Meng: The focus state priorities at time two hundred twenty four seconds included things like Ma L. and Fan Z.D., and a scoreboard visible at the bottom left, showing rally in progress and serve to return.
Tom: These results show they can achieve eighty-two point zero six nine one perception backward accuracy on OVO-Bench with Qwen three VL eight billion models, which is pretty strong performance against their baseline.
Jane: And on StreamingBench overall accuracy they hit seventy four point five, which is substantial when you consider the challenge of maintaining streaming memory.
Lu: The memory retrieval module computes a relevance score R of o divided by q which sums weighted matches between the parsed query fields and the entity’s identity fields, using direct matching first.
Meng: For multi-turn interactions, they include interaction relevance signals in the focus update based on matched entities and actions from prior queries.
Tom: This whole setup is about creating a focused semantic memory that records rich details for specific things while making sure the overall system remains efficient across the entire streaming prefix.
Jane: It’s about solving the core challenge of what information to retain and how to organize that history when it keeps growing, which is something traditional methods struggle with.
Lu: The related work section mentions video-LLMs typically assume access to the full video or query before selecting evidence, whereas FOLIO writes memory online from the observed prefix.
Conclusion: Tom: So looking at the FOLIO paper, it's about focused semantic memory for streaming video understanding and it’s a training-free system. It’s designed to record important parts of a stream in higher detail while keeping surrounding context compact.
Jane: The authors are focusing on how to handle the core challenge of deciding what information to retain and how to organize that history when the stream keeps growing, which leads them to introduce this focused semantic memory approach.
Lu: The implication is that future queries will depend heavily on specific entities, actions, and their temporal changes in a way that FOLIO structures its history around those exact elements for better retrieval.
Meng: From an engineering standpoint, it means they are using adaptive keyframe selection to reduce the visual input sent to the writer model while still getting high accuracy.
Tom: They achieved eighty-two point zero six nine one perception backward accuracy on OVO-Bench and seventy four point five overall accuracy on StreamingBench with this focused approach.
Jane: This work suggests a way to maintain efficient memory for streaming video understanding without having to keep every single visual detail indefinitely.
Haoyang Fan, Dhruv Parikh, Anvitha Ramachandran, Sameh Gobriel, Nilesh Jain, Rajgopal Kannan, Viktor Prasanna
University of Southern California (USC) · Intel Labs · DEVCOM Army Research Office
cs.CV
Submitted: 2026-07-14
Updated: 2026-11-07
Comments: 28 pages, 5 figures
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 75/100
The gist: In online streaming video understanding, FOLIO introduces a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context…
Key concepts
- Focused Semantic Memory Construction
- This is the core idea where FOLIO records important entities, actions, and events with rich details. It keeps the general surrounding context brief instead of recording every visual detail equally. This selective recording saves memory and focuses the system on what matters for later queries.
- Hybrid Memory Structure
- The memory is split into three parts: a short-term visual buffer, long-term semantic memory centered on entities, and a visual-evidence cache. At query time, it uses a lightweight retrieval method that checks direct matches first and expands the search if necessary.
- Multi-Turn Interaction Conditioned Focus
- For queries spanning multiple turns, FOLIO considers how previous queries matched entities or actions. These prior interactions create signals that guide the focus update, ensuring the memory prioritizes relevant information based on the ongoing conversation history.
Terminology
Summary
In online streaming video understanding, FOLIO introduces a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact because this system addresses the core challenge of deciding what information to retain and how to organize the maintained history as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes
Pruning redundant reasoning tokens
The core idea of FOLIO is focused memory construction where important entities, actions, and events are recorded with richer details while surrounding context is kept compact. As the stream arrives, FOLIO updates a hybrid memory at the segment level guided by a dynamic focus state. p. 0 This hybrid memory combines a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. p. 0
How the training signal is built
The focus state update uses two groups of features where the positive features include visibility, first appearance, reappearance, persistence, event participation, movement, state change, manipulable-target status and interaction-conditioned relevance. The negative features include absence and long disappearance and static background behavior. p. 2 The score update is defined as pi(o) = clip[0,1](γpi−1(o) + ∆i(o)) where ∆i(o) = α⊤ϕ+i(o) − β⊤ϕ−i(o). p. 2
Where the gains come from
FOLIO achieves state-of-the-art performance reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74. p. 05 overall accuracy on StreamingBench while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly. p. 0 The savings come from two system choices, specifically focused semantic memory construction which avoids writing every observed entity with the same level of detail, and adaptive keyframe selection which reduces the visual input sent to the writer VLM. p. 6 The merged memory banks remain compact for all variants because records are merged at the entity level, but focused writing still reduces the amount of text that must be generated, merged, and retrieved. p. 6
Hybrid Memory Structure and Retrieval
The hybrid memory state Mi is defined as (Si, Oi, Bi) where Si is the short-term visual buffer, Oi is the long-term semantic memory implemented as an entity-centered structured memory, and Bi is the visual-evidence cache. At query time FOLIO uses lightweight hybrid retrieval combining direct matching over the structured memory with semantic query expansion when direct matching is insufficient. p. 2 The retrieval module computes a relevance score R(o q) which sums weighted matches between the parsed query fields and the entity’s identity fields. p. 4
Multi-Turn Interaction Conditioned Focus
For a multi-turn query stream, matched entities and actions from prior queries contribute interaction-relevance signals to ϕ+i(o) in Sec. 3.2. The focus update uses two groups of features where the positive features include visibility, first appearance, reappearance, persistence, event participation, movement, state change and interaction-conditioned relevance. p. 2 Terms that do not yet match any entity are placed in a pending set Ppendi and retried after later segments are written. p. 4 The paper reader with query history performs best, but removing interaction focus drops accuracy under both reader prompts. p. 3
System Efficiency and Footprint
The text footprint remains manageable even before entity-level merging as the persisted segment-level JSON records are about 140–156 KB per video minute, and the final merged entity bank is smaller because records referring to the same entity are consolidated. Selected visual evidence is represented by timestamped keyframe references and re-extracted from the source video when needed. p. 4 The memory-writing cost is a significant systems cost but not the only source of latency, and batching or faster writer and answering backbones are the main systems optimizations for real-time deployment. p. 6 The final merged memory bank is smaller because records referring to the same entity are consolidated. p. 4
Query-Time Evidence Assembly
Evidence selection inside ASSEMBLE is parameterized by the query type χq from zq, and the assembled block always includes the parsed query context, target and answer-option support and a brief scene overview. Type-specific selection adds interaction action-chain events and participant links for the targets and anchors for Interaction type. p. 3 The assembled block always includes the parsed query context, target and answer-option support, a brief scene overview; type-specific selection adds: Interaction: action-chain events and participant links for the targets and anchors. p. 3 The assembled block always includes the parsed query context, target and answer-option support, a brief scene overview; type-specific selection adds: Current-location: the most recent location period of the target entity. p. 3 The assembled block always includes the parsed query context, target and answer-option support; type-specific selection adds: Historical-location: the full location chain of the target entity up to tq. p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Focused memory construction based on a dynamic focus state guides memory updates by assigning writing levels like
focus,
high priority,
support,
andcontext
to entities and actions, ensuring that important entities receive higher-detail records while surrounding context is written compactly. This mechanism directly addresses the challenge of deciding what information to retain and how to organize history. -
The hybrid memory structure combines a short-term visual buffer with a long-term semantic memory organized as an entity-centered structured memory, which stores
target states, locations, relations, and associated events over time.
This allows for both real-time grounding in the current scene and retrospective recall of specific entity histories. -
A lightweight hybrid retrieval pipeline combines direct matching over the structured memory with semantic query expansion when direct matching is insufficient. The semantic query expansion step uses a VLM to
identify conceptually relevant objects
when keyword matches fail, allowing for concept-level queries likeavid reader
to be answered by linking them to conceptually related entities. -
Multi-turn interactions are handled by updating the focus state based on query terms, where
matched entities and actions from prior queries contribute interaction-relevance signals to fi before being fed back into (a).
This ensures that later memory construction continues to track targets under discussion across a longer horizon. -
Memory writing cost is substantially reduced by focusing on important parts of the stream while keeping surrounding context compact, as demonstrated by FOLIO reducing writer input tokens by 33.2% and merged memory size by 32.4% compared to fixed-uniform variants. This makes online memory construction more efficient for continuous video streams.
-
Query-time evidence selection uses a query-type taxonomy to parameterize evidence assembly, ensuring that the retrieved block includes type-specific information like
action-chain events and participant links for the targets and anchors
when dealing with interaction queries. This prevents relying solely on raw retrieval, especially for complex questions. -
Entity matching preserves stable identities while allowing natural language variation across segments by using a score based on normalized names, aliases, head nouns, categories, and partial-string matches. The rule that "a match is accepted when m(x, o) >= τm" ensures that distinct entities with the same head noun are not merged accidentally.
-
The system includes an answer generation mode selection process where
MODE B -- CONCEPT REASONING (only when memory has CONCEPT QUESTION header)
is enabled by the rule-based retriever failing entirely on a question, allowing the model to useINDIRECT evidence IS valid
for abstract concepts.
Sources
- Qwen3-VL Technical Report
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- StreamingTOM: Streaming Token Compression for Efficient Video Understanding
- Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously
- Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams
- WAT: Online Video Understanding Needs Watching Before Thinking
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
- From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents
- OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning
- Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
- StreamChat: Chatting with Streaming Video
- Thinking in Streaming Video
- AURA: Always-On Understanding and Real-Time Assistance via Video Streams
- LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
- Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
- A Simple Baseline for Streaming Video Understanding
- DriveLM: Driving with Graph Visual Question Answering
- CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs