Beyond Reconstruction: What EEG-to-Video Decoding Actually Recovers
cs.HC
Submitted: 2025-05-27
Updated: 2026-10-31
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reconstructing dynamic visual stimuli from EEG recordings is challenging due to the noisy, non-stationary nature of EEG signals and the limited availability of EEG-video datasets.
Terminology
Abstract
Reconstructing dynamic visual stimuli from EEG recordings is challenging due to the noisy, non-stationary nature of EEG signals and the limited availability of EEG-video datasets. We present EEGVid, a framework that learns EEG representations using triplet loss and reconstructs dynamic videos with a temporally conditioned GAN. We study what these representations encode and how this information supports generation. First, visual representations retain emotional structure, while emotion-based supervision does not preserve the same fine-grained visual information. Second, triplet learning shifts EEG features away from subject-specific structure toward stimulus-related information. Third, analysis across brain regions, hemispheres, and time reveals consistent differences in visual and emotional encoding, with temporal regions contributing strongly across tasks. Finally, we evaluate video generation using three controlled diagnostics. The learned encoder separates unseen video classes better than a randomly initialized encoder, although the cross-subject gain is small, while mismatched EEG conditioning shifts generated content toward the substituted stimulus, showing that the generator actively uses EEG as a content signal. A ground-truth class label yields stronger reconstruction metrics, partly because the label is a cleaner conditioning target. Together, these results show that EEG exhibits a consistent visual and emotional structure that can support dynamic video generation, whereas current reconstruction primarily reflects coarse stimulus-level information rather than fine-grained, segment-specific decoding.
Sources
- Brain decoding: toward real-time reconstruction of visual perception
- Adam: A Method for Stochastic Optimization
- A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity
- DynaMind: Reconstructing Dynamic Visual Scenes from EEG by Aligning Temporal Dynamics and Multimodal Semantics to Guided Diffusion
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
- Decoding Natural Images from EEG for Object Recognition
- Graph Attention Networks
- MindCine: Multimodal EEG-to-Video Reconstruction with Large-Scale Pretrained Models
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems