A 1000-hour EEG-EMG-audio dataset of Japanese speech production
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "A 1000-hour EEG-EMG-audio dataset of Japanese speech production".
Title and authors: Ines: So we're talking about this paper called "A one thousand-hour EEG-EMG-audio dataset of Japanese speech production" <ref:2606.01264#pg1,A 1000-hour EEG-EMG-audio dataset of Japanese speech production>. It's essentially a massive collection of data from three healthy native Japanese speakers doing open-vocabulary overt speech.
Marcus: Yeah, it’s got a hundred and two hours of synchronized EEG, facial electromyography, and audio. The authors are Motoshige Sato, Ilya Horiguchi, Masakazu Inoue, Kenichi Tomeoka, Eri Hatakeyama, Yuya Kita, Atsushi Yamamoto and Ippei Fujisawa on the first page.
Yuki: That sounds like a lot of data. So what’s the main point here for us? What are they trying to prove with this collection of recordings?
Ines: They built this because they wanted to support both speech research and broader EEG studies. The motivation was speech decoding, but it expanded into artifact characterization and multimodal alignment too.
Marcus: Exactly. The big deal is that it’s longitudinal, cross-device data. That means you get a really robust look at how these signals behave over time across different machines and speakers.
Yuki: But the paper also points out some challenges, right? It mentions that overt speech has a characteristic challenge because articulatory muscle activity contaminates scalp recordings.
Ines: Right, and that's why they included facial EMG and audio simultaneously. They say this combination supports artifact characterization, multimodal alignment, and analyses of neural signals relative to the temporal structure of speech.
Marcus: So it’s not just adding extra data; they see it as a tool to separate the noise from the actual brain signals related to speaking.
Yuki: And they validated this by showing that pre-stimulus baseline intervals showed low variance, which suggests stable baselines and effective artifact reduction in their processing pipeline.
Ines: They also did power spectral density checks, showing a one over f trend characteristic of electrophysiological signals in both the raw and preprocessed data. Then they looked at the spectrum between eight hertz and thirteen hertz, which showed a relatively flattened profile consistent with task-related power attenuation in the alpha band during active tasks.
Marcus: That means when they are actively speaking, that specific frequency band gets quieter, which makes sense for something related to speech production.
Yuki: And their spatiotemporal validation of event-related potentials confirmed that evoked responses were time-locked to task onset. They found components consistent with known speech-related physiology, like an auditory N1 near one hundred milliseconds and a negative deflection between two hundred and three hundred milliseconds over central parietal regions during continuous listening.
Ines: The key finding here is that the preprocessed pipeline, which included adaptive filtering to attenuate eye movements and facial EMG, resulted in responses unlikely to be dominated by ocular or muscular activity.
Title and authors: Marcus: That’s a strong claim for their processing methods when dealing with the noise from overt speech. It shows that those signals are manageable when you have all three modalities together.
Yuki: So, what does this mean for us as a population geneticist? Can we really trust these findings to generalize beyond just Japanese native speakers?
Ines: The authors acknowledge that generalization across subjects, sessions, and recording hardware remains a central challenge in EEG research. They suggest foundation-style models for EEG twenty one through twenty five.
Marcus: That’s the core problem they are trying to solve with this dataset. The design matters not only in terms of participant count but also in terms of repeated recordings and device diversity, and the availability of synchronized auxiliary signals.
Yuki: It sounds like they are moving toward models that can adapt better to different recording environments, which is important when we look at how brain function relates to language development across populations.
Ines: The paper clearly states that public data for speech-related EEG remain relatively limited compared to other areas of research. Existing datasets have expanded the field substantially, but they are often focused on small prompt sets or imagined rather than overt speech recordings.
Marcus: And this dataset addresses that by including a hundred and two hours of diverse, real-world Japanese speech production across different EEG systems like g.Pangolin and g.SCARABEO.
Yuki: So, the implication is that by having this kind of multimodal data, we can start building foundation-style models that are more robust to variations in recording hardware and subject differences as they look at speech-related EEG21 through twenty five.
Ines: The overall conclusion is that this JapanEEG dataset addresses the need for synchronized data to improve artifact modeling, multimodal alignment, and representation learning for speech tasks. It shows how combining EEG with facial EMG and audio can lead to cleaner signals for studying speech physiology.
Marcus: For me, it confirms that having diverse hardware like eego sports alongside ultra-high-density systems is necessary if we want those foundation models to actually work reliably in the real world.
Yuki: From my side, it supports the idea that even within a specific population like native Japanese speakers, the variety of recording conditions and speech types provides enough material to build something useful for broader understanding.
Ines: So that’s what they show with this one thousand-hour EEG-EMG-audio dataset of Japanese speech production <ref:2606.01264#pg1,1000-hour EEG-EMG-audio dataset of Japanese speech production>. It’s a strong step forward in getting cleaner neural signals from complex tasks like overt speech.
Marcus: It is a lot of data, and it sets a new standard for how we should think about multimodal datasets in this field.
Yuki: I think the focus on overcoming those generalization challenges is what really matters for connecting these findings to larger questions about language and cognition.
The paper's summary: Ines: So, this paper presents a massive collection of data: one thousand two hours of synchronized brain waves, facial muscle movements, and speech audio from three healthy native Japanese speakers.
Marcus: It’s a big dataset because it includes simultaneous facial electromyography and audio during open-vocabulary speech. That combination is what they are really focusing on.
Yuki: The goal wasn't just to decode Japanese speech, but to improve how we understand the brain signals themselves related to speaking. They are trying to separate the brain activity from the physical noise of moving your face while you talk.
Ines: Exactly. They say this multimodal setup helps with artifact characterization and multimodal alignment. Think about it: you can see if a certain EEG pattern is caused by the muscles in your mouth or just by what your brain is actually doing for the speech itself.
Marcus: And they used this data to test some basic signal quality things first. They looked at power spectral density, which showed a one over f trend, which is normal for these kinds of signals.
Yuki: They also checked the alpha band, between eight hertz and thirteen hertz. The results showed that the power in that band flattened out during active tasks. That suggests a real change happening in the brain when someone is speaking.
Ines: And they did some event related potential analysis, looking at how brain activity changes right when speech starts. They found components time-locked to the onset of speech, like an auditory N1 near one hundred milliseconds and a negative deflection between two hundred and three hundred milliseconds over central parietal areas during continuous listening.
Marcus: The important part they highlight is that their preprocessed pipeline, which uses adaptive filtering to take out eye movements and facial EMG, resulted in responses that weren't dominated by those muscular or ocular activities. That’s a big win for cleaning up the data.
Yuki: From a population genetics standpoint, this dataset gives us more material from native Japanese speakers than we’ve had before, especially when you consider the longitudinal nature of the recordings and using different EEG systems like g.Pangolin and g.SCARABEO.
Ines: And they are trying to use this variety to build foundation style models for EEG twenty one through twenty five. The idea is that if a model learns how to handle noise across different hardware and subjects, it will be more useful when we try to apply it elsewhere.
Marcus: That’s the core challenge they are tackling: making sure these models generalize beyond just the specific recording session or device you used to get the data.
Yuki: So, what this means for us is that having a diverse and long set of multimodal speech data allows us to build better tools for studying how language and brain function vary across different people.
Ines: It’s a step toward having more robust methods for extracting clean neural signals from complex tasks like overt speech production. We need to keep looking at how these models handle those real-world noise challenges, though.
The paper's improvements: Tom: So we’re looking at how the authors of this dataset are thinking about making it even better, right? They aren't just handing over data; they're proposing ways to use that data that are smarter than just using it as is.
Ines: Right. The paper suggests three main areas for improvement, focusing on how we model the noise from speaking itself. First, they want to improve artifact modeling by using both facial electromyography and audio together.
Marcus: So, instead of just looking at one signal stream, they want to use the combination of face muscle activity and sound to better figure out what’s going on in the brain. That should help AI systems tell the difference between actual speech movements and extraneous noise from muscles.
Yuki: That makes sense for understanding how complex biological processes work. If you can isolate those specific neural signals related to articulation, you get a clearer picture of the underlying mechanism for speech production.
Ines: Exactly. They also push for better multimodal signal processing by using techniques like speech envelope alignment and EMG aware modeling. This lets AI models weave together information from EEG, facial electromyography, and audio all at the same time during the learning process.
Marcus: That sounds like it tackles the problem of getting representations that are accurate across different recording sessions and different devices. If a model learns to ignore device-specific noise patterns, that’s huge for generalization.
Yuki: And then there’s representation learning itself. They suggest incorporating longitudinal and cross device adaptation strategies directly into how we build these foundation style models for EEG twenty one through twenty five.
Ines: That means the models themselves should be built to adapt as they see more data from different sources, so they don't get stuck on what works only for that one specific recording setup or speaker.
Marcus: It’s about building something that is robust to those variations in hardware and subject differences. That directly addresses the challenge of getting those models to work reliably in a real world setting.
Yuki: So, it moves the focus from just collecting good data to actively designing systems that can handle messy, real-world biological signals consistently.
Ines: It’s about building tools that are less dependent on perfect pre-processing and more reliant on smart integration across all the available modalities.
Marcus: And that's where we need to look next: how do these improved models actually perform when we test them on new, unseen speech tasks?
Conclusion: Tom: So we’re wrapping up this discussion on "A one thousand-hour EEG-EMG-audio dataset of Japanese speech production". Basically, the paper shows that combining EEG with facial EMG and audio gives us a really powerful tool for studying speech signals.
Ines: Right. The main thing is that this multimodal data helps clean up the neural recordings by allowing us to better characterize the physical noise from speaking. It shows that you can get cleaner responses time-locked to the actual task onset, which is what we want for biological signal analysis.
Marcus: From a cohort perspective, it’s valuable because it addresses how different recording systems and subjects affect those signals. The study uses three different EEG systems, so they are testing if their findings hold up across various hardware setups.
Yuki: I think the significance is that this provides a much richer sample of overt speech data from native Japanese speakers than what we usually have for these specific tasks. It supports the idea that even within a single population, the variety of recording conditions gives us enough material to build something useful for broader understanding.
Ines: So, what this means is we can start building foundation style models that are more robust to those variations in recording hardware and subject differences when we look at speech-related EEG twenty one through twenty five.
Marcus: I agree. It’s a step toward making those AI tools more reliable outside of the controlled environment where the data was collected. But we still have to figure out how to make sure these models are actually accurate when applied to completely new populations or languages, right?
Yuki: That's definitely the next frontier. We need to see if this kind of robust multimodal signal processing can help us connect speech-related brain activity more deeply with broader questions about language development across different species.
Ines: Exactly. It’s a strong piece of data, but the real test is applying these improved methods to something completely different and seeing what they recover about the biology. That's what we need to keep chasing next.
Araya Inc. · University of California, San Francisco
q-bio.NC, cs.HC, cs.SD, eess.AS, eess.SP
Submitted: 2026-05-31
Updated: 2026-11-17
Comments: 22 pages, 4 figures. This version describes controlled access, voice-masked audio, and expanded technical validation. Dataset: https://doi.org/10.7910/DVN/5RM3OT
Code: https://github.com/Motoshige496/JapanEEG
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: A multimodal dataset of 1020 hours of simultaneously recorded scalp electroencephalography (EEG), facial electromyography (EMG)…
Key concepts
- Multimodal Dataset
- This refers to a dataset that combines different types of data—in this case, brainwaves (EEG), facial muscle activity (EMG), and sound recordings. Combining these sources provides a richer view of complex phenomena like speech production than using any single data type alone.
- Artifact Characterization
- This process involves identifying and understanding the non-speech signals in the recording, such as muscle noise from facial movements (EMG). Using simultaneous EMG and audio helps researchers precisely characterize these artifacts so they can be removed or accounted for during analysis.
- Speech-Related Physiology
- These are the specific brain responses that occur when a person produces speech. The study looked for known physiological markers, like an auditory N1 component near 100 ms, which helps confirm that the observed EEG changes are genuinely linked to the act of speaking.
- Cross-Device/Longitudinal Data
- The dataset is unique because it collects data across three different EEG systems and many sessions. This structure allows researchers to develop models that work well not just for one specific machine or one person, but can generalize across different recording hardware and subjects.
Terminology
Summary
A multimodal dataset of 1020 hours of simultaneously recorded scalp electroencephalography (EEG), facial electromyography (EMG), and speech audio from three healthy native Japanese speakers during open-vocabulary overt speech is presented to support both speech-related and broader EEG research. p. 0 A 1000-hour EEG–EMG–audio dataset of Japanese speech
production
Motoshige Sato1,†
, Ilya Horiguchi1
, Masakazu Inoue1
, Kenichi Tomeoka1
, Eri Hatakeyama1
, Yuya Kita1
,
Atsushi Yamamoto1
, Ippei Fujisawa1
, Shuntaro Sasai1,*
1Araya Inc. , Tokyo, Japan
Abstract
We present a multimodal dataset of 1020 hours of simultaneously recorded scalp electroencephalography (EEG), facial electromyography (EMG), and speech audio from three healthy
native Japanese speakers during open-vocabulary overt speech.Where the paper says it
Dataset Overview
The JapanEEG dataset consists of 1020 hours of synchronized EEG, facial EMG, and audio acquired during open-vocabulary overt Japanese speech from three healthy participants across many sessions and three EEG systems spanning 62–128 channels. The recordings were collected with speech decoding as a central motivation but their reuse value extends beyond speech decoding because they are longitudinal, multimodal, and cross-device. p. 1 The data were collected with speech decoding as a central
motivation, but their reuse value extends beyond speech decoding. Because the recordings are longitudinal, multimodal, and cross-device, they can support a broad range of studies, including EEG
preprocessing and artifact removal, EMG-aware modeling, speech-envelope alignment, representation learning, session and device adaptation, and robustness analyses. For this reason, simultaneous facial EMG and audio are valuable parts of the dataset rather than
ancillary metadata: they support artifact characterization, multimodal alignment, and analyses of
neural signals relative to the temporal structure of speech. JapanEEG (/dZæp@ni:dZi:/; Japanese + EEG) was assembled to address this combination of needs.Where the paper says it
Where the paper says it
Experimental Setup
The study involved three healthy male participants who were native Japanese speakers aged 22–44. The experimental paradigm required participants to perform overt and covert speech tasks, including playing a text-based television game while reading aloud every line of in-game dialogue, reading aloud from a book, or reading aloud sentences from a speech corpus for overt speech. p. 2 For overt speech sessions, the participant performed one of the following: (i)
playing a text-based television game while reading aloud every line of in-game dialogue, (ii) reading
aloud from a book, or (iii) reading aloud sentences from a speech corpus. For covert (imagined)
speech sessions, the participant’s own voice was first recorded; the participant then listened to this
recording and was instructed to reproduce it as imagined speech. For covert (imagined)
speech sessions, the participant’s own voice was first recorded; the participant then listened to this
recording and was instructed to reproduce it as imagined speech. No fixed time limit was imposed
on any session; participants continued at their own pace for as long as they were able. No fixed time limit was imposed
on any session; participants continued at their own pace for as long as they were able. EEG and EMG recording
Three scalp EEG systems were employed across the dataset (Table 3):
g. tec; 128 channels). An ultra-high-density system using a 128-channel subset selected
from a 1024-channel grid, targeting language and motor cortical areas. tec; 62 channels). A whole-brain EEG system sampling at 1200 Hz.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Technical Validation
Technical validation focused on basic checks of signal quality, including power spectral density (PSD) and event-related potential (ERP) analyses. PSD analysis showed a 1/f aperiodic trend characteristic of electrophysiological signals in both raw and preprocessed data, while the preprocessed spectra in the 8–13 Hz range showed a relatively flattened profile consistent with task-related power attenuation in the alpha band during active tasks. p. 6 The preprocessing pipeline attenuated non-neural artifacts,
including line noise visible as sharp peaks in the raw spectra, while preserving the broadband
spectral structure. In the 8–13 Hz range, the preprocessed spectra showed a relatively flattened
profile rather than the sharp peak typical of eyes-closed resting states, consistent with task-related
power attenuation in the alpha band during active tasks such as overt speech, covert speech, and
listening47. Spatiotemporal Validation of Event-Related Dynamics
To examine the temporal precision, spatial structure, and signal quality of the dataset, we analysed
the event-related potentials (ERPs) and their topographical voltage distributions time-locked to
task onset (Figures 3 and 4). Across participants, devices, and conditions, the pre-stimulus baseline
intervals showed low variance, indicating stable baselines and effective artifact reduction.Where the paper says it
Where the paper says it
Data Availability and Structure
The JapanEEG dataset is publicly available via OpenNeuro under the CC0 waiver in Brain Imaging Data Structure (BIDS) format. The total dataset size is approximately 955 GB, organized according to the BIDS-EEG specification with directories for raw data, processed data, and code. p. 5 The total dataset size is approximately 955
GB. Directory structure
The dataset follows the BIDS-EEG specification. Event files (events. tsv) include the onsets of speech segments, the
speech type (overt, covert, or listening), and the transcribed text generated by faster_whisper
(kotoba-whisper-v2.Where the paper says it
Where the paper says it
Technical Validation Details
The technical validation included power spectral density computation across participants, devices, and tasks to show the expected 1/f spectral profile and task-related alpha-band attenuation. Spatiotemporal validation of ERPs showed that evoked responses were observed time-locked to task onset, with components consistent with known speech-related physiology such as an auditory N1 near 100 ms and a negative deflection between 200 and 300 ms over central-parietal regions during continuous listening. p. 8 During the continuous listening task, Voice Activity Detection (VAD)-aligned averaging yielded a multiphasic response. Early
sensory components were present, including an auditory N1 near 100 ms 48, followed by a negative
deflection between 200 and 300 ms over central-parietal regions (Figures 3 and 4), consistent with
acoustic-phonological processing and the auditory representation of speech sounds within sensorimotor networks 49,50
. Because the preprocessing pipeline included adaptive
filtering to attenuate EOG and facial EMG, these components are unlikely to be dominated by
ocular or muscular activity. Together, these checks—stable baselines, smooth spatiotemporal evolution, and responses timelocked to task onset that are consistent with known speech-related physiology—indicate that the
recordings are of suitable quality for reuse. Technical validation included power spectral density and event-related potential analyses across
participants, devices, and tasks, which showed the expected 1/f spectral profile, task-related
alpha-band attenuation, and time-locked evoked responses. The dataset is released in Brain
Imaging Data Structure (BIDS) format via OpenNeuro under a CC0 waiver to support both
speech-related and broader EEG research.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Artifact modeling for speech production can be improved by using simultaneous facial EMG and audio to support
artifact characterization
andmultimodal alignment.
This allows AI systems to better distinguish between neural signals related to speech articulatory kinematics versus extraneous muscular noise, leading to more robust signal processing pipelines. -
Multimodal signal processing can be enhanced by employing techniques such as
speech-envelope alignment
andEMG-aware modeling.
This enables AI models to integrate information from EEG, facial EMG, and audio concurrently, improving the accuracy of representations learned for speech tasks across different recording sessions and devices. -
Representation learning in EEG systems can be improved by incorporating longitudinal and cross-device adaptation strategies derived from the dataset's structure. This allows for building
foundation-style models
that are more robust to variations in recording hardware and subject differences, addressing the challenge ofgeneralization across subjects, sessions, and recording hardware.
Abstract
We present a multimodal dataset of 1020 hours of simultaneously recorded scalp electroencephalography (EEG), facial electromyography (EMG), and speech audio from three healthy native Japanese speakers during open-vocabulary overt speech, covert repetition and speech listening. Recordings were acquired in many sessions over several months with three EEG systems of 61-128 channels, an ultra-high-density system (g.Pangolin) and two cap-type systems (g.SCARABEO and eego sports). Each session provides time-synchronized EEG, facial EMG, and audio, together with speech-event annotations and transcriptions. Although collected with speech decoding as a primary motivation, the dataset also supports work on multimodal signal processing, artifact modeling, longitudinal and cross-device adaptation, and EEG representation learning. Technical validation used power spectral density and event-related potential analyses to characterise spectral profiles and time-locked responses across participants, devices, and tasks. The dataset is available in Brain Imaging Data Structure (BIDS) format from Harvard Dataverse under controlled access for research use. Its speech audio is voice-masked.
Related papers
- Self-Supervised Learning of Local Predictive Directions as a Neural Computation
- Synchronized Hearts, Desynchronized Brains: A Nonlinear Dynamics Perspective for Function and Failure
- Message interaction dynamics covary with brain volume across mammalian connectomes
- Planning as Dynamics Relaxation: Hippocampal Recurrent Network Realizes Optimal Goal-Directed Navigation
- The Platonic brain bridge hypothesis: brain networks as architectural priors for multimodal large language models
- BrainPFN: Amortized Brain-Behaviour Prediction via Prior-Data Fitted Networks