CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings

arXiv:2605.10337 · cs.AI, eess.SP · Submitted 2026-05-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings".

Jane: The gist The CORTEG framework introduces a cross-modality transfer method that adapts large pretrained EEG foundation models to ECoG, enabling efficient, cross-patient learning for intracranial brain interfaces.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re wrapping up on this paper, "CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings." It’s about taking those large, general scalp EEG models and making them work for brain recordings from inside the skull, ECoG.

Jane: It’s really neat how they tackle that modality gap by using a cross-modality transfer framework to bridge the difference between what you see on the scalp and what's happening deep inside.

Lu: The authors used a pretrained EEG foundation model as a backbone and then hooked it up with an electrode-aware spatial adapter to map those ECoG coordinates into that learned space.

Meng: From my side, I’m looking at how they handle the data streams, decomposing the ECoG signal into low frequency stuff and high gamma activity. That’s important for deployment because you can’t just feed raw data in there.

Lalam: What I see is a dual-stream tokenizer that handles both those low frequency signals and the high gamma activity separately before they get merged for decoding.

Tom: And they fine-tuned this whole thing using a parameter-efficient strategy called leave-one-subject-out with LoRA to adapt it to a new patient really fast. That’s what makes it practical.

Jane: The results show that this setup achieves state of the art mean performance on both their regression tasks, which were finger trajectory and audio envelope decoding.

Lu: They found that the pretraining from scalp EEG actually gave them gains over just starting with a random model architecture for ECoG.

Meng: But I’m watching those numbers too. The gain is bigger on the audio task and when you only have a small number of subjects to calibrate to, which is where I care about deployment at scale.

Tom: It means we don't have to build entirely new decoders from scratch for every single patient; we can repurpose existing knowledge.

Jane: So what does this mean for us? It suggests that pretraining on one type of brain signal could actually be a really powerful starting point for understanding another, different recording modality.

Conclusion: Tom: So we’re wrapping up on CORTEG, which is about taking those big, general scalp EEG models and making them work for brain recordings from inside the skull, ECoG.

Lu: The authors used a pretrained EEG foundation model as a backbone and then hooked it up with an electrode-aware spatial adapter to map those ECoG coordinates into that learned space.

Lalam: What I see is a dual-stream tokenizer that handles both those low frequency signals and the high gamma activity separately before they get merged for decoding.

Tom: And they fine-tuned this whole thing using a parameter-efficient strategy called leave-one-subject-out with LoRA to adapt it to a new patient really fast. That’s what makes it practical.

Jane: The results show that this setup achieves state of the art mean performance on both their regression tasks, which were finger trajectory and audio envelope decoding.

Lu: They found that the pretraining from scalp EEG actually gave them gains over just starting with a random model architecture for ECoG.

Tom: It means we don't have to build entirely new decoders from scratch for every single patient; we can repurpose existing knowledge.

Lu: The latent representations they found suggest the pretrained model learns a lower-dimensional structure for finger movements, which is more compact than what random training would yield.

Meng: That compression is good because it means the model needs fewer parameters to learn from just one new patient’s data.

Tom: We still have limitations on things like MNI space localization and they didn't show how well it transfers across different types of tasks.

Jane: It’s a solid step, definitely showing that scalp EEG pretraining isn't just some isolated technique; it can be repurposed for intracranial recording decoding in a way that makes adaptation much more efficient.

Liuyin Yang, Qiang Sun, Bob Van Dyck, Eva Calvo Merino, Marc M. Van Hulle

Laboratory for Neuro- & Psychophysiology, Department of Neurosciences, KU Leuven

cs.AI, eess.SP

Submitted: 2026-05-11

Updated: 2026-10-14

Comments: Camera-ready version with new BrainTree Bank experiments and comparisons with iEEG FMs

Code: https://github.com/LiuyinYang1101/CORTEG

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The CORTEG framework introduces a cross-modality transfer method that adapts large pretrained EEG foundation models to ECoG, enabling efficient, cross-patient learning for intracranial brain

Key concepts

EEG FM Backbone: This is a large, pretrained foundation model specifically trained on scalp EEG data. It learns general temporal representations of brain activity patterns. This pretraining provides a strong starting point for understanding neural dynamics across different recording modalities.
KNNSoftFourier Spatial Adapter: This component maps the physical locations of ECoG electrodes onto the learned representation space of the pretrained EEG model. It ensures that the model can effectively utilize electrode-specific spatial information from ECoG recordings by aligning it with its existing knowledge base.
Dual-Stream Tokenizer: Raw ECoG signals are split into two streams: low-frequency signals and high-gamma activity. The tokenizer processes these separately, allowing the model to leverage the EEG FM's strength in handling low frequencies while incorporating the high-gamma dynamics crucial for decoding intracranial data.
Leave-One-Subject-Out Fine-Tuning (LOOFT): This is a parameter-efficient strategy where a population model is adapted to a single new patient by fine-tuning only a small subset of parameters. This allows the system to calibrate to an individual in minutes, using existing knowledge from many patients without needing extensive retraining.

Terminology

Foundation Model — 2× in this paper · explained in 3 episodes
Foundation Models — 2× in this paper · explained in 7 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Calibration — 1× in this paper · explained in 7 episodes
Decoder — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes

Summary

The gist The CORTEG framework introduces a cross-modality transfer method that adapts large pretrained EEG foundation models to ECoG, enabling efficient, cross-patient learning for intracranial brain interfaces.

How it works

CORTEG is a CrOss-Modality Representation Transfer framework that adapts a pretrained EEG FM to ECoG. It combines several key components to bridge the modality gap between scalp EEG and ECoG.

The architecture involves several technical innovations:

Pretrained Backbone and Spatial Adapter: The framework uses ST-EEGFormer as the pretrained EEG FM backbone, which learns temporal representations, coupled with a KNNSoftFourier spatial adapter that maps ECoG electrode coordinates into the pretrained EEG channelembedding space.
Dual-Stream Tokenizer: Raw ECoG is decomposed into low-frequency signals (LFS) and high-gamma activity (HGA), which are then patch-embedded, combined with adapted spatial embeddings, and passed through the pretrained EEG encoder. The dual-stream design utilizes the EEG FM’s ability on low-frequency signals while augmenting them with high-gamma dynamics essential for ECoG decoding.
Late Fusion and LoRA Adaptation: The low stream traverses the first L−K transformer layers (frozen), while at block L−K, the high-gamma tokens are averaged with the encoder-processed low-stream tokens, Zmerged= 1/2(Z(L−K)lo +Zhi), and the merged sequence traverses the remaining K layers jointly. The model adapts those K shared layers via LoRA, where trainable parameters are LoRA A, B, the spatial adapter, and the regression head.

Key Contributions

The paper introduces CORTEG with three primary contributions based on empirical strength:

Cross-subject deployment regime: This involves parameter-efficient leave-one-subject-out finetuning (LOOFT) with LoRA, which adapts a population CORTEG model to a new patient in 10–30 minutes on a single GPU while matching pooled training performance.
Cross-modality pretraining signal: Under matched per-subject baselines, scalp-EEG pretraining produces task-dependent gains over a randomly initialized backbone of the same architecture.
Best mean correlation among compared methods: CORTEG reaches r=0.554 on the public Stanford finger benchmark and r=0.339 on the private audio-envelope dataset, matching or exceeding prior decoders.

Performance and Analysis

The evaluation was conducted on two challenging regression tasks: public finger trajectory regression (n=9) and private audio envelope regression (n=16).

**: CORTEG pooled training achieves the highest mean Pearson r on both tasks, reaching r=0.554 on the finger task and r=0.339 on the audio task. The gain is not statistically significant on n=9 subjects for finger decoding, but larger gains are seen in the audio task and in low-data per-patient calibration. LOO-FT reaches pooled-level performance on both tasks, suggesting an existing population model can be calibrated to a new patient via Stage-2 finetuning without retraining the multi-subject pooled model. The adaptation is parameter-efficient, updating only 297 K parameters through LoRA and the spatial adapter. The analysis of neural manifolds shows that pretrained representations form a lower-dimensional, finger-separable manifold (participation ratio PR=8.6) compared to random initialization (PR=15.2). This suggests that pretraining provides a more compact, task-aligned subspace. The channel importance maps align with neurophysiology, separating weights by task, such as clustering over the sensorimotor strip for finger trajectory weights. The results provide systematic evidence that scalpEEG pretraining can be repurposed for ECoG decoding. The main advantage is a more practical deployment strategy where a new patient can be calibrated in minutes rather than hours. The limitations include MNI-space localization and the absence of cross-task transfer. The paper concludes that pretraining provides a structured, low-dimensional prior alignable to ECoG, rather than merely a favorable initialization. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr. Dirk Van Roost, and Kato Van Rooy — for performing the electrode implantations and for their assistance in recording the intracranial audio-envelope data from the participating patients. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr. Dirk Van Roost, and Kato Van Rooy — for performing the electrode implantations and for their assistance in recording the intracranial audio-envelope data from the participating patients. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr. Dirk Van Roost, and Kato Van Rooy — for performing the electrode implantations and for their assistance in recording the intracranial audio-envelope data from the participating patients. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr. Dirk Van Roost, and Kato Van Rooy — for performing the electrode implantations and for their assistance in recording the intracranial audio-envelope data from the participating patients. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr. Dirk Van Roost, and Kato Van Rooy — for performing the electrode implantations and for their assistance in recording the intracranial audio-envelope data from the participating patients. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr. Dirk Van Roost, and Kato Van Rooy — for performing the electrode implantations and for their assistance in recording the intracranial audio-envelope data from the participating patients. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. The neurosurgery and neurology team at Ghent University Hospital — in particular Dr. Alfred Meurs, Dr. Evelien Carrette, Dr.

Improvements for AI systems

Cross-modality Adaptation for New Patients: The CORTEG framework enables a population model can be trained once and efficiently adapted to new patients, with a new patient calibrated in 10–30 minutes on a single GPU. This allows BCIs to become more practical by avoiding the need to retrain a full per-subject decoder from scratch.
Enhanced Decoding Performance Across Modalities: The system achieves performance metrics that match or exceed the strongest task-specific baselines on both tasks, specifically reaching r=0.554 on the public finger benchmark and an improved correlation of r=0.339 on the private audio-envelope dataset compared to CNN-LSTM (0.261).
Task-Specific Feature Extraction: The dual-stream tokenizer allows the model to utilize both low-frequency and high-gamma activity, as demonstrated by the finding that Dual-stream fusion beats either stream alone, suggesting a more comprehensive understanding of neural signals relevant to different decoding tasks.
Parameter Efficient Adaptation: By employing Parameter-efficient leave-one-subject-out fine-tuning (LOOFT) with LoRA, the model adapts efficiently, as it updates only 297 K parameters through LoRA and the spatial adapter, fewer than all per-subject deep baselines (334 K–1.16 M).
Task-Aligned Representation Learning: The analysis suggests that pretraining provides a more compact, finger-separable manifold, as the pretrained model exhibits a lower participation ratio (PR=8.6 vs. 15.2 for random init), implying that representations are task-appropriate through the regression objective alone.