Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learning
stat.ML, cs.AI, cs.LG
Submitted: 2026-02-04
Updated: 2026-11-15
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks.
Terminology
Abstract
Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks. However, existing results focus exclusively on unimodal data; in contrast, the theoretical underpinnings of in-context learning for multi-modal data remain poorly understood. We introduce a mathematically tractable framework for studying multi-modal learning and explore when transformer-like architectures can recover Bayes-optimal performance in-context. To model multi-modal problems, we assume the observed data arises from a latent factor model. Our first result comprises a negative take on expressibility: we prove that single-layer self-attention models fail to recover the Bayes-optimal predictor uniformly over the task distribution. To address this limitation, we introduce a novel, linearized cross-attention mechanism, which we study in the regime where both the number of cross-attention layers and the context length are large. We show that this cross-attention mechanism is provably Bayes optimal when optimized using gradient flow. Our results underscore the benefits of depth for in-context learning and establish the provable utility of cross-attention for multi-modal distributions.
Sources
- High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification
- Contrastive Learning on Multimodal Analysis of Electronic Health Records
- Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
- Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
- In-Context Convergence of Transformers
- Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
- Continual Multimodal Contrastive Learning
- Optimal In-context Adaptivity and Distributional Robustness of Transformers
- Spectral Thresholds in Correlated Spiked Models and Fundamental Limits of Partial Least Squares
- Multimodal data integration and cross-modal querying via orchestrated approximate message passing
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference