Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanism
cs.LG
Submitted: 2026-09-28
Updated: 2026-09-28
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- The Bayesian Geometry of Transformer Attention
- Transformers Simulate MLE for Sequence Generation in Bayesian Networks
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective
- Amortizing intractable inference in large language models
- A Latent Space Theory for Emergent Abilities in Large Language Models
- CLadder: Assessing Causal Reasoning in Language Models
- Finding Belief Geometries with Sparse Autoencoders
- What Does it Mean for a Neural Network to Learn a "World Model"?
- Dual Traits in Probabilistic Reasoning of Large Language Models
- Transformers Can Do Bayesian Inference
- How Transformers Learn Causal Structure with Gradient Descent
- Constrained belief updates explain geometric structures in transformer representations
- Transformers learn factored representations
- The information bottleneck method
- Chess as a Testbed for Language Model State Tracking
- Revisiting the Othello World Model Hypothesis
- Deep de Finetti: Recovering Topic Distributions from Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks