Token Space: A Category Theory Framework for AI Computations

arXiv:2404.11624 · math.GM, cs.LG · Submitted 2024-04-11 · Read on arXiv

math.GM, cs.LG

Submitted: 2024-04-11

Updated: 2026-10-02

Terminology

• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• Categories of Optics
• Category Theory in Machine Learning
• Attention Is All You Need
• Training language models to follow instructions with human feedback
• The Llama 3 Herd of Models
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• LLaMA: Open and Efficient Foundation Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Root Mean Square Layer Normalization
• GLU Variants Improve Transformer
• Distilling the Knowledge in a Neural Network
• A Knowledge Compilation Map
• Monoidal computer I: Basic computability by string diagrams
• A formally verified compiler back-end
• egg: Fast and Extensible Equality Saturation
• Functorial Data Migration
• Eliciting Latent Predictions from Transformers with the Tuned Lens
• Exponentiability in categories of relational structures
• On the axioms for adhesive and quasiadhesive categories
• Circuits via topoi