ZetaGPT: A Reference Implementation of Positional–Encoding–Free State–Space–Attention Language Models

Name: ZetaGPT: A Reference Implementation of Positional–Encoding–Free State–Space–Attention Language Models
Uploaded: 2026-08-11T12:22:50.099879+00:00
Duration: 1661 s
Description: The hosts discuss ZetaGPT, a small language model without explicit positional encodings, using a state-space module before attention to encode order implicitly. They cover the architecture, training pipeline, tokenizer dynamics, and comparisons to other models, concluding it's a reproducible reference implementation for research.

The hosts discuss ZetaGPT, a small language model without explicit positional encodings, using a state-space module before attention to encode order implicitly. They cover the architecture, training pipeline, tokenizer dynamics, and comparisons to other models, concluding it's a reproducible reference implementation for research.

Open the full episode