Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

Autoregressive language models are trained by next-token cross-entropy, yet several recent systems also iterate a hidden state without emitting a token: continuous thoughts, pause steps, and recurrent depth. This note separates three risks on one latent process. The token risk R tok scores a readout. The teacher-forced latent risk R 1 scores a one-step map that sees the true hidden state. The open-loop risk R H scores the same map after it has been iterated on its own outputs. If the Bayes next-token posterior is a function of a k-dimensional statistic with full-rank covariance, a width- k latent, a linear least-squares transition, and a refit unembedding match the Bayes token predictor. The statistic is identified only up to an invertible gauge: whitening the residual stream does not change the token law once the unembedding is refit. The matching obstructions are sharp. A constant latent drives R 1 to zero while token cross-entropy falls back to the marginal entropy. A perfect token predictor can still hide a null-space walk whose open-loop error grows linearly in the horizon. A small next-latent loss does not certificate next-token prediction, and a small next-token loss does not certificate free latent rollout, unless a rank or injectivity condition is stated with the number.