Prediction Is Not a Sufficient Memory State

The gist The minimal exact state of a learner is governed by its continuation-identifiable evidence and not the size of its current predictor.

How it works

The paper characterizes the distinction between a memory that reproduces every current prediction and one that can learn from the next observation, showing this distinction for exact cumulative learning and attention The central results are a native, unweighted loss classification and a quantitative finite-bit law

Ridge Regression Analysis

For ridge regression, the hidden evidence is the design matrix Theorem 1 establishes that the minimum continuous read-only dimension is dp, while supporting all present readouts and one arbitrary new ridge observation requires exactly m = dp + sd, where sd = d(d + 1)/2 On zero-label prefixes, the exact continuationsummary dimension is sd The attaining algorithm is recursive least squares which is not a new architecture

Native State Identification

For ordinary, unweighted residual observations, the required memory dimension depends on the function space of gradient functions Theorem 2 states that two histories have identical optima after every single additional observation if and only if their accumulated gradient functions agree The exact minimum continuous recurrent state dimension is r, where r = dim Gρ, or no fixed finite dimension suffices if r = ∞

Polynomial Closure and Loss Types

Theorem 3 shows that an exact continuous recurrent state of size independent of history length exists if and only if the loss is polynomial For degree D ≥ 1, its minimum dimension is rD(d) = d + D/(D − 1) For every nonpolynomial smooth convex residual loss, the required dimension is unbounded even on equal-prediction histories

Attention and Kernel Evidence

Kernel attention provides a second exact closure classification Theorem 8 identifies the exact minimum continuous continuation-state dimension as rκ(p+1) when finite, otherwise no fixed finite dimension suffices For unweighted exponential attention, with queries in a nonempty open subset of R d, a length-N prefix requires exactly N(d + p) coordinates for one-token continuation

Finite-Bit Memory Bounds

Theorem 6 provides a quantitative finite-bit law for matching average finite-bit risk The lower bound allows private randomization and independent shared randomness, while the upper bound is an offline prefix quantizer with a coherent ridge decoder

Evidence Preservation

The results specify what future updates reveal, rather than proposing a new regression algorithm or asserting trained-model gains Proposition 9 provides a gradient-profile certificate that survives common updates under certain conditions

Conclusion on State Size

The minimal exact state of a learner is governed by its continuation-identifiable evidence and not the size of its current predictor Ordinary residual observations give a polynomial/nonpolynomial closure dichotomy and an exact count of evidence hidden at a fixed predictor Ridge regression makes this information quantitative at finite precision under random future queries Attention exposes the same distinction through its numerator and normalization mass The minimal exact state of a learner is governed by its continuation-identifiable evidence and not the size of its current predictor.

Conclusion on State Size

The minimal exact state of a learner is governed by its continuation-identifiable evidence and not the size of its current predictor. Ordinary residual observations give a polynomial/nonpolynomial closure dichotomy and an exact count of evidence hidden at a fixed predictor Ridge regression makes this information quantitative at finite precision under random future queries Attention exposes the same distinction through its numerator and normalization mass The minimal exact state of a learner is governed by its continuation-identifiable evidence and not the size of its current predictor.

Conclusion on State Size

The minimal exact state of a learner is governed by its continuation-identifiable evidence and not the size of its current predictor.

1. Complete state identification for native learning updates: The system can identify exactly which evidence is required for continuation, as The results identify what future updates reveal, rather than proposing a new regression algorithm or asserting trained-model gains. This allows the system to determine if a history is sufficient to update the predictor without needing all past data.

2. Exact dimension characterization for polynomial losses: For smooth convex residual losses with unrestricted real features and labels, the system can determine exactly when the loss is polynomial and calculate its minimum dimension as rD(d) = d + D / (D - 1). This provides a theoretical upper bound on the memory required for exact cumulative learning.

3. Optimal finite-bit compression for ridge regression: The system can implement an optimal B-bit compression with a guaranteed error of order d(-12 - 2B/sd), where sd = d(d + 1)/2, even with arbitrary randomized codes. This allows for highly efficient, provably accurate memory storage under specific conditions.

4. Native state identification via gradient profiles: The system can determine if two histories have identical optima after every single additional observation if and only if their accumulated gradient functions agree, characterizing an update-closed state as one where h ∼1 h′ means equality of all present predictions, and h ∼1 h′ equality after every single native observation.

5. Distinguishing nonpolynomial losses: The system can identify that Nonpolynomial losses have unbounded requirements on such equal-prediction histories, providing a clear criterion for when fixed-size memory is insufficient for exact learning.

6. Kernel attention analysis: The system can use kernel attention to identify the parallel role of unnormalized kernel evidence by extracting the difference between two continued outputs, which is used to recover Z and B from a single token appended.

7. Exact reconstruction of ridge states: For ridge regression, the system can store the full state required for exact continuation, which includes (C + kk⊤, M + yk⊤) and maintains a cost of O(d2 + pd) for updates.

8. Adaptive query selection based on evidence geometry: The system can use K exact response geometry to select an active basis from the current values and query, which is computed from the observed geometry to identify the necessary coordinates.

A memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial. With ordinary, unweighted observations, degree D requires precisely r= D-1 coordinates; even among histories with one fixed current predictor, exactly r-d coordinates can remain necessary for continuation. Nonpolynomial losses have unbounded requirements on such equal-prediction histories. Bounded scalar logistic examples retain this obstruction after fixing any finite derivative jet. For ridge regression, the hidden evidence is the design matrix: the exact state costs dp+d(d+1)/2, versus dp for the predictor. On a well-conditioned zero-prediction family, optimal B-bit compression has average one-update error of order d-12-2B/s d, s d=d(d+1)/2, even with arbitrary randomized codes. Kernel attention gives a second exact closure classification. The results identify what future updates reveal, rather than proposing a new regression algorithm or asserting trained-model gains.