A Geometry-Based Capacity Theory for Finite-Feature Associative Memory

The gist: For exact-key retrieval in compressed random-feature Hebbian memory, retrieval interference separates into a finite-feature component and a structural component determined by kernel overlap among stored keys.

How it works

The theory develops a geometry-based capacity theory for exact-key retrieval in compressed finite-feature Hebbian associative memory The framework links representation geometry directly to memory capacity and distinguishes when performance can be improved by increasing the feature budget and when the representation itself must be changed For random or approximately isotropic values, retrieval interference separates into finite-feature noise, which decreases with feature dimension, and structural interference, which is determined by squared kernel overlap among stored keys. This yields a fit-free prediction of retrieval quality, reveals a geometry-dependent capacity ceiling, and predicts the feature budget required for a target retrieval quality.

Key Theoretical Components

The analysis separates two sources of error: finite-feature noise and structural interference caused by the geometry of the stored representations. For random or approximately isotropic values, the leading approximation separates retrieval interference into a finite-feature term, approximately (N − 1)/R, and a structural term Pj!=i K2ij. Intuitively, this structural term measures how strongly the other stored items overlap with item i under the chosen kernel. Once the kernel, stored keys, number of items N, and feature budget R are specified, no coefficient is fitted to the measured retrieval curve.

Capacity Laws and Predictions

The simple law separates the leading finite-feature and structural terms: Itotal,i ≈ (N − 1)/R [z] finite-feature + Xj!=i K2ij [z] structural. As R → ∞, consistency removes the finite-feature term and gives the key-only mean-field ceiling Ci,∞ ≈ [1 + Xj!=i K2ij]−1/2. Increasing R reduces finite-feature interference but cannot remove the structural ceiling set by key geometry. For a target cosine C⋆, a memory-design boundary is obtained by solving C(R/N) = C⋆ for R/N.

Correlated Values and Joint Geometry

When values are correlated, the infinite-feature retrieval vector becomes a kernel-weighted mixture of stored values, so its target cosine depends jointly on the key kernel and the value Gram matrix. The deterministic key–value geometry is characterized exactly, and only the finite-R perturbation is approximated. The infinite-feature retrieval cosine is exactly Ci,∞ = ai/sqrt(bi).

Experimental Validation

The theory was validated on synthetic, visual, and medical-image representations. Experiments tested ResNet50 and ViT-B/16 embeddings under RAW, CENTERED, and PCAWHITENED geometry interventions. The framework successfully separates two different remedies for poor retrieval: increasing the finite feature budget when approximation noise dominates, or changing the representation when structural interference sets the limiting ceiling.

Practical Implications

The theory suggests a practical sequence for system design: first measure kernel overlap in the intended representation, then use the predicted capacity curve to determine whether the target retrieval quality is reachable at the available feature budget. The framework demonstrates that measured kernel geometry is a more informative guide to memory sizing and representation selection than nominal dimension or effective rank alone.

Limitations

The analysis is restricted to exact-key queries; robustness to corrupted, partial, or shifted cues is not modeled. Capacity is defined operationally through target retrieval cosine and feature budget R/N, not as an asymptotic patterncount capacity for other memory mechanisms. The covariance-aware correction remains approximate and retains residual crossover bias, while the simple finite-R law is less accurate for strongly correlated values at very low R, where expectation-of-ratio effects are non-negligible. The medical stress tests use 2-D axial slices and generalpurpose encoders rather than 3-D or medically specialized representations, and they evaluate representation geometry and memory interference rather than clinical prediction.

Conclusion

For exact-key retrieval in compressed random-feature Hebbian memory, retrieval interference separates into a finitefeature component and a structural component determined by kernel overlap among stored keys. The resulting law predicts target-cosine capacity across synthetic, naturalimage, and medical representation geometries, including a public matched-N SynthRAD2023 MRI–CT stress test and an independent NCCT–CTA confirmation. For correlated values, the deterministic retrieval geometry is governed jointly by the key kernel and value Gram matrix, while only the finiteR perturbation is approximated. The same quantities can also estimate the memory budget required for a target retrieval quality, rank candidate representation and kernel configurations, and localize design crossovers. Within this storage regime, the practical value of the theory is therefore not only to describe capacity but also to distinguish when a system is limited by insufficient finite-feature budget from when it is limited by representation geometry itself. The paper's main contribution is analytical rather than architectural: a geometry-dependent capacity law for compressed finite-feature memory, an extension to correlated values through joint key–value geometry, and a test of whether the same quantities can guide memory sizing and representation selection.

Data and Code Availability

The SynthRAD2023 Task-1 training data used for the public medical validation are publicly available from the challenge dataset release. Reproduction code and aggregate source tables for the synthetic, natural-image, SynthRAD2023, and independent cohort analyses will be released publicly upon publication at https://github.com/jianhai-zhang/memory. The independent cohort analysis uses paired NCCT–CTA examinations from the AcT (Alteplase compared to Tenecteplase) randomized controlled trial. Analyses used de-identified data under institutional datagovernance requirements. The public experiment is limited to the SynthRAD2023 Task-1 training split. The independent 445-pair NCCT–CTA cohort gives a Raw rank=6.5 C∞=0.652 Structural interference Istruct=1.66. The independent NCCT/CTA cohort gives a Raw rank=6.5 C∞=0.732 Structural interference Istruct=1.354. The independent NCCT/CTA cohort gives a Raw rank=768 C∞=0.779 Structural interference Istruct=1.26.

Appendix A

For a normalized real shift-invariant positive-definite kernel K(x, z) = κ(x − z) with K(x, x) = 1, Bochner’s representation implies that κ(δ) can be written as an expectation of Fourier features. The broad capacity validations use random-phase cosine features followed by row normalization, as specified in Sec. 4.4. For the covariance analysis uses a paired sine/cosine construction because its second moments admit the clean expression below.

Appendix B

The exact infinite-feature geometry is defined by y∞,i = XjKijvj. The finite-R cosine is exactly Ci(R) = ai + ⟨ϵi, vi⟩p/bi + 2⟨y∞,i, ϵi⟩ + ϵi squared.

1. Stable retrieval quality prediction based on representation geometry: The system can determine when increasing the feature budget stop helping because the remaining interference is caused by the representation geometry itself, thus guiding whether to increase memory or change representation, as stated in Section 6.1.

2. Geometry-dependent capacity ceiling estimation: The AI can predict the feature budget required for a target retrieval quality, revealing a geometry-dependent capacity ceiling, allowing for targeted memory provisioning based on the measured kernel geometry rather than just nominal dimension or effective rank alone, as noted in Section 9.

3. Correlated value handling via joint geometry analysis: For systems dealing with correlated data, the model can determine the deterministic key–value geometry and use this to derive a joint finite-R law, which is more accurate than the key-only approximation when values are correlated, as detailed in Section 6.4.

4. Optimal representation selection guided by theory: The system can select between different preprocessing and kernel choices by ranking them using their predicted capacity curves, providing a quantitative measure of which representation—RAW, CENTERED, or WHITENED—is best for a fixed memory budget, as suggested in Section 5.8.

5. Crossover prediction for design choices: The AI can estimate the RAW-to-WHITENED crossover by utilizing the covariance-aware theory, which is shown to be closer to empirical crossovers and reduces overall MAE by 84.5% compared to the simple theory, as summarized in Section 5.9.