Beyond the Parameter Monolith: Reconstructive Memories, Executable Skills, and Residual Assembly for Language Models
cs.AI
Submitted: 2026-10-02
Updated: 2026-10-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language-model systems can separate contextual computation, persistent storage, and exact execution instead of updating all capabilities through one shared parameter system.
Abstract
Language-model systems can separate contextual computation, persistent storage, and exact execution instead of updating all capabilities through one shared parameter system. We investigate FEM-ASM, a finite-element-method-inspired organization in which independently constructed document states and deterministic executable skills contribute typed proposals to a shared language-model state. An explicit residual operator reconciles proposals attached to common interface nodes. We evaluate this organization through controlled experiments and negative results rather than claiming a physical finite-element formulation of language. An attention-free Multi-Mesh prototype learns causal language modeling but does not establish competitive general capability. A versioned store contains 52,809 reconstructive memory elements near a 1.7-billion-floating-value budget; reconstruction is incomplete, with approximately 75% token accuracy. Support-aware lexical indices make these elements addressable under provenance-controlled query construction. For executable arithmetic, positional result observations substantially improve neural rendering relative to a repeated global result vector, and output substitutions change the model's preferred answer. A bounded attachment demonstration further measures the effect of making selected evidence available, without establishing the utility of loading an entire multi-billion-value store. The results support a separation of storage, execution, and neural coordination, while identifying unresolved limitations in question-only retrieval, unrestricted answer generation, and end-to-end efficiency.
Sources
- Deep Equilibrium Models
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Measuring Massive Multitask Language Understanding
- Generalization through Memorization: Nearest Neighbor Language Models
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Compressive Transformers for Long-Range Sequence Modelling
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Attention Is All You Need
- Memorizing Transformers
- Deep Sets
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems