A Tool in the Forward Pass: Frozen Tape Modules as Layers of a Small Language Model
cs.AI, cs.NE
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: 12 pages, 1 figure, 9 tables. Code, configurations, results and frozen modules: https://doi.org/10.5281/zenodo.23831494
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Calling a tool through generated text adds decoding steps, even when the computation itself is short.
Terminology
Abstract
Calling a tool through generated text adds decoding steps, even when the computation itself is short. We study an alternative: a small neural module that runs inside a language model's forward pass. A learned writer turns hidden states into a symbolic tape, a short sequence of discrete symbols that encodes the computation. The module, trained separately and then frozen, reduces the tape to a result, and a learned reader returns that result to the model. We evaluate arithmetic and object tracking in gemma-3-1b-it. With a writer trained on labelled tapes, the layer answers all 750 items of BIG-Bench Hard's tracking shuffled objects correctly. At seven people it needs about 1/10 of the GPU kernel time of a text-tool baseline that uses the same host model and exact code, with accuracy 1.00 against 0.99. The writer can also be learned from answers alone, guided by a hand-written copying prior, but less reliably: 5 of 6 runs succeed in the 1B host and 2 of 4 in the 4B host. Two independently trained layers keep their task accuracy when attached to one model, and a learned gate keeps them silent on the tested general-text and look-alike prompts. The results support a narrow use case: a bounded computation that returns one symbol can avoid the decoding cost of a textual tool call. Extracting the right input reliably, and deciding when to act, remain the main open problems.
Sources
- LLM Augmented LLMs: Expanding Capabilities through Composition
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Transformers meet Neural Algorithmic Reasoners
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
- Training Verifiers to Solve Math Word Problems
- Implicit Chain of Thought Reasoning via Knowledge Distillation
- IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently
- OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step
- PAL: Program-aided Language Models
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Gemma 3 Technical Report
- From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
- Training Large Language Models to Reason in a Continuous Latent Space
- LoRA: Low-Rank Adaptation of Large Language Models
- Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
- Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
- Bridging Discrete and Backpropagation: Straight-Through and Beyond
- Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents