Program-as-Weights: A Programming Paradigm for Fuzzy Functions
cs.LG, cs.AI, cs.CL
Submitted: 2026-07-02
Updated: 2026-11-07
Comments: NeurIPS 2026 camera-ready version
Code: https://github.com/ggerganov/llama.cpp
License: http://creativecommons.org/licenses/by/4.0/
The gist: Many everyday programming tasks resist clean rule-based implementation, such as alerting on important log lines, repairing malformed JSON, or ranking search results by intent…
Terminology
Abstract
Many everyday programming tasks resist clean rule-based implementation, such as alerting on important log lines, repairing malformed JSON, or ranking search results by intent, and are increasingly outsourced to large language model APIs at the cost of locality, reproducibility, and price. We propose fuzzy-function programming: compiling such a function from a natural-language specification into a compact, locally executable neural artifact. We instantiate this paradigm with Program-as-Weights (PAW), in which a 4B compiler trained on FuzzyBench, a 10M-example dataset we release, emits parameter-efficient adapters for a frozen, lightweight interpreter. A 0.6B Qwen3 interpreter executing PAW programs matches the performance of direct prompting of Qwen3-32B, while using roughly one fiftieth of the inference memory and running at 30 tokens/s on a MacBook M3. PAW reframes the foundation model from a per-input problem solver into a tool builder: invoked once per function definition, it produces a small reusable artifact whose subsequent calls per function application are cheap and offline.
Sources
- Phi-4 Technical Report
- AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model
- Qwen3-VL Technical Report
- Small Language Models are the Future of Agentic AI
- Doc-to-LoRA: Learning to Instantly Internalize Contexts
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Gemma 3 Technical Report
- Neural Turing Machines
- Textbooks Are All You Need
- HyperNetworks
- Parameter-Efficient Transfer Learning for NLP
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
- The Power of Scale for Parameter-Efficient Prompt Tuning
- Latent Context Compilation: Distilling Long Context into Compact Portable Memory
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
- SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis