The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era
cs.CL, cs.AI, cs.LG
Submitted: 2026-08-29
Updated: 2026-09-22
Comments: 75 pages, 11 tables, 6 figures
Code: https://github.com/kiyan-rezaee/language-vs-structure
License: http://creativecommons.org/licenses/by/4.0/
The gist: Specialized machine learning architectures encode structural assumptions (equivariance, permutation invariance, relational structure) that language-based foundation models lack by design.
Terminology
Abstract
Specialized machine learning architectures encode structural assumptions (equivariance, permutation invariance, relational structure) that language-based foundation models lack by design. This review asks whether such assumptions can instead be acquired through language, drawing on a corpus of 186 papers published between 2016 and 2026 across nine modalities: tabular data, graphs, time series, vision, chemistry, code, knowledge graphs, point clouds, and protein structure. Methods are organized into eight representational regimes, ranging from language-only prompting to fully specialized architectures, and are assessed not only on predictive accuracy but on whether structural information is preserved by the representation and computed by the model. Language-mediated models prove competitive in extreme few-shot prediction, discretized symbolic tasks, and textually annotated knowledge graphs. No evidence of general replacement survives once structure itself is evaluated: where direct tests exist, apparent parity is traceable to information asymmetry, benchmark contamination, or computation performed outside the language model. Across independent research communities, missing structural inductive biases are reintroduced through graph modules, structure-aware tokens, or specialized attention, indicating that specialization is relocated rather than eliminated. The review closes by specifying a falsifiable controlled experiment that would measure the remaining gap directly. The derived data supporting the findings of this review are openly available in the repository at https://github.com/kiyan-rezaee/language-vs-structure.
Sources
- What learning algorithm is in-context learning? Investigations with linear models
- How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off
- Relational inductive biases, deep learning, and graph networks
- DeepOSets: Non-Autoregressive In-Context Learning with Permutation-Invariance Inductive Bias
- When can in-context learning generalize out of task distribution?
- Transformers Don't In-Context Learn Least Squares Regression
- Towards Neural Scaling Laws on Graphs
- TabDPT: Scaling Tabular Foundation Models on Real Data
- In-context Learning and Induction Heads
- What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
- In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
- Tight Sample Complexity of Transformers
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering