Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features

arXiv:2610.37976 · cs.LG · Submitted 2026-10-27 · Read on arXiv

cs.LG

Submitted: 2026-10-27

Updated: 2026-10-27

License: http://creativecommons.org/licenses/by/4.0/

The gist: We empirically investigate how MLPs and Transformers differ when trained on data sets containing one or more features encoded by noisy k-parity.

Abstract

We empirically investigate how MLPs and Transformers differ when trained on data sets containing one or more features encoded by noisy k-parity. Noisy k-parity gives us hard synthetic data sets in which we can control the predictivity and availability of every feature. Across a range of data distributions, data scales and model sizes, we find large differences in the training dynamics and inductive biases of the two architectures: Transformers are more robust to noisy features but struggle to learn complex, low-availability features, whereas MLPs show the opposite trend and improve steadily with more data. We also find that, under a fixed training budget, increasing the number of unique training examples can slow Transformer optimisation enough to reduce final accuracy, so that a smaller data set performs better. Training-accuracy results show this is an optimisation-speed effect; it shrinks, but does not vanish, when the budget is scaled with the data set size. By giving precise control over feature predictivity and availability, this testbed offers a clear starting point for understanding when and why Transformers and MLPs learn differently, and for designing data and training budgets that exploit those differences.

Related papers