Script Choice in LLMs: Evidence for Late-Layer Commitment

arXiv:2609.28784 · cs.CL · Submitted 2026-09-23 · Read on arXiv

cs.CL

Submitted: 2026-09-23

Updated: 2026-09-23

Code: https://github.com/IDSIA-NLP/LLMScriptCommitment

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: In this paper, we investigate how script knowledge is distributed across the layers of LLMs using two complementary interpretability methods: logistic regression probing and logit-lens analysis.

Terminology

Abstract

In this paper, we investigate how script knowledge is distributed across the layers of LLMs using two complementary interpretability methods: logistic regression probing and logit-lens analysis. Our probing experiments reveal a clear asymmetry: both the input script and the instructed output script are encoded in the earliest layers of the network, while, in contrast, commitment to the actual output script emerges only in the final layers, with the model's intermediate representations defaulting to Latin throughout most of the layers. This two-stage process is confirmed by logit-lens analyses, which show that script commitment consistently occurs at the very last layers of the LLMs. Together with the weaker script-following performance observed in smaller models, these results form a converging body of evidence linking script commitment to model depth, with broader implications for the design of sufficiently deep, inclusive multilingual architectures.

Sources

Related papers