Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

arXiv:2604.17633 · cs.CL · Submitted 2026-04-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Copy First, Translate Later".

Tom: The gist The model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases:

Jane: First, who's behind it and why it matters.

Paper summary: Tom: To recap, this paper is about studying how cross-lingual generalization emerges during multilingual pretraining by capturing checkpoints at a finer granularity #pg1.

Jane: The core claim is that we need to look at the early phases of learning because prior work looked too isolated or only at sparse points in training #pg1.

Lu: By pretraining a 1 point 7B model on nine languages and using word-level translation as a testbed, they trace how translation develops over training through behavioral analyses and other methods #pg2 <ref:2604.17633#pg1,word-level translation as a testbed>.

Meng: The main point they are driving home is that the copying mechanism emerges in parallel with basic linguistic capabilities #pg3.

Lalam: And then translation itself develops in two distinct phases, starting with a phase dominated by copying and surface-level similarities, followed by a second phase where more generalizing translation mechanisms develop while copying is refined #pg2.

Tom: It really matters because it gives us a fine-grained view of the entire process of how cross-lingual generalization builds up during this massive pretraining phase #pg3.

Jane: It helps us understand the underlying mechanics instead of just seeing a final result, which is what they are aiming for with this study #pg1.

Lu: They are providing two hundred snapshots across training for their analysis, which is much more detailed than any other publicly available multilingual checkpoints #pg2 <ref:2604.17633#pg2,than any other publicly available multilingual checkpoints>.

Meng: It’s interesting because they do it without a cross-lingual objective or targeted parallel data, just random sampling from each language partition at every step #pg3.

Lalam: That lack of explicit cross-lingual training signals is important because it mirrors more realistic training conditions and helps mitigate order-based effects #pg3.

Tom: So we're looking at a very specific window in time where these fundamental mechanisms are being forged, which is what makes this paper so relevant right now.

Conclusion: Tom: Looking at the full scope of Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining by Körner and colleagues #pg1, it’s clear they are focusing intensely on the dynamics of word-level translation during early training.

Jane: The paper’s main contribution is offering a unified view that links token-level copying to the development of more complex, generalizing translation mechanisms #pg2.

Lu: It shows that copying isn't just an error; it's a strategy that gets refined as the model learns better ways to suppress it #pg3.

Meng: From an engineering standpoint, understanding this interaction helps us design models that might be more robust during the initial learning stages #pg2.

Lalam: For culture, this suggests that language acquisition in AI isn't just about memorizing words; it involves a dynamic interplay between simple repetition and developing deeper understanding #pg3.

Tom: It really puts the focus on those intermediate layers, showing that most of the crucial translation mechanisms are formed between four billion and eighteen point eight billion tokens processed #pg3.

Jane: So, essentially, this work gives us a very detailed map of how incidental word-level translation develops in multilingual models during pretraining #pg3.

Felicia Körner, Maria Matveev, Florian Eichin

MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Department of Mathematics, LMU Munich, Germany · Department of Physics and Technology, University of Tromsø, Norway · DLR-German Aerospace Center

cs.CL

Submitted: 2026-04-19

Updated: 2026-10-08

Comments: 9 pages, accepted to EMNLP Main 2026, code, model checkpoints, and dataset available

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The gist The model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and

Key concepts

Token-Level Copying: This refers to the model's tendency to reproduce specific words or tokens from the source language directly in the target language during early training. It acts as a primary, though imperfect, strategy for initial cross-lingual alignment.
Generalizing Mechanisms: These are more sophisticated translation capabilities that allow the model to translate without relying solely on direct token overlap. These mechanisms develop later in training and are crucial for achieving true cross-lingual understanding.
Two Distinct Phases of Translation: The paper identifies two distinct stages in how translation capability evolves. Phase I is characterized by copying and surface similarities, while Phase II involves the development of more abstract, generalizing translation methods that refine the process.

Terminology

Generalization — 2× in this paper · explained in 13 episodes
Ablation — 1× in this paper · explained in 1 episode
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist The model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. <ref:2604.17633#pg2>

How it works

The study pretrains a multilingual 1.7B model on nine diverse languages to capture checkpoints at a finer granularity than prior work, allowing for the tracing of linguistic and translation capabilities over training through behavioral analyses, model-component analysis, and parameter-based ablations. <ref:2604.17633#pg3> The evaluation uses word-level translation (WLT) as a testbed to trace how translation develops over training through these methods. <ref:2604.17633#pg3>

The researchers identify two distinct phases of learning in WLT, hypothesizing that these phases reflect the evolution and interaction of two mechanisms: token-level copying and a generalizing mechanism implementing shared representation spaces. <ref:2604.17633#pg3> Phase I is characterized by the model rapidly acquiring linguistic knowledge in parallel with a useful, but overeager copying mechanism. <ref:2604.17633#pg3> Phase II is where the model develops translation aided by token overlap and, more slowly, the ability to translate without token overlap. <ref:2604.17633#pg3>

Key findings on mechanisms

The analysis reveals that copying is the dominant error mode in early training, peaking at 53.2% of errors attributable to erroneous copying across all language pairs. <ref:2604.17633#pg2> Copying becomes more targeted during Phase I, with context copying rising to 33.6% of outputs until 6.7B tokens processed, and source word copying overtaking context copying after 7.5B tokens in the repetition task. <ref:2604.17633#pg2>

The generalizing mechanisms enabling translation without token overlap are largely developed by the halfway point of pretraining, with final bottom and top layers enabling translation at this point. <ref:2604.17633#pg3> The logit lens analysis shows that valid translations with partial or full token overlap are increasingly downranked in the upper layers during Phase II, even though the model ultimately prefers these outputs. <ref:2604.17633#pg3> This suggests that generalizing mechanisms help suppress erroneous copying. <ref:2604.17633#pg3>

Interpretability and Ablations

The study employs two complementary interpretability methods, logit lens and ExPLAIND, to understand how token-level copying and generalizing mechanisms develop and interact over the model’s learning phases. <ref:2604.17633#pg3> ExPLAIND identifies attention value projections in layers 0–2 as the top contributors of copy-promoting influence in Phase I, while parameters in layers 6–9 are top contributors of copy-suppressing influence in Phase II. <ref:2604.17633#pg3>

Layer-swapping experiments confirm that fixing the bottom and top blocks allows for significant improvement in WLT performance, with the final bottom and top blocks enabling translation at around 18.8B tokens processed. <ref:2604.17633#pg3> The logit lens margin plot shows that copy-promoting layer transitions appear as blue stripes, while translation accumulates a small relative gain across most layers, increasing towards model output and as training progresses. <ref:2604.17633#pg3>

Conclusion

The paper concludes that token-level copying is a strong early strategy learned in parallel with basic linguistic capabilities, and as training progresses, the generalizing mechanism develops, coinciding with a decline in copying as an error mode and more selective token-level copying. <ref:2604.17633#pg3> This provides insight into how incidental word-level translation develops in multilingual language models. <ref:2604.17633#pg3>

The findings suggest that the mechanisms relevant for WLT in the intermediate layers are largely formed between 4B and 18.8B tokens, with smaller refinements until 25B tokens. <ref:2604.17633#pg3> This indicates that the mechanisms relevant for WLT in the intermediate layers are largely formed between 4B and 18.8B tokens, with smaller refinements until 25B tokens. <ref:2604.17633#pg3> The final bottom and top blocks allow mapping in and out of the generalizing mechanism in the intermediate block, supporting a three-stage hypothesis for multilingual processing. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3>

The paper's work contributes to the broader goal of algorithmic transparency by studying the emergence and interaction of two central mechanisms in word-level translation. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The analysis is restricted to early pretraining dynamics, which is argued to be the most relevant window for the emergence of these mechanisms. <ref:2604.17633#pg3> The paper's work contributes to the broader goal of algorithmic transparency by studying the emergence and interaction of two central mechanisms in word-level translation. <ref:2604.17633#pg3>

The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> The paper's work contributes to the broader goal of algorithmic transparency by studying the emergence and interaction of two central mechanisms in word-level translation. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.17633#pg3> The study provides a finegrained view of how cross-lingual generalization develops during multilingual pretraining. <ref:2604.17633#pg3> This research can inform future work on the inclusion of low-resource languages in global AI development and help bridge the digital divide. <ref:2604.

Improvements for AI systems

This system can develop a generalizing mechanism implementing shared representation spaces by allowing it to translate without token overlap, which is achieved by developing mechanisms that develop in parallel with basic linguistic capabilities.
The improved system will exhibit better cross-lingual transfer performance because the analysis shows that the final bottom and top layers enable translation at this point, suggesting these blocks are crucial for mapping in and out of a shared space.
The system will suppress erroneous copying by developing mechanisms where token overlap supports source word copying as a more targeted copying, while simultaneously observing that when token overlap is present, the outermost two layers’ log probabilities continue to increase over training.
The system can distinguish between learning phases: during Phase I, it will develop a useful, albeit overeager copying mechanism, and in Phase II, it will develop the generalizing mechanism that enables translation without overlap.
The system can optimize its copying behavior by selectively inhibiting parameters; for instance, inhibiting them results in even higher context copying (in Phase II) or reduces copying with only "minor impact on translation performance (< 4 percentage points)" if parameters in layers 0–2 are inhibited during Phase I.