Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
The study pretrained a 1.7B multilingual model across nine languages to observe early cross-lingual generalization. It found that the model first learns basic linguistic skills alongside token-level copying. Translation then develops in two phases: an initial phase dominated by surface-level copying, followed by a second phase where more general translation mechanisms emerge while copying is refined.