Bi-Phase Training: Learning Efficiently From Scratch

arXiv:2610.37934 · cs.LG · Submitted 2026-10-27 · Read on arXiv

cs.LG

Submitted: 2026-10-27

Updated: 2026-10-27

Comments: Findings of AACL-IJCNLP 2026

License: http://creativecommons.org/licenses/by/4.0/

The gist: Pre-training foundational models is computationally expensive as models scale.

Abstract

Pre-training foundational models is computationally expensive as models scale. In this work, we introduce Bi-Phase Training (BPT), a parameter-efficient pre-training method that preserves the high-rank learning dynamics observed in full-parameter optimization with substantially reduced trainable parameters. BPT parameterizes each weight update as a composition of constrained high-rank left/right transformations via trainable diagonal matrices and low-rank adapters whose updates are periodically merged into the frozen base weights, resulting in a compounding expansion of accessible directions during training. Across language-model pre-training from scratch (Qwen2.5 models from 100M to 1.5B parameters and OLMo-2-1B) on FineWeb-Edu (10B and 100B tokens), BPT achieves validation perplexity close to full-parameter baselines while using far fewer trainable parameters; e.g., at the 1.5B scale, BPT uses 66% fewer trainable parameters with only a small validation-perplexity gap. Additionally, a comprehensive evaluation across diverse downstream tasks shows that BPT preserves much of the full-parameterized model's downstream performance.

Related papers