BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

Name: BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training
Uploaded: 2026-10-18T23:23:04.364445+00:00
Duration: 961 s
Description: BSIPipe is a new technique for training large language models with long sequences by using bidirectional sequence-level interleaved pipeline parallelism. It achieves speedups of 1.11x to 1.13x over existing methods like GPT and LLaMA3. The method works by scheduling two opposing pipelines to minimize execution bubbles and uses specific communication optimizations to improve training throughput.

BSIPipe is a new technique for training large language models with long sequences by using bidirectional sequence-level interleaved pipeline parallelism. It achieves speedups of 1.11x to 1.13x over existing methods like GPT and LLaMA3. The method works by scheduling two opposing pipelines to minimize execution bubbles and uses specific communication optimizations to improve training throughput.

Open the full episode