Multi-Bin Batching for Increasing LLM Inference Throughput

Name: Multi-Bin Batching for Increasing LLM Inference Throughput
Uploaded: 2026-08-20T03:44:42.595171+00:00
Duration: 571 s
Description: The episode discusses 'Multi-Bin Batching for Increasing LLM Inference Throughput,' a method that groups requests by their predicted service time to reduce wasted GPU time. Hosts analyze how this technique improves throughput, showing gains up to 70% with perfect length predictions, while also discussing the trade-off of increased latency.

The episode discusses 'Multi-Bin Batching for Increasing LLM Inference Throughput,' a method that groups requests by their predicted service time to reduce wasted GPU time. Hosts analyze how this technique improves throughput, showing gains up to 70% with perfect length predictions, while also discussing the trade-off of increased latency.

Open the full episode