Studying quantization trade-offs for efficient inference deployment in machine translation
cs.CL, cs.PF
Submitted: 2026-07-31
Updated: 2026-09-12
Code: https://github.com/NVIDIA/TensorRT-LLM
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency.
Terminology
Abstract
Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common approach to reduce the memory footprint and improve inference efficiency, yet its impact on latency and throughput is rarely evaluated under controlled, orchestration-level workloads. In this work we study the quantization trade-offs of two translation model families, EuroLLM and Hy-MT2 across five models ranging from 1.7B to 22B for efficient deployment on a single A100 or H100 GPU. We demonstrate that combining a document-chunking strategy with W4A8 or W8A8 quantization improves the latency-throughput Pareto-curve under a wide range of workloads. Furthermore, since standard machine translation (MT) benchmarks rely on isolated sentences and fail to capture long-context dynamics, we introduce a document-level evaluation from WMT24++ to assess how text chunking strategies affect translation quality under quantization. Our results reveal that standard segment-level evaluation can fail to predict the interaction between quantization and long-context document translation. While Hy-MT2 remains robust under quantization, EuroLLM shows strong sensitivity and translation quality collapses rapidly for all considered quantization formats. Overall, our experiments show that the trade-off between inference efficiency and translation quality depends not only on the quantization format, but also on the choice of text chunking strategy.
Sources
- Pretraining Large Language Models with NVFP4
- GPT-4 Technical Report
- Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters
- Evaluating Quantized Large Language Models
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
- Microscaling Data Formats for Deep Learning
- The Uneven Impact of Post-Training Quantization in Machine Translation
- BitNet: Scaling 1-bit Transformers for Large Language Models
- 1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
- BitNet Distillation
- Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
- Training language models to follow instructions with human feedback
- QQQ: Quality Quattuor-Bit Quantization for Large Language Models
- Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering