HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs".
Jane: Here's the gist of the paper. Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult The authors present HALO, a new quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining strategic placement of Hadamard rotations in both forward and backward passes, high-performance kernel support, and FSDP integration Applied to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks, while delivering up to 1.41× end-to-end speedup for full fine-tuning on RTX 4090 GPUs Problem Context and Motivation The challenge in quantized training is that for training, all three matrix multiplications (one during the forward pass and two during back-propagation) must be executed in lower precision, which creates critical issues both in terms of accuracy—as quantizing weights, activations, and errors can induce significant training instability—and performance—as the overheads of switching between representations can negate the performance gains of lower-precision computation This difficulty is especially pronounced when fine-tuning pre-trained models because these models often have large outlier values in their weight, activation, and error distributions
37; 42; 34; 28: .
Tom: Proposed HALO Methodology HALO addresses the challenges by modifying the structure of Transformer-based models in a transparent fashion to allow fine-tuning in lower precision (INT8, or even FP6) while performing all large matrix multiplications in lower precision.
Jane: The approach starts from an analysis of quantization error sensitivity, identifying the forward pass and output gradients as key sources of sensitivity to quantization.
Tom: The authors propose using left- and right-hand-side Hadamard transforms to address outliers over matrix rows and columns, respectively.
Paper summary: Tom: So looking at the title of "HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs" and reading through all this, it seems like they’re really focused on solving that core problem we talked about earlier—making low precision training stable and fast.
Jane: And what does that mean in simpler terms? It means they found a way to handle those messy outlier values in the weights and activations during fine tuning without sacrificing too much accuracy when you use lower precision formats like INT8 or FP6.
Lu: The implication is that we don't have to stick strictly to full precision training just because we want speed, especially for large pre-trained models where those outlier structures are already present <ref:2501.02625#pg2>.
Meng: From a deployment standpoint, if this works well on the training side, it suggests that fine tuning massive models can be done much more efficiently before we even get to the inference stage three <ref:2501.02625#pg1>.
Tom: It’s about making that training process itself much faster and more robust, which is what they claim by tracking accuracy with full precision variants across a wide series of tasks <ref:2501.02625#pg3>.
Jane: So for someone just listening who isn't looking at the math, this paper suggests that we can get good results from using lower precision training formats, provided you use smart techniques to manage the training signal itself.
Lu: They also mentioned in their final thoughts that they plan to investigate HALO for pre-training accurate models from scratch and extend it for additional GPU hardware types <ref:2501.02625#pg3>.
Meng: It sounds like they're not just stopping at fine tuning; they’re thinking about applying this concept earlier in the model lifecycle, which would be very interesting for production systems.
Tom: Yeah, so the work with HALO is showing that we can get fast and accurate fine tuning where all the linear modules are in lower precision, even with initial outlier structures in weights and errors <ref:2501.02625#pg3>.
Jane: That's a pretty concrete statement. So basically, this paper gives us a new framework for training LLMs that balances speed and accuracy by cleverly managing the signal during the training process <ref:2501.02625#pg3>.
Conclusion: Tom: So we're wrapping up on HALO, which is this paper about using Hadamard transforms to make training large language models in lower precision more accurate and faster <ref:2501.02625#pg1>.
Jane: Right, so it tackles that tricky problem where you want speed but you don't want your model to get weird or inaccurate during the training process <ref:2501.02625#pg3>.
Tom: The authors present this HALO method, and they show that by strategically placing these Hadamard rotations in both the forward pass and the backward pass, you can keep things pretty accurate even when you're doing all your big matrix multiplications in lower precision formats like INT8 <ref:2501.02625#pg3>.
Lu: What’s interesting is how they break it down into two levels, HALO-one and HALO-two depending on how much rotation you use to handle those outliers in the data distributions <ref:2501.02625#pg4>.
Meng: From a deployment standpoint, if this works well during fine tuning for models like Llama or Qwen, it suggests that we can significantly cut down the compute cost before we even get to inference <ref:2501.02625#pg3>.
Jane: So basically, they've found a way to make the training signal more stable when you're running all those heavy computations at lower precision <ref:2501.02625#pg3>.
Tom: The results are solid, showing near full precision accuracy during fine tuning on various tasks and even achieving speedups up to one point four one times for full fine tuning on an RTX forty nine zero GPU <ref:2501.02625#pg3>.
Lu: They compared it against other methods like SwitchBack and Jetfire, and HALO-two came out as the most accurate scheme for integer formats like INT8 <ref:2501.02625#pg4>.
Meng: But what about the real world? Does this translate to something that actually runs fast on our production hardware with real traffic?
Jane: That's a big question, Meng, because while they show great speedups during fine tuning, we still need to see how it behaves when the model is actually being used <ref:2501.02625#pg3>.
Tom: It seems like this paper lays a foundation for making low precision training much more practical and robust for the next generation of large language models.
Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L. Castro, Torsten Hoefler, Dan Alistarh
ETH Zurich · IST Austria
cs.LG
Submitted: 2025-01-05
Updated: 2025-11-05
Comments: 19 pages, 6 figures
Journal ref: Advances in Neural Information Processing Systems 38 (NeurIPS 2025), pp. 118871-118896
DOI: 10.52202/085713-3966
Code: https://github.com/IST-DASLab/HALO
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 97/100
The gist: Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult The authors…
Key concepts
- Quantized Training Challenge
- Training LLMs in low precision is hard because all three matrix multiplications (forward pass and two backpropagation steps) must use lower precision. This causes accuracy issues due to instability and performance losses from switching between representations.
- Hadamard Rotations
- These are mathematical rotations applied to weight gradients during the forward pass. They are used as a technique to mitigate outliers in the weight distributions, helping the model maintain accuracy when using lower precision formats like INT8.
- HALO Levels (HALO-1 and HALO-2)
- The paper defines two levels based on Hadamard placement. HALO-1 balances accuracy and performance for moderate dynamic ranges (like FP6). HALO-2 uses rotations to protect all multiplications during both forward and backward passes, making it suitable for integer formats like INT8.
Terminology
Summary
Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult The authors present HALO, a new quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining strategic placement of Hadamard rotations in both forward and backward passes, high-performance kernel support, and FSDP integration Applied to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks, while delivering up to 1. p. 041× end-to-end speedup for full fine-tuning on RTX 4090 GPUs
Problem Context and Motivation
The challenge in quantized training is that for training, all three matrix multiplications (one during the forward pass and two during back-propagation) must be executed in lower precision, which creates critical issues both in terms of accuracy—as quantizing weights, activations, and errors can induce significant training instability—and performance—as the overheads of switching between representations can negate the performance gains of lower-precision computation This difficulty is especially pronounced when fine-tuning pre-trained models because these models often have large outlier values in their weight, activation, and error distributions [37; 42; 34; 28].
Proposed HALO Methodology
HALO addresses the challenges by modifying the structure of Transformer-based models in a transparent fashion to allow fine-tuning in lower precision (INT8, or even FP6) while performing all large matrix multiplications in lower precision. The approach starts from an analysis of quantization error sensitivity, identifying the forward pass and output gradients as key sources of sensitivity to quantization. p. 1 The authors propose using left- and right-hand-side Hadamard transforms to address outliers over matrix rows and columns, respectively. p. 1
HALO Levels and Implementation
The paper defines two levels of HALO based on the number and placement of Hadamard transforms during training:
-
HALO-1 strikes a trade-off between accuracy and performance that is well-suited for distributions with moderate dynamic ranges (such as FP6). p. 1
-
HALO-2 employs rotations to protect all multiplications during both the forward and backward passes, making it more suitable for integer representations (such as INT8). p. 1
The method supports both full fine-tuning (FFT) and parameter-efficient fine-tuning (PEFT) methods, with HALOPEFT being a variant that quantizes the majority of the computation while retaining low-rank computations in high precision. p. 6
Performance and Communication Optimization
HALO integrates with the Fully Sharded Data Parallel (FSDP) scheme via Hadamard-Quantized FSDP (HQ-FSDP) to enable further savings via low-precision communication. This approach significantly reduces communication overhead while distributing the quantization and, possibly, the Hadamard transformation overhead across processes Furthermore, activation quantization is used to minimize memory usage by storing quantized activations for the backward pass. p. 6
Experimental Validation
The results show that HALO closely tracks the accuracy of full-precision variants across a wide series of tasks, improving upon the best known prior methods on the more challenging INT8 and FP6 formats For INT8 quantization, HALO achieves up to a 1.36× end-to-end speedup which is comparable to the 1. p. 9 p. 941× speedup of FP8 without any Hadamard transformation. p. 9 The authors also demonstrate that for larger models like Qwen, HALO-2 outperforms both JetFire and BF16 across both model sizes. p. 7
Conclusion
The authors conclude that it is possible to perform fast and accurate fine-tuning while the majority of the forward-backward computation (all linear modules) is in lower precision, even if the model initially has an outlier structure in weights, activations, and errors. HALO uses simple tensor-wise quantization for all weights, inputs, and errors, utilizes low-precision communication (HQ-FSDP), and reduces memory usage by storing quantized activations. p. 9 The paper suggests that in future work, they plan to investigate HALO for pre-training accurate models from scratch and extend it for additional GPU hardware types. p. 9
Pruning redundant reasoning tokens
The high performance of large language models (LLMs) across a wide series of tasks comes with considerable computational costs; reducing them is one of the key directions in Machine Learning Systems research [9; 16; 21] For LLM inference, a standard acceleration approach has been the quantization of weights and activations, which reduces the precision at which they are stored and potentially also computed over, e.g. [17; 42; 1; 35; 3]. By contrast, much less is known about quantization in the context of LLM training While the recent DeepSeek breakthrough [11] has showed that 8-bit floating-point (FP8) pre-training of large models can be done both accurately and efficiently, it is not known how to extend this result using the more widely-supported 8-bit integer (INT8) format, or lower-precision 6-bit formats such as MXFP6 [38; 41]. p. 0 Intuitively, accurate and efficient quantized training is much more challenging than quantized inference: while in inference a single large matrix multiplication occurs in lower precision, for training, all three matrix multiplications (one during the forward pass and two during back-propagation) must be executed in lower precision. p. 0 This creates critical issues both in terms of accuracy—as quantizing weights, activations, and errors can induce significant training instability—and in terms of performance—as the overheads of switching between representations can negate the performance gains of lower-precision computation. p. 0
How the training signal is built
The authors analyze the key challenges of applying Hadamard transformations to reduce quantization error while performing forward and backward passes in low precision They study the cosine similarity between the weight gradients of the baseline model (no quantization) and the quantized model when quantization is applied during the forward pass or backward pass. They conclude that mitigating outliers is key during the forward pass, and they can recover most of the averaged cosine similarity (0. p. 5883) by applying Hadamard transformations during the forward pass on the weights and inputs. p. 5
Where the gains come from
The speedup analysis shows that HALO-2 attains additional 20-40% speedups compared to SwitchBack, primarily due to the higher precision matrix multiplication for weight gradient calculations in SwitchBack The effect of HQ-FSDP shows that it achieves 1.37× to 1.43× speedups, delivering 10-40% higher speedups compared to 16-bit communication. p. 8 In Table 9, INT8 HALO-2 shows a speedup of 1. p. 745× on four RTX 4090 GPUs with HQ-FSDP. p. 0
The authors compare HALO against three main baselines for full fine-tuning: BF16; SwitchBack [38], and Jetfire [41] SwitchBack fails to recover accuracy on both the ViGGO and GSM8K datasets, despite using row-wise and columnwise quantization for inputs and weights, along with high-precision weight gradient calculations. HALO-1 shows a 2. p. 78% accuracy loss compared to the BF16 model on GSM8K, whereas HALO-2 achieves within 1% of the BF16 accuracy on INT8. p. 7
The authors define two levels within HALO corresponding to data formats with moderate and narrow dynamic ranges, respectively as follows:
-
At the first level, denoted by HALO-1(H F, E H, GH), they employ the middle case during the forward pass to mitigate issues with weight gradients. p. 5
-
Finally, to mitigate outliers in errors, they define the highest level by applying a left-hand-side Hadamard transformation to E, denoted by HALO-2(H F, HE H, GH). p. 6
The authors show that for INT8 precision, HALO-0 fails to recover accuracy, leading to an approximate 40% drop in accuracy on their datasets At the next level, HALO-1 recovers approximately 37% of the above accuracy gap by applying right-hand-side Hadamard transformations on the weights and inputs during both the forward and backward passes.
The authors present results from integrations with LoRA and QLoRA, but their focus is on accelerating the actual computations during fine-tuning, which remains a less-explored area In Table 4, for INT8 and FP8, HALOPEFT is always within the standard deviation of baseline. p. 8 On GSM8K, FP6 has approximately a 2% accuracy degradation when using LoRA [19] compared to BF16. p. 7
The authors describe the HQ-FSDP implementation, noting that they save only the global quantization scales during the forward pass and recompute the quantization and Hadamard transform during the backward pass This approach eliminates the need for a second quantization and potential Hadamard transform during the backward pass.
The authors present HALO as a low-precision fine-tuning method that can be presented as a QAT scheme where inference of the fine-tuned model will be done in low precision They note that since FP6 has the same TensorCore peak performance as FP8 (with less read/write overhead), the speedups achieved with FP8 can be considered a lower bound for the potential speedups with FP6 as well.
In Table 7, using INT8, the most accurate HALO level, HALO-2, achieves speedups of 1.15× to 1.38× compared to BF16 when using their kernels. p. 7 For FP8, they use HALO-0, which achieves speedups of 1. p. 827× to 1.32×, coming within 5% of the ideal speedup. p. 8 The speedup increases with larger batch sizes as the quantization and Hadamard overheads become less significant, and the matrix multiplications become the primary bottleneck. p. 4 p. 4
The authors apply HALO to pre-training of TinyLlama-1.1B on the C4 dataset, where FP8 closely matches the base BF16 training even with HALO level 0, both achieving an evaluation loss of 2. p. 155. For FP6, only HALO level 2 converges with a final evaluation cross-entropy of 2. p. 770. INT8 and FP4 diverge, no matter the HALO level. p. 7
HALO-1 achieves a better trade-off between accuracy and the number of Hadamard transformations for FP6 precision HALO-2 closes the accuracy gap with BF16 on the GSM8k dataset when using MXFP6. The authors note that in Appendix A. p. 19, they provide inference speed results for the model which is fine-tuned using HALO, showing that the Hadamard transformations have minimal impact on inference performance. p. 9 p. 9 p. 9
The authors implement HALO in PyTorch [31] based on the llm-foundry codebase [25] for FFT, and the standard HuggingFace PEFT library for HALOPEFT They use tensor-wise symmetric quantization (a single scale for the entire tensor) for all data types, and Round-to-Nearest (RTN) quantization across all our experiments. They implement their own low-precision matrix multiplications using the CUTLASS library [29] for all linear modules (except for the LM head and embeddings) and keep the rest of the model in the original precision (BF16) during fine-tuning. p. 7
In Table 3, they summarize all HALO levels for full fine-tuning (FFT), showing that HALO-2 is the most accurate scheme. In Table 4, for INT8 and FP8, HALOPEFT is always within the standard deviation of baseline.
The authors show that in Table 9, INT8 HALO-2 achieves speedups of 1.37× on four RTX 4090 GPUs with HQ-FSDP. p. 0 The speedup increases with larger batch sizes as the quantization and Hadamard overheads become less significant, and the matrix multiplications become the primary bottleneck. They also note that in Appendix A. p. 19, they provide inference speed results for the model which is fine-tuned using HALO, showing that the Hadamard transformations have minimal impact on inference performance.
The authors compare HALO against three main baselines for full fine-tuning: BF16; SwitchBack [38], and Jetfire [41] However, SwitchBack fails to recover accuracy on both the ViGGO and GSM8K datasets, despite using row-wise and columnwise quantization for inputs and weights, along with high-precision weight gradient calculations. In Table 1, HALO-2 achieves relative accuracy within 1% of the BF16 model on INT8. p. 7
The authors describe the HQ-FSDP implementation, noting that they save only the global quantization scales during the forward pass and recompute the quantization and Hadamard transform during the backward pass This approach eliminates the need for a second quantization and potential Hadamard transform during the backward pass.
HALO achieves up to a 1.36× end-to-end speedup which is comparable to the 1.41× speedup of FP8 without any Hadamard transformation This holds during fine-tuning of LLAMA3-8B on four and eight commodity GPUs, while maintaining accuracy close to the baseline. p. 9 The authors also note that in Appendix A. p. 19, they provide inference speed results for the model which is fine-tuned using HALO, showing that the Hadamard transformations have minimal impact on inference performance.
The authors implement HALO in PyTorch [31] based on the llm-foundry codebase [25] for FFT, and the standard HuggingFace PEFT library for HALOPEFT They use tensor-wise symmetric quantization (a single scale for the entire tensor) for all data types, and Round-to-Nearest (RTN) quantization across all our experiments They implement their own low-precision matrix multiplications using the CUTLASS library [29] for all linear modules (except for the LM head and embeddings) and keep the rest of the model in the original precision (BF16) during fine-tuning.
In Table 3, they summarize all HALO levels for full fine-tuning (FFT), showing that HALO-2 is the most accurate scheme.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive low-precision training for pre-trained LLMs using HALO allows for accurate fine-tuning in INT8 and FP6 precision while maintaining accuracy close to BF16 baseline, achieving up to 1.36x end-to-end speedup on RTX 4090 GPUs.
-
The system can perform all large matrix multiplications during the forward and backward passes in lower precision by strategically placing Hadamard rotations in both passes, which mitigates outliers in weight, activation, and error distributions.
-
Implement HALO-1 for moderate dynamic ranges (like FP6) to strike a trade-off between accuracy and performance while employing right-hand-side Hadamard transformations on weights and inputs during the forward pass.
-
For narrow dynamic ranges like INT8, use HALO-2 to mitigate outliers in error gradients during the backward pass by applying left-hand-side Hadamard transformations on errors, achieving accuracy within 1% of BF16 results.
-
Integrate HQ-FSDP to enable low-precision communication during sharded training, significantly reducing communication overhead while distributing quantization and Hadamard transformation costs across processes.
-
Utilize Activation Checkpointing combined with HQ-FSDP for memory reduction, storing only quantized activations for the backward pass and reusing them during the second forward pass, minimizing activation storage requirements.
-
Apply HALOPEFT to parameter-efficient fine-tuning schemes like LoRA, quantizing most computations while retaining low-rank computations in high precision to maintain performance with minimal accuracy loss.
Sources
- QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- FP8 versus INT8 for efficient deep learning inference
- Incoherence-Optimal Matrix Completion
- Exploring Quantization for Efficient Pre-Training of Transformer Language Models
- Training Verifiers to Solve Math Word Problems
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- The Llama 3 Herd of Models
- Extreme Compression of Large Language Models via Additive Quantization
- Scaling FP8 training to trillion-token LLMs
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Training Compute-Optimal Large Language Models
- LoRA: Low-Rank Adaptation of Large Language Models
- SpinQuant: LLM quantization with learned rotations
- Mixed Precision Training
- RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation
- Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
- FP8-LM: Training FP8 Large Language Models
- Massive Activations in Large Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis