Improved Scaling for Fast Mode of Ozaki Scheme II

arXiv:2606.29129 · cs.MS, cs.DC, cs.NA, math.NA · Submitted 2026-06-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Improved Scaling for Fast Mode of Ozaki Scheme II".

Jane: Here's the gist of the paper. Ozaki scheme II emulates high-precision matrix multiplication using low-precision integer matrix operations based on the Chinese remainder theorem (CRT) The proposed method achieves accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode Core Concept and Motivation The paper addresses the lack of scale invariance in the existing fast mode scaling formula, which causes accuracy degradation or CRT recovery failure when input matrices are scaled by a constant factor.

Tom: The authors propose a revised scaling formula derived from the CRT uniqueness condition via the Cauchy–Schwarz inequality to ensure scale invariance and guarantee that entries of the scaled integer matrix product never exceed the CRT recovery threshold P/2.

Jane: This improved method is designed to overcome both the accuracy limitation of fast mode and the throughput constraint of accurate mode, offering a superior balance of accuracy and performance.

Tom: Theoretical Limitation Analysis The theoretical analysis demonstrates that in fast mode, the effective precision of the i-th row of the scaled integer matrix A' depends on a scalar factor α when input matrices are scaled by α.

Title and authors: Tom: So we're looking at this paper, "Improved Scaling for Fast Mode of Ozaki Scheme II." It sounds like they’re tackling a problem where you want fast matrix multiplication but you can't lose too much accuracy when you scale your matrices.

Jane: Exactly. It’s all about making sure that when you multiply two big matrices using these integer operations based on the Chinese remainder theorem, scaling them by some constant doesn't mess up the result or cause the math to break down.

Lu: What I see is that they noticed a problem with the existing fast mode formula because it wasn't scale invariant. If you multiply your input matrices by some factor alpha, the precision of those intermediate integer matrices changes as alpha gets bigger, which leads to accuracy dropping or even CRT recovery failing entirely.

Meng: From a deployment standpoint, that kind of instability is a nightmare at scale. If the scaling isn't consistent when the input changes slightly, you can’t reliably ship it to production without constant re-calibration.

Lalam: I see this paper proposes fixing that by deriving a new scaling formula directly from the CRT uniqueness condition using something called the Cauchy–Schwarz inequality, and this new formula is scale-invariant by construction.

Tom: That sounds promising. So instead of having a separate accurate mode for estimation and a fast mode for multiplication, they're trying to bridge that gap with one better scaling method?

Jane: Right. The paper explains that this revised formula guarantees the entries of the scaled integer matrix product never exceed P over two, which is what keeps the CRT recovery working reliably.

Lu: And it specifically addresses the issue where in a certain regime, even though your effective precision might seem fixed at floor of P fast minus one plus one bits, those products can still blow past that threshold if you use the old scaling.

Meng: That’s good to hear because it means we don't have to worry about catastrophic failure when we push the inputs to their limits. It keeps the system stable across different input scales.

Lalam: The authors define two sets of integers, mu and nu, right from that Cauchy–Schwarz bound, which ensures that these scaling factors are sufficient for the desired behavior.

Jane: So, what’s the immediate result of this change for someone who just wants to use this method? It’s a better balance between speed and correctness.

Title and authors: Tom: Apparently, they show experiments on an NVIDIA GH200 GPU where they achieve accuracy comparable to their accurate mode while keeping the throughput pretty close to what fast mode achieves when you increase the number of moduli.

Lu: That’s the main win, Tom. They're solving a trade-off that used to exist: you could either have high accuracy at a slow speed or high speed with lower accuracy when things got big.

Meng: If it maintains throughput comparable to fast mode, that means we can still run this relatively quickly on hardware without needing the massive computational cost of the accurate mode. That’s something I care about for deployment.

Jane: It really does offer a superior trade-off management system, overcoming both the accuracy limitations of fast mode and the throughput constraints of accurate mode simultaneously.

Tom: So they essentially found a way to make fast mode behave more reliably even when you're not using the exact scaling parameters they originally assumed.

Lu: Precisely. The gains come from deriving that revised scaling formula directly from the CRT uniqueness condition via the Cauchy–Schwarz inequality, which is what ensures those products stay bounded by P over two, preventing CRT recovery failures.

Lalam: And just to add a bit more detail on how they set up those scaling factors, they define mu i and nu j using a formula derived from that bound, which is what you see in Algorithm two.

Jane: So for the listener driving or cooking right now, the big picture is that this paper gives us a way to scale these high-precision matrix operations robustly without losing accuracy or slowing down too much.

Tom: It’s a solid engineering fix because it doesn't add any extra computational cost to the original fast mode; the complexity stays the same.

Lu: That's a crucial point—no additional overhead on top of what you already have, which makes it practical for real-world use.

Meng: If there’s no added overhead, that’s great for my team because we don't have to worry about slowing down the inference pipeline when we switch from fast mode to this improved version.

Lalam: So, to wrap up on the methodology, they show that by choosing mu and nu this way, they ensure the effective bit width of the scaled integer matrix is constant and independent of alpha.

Title and authors: Jane: That constant effective bit width is what locks in that accuracy across different input scales we might encounter in real applications.

Tom: And that stability is what allows them to claim accuracy comparable to accurate mode for all values of phi tested in their experiments.

Lu: It means you can rely on this scaling method much more than the previous fast mode version when dealing with larger matrix operations.

Meng: I still have a question about the limitations, though. The paper mentions that they need to do a rigorous error analysis analogous to what they did for accurate mode in future work. What’s the caveat there?

Lalam: They acknowledge that a full rigorous error analysis, just like for accurate mode, is something that needs to be done later on by the authors themselves.

Jane: So while they get the immediate performance and stability win now, they are flagging that a deeper theoretical error check is still needed down the line.

Tom: It sounds like a complete picture then: we have a method that works better than before, maintaining speed and accuracy, but it’s not yet fully proven to be theoretically airtight in every single edge case.

Lu: That’s the honest assessment from their side. The idea is sound, the implementation works well on the GH200 GPU data they used.

Meng: For production deployment, that means we can move forward with this improved fast mode formula now, knowing it's better than what we were using before for large phi values.

Lalam: So in summary, "Improved Scaling for Fast Mode of Ozaki Scheme II" proposes a revised scaling formula derived from the CRT uniqueness condition via Cauchy–Schwarz to ensure scale invariance and guaranteed error bounds.

Jane: It’s a significant improvement over the previous version because it fixes the issue where scaling inputs changed the effective precision of intermediate matrices.

Tom: We got speed, we got accuracy comparable to accurate mode, and we kept the original fast mode throughput. That’s a strong result for this type of emulation work.

Lu: It shows that you can use low-precision integer operations effectively while staying within the bounds required by the CRT structure.

Meng: It means more efficient matrix multiplication is possible without sacrificing reliability for large inputs, which is what we need when we’re running complex models.

Lalam: That’s the core idea: a scale-invariant scaling that prevents failures and keeps you right where you want it on the performance curve.

The paper's summary: Tom: So we’re looking at how they fixed that scaling problem in Ozaki Scheme II fast mode. Basically, they realized the old way of scaling things wasn't stable when you changed the input size, so it could cause errors or even make the math completely fail during CRT recovery.

Jane: Right, so if you scale your matrices by some number alpha, the precision of those intermediate steps kept changing as alpha got bigger—that’s what they were seeing. This new method uses something called the Cauchy–Schwarz inequality to derive a different scaling formula.

Lu: The big idea is that this new formula is scale invariant by construction. That means no matter how much you scale your input matrices, the effective precision of the scaled integer matrix stays constant and doesn't depend on that alpha factor anymore.

Meng: So what does that mean for me in production? It means we can push the inputs to their limits, and it won't suddenly break because of a scaling issue we didn't anticipate. That’s crucial for stable deployment.

Tom: Exactly, and they guarantee that the entries of the final scaled matrix product will never exceed P over two, which keeps that Chinese remainder theorem recovery working reliably without failure. That’s a hard bound on the error.

Jane: And what they achieve in their experiments is pretty strong. They manage to get accuracy comparable to their accurate mode while keeping the speed roughly where fast mode is supposed to be—that’s a really tough trade-off for matrix multiplication systems.

Lu: The paper shows that this approach successfully solves that old problem of having to choose between high accuracy at a slow speed or high speed with lower accuracy when things get big. They found a better balance.

Tom: And the good news is they didn't add any extra computational work to the original fast mode structure; it’s the same complexity as before, which makes it practical for real-world use. No new overhead there.

Jane: But they aren't saying this is perfect yet either; they admit that a full rigorous error analysis, just like what you do for accurate mode, is still a piece of work left to do in future research.

Lu: So the immediate implication is that we have a much more robust fast mode scaling mechanism now, one that doesn't suffer from those scale-related precision drops, but they need more formal proof on the error bounds later.

Tom: For someone listening who just wants to know what this means for their work, it means you can rely on this method to handle larger matrix operations with much greater stability than before. It’s a solid engineering fix for reliability under scaling.

Jane: So, while they've solved the immediate issue of scale invariance and guaranteed recovery, the next step is that deeper theoretical error checking we need to see more formal analysis on how it behaves in every possible edge case.

The paper's improvements: Tom: So, beyond just fixing the scaling formula itself, they are proposing a whole new way to think about how this works for more complex stuff later on. They want to extend this concept to complex matrix multiplication, things like ZGEMM and CGEMM.

Jane: That makes sense because right now it's all focused on real numbers, but if you want to run deep learning models that involve complex numbers, the current method might not hold up as well.

Lu: Exactly. They’re looking at how this scaling logic translates to those complex multiplication problems and trying to see if the same principles of scale invariance still apply there. It's an ambitious step for a fast mode technique.

Meng: If it works for complex numbers, that opens up a lot more possibilities for using these low-precision integer operations in advanced AI tasks where we already need those complex structures. That’s a big win for deployment versatility.

Tom: And they're also looking at hardware, specifically FP8 matrix engines. They want to see if this scaling logic can be integrated into those specialized accelerators to make the whole system faster and more efficient than standard fast mode implementations.

Jane: It’s like they’re trying to bridge the gap between the theoretical math of scaling and actual high-speed hardware implementation. If you can get that right, it means we could have even better performance on new chips.

Lu: The authors are also suggesting a more thorough error analysis for this improved version, something that matches how they analyzed the accurate mode before. They know that while the scaling is better now, a deeper proof of its stability across all conditions is needed to be completely certain.

Meng: I get that, and honestly, since they're already shipping on GH200 hardware with good results, a more robust scaling formula that works reliably in practice is what matters most for us right now. A solid result today trumps a perfect theoretical proof tomorrow for deployment decisions.

Tom: So the immediate implication is we have a clearer roadmap: first, they tackle complex multiplication and hardware integration, and second, they do the deeper error analysis later to lock down those guarantees even more tightly.

Jane: It shows they’re not just stopping at the scaling fix; they see this as a foundation for building a more versatile low-precision math toolkit for AI applications across different types of data.

Conclusion: Tom: So to wrap up, "Improved Scaling for Fast Mode of Ozaki Scheme II" basically fixes the core problem where scaling your input matrices messes up the precision of fast mode calculations by using a scale-invariant formula derived from Cauchy–Schwarz inequality.

Jane: Right, so the big picture is that they managed to keep the speed while ensuring accuracy comparable to accurate mode for these scaled integer matrix operations, which is a huge win for practical AI systems.

Lu: It really shows that even in low-precision integer math based on the Chinese remainder theorem, you can maintain stability when you scale inputs by any constant factor alpha. That's a solid foundation for more complex problems like ZGEMM later.

Meng: For us engineers, it means we can deploy this faster method with confidence knowing it won't suddenly break because of how large our input matrices are. It’s a reliable bump in performance for production pipelines.

Lalam: I'm confident in saying that the scaling factors they derived from that bound are sufficient conditions; they ensure the entries of the final product stay under P over two, which means CRT recovery is guaranteed to never fail under these conditions.

Tom: So we got speed, we got accuracy comparable to accurate mode, and we kept the original fast mode throughput without adding any extra computational cost. That’s a strong result for this type of emulation work.

Jane: It's a solid engineering fix because it doesn't add any new overhead to the original fast mode structure; that keeps things practical for real-world use.

Lu: And while they did get this stability win, they flagged that a full rigorous error analysis, similar to what they did for accurate mode, is still something that needs to be done later on by the authors themselves.

Meng: So we have a method that works better than before now, maintaining speed and accuracy for large inputs but it’s not yet fully proven to be theoretically airtight in every single edge case.

Tom: That's the honest assessment from their side; they got the immediate performance and stability win now, but they're flagging that a deeper theoretical error check is needed down the line.

Jane: It sounds like a complete picture then: we have an improved fast mode scaling formula that’s better than before, maintaining speed and accuracy, but it’s not yet fully proven to be theoretically airtight in every single edge case.

Lu: The idea is sound; the implementation works well on the GH200 GPU data they used. They've built a very strong piece of machinery here.

Meng: For production deployment, that means we can move forward with this improved fast mode formula now, knowing it’s better than what we were using before for large phi values.

Lalam: So in summary, "Improved Scaling for Fast Mode of Ozaki Scheme II" proposes a revised scaling formula derived from the Chinese remainder theorem uniqueness condition via Cauchy–Schwarz to ensure scale invariance and guaranteed error bounds.

Tom: It’s a significant improvement over the previous version because it fixes the issue where scaling inputs changed the effective precision of intermediate matrices.

Jane: So while they've solved the immediate issue of scale invariance and guaranteed recovery, the next step is that deeper theoretical error checking we need to see more formal analysis on how it behaves in every possible edge case.

Lu: It really shows that even in low-precision integer math based on the Chinese remainder theorem, you can maintain stability when you scale inputs by any constant factor alpha.

Meng: If there’s no added overhead, that’s great for my team because we don't have to worry about slowing down the inference pipeline when we switch from fast mode to this improved version.

Lalam: That’s the core idea: a scale-invariant scaling that prevents failures and keeps you right where you want it on the performance curve.

Jane: So in summary, "Improved Scaling for Fast Mode of Ozaki Scheme II" proposes a revised scaling formula derived from the Chinese remainder theorem uniqueness condition via Cauchy–Schwarz to ensure scale invariance and guaranteed error bounds.

Shota Kawakami, Daisuke Takahashi

Graduate School of Science and Technology, University of Tsukuba

cs.MS, cs.DC, cs.NA, math.NA

Submitted: 2026-06-28

Updated: 2026-11-09

Comments: Corrected the equations on p. 7 and fixed minor typos. The proposed method and the experimental results are unchanged

Code: https://github.com/RIKEN-RCCS/GEMMul8

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: Ozaki scheme II emulates high-precision matrix multiplication using low-precision integer matrix operations based on the Chinese remainder theorem (CRT) The proposed method achieves accuracy…

Key concepts

Scale Invariance
This property ensures that the performance and accuracy of the matrix multiplication method do not degrade when input matrices are multiplied by a constant scaling factor (alpha). The original fast mode failed this test, meaning scaling inputs caused precision issues. The new method is designed to be scale-invariant, meaning it works reliably regardless of how much the input matrices are scaled.
Chinese Remainder Theorem (CRT)
The CRT is a mathematical principle used here to ensure that the results of low-precision integer operations accurately represent the high-precision result. The authors use it to define conditions for scaling factors, guaranteeing that the entries of the resulting scaled matrix product stay within a threshold (P/2), which prevents errors during recovery.
Cauchy–Schwarz Inequality
This inequality is used as a tool to derive the necessary bounds for defining the scaling parameters (mu and nu). By applying this mathematical bound, the authors create a revised scaling formula that directly addresses the limitations of fast mode, ensuring that entries in the final matrix product are always bounded by P/2.
CRT Recovery Threshold (P/2)
This is a critical limit derived from the CRT. If any entry in the scaled integer matrix product exceeds this threshold, the method cannot reliably recover the original high-precision result. The proposed method guarantees that all entries remain below P/2, ensuring successful and reliable CRT recovery.

Terminology

Summary

Ozaki scheme II emulates high-precision matrix multiplication using low-precision integer matrix operations based on the Chinese remainder theorem (CRT) The proposed method achieves accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode

Core Concept and Motivation

The paper addresses the lack of scale invariance in the existing fast mode scaling formula, which causes accuracy degradation or CRT recovery failure when input matrices are scaled by a constant factor. The authors propose a revised scaling formula derived from the CRT uniqueness condition via the Cauchy–Schwarz inequality to ensure scale invariance and guarantee that entries of the scaled integer matrix product never exceed the CRT recovery threshold P/2. p. 0

Where the paper says it

We show that the existing formula lacks scale invariance; multiplying the input matrices by a constant changes the effective bit width of the integer matrices in the scaling step, causing accuracy degradation or CRT recovery failure. To address this, we propose a revised scaling formula derived from the CRT uniqueness condition via the Cauchy– Schwarz inequality.

This improved method is designed to overcome both the accuracy limitation of fast mode and the throughput constraint of accurate mode, offering a superior balance of accuracy and performance. p. 0
Where the paper says it

In the accuracy–throughput trade-off, the proposed method overcomes the accuracy limitation of fast mode and the throughput constraint of accurate mode, offering a superior accuracy and performance. Keywords matrix multiplication, floating-point arithmetic, matrix engine, Ozaki scheme 1 Introduction Modern accelerators such as graphics processing units (GPUs) have been designed with increasingly powerful low-precision arithmetic units, driven by growing demand from the artificial intelligence community.

Theoretical Limitation Analysis

The theoretical analysis demonstrates that in fast mode, the effective precision of the i-th row of the scaled integer matrix A' depends on a scalar factor α when input matrices are scaled by α. Specifically, this dependence leads to an effective precision that decreases as α increases, meaning the fast mode scaling lacks scale invariance. p. 4

Where the paper says it

+ 2 log2 (α) ! , this expression depends on α, and the effective precision of A′ decreases as α increases; hence, the fast mode scaling lacks scale invariance.

When 1 ≥ δ · logg2 X k=1 h=1 aˆ2 ih! p. 3
Where the paper says it

Thus, defining δ := single△ 0. 5 1 − 4u32 , (6) and evaluating log2 (x) as logg2 (x) := single△ log2f x 2 ⌊log2 (x)⌋ + ⌊log2 (x)⌋ , (7) (4) and (5) can be computed as follows: µi := µ ′−1 i · 2 ⌊P ′ accu−δ·logg2(max1≦h≦n c¯ih)⌋ , (8) νj := ν ′−1 j · 2 ⌊P ′ accu−δ·logg2(max1≦h≦m c¯hj )⌋ .

+ 2 log2(α)! p. 2
Where the paper says it

In accurate mode, an auxiliary INT8 GEMM is used to evaluate condition (1). First, we define µ ′ = (µ ′ i ) ∈ F m and ν ′ = ν ′ j ∈ F n from A and B, respectively, as follows: µ ′ i := 25−⌊log2(max1≦h≦k |aih|)⌋ , ν ′ j := 25−⌊log2(max1≦h≦k |bhj |)⌋ .

p. 2
Where the paper says it

In accurate mode, an auxiliary INT8 GEMM is used to evaluate condition (1). First, we define µ ′ = (µ ′ i ) ∈ F m and ν ′ = ν ′ j ∈ F n from A and B, respectively, as follows: µ ′ i := 25−⌊log2(max1≦h≦k |aih|)⌋ , ν ′ j := 25−⌊log2(max1≦h≦k |bhj |)⌋ .

p. 2
Where the paper says it

In accurate mode, an auxiliary INT8 GEMM is used to evaluate condition (1). First, we define µ ′ = (µ ′ i ) ∈ F m and ν ′ = ν ′ j ∈ F n from A and B, respectively, as follows: µ ′ i := 25−⌊log2(max1≦h≦k |aih|)⌋ , ν ′ j := 25−⌊log2(max1≦h≦k |bhj |)⌋ .

, the effective precision does not depend on α and is fixed at floor(P'fast - 1) + 1 bits, but A'B' may exceed P/2, causing CRT recovery to fail.

Proposed Solution Details

The proposed method defines µ and ν directly from the Cauchy–Schwarz bound (10), leading to sufficient conditions for the scaling factors. The authors define µi and νj using a formula derived from this bound, which is implemented in Algorithm 2. p. 6

Where the paper says it

4 Proposed Method We define µ and ν directly from the Cauchy–Schwarz bound (10). With the requirement of 2µi ∥ai,:∥2 ∥b:,j∥2 νj ≤ P − 1 < P to hold for all i, j, a sufficient condition is µi ≤ s P − 1 2 · 1 ∥ai,:∥2 , νj ≤ s P − 1 2 · 1 ∥b:,j∥2 .

This revised formula ensures that the effective bit width of the scaled integer matrix is constant and independent of α, and it guarantees that entries of A'B' are always bounded by P/2, ensuring CRT recovery never fails.

Performance Evaluation

Experiments on an NVIDIA GH200 GPU confirm that the proposed method achieves accuracy comparable to accurate mode for all values of ϕ. Furthermore, the throughput approaches that of the original fast mode as the number of moduli increases. p. 0

Where the paper says it

The proposed formula is scale-invariant by construction, guarantees that the CRT uniqueness condition is always satisfied, and introduces no additional overhead over the original fast mode. Experiments on an NVIDIA GH200 GPU show that the proposed method achieves accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode.

In terms of trade-off, OS II-prop is shown to achieve higher throughput than OS II-accu at any given accuracy level, resolving the accuracy degradation issue observed in OS II-fast for large ϕ. p. 8
Where the paper says it

Since the accuracy evaluation showed that OS II-prop maintains accuracy comparable to that of OS II-accu, OS II-prop delivers higher throughput at equivalent accuracy. Both OS II-prop and OS II-fast achieve similar throughput for ϕ = 0.

Conclusion and Future Work

The paper successfully proposes an improved fast mode scaling formula that maintains accuracy without sacrificing throughput. The proposed method's computational complexity is the same as that of the original fast mode, introducing no additional overhead. Future work includes extending this scaling formula to complex matrix multiplication (ZGEMM and CGEMM) and methods exploiting FP8 matrix engines. p. 1

Where the paper says it

Throughout this paper, we focus on DGEMM and SGEMM using INT8 matrix engines. However, since an analogous scaling formula is used in the fast mode of DGEMM and SGEMM with FP8 matrix engines and of ZGEMM and CGEMM with INT8 matrix engines, the proposed improvement is applicable to these cases as well.

A rigorous error analysis analogous to that performed for accurate mode is also suggested as future work. p. 3
Where the paper says it

A detailed error analysis of accurate mode is provided by Uchino et al. In fast mode, the estimation is performed using the Cauchy–Schwarz inequality.

Pruning redundant reasoning tokens

The authors show that the existing scaling formula for fast mode lacks scale invariance, which can cause accuracy degradation or even CRT recovery failure. The proposed method is scale-invariant by construction and guarantees that the entries of the scaled integer matrix product A'B' never exceed the CRT recovery threshold P/2.

How the training signal is built

The authors define µi as 2floor(P'prop - 0.5·(logg2(Pk h=1 a squared ih)+4u32)) <= s P-1/2 · 1 ai, squared, and analogously for νj. p. 6

Where the paper says it

We therefore define µi as the largest power of two satisfying this bound: µi := 2⌊P ′ prop−0. 5·(logg2( Pk h=1 a 2 ih)+4u32)⌋ ≤ s P − 1 2 · 1 ∥ai,:∥2 , (17) and analogously for νj : νj := 2⌊P ′ prop−0.

This definition is based on the bound derived from the Cauchy–Schwarz inequality. p. 0
Where the paper says it

We show that the existing formula lacks scale invariance; multiplying the input matrices by a constant changes the effective bit width of the integer matrices in the scaling step, causing accuracy degradation or CRT recovery failure. To address this, we propose a revised scaling formula derived from the CRT uniqueness condition via the Cauchy– Schwarz inequality.

Where the gains come from

The gains come from deriving a revised scaling formula directly from the CRT uniqueness condition via the Cauchy–Schwarz inequality. This approach ensures that entries of A'B' are always bounded by P/2, ensuring CRT recovery never fails. The proposed method achieves higher throughput than that of OS II-accu at any given accuracy level, resolving the accuracy degradation issue observed in OS II-fast for large ϕ. p. 0

Where the paper says it

The proposed formula is scale-invariant by construction, guarantees that the CRT uniqueness condition is always satisfied, and introduces no additional overhead over the original fast mode. Experiments on an NVIDIA GH200 GPU show that the proposed method achieves accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode.

The authors show that in the regime where 1 ≥ δ · logg2 X k=1 h=1 a squared ih! + 2 log2(α)!, the effective precision does not depend on α and is fixed at floor(P'fast - 1) + 1 bits; however, A'B' may exceed P/2, causing the CRT recovery to fail.

The gains come from deriving a revised scaling formula directly from the CRT uniqueness condition via the Cauchy–Schwarz inequality. This approach ensures that entries of A'B' are always bounded by P/2, ensuring CRT recovery never fails.

The authors show that in the regime where 1 ≥ δ · logg2 X k=1 h=1 a squared ih! + 2 log2(α)!, the effective precision does not depend on α and is fixed at floor(P'fast - 1) + 1 bits, but A'B' may exceed P/2, causing the CRT recovery to fail.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Revised fast mode scaling formula: The proposed formula is scale-invariant by construction and guarantees that the CRT uniqueness condition is always satisfied, which prevents accuracy degradation or CRT recovery failure when input matrices are scaled by a constant factor alpha.

  2. Improved accuracy for high-precision emulation: The system can achieve accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode across experiments, overcoming the accuracy limitation of fast mode and the throughput constraint of accurate mode.

  3. Superior trade-off management: The proposed method overcomes both the accuracy limitation of OS II-fast and the throughput constraint of OS II-accu, ensuring it achieves higher throughput than that of OS II-accu at equivalent accuracy across various matrix multiplication scenarios.

Abstract

Ozaki scheme II emulates high-precision matrix multiplication using low-precision integer matrix operations based on the Chinese remainder theorem (CRT). It first scales the high-precision matrices to convert them into integer matrices. For this scaling step, Ozaki scheme II provides two modes: accurate mode, which uses INT8 matrix multiplication to estimate scaling factors, and fast mode, which applies the Cauchy--Schwarz inequality at lower computational cost. We show that the existing formula lacks scale invariance; multiplying the input matrices by a constant changes the effective bit width of the integer matrices in the scaling step, causing accuracy degradation or CRT recovery failure. To address this, we propose a revised scaling formula derived from the CRT uniqueness condition via the Cauchy--Schwarz inequality. The proposed formula is scale-invariant by construction, guarantees that the CRT uniqueness condition is always satisfied, and introduces no additional overhead over the original fast mode. Experiments on an NVIDIA GH200 GPU show that the proposed method achieves accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode. In the accuracy--throughput trade-off, the proposed method overcomes the accuracy limitation of fast mode and the throughput constraint of accurate mode, offering a superior accuracy and performance.

Sources

Related papers