BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX

summary

The gist

Modern CPUs increasingly integrate high-throughput matrix engines optimized for low-precision AI workloads…

In short

This work uses Intel AMX and BF16 matrix products to emulate high-precision FP32 and FP64 matrix multiplication on CPUs. It decomposes these high-precision operations into controlled collections of low-precision BF16 products. The method achieves FP32 accuracy for AMX-FP32 paths while allowing a tunable accuracy/cost tradeoff for AMX-FP64.

Key concepts

BF16 Component Decomposition
This technique breaks down an FP32 or FP64 matrix multiplication into several smaller, lower-precision BF16 matrix products. For example, an FP32 operation is represented by a combination of three BF16 components. This decomposition allows the high-precision calculation to be approximated using the faster low-precision hardware available in AMX.
AMX Tile Engine
Intel Advanced Matrix Extensions (AMX) provides a dense tile matrix engine within a general-purpose CPU. The paper leverages this engine, specifically its BF16 instructions, to perform the core low-precision matrix multiplications efficiently. This hardware is key to achieving speedups over standard software implementations.
Accuracy/Cost Tradeoff
The method allows users to control the accuracy of the emulation by choosing how many BF16 component products are retained in the calculation. For FP64, increasing this number improves agreement with DGEMM accuracy but increases computational cost. This lets engineers select a schedule that balances required numerical precision against performance needs.
Operand Reuse Schedule
For the AMX-FP32 path, a specific scheduling strategy is used to keep the FP32 AMX accumulator tiles active across multiple scheduled component products. This reuse of data minimizes costly data movement and tile loading, significantly improving efficiency for large matrix operations.

Terminology used across episodes

This episode discusses

The paper

BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX · Read on arXiv

Maginfra Co., Ltd.

Modern CPUs increasingly integrate high-throughput matrix engines optimized for low-precision AI workloads, while many scientific computing applications still rely on FP32 and FP64 GEMM to meet their numerical accuracy requirements. This paper presents a method that combines BF16 matrix products on Intel Advanced Matrix Extensions (AMX) to emulate real square FP32 and FP64 GEMM. For FP32 GEMM, each element in both input matrices is decomposed into three BF16 components, and six selected component products are evaluated to target FP32-level accuracy comparable to oneMKL SGEMM. For FP64 inputs within the supported BF16 exponent range, the method uses a simplified fixed six-slice Ozaki decomposition. Each retained BF16 product is first produced in FP32, then widened and accumulated in FP64. Four product-count settings retain 6, 10, 15, or 21 component products, exposing the accuracy-performance tradeoff relative to oneMKL DGEMM. The FP32 schedule reuses component data and retains partial sums in AMX tiles. AMX-FP32 outperforms oneMKL SGEMM at tested orders N at least4096 on 64 threads and also achieves gains for smaller matrices at lower thread counts. AMX-FP64 variants retaining fewer component products can outperform oneMKL DGEMM at sufficiently large orders; more products improve accuracy at additional cost.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX".

Jane: Here's the gist of the paper. Modern CPUs increasingly integrate high-throughput matrix engines optimized for low-precision AI workloads, while many scientific computing applications still rely on FP32 and FP64 GEMM to meet their numerical accuracy requirements <ref:2609.04663#pg2> This paper presents a CPU-oriented method based on Intel Advanced Matrix Extensions (AMX) and BF16 matrix products <ref:2609.04663#pg2> The core idea is to represent a high-precision matrix product as a controlled collection of low-precision matrix products <ref:2609.04663#pg2> Emulation Strategy The method investigates an algorithmic path across the precision barrier on CPUs by decomposing each FP32 or FP64 operand into several BF16 component matrices <ref:2609.04663#pg2>.

Tom: For FP32, it uses a three-component BF16 residual decomposition and evaluates six selected component products targeting FP32-level accuracy relative to oneMKL SGEMM <ref:2609.04663#pg2> For FP64 inputs within the supported BF16 exponent range, it applies a simplified fixed six-slice Ozaki decomposition where each retained BF16 product is produced in FP32 and then widened and accumulated in FP64 <ref:2609.04663#pg2>.

Jane: The product count is a tunable accuracy–cost control, with four settings retaining 6, 10, 15, or 21 component products to expose the accuracy–performance tradeoff relative to oneMKL DGEMM <ref:2609.04663#pg2>.

Tom: Hardware and Performance Analysis The implementation utilizes Intel Advanced Matrix Extensions (AMX) which provides a dense tile matrix engine into a general-purpose CPU <ref:2609.04663#pg2>.

Title and authors: Tom: So, we're looking at this paper now called "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX". It’s about taking those high-throughput matrix engines that modern CPUs have for low precision AI stuff and using them to actually do the heavy lifting for scientific computing that still needs high precision.

Jane: That sounds like a big bridge, Tom. So, what is the basic problem they're trying to solve here? Why can't we just use the low-precision matrix units for everything?

Tom: Well, many scientific applications need FP32 or FP64 accuracy for things like climate modeling or fluid dynamics, but these new AI matrix engines are optimized for lower precision, like BF16. This paper proposes an algorithmic bridge that uses those lower precision products to emulate the higher precision GEMM operations.

Lu: The core idea is to represent a high-precision matrix product by breaking down each FP32 or FP64 operand into several smaller BF16 component matrices and then evaluating a controlled collection of these low-precision products. This decomposition is the main way they try to get that higher precision back.

Meng: I'm thinking about what this means for deployment. If we can emulate FP32 or even FP64 using AMX, that’s huge for performance in those domains, but what’s the actual cost of this emulation?

Tom: The paper shows they use Intel Advanced Matrix Extensions, which is a tile matrix engine inside a general CPU. For FP32, they use a three component BF16 residual decomposition and evaluate six selected component products to aim for FP32 level accuracy compared to oneMKL SGEMM.

Jane: Three components and six products sounds like they are trying to find the right balance between computation time and keeping the final result accurate enough for scientific use. Are they claiming perfect elementwise identity here?

Lu: They specifically state they are not claiming bitwise or elementwise identity, which is important because it means they aren't trying to perfectly replicate a standard GEMM. Instead, they are targeting FP32-level accuracy relative to oneMKL SGEMM.

Meng: So for the FP64 inputs, what do we know about that path? Can this work for double precision data?

Tom: For FP64 inputs within the supported BF16 exponent range, they use a simplified fixed six slice Ozaki decomposition. Each BF16 product they retain is first produced in FP32, and then it gets widened and accumulated in FP64.

Jane: So for double precision data, they're doing a materialization step—converting the BF16 products to FP32 before accumulating them as FP64. That sounds like it adds some overhead on top of the low-precision computation, right?

Title and authors: Lu: Exactly. That conversion and accumulation step is what introduces the cost, but it lets them achieve that higher precision result for FP64 inputs when using AMX. They tune how many component products they keep, from six up to twenty-one, to find the sweet spot between speed and accuracy compared to oneMKL DGEMM.

Tom: And in terms of performance, they found that the AMX-FP32 path is faster than oneMKL SGEMM for all square sizes tested, but the speedup isn't smooth as you increase the matrix size.

Meng: Non-monotonic speedup? That’s a bit concerning for production. Does that mean if I scale up my problem, I might actually slow down compared to the standard library?

Jane: It means that while it works well for certain sizes, you have to watch how many component products you keep because the performance gain isn't just about scaling up the matrix size.

Lu: And they found that for AMX-FP64, low product-count variants can actually be faster than DGEMM at sufficiently large matrix orders. But as soon as you add more products to get better accuracy, the cost of storing and converting those extra FP32 results in FP64 starts to eat up the performance gain.

Tom: So it’s a clear tradeoff: you can get speed with fewer components, but if you need more precision, you have to pay for the extra accumulation and conversion work.

Jane: What about what this means for someone who just wants to run a simulation? Does this method actually give them usable FP32 or FP64 accuracy without having to use the full DGEMM routine?

Lu: It gives them a way to get close, but it’s not a free pass. You need that controlled collection of low-precision products and you have to manage how many you keep based on what your application can tolerate.

Meng: From an engineering standpoint, the implementation details matter here. They use microkernel dimensions like two fifty-six by two fifty-six by two fifty-six by thirty two by thirty two. That’s a pretty specific setup for how they are tiling things in the hardware.

Tom: And for the FP32 path, they kept those FP32 AMX accumulator tiles live across all six scheduled component products, which helped with data movement and tile load efficiency.

Jane: So that keeping the accumulators live is a key part of making it run efficiently on the CPU hardware. That’s a practical detail for someone trying to get this running in production.

Lu: And for the FP64 variants, they store each selected AMX component result as FP32, convert it to FP64, and then accumulate everything in FP64 outside of the AMX engine. That’s a very different data flow than the first path.

Title and authors: Tom: So we have two distinct paths here: one where we keep things in FP32 tiles for efficiency, and one where we materialize everything into FP64 for the final result.

Jane: This whole paper, "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX", shows how you can use hardware designed for low precision AI to bridge the gap to high precision scientific computing using decomposition.

Lu: It's a framework that uses the component-expansion principle, which they note isn't just specific to xeighty-six AMX, suggesting this idea could potentially be adapted to other CPUs that have high-throughput low-precision matrix hardware.

Meng: I’m interested in the limitations they mention. They state the attainable speedup is constrained by the current AMX tile register resources. If we need a bigger matrix than what fits in those registers, this method hits a wall and just won't scale further on that specific hardware setup.

Tom: Right, so it’s not just about the algorithm; it’s constrained by the physical resources of the specific chip you're running it on right now.

Jane: So what we have here is a technique that allows scientific computing to use low-precision matrix engines for speed while still hitting those high precision targets, provided you manage the decomposition and accept that there are tradeoffs in accuracy versus performance.

Lu: It’s less about getting perfect FP64 than it is about finding a controlled way to reconstruct a high-precision result from lower precision parts.

Meng: For deployment, it means we can use this for specific workloads where you can tune those product counts and accept the resulting accuracy profile for your needs.

Tom: So to wrap up, "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX" shows how to use AMX with BF16 products to emulate FP32 and FP64 matrix multiplication through controlled component decomposition and reconstruction.

Jane: It’s a solid piece of research because it gives a concrete, hardware-based method for people who need high precision but can't always afford the full DGEMM runtime.

Lu: The future work they suggest is extending this to rectangular or skinny matrices, which would be important for real-world applications that don't always use perfectly square matrices.

Meng: I hope we see more of this being ported to other platforms because being hardware specific like Intel AMX can limit where we can actually deploy these kinds of acceleration tricks.

Tom: That’s the main practical concern, right? So, as you go through these ideas for the next paper, keep an eye on how they expand that component-expansion idea into wider architectures.

The paper's summary: Tom: So, we're looking at this paper now called "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX". They’re basically taking these fast low-precision matrix engines meant for AI and forcing them to do the heavy lifting for high precision scientific math, like what you need for climate models.

Jane: It’s about representing a really big multiplication problem by breaking it down into smaller, lower-precision pieces and then putting those pieces back together using Intel's AMX instructions.

Lu: The core idea is that you take your FP32 or FP64 matrix product, split it into several BF16 component matrices, and evaluate a controlled selection of those products to try and get the final answer close to what you'd get from a standard high-precision operation.

Meng: So it’s an emulation strategy. They aren't just using the AI hardware for AI; they’re making it act like a more precise calculator for science. But how do you actually make that work on the chip?

Tom: For FP32, they use a three component BF16 residual decomposition and schedule six selected products to hit FP32 level accuracy when compared to the standard oneMKL SGEMM.

Jane: That means they are aiming for a very specific, known level of accuracy, not just some vague approximation. And for FP64 inputs, they use a fixed six slice Ozaki decomposition where each BF16 product is first made into FP32 and then widened into FP64 for accumulation.

Lu: The crucial part is the tuning. They test four settings where they keep six, ten, fifteen, or twenty-one component products to show exactly how much accuracy you get for every bit of performance you lose compared to the standard oneMKL DGEMM.

Meng: That sounds like a real engineering challenge because you have to manage that trade-off between how fast it runs and how close the final number is to the true answer.

Tom: And they found that for FP32, this AMX path is faster than oneMKL SGEMM across all square matrix sizes, but that speedup isn't smooth; it changes depending on how big your matrix gets.

Jane: So the implication here is that if you have a scientific workload where you can tolerate those specific accuracy trade-offs, you could get significant speedups by using this hardware instead of just running the standard double precision routines.

Lu: For FP64, they found that for very large matrix orders, keeping fewer products might actually give you a speed advantage over DGEMM, but as soon as you need more accuracy, the cost of converting those intermediate results to FP64 starts to take back the performance gain.

Meng: So it’s not a free way to get high precision; there’s always a measurable cost in terms of extra storage and conversion work.

Tom: Exactly. They also pointed out that they keep the FP32 accumulator tiles live across those six products for the FP32 path, which helps with data movement efficiency on the CPU side.

Jane: That detail about keeping those tiles live is important because it shows they optimized the data flow specifically for how Intel AMX handles its tile registers.

Lu: And they also noted that this decomposition idea isn't just for xeighty-six AMX; it’s a component expansion principle, so theoretically you could adapt this to other CPUs that have high-throughput low-precision matrix hardware.

Meng: That’s the big picture—if future hardware has better tile registers, maybe this whole framework can scale up even further.

Tom: It means the performance ceiling is currently set by how many tiles the AMX engine has available on your specific chip today.

Jane: So what this paper changes for us is showing a concrete way to use AI hardware for tasks that demand scientific precision, provided we understand exactly what accuracy we're getting versus what performance we're sacrificing.

Lu: It’s less about replacing DGEMM entirely and more about building a specific, precision-aware toolset on top of the existing low-precision matrix capabilities.

The paper's improvements: Tom: So, we’re talking about what these authors suggest as improvements for their method of emulating high precision math using low precision matrix engines on Intel AMX. They aren't just happy with the initial setup, they see a way to make it better.

Jane: Okay, so if the initial version was a proof of concept, what’s the next step in making this technique actually useful for bigger science problems?

Lu: They suggest extending these kernels and evaluations to handle more general matrix multiplication interfaces. Right now it’s mostly focused on square matrices, but they want to see how this decomposition works for rectangular or skinny matrices too.

Meng: That would be important because real-world simulations rarely use perfectly square problems; they often deal with different sized data sets.

Tom: And they also look at what happens when we move beyond the current limitations of the AMX tile registers. They explore what happens if future CPU matrix engines provide more tile registers or a larger tile capacity.

Jane: So if those future processors have bigger resources, this decomposition framework could potentially use bigger microkernels and keep more data on the chip for even better performance.

Lu: That’s the ambition; they see that component-expansion principle as something that isn't just specific to xeighty-six AMX hardware, suggesting it might work on other specialized low-precision matrix hardware as well.

Tom: So the authors are essentially laying out a roadmap for how this idea can be generalized beyond just Intel chips.

Jane: It’s interesting because it moves the discussion from a specific chip implementation to a more general mathematical representation of how these precision levels interact across different types of processors.

Meng: From an engineering standpoint, that means if we build our own accelerator hardware later, we have this existing idea they can use as a blueprint for how to handle precision mismatches.

Lu: Exactly. And they are also focusing on making sure the method remains robust when dealing with inputs outside the normal BF16 range, like handling potential overflow or underflow in FP64 inputs.

Tom: So they are trying to make it more reliable for real-world scientific data, not just perfect test cases.

Jane: It’s a step toward making these emulation techniques practical for actual high-performance computing where data isn't always perfectly behaved.

Conclusion: Tom: So we’re wrapping up this look at "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX". Basically, they showed how you can use low precision matrix engines to get high precision results for scientific computing.

Jane: It’s a solid framework because it shows a concrete way to bridge the gap between fast AI hardware and demanding numerical accuracy requirements.

Lu: The core finding is that you get this controlled collection of low-precision products, and by tuning how many you keep, you can find an accuracy cost tradeoff for FP64 inputs.

Meng: For production, it means we can use this to tailor the precision to the task at hand instead of just running a single standard routine.

Tom: Right. They found that for FP32, the AMX path is faster than its baseline SGEMM across all square sizes tested, even if that speedup isn't smooth as your matrix gets bigger.

Jane: That means for certain workloads, you might actually see a speed boost by using this hardware instead of the standard library code.

Lu: And for FP64 inputs, they found that low product counts can actually beat DGEMM at large matrix orders, but adding more products just increases the overhead from materialization and conversion to FP64.

Meng: So it’s a clear constraint: performance gains come with a direct cost in terms of extra work outside of the main AI engine.

Tom: The method relies heavily on Intel AMX instructions, specifically its BF16 tile instructions, to manage this emulation.

Jane: It’s important to remember that this specific setup is constrained by the current tile register resources on the chip you are using right now.

Lu: And they pointed out that for future work, they plan to extend this framework to rectangular or skinny matrices, which would be crucial for real-world applications that don't always use perfectly square problems.

Meng: If future CPU matrix engines provide more tile registers or bigger capacity, this whole decomposition idea could scale up even further on new hardware platforms.

Tom: So the main thing is that this component expansion principle isn't just locked into xeighty-six AMX, it’s a concept we can adapt to other high-throughput low precision matrix hardware.

Jane: It’s a good reminder that the potential for using AI hardware in scientific computing is bigger than we initially thought.

Lu: I think what this paper really changes for the field is showing a generalized way to use low precision products as building blocks for higher precision computation, regardless of the specific chip architecture.

Meng: I just hope we see more of this being ported to other platforms because being hardware specific like Intel AMX can limit where we can actually deploy these kinds of acceleration tricks.

Tom: Well said. So that’s the summary for "BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX".

Jane: It’s a solid piece of research showing how you can use low precision matrix engines to emulate high precision math through decomposition.

Lu: This is less about perfect emulation and more about finding a controlled way to reconstruct high precision results from lower precision parts.

Meng: It’s a framework that lets us tailor the precision to the task at hand while managing the performance cost.

More episodes

← Home