GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC

summary

The gist

GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC presents a porting of DG-SWEM to NVIDIA GPUs using OpenACC and Unified Memory…

In short

The study ported a Discontinuous Galerkin Shallow Water Equations model to NVIDIA GPUs using OpenACC and Unified Memory. This approach successfully achieved significant speedups, around 18 times faster than the CPU version, on a large Hurricane Harvey scenario running on Grace Hopper chips. It demonstrates that exploiting data parallelism in explicit DG methods simplifies porting and enhances maintainability.

Key concepts

Discontinuous Galerkin (DG) Formulation
This is a numerical method used to solve the shallow water equations by dividing the physical domain into small elements. The solution within each element is approximated using special basis functions, which allows for high accuracy in modeling fluid flow and wave propagation.
OpenACC with Unified Memory
OpenACC is a programming model that allows developers to specify parallel regions of code on GPUs using simple directives. Unified Memory lets the GPU access data stored on the main computer memory (host) without needing manual copying, automatically handling data migration when needed.

Terminology used across episodes

This episode discusses

The paper

GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC · Read on arXiv

Chayanon Wichitrnithed, Eirik Valseth, Ethan J. Kubatko, Shintaro Bunya, Clint Dawson

Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin · The Norwegian University of Life Sciences · Simula Research Laboratory · The Department of Civil, Environmental, and Geodetic Engineering, The Ohio State University · Coastal Resilience Center, The University of North Carolina at Chapel Hill · Institute of Marine Sciences, The University of North Carolina at Chapel Hill

DOI: 10.1061/JWPED5.WWENG-2426

Transcript

Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.

Kai: We're looking at the paper: "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC".

Mira: Here's the gist of the paper. GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC presents a porting of DG-SWEM to NVIDIA GPUs using OpenACC and Unified Memory, simplifying the porting process and maintaining a single codebase for both CPU and GPU versions.

Kai: The paper tests this code on a large Hurricane Harvey scenario on NVIDIA’s Grace Hopper chip, comparing its performance against the CPU version <ref:2508.21208#pg2>.

Mira: Model Formulation The governing model in DG-SWEM is the two-dimensional shallow water equations which consists of mass and momentum conservation in each direction <ref:2508.21208#pg5>.

Kai: The problem is formulated as finding (zeta, u) in Ω such that specific conservation laws are satisfied <ref:2508.21208#pg6>.

Title and authors: Mira: So, to recap what we just covered, the DG-SWEM paper is about taking a model used for storm surge simulation and getting it onto NVIDIA GPUs using OpenACC and Unified Memory, which turned out to be a pretty clean way to port it.

Kai: Right. The core of the model uses discontinuous Galerkin formulation on the primitive shallow water equations, which means they approximate the solution using polynomial basis functions local to each element, ensuring local conservation even though they don't have a single value at every node.

Lev: They are essentially synthesizing Galerkin and finite volume methods here, so you get that good local conservation property where coupling between adjacent elements is handled by a numerical flux. This is what makes it work for hyperbolic conservation laws in these kinds of physics problems.

Mira: The main problem they're tackling with DG-SWEM is the increased computational cost compared to the continuous formulation, which they state can be a primary disadvantage when trying to use it for operational forecasting like ADCIRC.

Kai: And this paper directly addresses that cost by mapping it onto the GPU, exploiting the fact that these methods have a large amount of data parallelism because each element computes its own terms locally. The authors show that this inherent structure is naturally suited for GPU architectures.

Lev: They use the time-explicit scheme, an optimized version of the strong stability preserving runge-kutta scheme, which they tune specifically with respect to the DG spatial discretizations to keep things stable while being efficient.

Mira: The results show that this mapping works well; they achieved about eighteen times speedup on a Hurricane Harvey scenario when running on the Grace Hopper chip. That's a big number in terms of practical simulation time reduction.

Kai: And they didn't just get a good result for one test case, they also provided some empirical performance metrics, showing that the AREA kernel was capable of achieving one point nine Teraflops per second in compute throughput on that hardware <ref:2508.21208#pg1>.

Lev: That throughput figure is interesting because it gives us a concrete measure of what the actual hardware can do when you feed it data structured this way through these DG operations.

Mira: So, essentially, they've shown that this specific approach—DG-SWEM ported with OpenACC and Unified Memory—is a viable pathway to making computationally expensive storm surge models much more accessible for faster use.

The paper's summary: Kai: Now let's talk about what the authors suggest they should do next, because they didn't just stop at getting the speedup. They identified a few areas for improvement in their work.

Mira: They point out that while data parallelism is present, achieving peak performance still depends heavily on things like how well you structure those edge integrations to exploit GPU shared memory better.

Lev: That ties into my point about poor spatial locality; the EDGE kernel showed a markedly lower intensity than they estimated, which means we need to focus on making those boundary calculations more efficient in terms of data reuse.

Kai: So, the suggestion is to look at how they are packing and unpacking ghost elements, because parallelizing that part can take advantage of the GPU's higher memory bandwidth better. They want to see if they can get a better return on investment from that specific data movement.

Mira: I think their final point is about improving communication strategies for multi-GPU setups. They suggest using MPI persistent communication with optimized packing and unpacking subroutines to handle data exchange between different domains more effectively.

Lev: That addresses the issue of data being transferred to the host and back to the device during MPI communication, which is a real bottleneck when you scale up across multiple nodes. It's about making that exchange smoother for a larger system.

Kai: So, it sounds like the path forward isn't just about getting one big speedup on one machine, but about refining how we manage data locality and inter-node communication when scaling up.

The paper's improvements: Mira: We've covered the full picture of this paper on "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC." They showed that porting DG-SWEM to GPUs is achievable with minimal code changes using OpenACC and Unified Memory.

Kai: And they quantified it by showing an eighteen times speedup on a large scenario, which gives us a concrete benchmark against the CPU version. The key finding for me is that the explicit time discretization in DG methods naturally lends itself to data parallelism through element and edge computations.

Lev: For real hardware, what I'd add is that while they got a big speedup on one node, scaling up across many GPUs still has overheads in communication that need deeper investigation.

Mira: So the DG-SWEM paper proves that this approach simplifies the programming effort for complex models by turning a complex problem into something more accessible through compiler directives.

Kai: It's about showing that the explicit time discretization in DG methods is inherently parallel, but realizing peak performance requires us to focus on data locality and better inter-node communication when we scale things up.

Lev: That's it for this paper. We'll be ready for the next one when we are.

Conclusion: Kai: So we looked at the paper "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC." Basically, they took a model for storm surge, which is super slow on a regular computer, and got it running much faster on an NVIDIA GPU.

Mira: Yeah. They used discontinuous Galerkin methods to solve the shallow water equations and then ported that whole thing onto the hardware using OpenACC directives.

Lev: The main takeaway for me is how they handled the data parallelism inherent in explicit time stepping, which means each element does its own math locally, making it a good fit for GPUs.

Kai: They got a speedup of around eighteen times when testing on a large Hurricane Harvey scenario running on their Grace Hopper chip. That’s not just an academic number; that’s a massive reduction in simulation time for operational forecasting.

Mira: But I have to pin that claim to the fact that they used Unified Memory, which means the GPU kernels can just access the data on the host without needing to manually copy everything back and forth.

Lev: That simplifies programming effort a lot, I agree. It cuts down on those painful manual data transfers we usually have to manage for real hardware.

Kai: The authors noted that OpenACC lets you keep a single code base; you just turn on the compilation flag and the GPU code gets generated automatically. That’s really clean for maintenance compared to setting up separate CUDA projects.

Mira: It’s about minimizing integration efforts, which is what they were aiming for when they chose OpenACC over more manual thread management approaches.

Lev: I just think that one hundred forty-seven lines of code added by the OpenACC directives is a trade-off for the massive simplification in the rest of the pipeline, especially with Unified Memory handling things automatically.

Kai: So to wrap up, "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC" shows that we can successfully port these complex fluid models to GPUs using standard tools and get significant speedups on real hardware.

Mira: It proves that exploiting the natural data parallelism in explicit DG methods is a solid path for making high-fidelity simulations more accessible for things like storm surge prediction.

Lev: For scaling up, the authors flagged that they still need to look deeper into how they structure edge integrations and how to optimize multi-GPU communication when you move beyond a single machine setup.

Kai: Exactly. It’s not just about the raw FLOPS number; it’s about making sure that data movement between those elements is as efficient as possible when you run big, real-world scenarios.

Mira: So the next step is figuring out how to push those boundary calculations even further toward exploiting shared memory on the GPU itself.

Lev: That sounds like a solid direction for future work if we want to move from single-node speedups to true multi-node scaling efficiency.

More episodes

← Home