GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC

arXiv:2508.21208 · physics.comp-ph · Submitted 2025-08-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.

Kai: We're looking at the paper: "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC".

Mira: Here's the gist of the paper. GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC presents a porting of DG-SWEM to NVIDIA GPUs using OpenACC and Unified Memory, simplifying the porting process and maintaining a single codebase for both CPU and GPU versions.

Kai: The paper tests this code on a large Hurricane Harvey scenario on NVIDIA’s Grace Hopper chip, comparing its performance against the CPU version <ref:2508.21208#pg2>.

Mira: Model Formulation The governing model in DG-SWEM is the two-dimensional shallow water equations which consists of mass and momentum conservation in each direction <ref:2508.21208#pg5>.

Kai: The problem is formulated as finding (zeta, u) in Ω such that specific conservation laws are satisfied <ref:2508.21208#pg6>.

Title and authors: Mira: So, to recap what we just covered, the DG-SWEM paper is about taking a model used for storm surge simulation and getting it onto NVIDIA GPUs using OpenACC and Unified Memory, which turned out to be a pretty clean way to port it.

Kai: Right. The core of the model uses discontinuous Galerkin formulation on the primitive shallow water equations, which means they approximate the solution using polynomial basis functions local to each element, ensuring local conservation even though they don't have a single value at every node.

Lev: They are essentially synthesizing Galerkin and finite volume methods here, so you get that good local conservation property where coupling between adjacent elements is handled by a numerical flux. This is what makes it work for hyperbolic conservation laws in these kinds of physics problems.

Mira: The main problem they're tackling with DG-SWEM is the increased computational cost compared to the continuous formulation, which they state can be a primary disadvantage when trying to use it for operational forecasting like ADCIRC.

Kai: And this paper directly addresses that cost by mapping it onto the GPU, exploiting the fact that these methods have a large amount of data parallelism because each element computes its own terms locally. The authors show that this inherent structure is naturally suited for GPU architectures.

Lev: They use the time-explicit scheme, an optimized version of the strong stability preserving runge-kutta scheme, which they tune specifically with respect to the DG spatial discretizations to keep things stable while being efficient.

Mira: The results show that this mapping works well; they achieved about eighteen times speedup on a Hurricane Harvey scenario when running on the Grace Hopper chip. That's a big number in terms of practical simulation time reduction.

Kai: And they didn't just get a good result for one test case, they also provided some empirical performance metrics, showing that the AREA kernel was capable of achieving one point nine Teraflops per second in compute throughput on that hardware <ref:2508.21208#pg1>.

Lev: That throughput figure is interesting because it gives us a concrete measure of what the actual hardware can do when you feed it data structured this way through these DG operations.

Mira: So, essentially, they've shown that this specific approach—DG-SWEM ported with OpenACC and Unified Memory—is a viable pathway to making computationally expensive storm surge models much more accessible for faster use.

The paper's summary: Kai: Now let's talk about what the authors suggest they should do next, because they didn't just stop at getting the speedup. They identified a few areas for improvement in their work.

Mira: They point out that while data parallelism is present, achieving peak performance still depends heavily on things like how well you structure those edge integrations to exploit GPU shared memory better.

Lev: That ties into my point about poor spatial locality; the EDGE kernel showed a markedly lower intensity than they estimated, which means we need to focus on making those boundary calculations more efficient in terms of data reuse.

Kai: So, the suggestion is to look at how they are packing and unpacking ghost elements, because parallelizing that part can take advantage of the GPU's higher memory bandwidth better. They want to see if they can get a better return on investment from that specific data movement.

Mira: I think their final point is about improving communication strategies for multi-GPU setups. They suggest using MPI persistent communication with optimized packing and unpacking subroutines to handle data exchange between different domains more effectively.

Lev: That addresses the issue of data being transferred to the host and back to the device during MPI communication, which is a real bottleneck when you scale up across multiple nodes. It's about making that exchange smoother for a larger system.

Kai: So, it sounds like the path forward isn't just about getting one big speedup on one machine, but about refining how we manage data locality and inter-node communication when scaling up.

The paper's improvements: Mira: We've covered the full picture of this paper on "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC." They showed that porting DG-SWEM to GPUs is achievable with minimal code changes using OpenACC and Unified Memory.

Kai: And they quantified it by showing an eighteen times speedup on a large scenario, which gives us a concrete benchmark against the CPU version. The key finding for me is that the explicit time discretization in DG methods naturally lends itself to data parallelism through element and edge computations.

Lev: For real hardware, what I'd add is that while they got a big speedup on one node, scaling up across many GPUs still has overheads in communication that need deeper investigation.

Mira: So the DG-SWEM paper proves that this approach simplifies the programming effort for complex models by turning a complex problem into something more accessible through compiler directives.

Kai: It's about showing that the explicit time discretization in DG methods is inherently parallel, but realizing peak performance requires us to focus on data locality and better inter-node communication when we scale things up.

Lev: That's it for this paper. We'll be ready for the next one when we are.

Conclusion: Kai: So we looked at the paper "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC." Basically, they took a model for storm surge, which is super slow on a regular computer, and got it running much faster on an NVIDIA GPU.

Mira: Yeah. They used discontinuous Galerkin methods to solve the shallow water equations and then ported that whole thing onto the hardware using OpenACC directives.

Lev: The main takeaway for me is how they handled the data parallelism inherent in explicit time stepping, which means each element does its own math locally, making it a good fit for GPUs.

Kai: They got a speedup of around eighteen times when testing on a large Hurricane Harvey scenario running on their Grace Hopper chip. That’s not just an academic number; that’s a massive reduction in simulation time for operational forecasting.

Mira: But I have to pin that claim to the fact that they used Unified Memory, which means the GPU kernels can just access the data on the host without needing to manually copy everything back and forth.

Lev: That simplifies programming effort a lot, I agree. It cuts down on those painful manual data transfers we usually have to manage for real hardware.

Kai: The authors noted that OpenACC lets you keep a single code base; you just turn on the compilation flag and the GPU code gets generated automatically. That’s really clean for maintenance compared to setting up separate CUDA projects.

Mira: It’s about minimizing integration efforts, which is what they were aiming for when they chose OpenACC over more manual thread management approaches.

Lev: I just think that one hundred forty-seven lines of code added by the OpenACC directives is a trade-off for the massive simplification in the rest of the pipeline, especially with Unified Memory handling things automatically.

Kai: So to wrap up, "GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC" shows that we can successfully port these complex fluid models to GPUs using standard tools and get significant speedups on real hardware.

Mira: It proves that exploiting the natural data parallelism in explicit DG methods is a solid path for making high-fidelity simulations more accessible for things like storm surge prediction.

Lev: For scaling up, the authors flagged that they still need to look deeper into how they structure edge integrations and how to optimize multi-GPU communication when you move beyond a single machine setup.

Kai: Exactly. It’s not just about the raw FLOPS number; it’s about making sure that data movement between those elements is as efficient as possible when you run big, real-world scenarios.

Mira: So the next step is figuring out how to push those boundary calculations even further toward exploiting shared memory on the GPU itself.

Lev: That sounds like a solid direction for future work if we want to move from single-node speedups to true multi-node scaling efficiency.

Chayanon Wichitrnithed, Eirik Valseth, Ethan J. Kubatko, Shintaro Bunya, Clint Dawson

Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin · The Norwegian University of Life Sciences · Simula Research Laboratory · The Department of Civil, Environmental, and Geodetic Engineering, The Ohio State University · Coastal Resilience Center, The University of North Carolina at Chapel Hill · Institute of Marine Sciences, The University of North Carolina at Chapel Hill

physics.comp-ph

Submitted: 2025-08-28

Updated: 2026-11-12

DOI: 10.1061/JWPED5.WWENG-2426

Code: https://github.com/foci/dgswem

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 75/100

The gist: GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC presents a porting of DG-SWEM to NVIDIA GPUs using OpenACC and Unified Memory…

Key concepts

Discontinuous Galerkin (DG) Formulation
This is a numerical method used to solve the shallow water equations by dividing the physical domain into small elements. The solution within each element is approximated using special basis functions, which allows for high accuracy in modeling fluid flow and wave propagation.
OpenACC with Unified Memory
OpenACC is a programming model that allows developers to specify parallel regions of code on GPUs using simple directives. Unified Memory lets the GPU access data stored on the main computer memory (host) without needing manual copying, automatically handling data migration when needed.

Terminology

Summary

GPU-acceleration of the Discontinuous Galerkin Shallow Water Equations Model (DG-SWEM) with OpenACC presents a porting of DG-SWEM to NVIDIA GPUs using OpenACC and Unified Memory, simplifying the porting process and maintaining a single codebase for both CPU and GPU versions. p. 0 The paper tests this code on a large Hurricane Harvey scenario on NVIDIA’s Grace Hopper chip, comparing its performance against the CPU version <ref:2508. p. 021208#pg2>.

Model Formulation

The governing model in DG-SWEM is the two-dimensional shallow water equations which consists of mass and momentum conservation in each direction <ref:2508.21208#pg5>. The problem is formulated as finding (zeta, u) in Ω such that specific conservation laws are satisfied <ref:2508. p. 321208#pg6>. The state variables to be solved are defined as w = h zeta, uH, vH T <ref:2508. p. 421208#pg6>. These equations can be written in a conservative form as d w/d t + ∇ · F(w) = s for i = 1, 2, 3 <ref:2508. p. 421208#pg7>. The domain Ω is partitioned into non-overlapping elements Ω = Ø eΩe <ref:2508. p. 521208#pg8>.

Discretization and Parallelism

The Discontinuous Galerkin Formulation approximates the solution w in space using a set of basis functions based on the orthogonal Dubiner basis <ref:2508.21208#pg11>. This leads to a system of ODEs in the form d we/d t = Lh (we) on each element e, where terms from (3) are collected into the RHS Lh <ref:2508. p. 621208#pg13>. Time integration uses an optimized version of the explicit Strong Stability Preserving Runge-Kutta scheme (SSPRK) to discretize (5), which is optimized with respect to DG spatial discretizations <ref:2508. p. 621208#pg14>. The code supports parallel execution on CPUs through domain decomposition with METIS using MPI <ref:2508. p. 621208#pg15>.

GPU Programming Approach

Graphics processing units can be viewed as a multithreaded vector processor where data parallelism is particularly applicable due to the inherently local computation of the elements <ref:2508.21208#pg16>. The authors choose to use OpenACC in this work because it has better support on the NVHPC compilers, aiming to minimize integration and maintenance efforts <ref:2508. p. 721208#pg18>. Unified Memory allows GPU kernels to access data on the host without needing to explicitly copy data, with migration performed automatically once a page fault is triggered <ref:2508. p. 921208#pg11>.

Performance Results

The test case involves a large Hurricane Harvey dataset with 6,675,517 elements and 3,352,598 nodes <ref:2508.21208#pg3>. A node-by-node comparison observed a speedup of GPU code over CPU of around 18× <ref:2508. p. 821208#pg3>. Performance evaluation using the empirical Roofline showed that the AREA kernel achieves a compute throughput of 1. p. 69 TFLOPs/s <ref:2508. p. 0 p. 021208#pg6>. The EDGE kernel, due to poor spatial locality, has a markedly lower intensity than estimated, achieving 1.3 TFLOPs/s <ref:2508. p. 021208#pg6>.

Conclusion

The porting successfully exploited the data parallelism inherent in the explicit-time DG method through element and edge computations <ref:2508.21208#pg25>. The use of CUDA Unified Memory simplifies programming effort and enhances maintainability by requiring very few code modifications compared to previous frameworks <ref:2508. p. 921208#pg25>. Potential future research directions include a more detailed benchmark on multi-node scaling and exploring alternative ways to structure edge integration to exploit GPU shared memory <ref:2508. p. 221208#pg26>. The authors also note that using linear (p = 1) elements is typically insufficient to reach close to peak performance of the GPU <ref:2508. p. 021208#pg6>. The total lines of code increased by 147 when incorporating OpenACC directives <ref:2508. p. 021208#pg24>.

Data Availability

Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request <ref:2508.21208#pg25>. The Flood datasets used are also available for those interested in the results <ref:2508.21208#pg25>. The DG-SWEM code is available at https://github. p. 7com/foci/dgswem <ref:2508. p. 021208#pg25>.

Acknowledgements

This work has been supported by the United States National Science Foundation NSF PREEVENTS Track 2 Program, under NSF Grant Numbers 1855047 <ref:2508.21208#pg25>. This material is also based on work supported by the US Department of Homeland Security under Grant No. p. 42015-ST-061-ND0001-01 <ref:2508. p. 021208#pg25>. The authors also would like to gratefully acknowledge the use of the ”ADCIRC” allocation on the Vista supercomputer, and “ADCIRC”, “DMS23001”, and “DMS21031” allocations on the Frontera supercomputer at the Texas Advanced Computing Center at the University of Texas at Austin <ref:2508. p. 8 p. 821208#pg25>. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the US Department of Homeland Security <ref:2508.21208#pg25>. The authors also would like to gratefully acknowledge the use of the ”ADCIRC” allocation on the Vista supercomputer, and “ADCIRC”, “DMS23001”, and “DMS21031” allocations on the Frontera supercomputer at the Texas Advanced Computing Center at the University of Texas at Austin <ref:2508.21208#pg25>. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the US Department of Homeland Security <ref:2508.21208#pg25>.

References

The references listed support the methodology used in this study <ref:2508.21208#pg2>. The references include works on DG methods for storm surge modeling <ref:2508. p. 0 p. 021208#pg2>, and GPU acceleration frameworks such as CUDA and OpenACC. p. 0 p. 0 Specific related works include those by Dawson et al. p. 4 p. 4 (2011) on DG methods for hurricane storm surge <ref:2508. p. 0 p. 021208#pg2>, and Pachev (2024) on automated GPU acceleration of stabilized shallow water solvers with FEniCSx <ref:2508. p. 021208#pg9>. The performance analysis is supported by works like Yang (2020) on hierarchical roofline analysis <ref:2508. p. 0 p. 021208#pg6> and Hager and Wellein (2010) on high performance computing for scientists and engineers. p. 7 p. 7 The specific DG formulation relies on work by Kubatko et al. p. 4 p. 4 (2006) on hp discontinuous Galerkin methods for advection dominated problems in shallow water flow <ref:2508. p. 1 p. 121208#pg2>. The paper also references foundational work such as Tan (1992) on shallow water hydrodynamics <ref:2508. p. 3 p. 321208#pg5> and Cockburn (1998) on an introduction to the discontinuous Galerkin method for convectiondominated problems <ref:2508. p. 2 p. 221208#pg2>. The authors cite several other relevant papers concerning DG methods, including those by Dawson et al. p. 4 p. 4 (2024) on Swemnics <ref:2508. p. 0 p. 0 p. 021208#pg2>, and DuChene et al. p. 7 (2011) on a framework for running the ADCIRC discontinuous galerkin storm surge model on a GPU <ref:2508. p. 0 p. 021208#pg9>. The performance evaluation is further supported by empirical roofline analysis procedures described in Yang (2020) <ref:2508. p. 0 p. 0 p. 021208#pg6> and work by Hennessy and Patterson (2018) on computer architecture. p. 6 The authors also reference works concerning data movement in tightly coupled heterogeneous systems, such as Fusco et al. p. 9 (2024) on the grace hopper superchip <ref:2508. p. 021208#pg10>. The paper acknowledges the use of specific supercomputer allocations at TACC and Frontera <ref:2508. p. 821208#pg25>. The references also cover historical models like SLOSH by Jelesnianski (1992) <ref:2508. p. 121208#pg2>. The authors cite works on parallel programming interfaces such as OpenMP (Openmp. p. 7org 2021) <ref:2508. p. 721208#pg9>. The references provide context for the numerical methods used, including those by Hesthaven and Warburton (2008) on Nodal Discontinuous Galerkin Methods <ref:2508. p. 521208#pg2>. The paper also includes references to specific GPU implementations like those by Klockner et al. p. 2 (2009) on nodal discontinuous Galerkin methods on GPUs <ref:2508. p. 021208#pg2>. The authors reference the use of MPI for multi-node communication, as detailed in Karypis and Kumar (1997) on METIS <ref:2508. p. 621208#pg2>. The paper also includes references to works concerning data prefetching for unstructured mesh applications, such as Hadade et al. p. 0 (2020) <ref:2508.21208#pg6>. The authors cite Pringle et al. p. 1 (2020) on global storm tide modeling with adcirc v55 <ref:2508. p. 121208#pg9>. The references also include work by Shu and Osher (1988) on efficient implementation of essentially non-oscillatory shockcapturing schemes <ref:2508. p. 621208#pg2>. The paper also includes references to specific GPU programming guides from NVIDIA (2020) and CUDA C++ programming guide (NVIDIA 2024) <ref:2508. p. 721208#pg8>. The references also include works on the explicit Strong-Stability-Preserving Runge–Kutta time discretizations for discontinuous galerkin methods, such as Kubatko et al. p. 6 (2014) <ref:2508. p. 021208#pg2>. The paper also includes references to specific DG solvers and their GPU acceleration, including those by Dawson et al. p. 0 (2024) on Swemnics <ref:2508.21208#pg9> and Xia et al. p. 2 (2014) on OpenACC-based GPU acceleration of a 3-D unstructured discontinuous Galerkin method <ref:2508. p. 021208#pg9>. The paper also includes references to the specific hardware used, such as Singer (2024) on Vista supercomputer <ref:2508. p. 821208#pg10> and Fusco et al. p. 8 (2024) on understanding data movement in tightly coupled heterogeneous systems <ref:2508. p. 921208#pg10>. The references also include works by Hope et al. p. 4 (2013) on hindcast and validation of hurricane ike waves, forerunner, and storm surge <ref:2508. p. 121208#pg2>. The references also include foundational texts such as those by Hennessy and Patterson (2018) on computer architecture <ref:2508. p. 621208#pg2>. The references also include works by Veeramony et al. p. 1 (2017) on forecasting storm surge and inundation <ref:2508. p. 121208#pg5>. The references also cover the mathematical theory of shallow water equations from Tan (1992) <ref:2508. p. 321208#pg5>. The references also include works by Jelesnianski (1992) on SLOSH <ref:2508. p. 1 p. 121208#pg2>.

List of Figures

Figure 1 defines the definition of shallow water elevations where the horizontal line denotes the geoid, where zeta = hb = 0 <ref:2508.21208#pg5>. Figure 2 shows the TX2008 mesh used in the Harvey scenario, with color representing bathymetry, negative downwards <ref:2508. p. 021208#pg19>. Figure 3 compares the number of degrees of freedom computer per second for different numbers of compute nodes <ref:2508. p. 221208#pg20>. Figure 4 breaks down time spent in each computational kernel when running for 2,880 time steps <ref:2508. p. 221208#pg21>. Figure 5 shows the empirical DRAM single-precision roofline for the integration kernels <ref:2508. p. 921208#pg23>.

List of Tables

Table 1 lists a Grace Hopper (GH) GPU node with its specifications <ref:2508.21208#pg2>. Table 2 lists a Grace Grace (GG) CPU node with its specifications <ref:2508. p. 821208#pg11>. Table 3 lists the compiler flags used for the single-precision option <ref:2508. p. 921208#pg11>. Table 4 summarizes the primary data arrays in the code, detailing array meanings such as ZE[k,e,irk] and PHI AREA[k,i] <ref:2508. p. 921208#pg12>. Table 5 shows estimated and experimental DRAM arithmetic intensities for several kernels <ref:2508.21208#pg22>. Table 6 shows occupancy and instruction breakdown of the integration kernels <ref:2508.21208#pg24>. Table 7 details the line count of OpenACC directives used in the implementation <ref:2508. p. 921208#pg24>.

List of Algorithms

Algorithm 1 describes the Area integration (AREA) procedure, which computes terms from (3) and accumulates them into RHS vectors <ref:2508.21208#pg14>. Algorithm 2 details the Interior edge integration (EDGE) procedure for computing line integrals <ref:2508.21208#pg16>. Algorithm 3 outlines the Flux gathering kernel (GATHER), which accumulates contributions to the RHS vectors <ref:2508.21208#pg17>. The algorithms detail how parallel execution is structured, including gang-vector parallelism and sequential loops for inner computations. p. 6 The data transfers between host and device are managed using cudaMemPrefetchAsync to avoid large overheads <ref:2508. p. 921208#pg21>. The packing and unpacking of ghost elements are parallelized to take advantage of the higher memory bandwidth of the GPU. p. 6 The procedures illustrate how numerical fluxes are calculated and added back to the RHS vectors using weights derived from quadrature points. p. 6 The results show that parallel communication time in the GPU case increases with more GPUs to around 6 s at 8 nodes, resulting in a scaling plateau <ref:2508.21208#pg5>. The empirical roofline shows that the AREA kernel achieves a compute throughput of 1. p. 69 TFLOPs/s <ref:2508.21208#pg6>. The paper concludes that there are potential performance issues like overlapping computation and communication that can be further investigated <ref:2508.21208#pg26>.

The authors successfully ported DG-SWEM to NVIDIA GPUs using OpenACC Fortran by exploiting the data parallelism inherent in the explicit-time DG method through element and edge computations <ref:2508.21208#pg2>. When combined with the CUDA Unified Memory System, the porting requires very few code modifications and can be maintained in the same code base <ref:2508. p. 921208#pg2>. Significant speedups over the original CPU implementation were observed when tested on multiple nodes on the NVIDIA Grace Hopper system <ref:2508. p. 021208#pg3>. The work demonstrates that for explicit DG methods, high degrees of data parallelism are inherent due to local computation of elements <ref:2508. p. 721208#pg16>.

The references listed support the methodology used in this study <ref:2508.21208#pg2>. The references include works on DG methods for storm surge modeling <ref:2508.21208#pg2>, and GPU acceleration frameworks such as CUDA and OpenACC. Specific related works include those by Dawson et al. (2011) on DG methods for hurricane storm surge <ref:2508.21208#pg9>, and Pachev (2024) on automated GPU acceleration of stabilized shallow water solvers with FEniCSx <ref:2508. p. 021208#pg9>. The performance analysis is supported by works like Yang (2020) on hierarchical roofline analysis <ref:2508.21208#pg6> and Hager and Wellein (2010) on high performance computing for scientists and engineers. The specific DG formulation relies on work by Kubatko et al. (2006) on hp discontinuous Galerkin methods for advection dominated problems in shallow water flow <ref:2508.21208#pg2>. The paper also references foundational work such as Tan (1992) on shallow water hydrodynamics <ref:2508.21208#pg5> and Cockburn (1998) on an introduction to the discontinuous Galerkin method for convectiondominated problems <ref:2508.21208#pg2>. The authors cite several other relevant papers concerning DG methods, including those by Dawson et al. (2024) on Swemnics <ref:2508.21208#pg9>, and DuChene et al. p. 7 (2011) on a framework for running the ADCIRC discontinuous galerkin storm surge model on a GPU <ref:2508.21208#pg9>. The performance evaluation is further supported by empirical roofline analysis procedures described in Yang (2020) <ref:2508.21208#pg6> and work by Hennessy and Patterson (26pg1, 26pg1>. p. 6 The references also cover the mathematical theory of shallow water equations from Tan (4pg4). p. 3 The references also include works by Jelesnianski (1992) on SLOSH <ref:2508.21208#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. GPU acceleration of Discontinuous Galerkin Shallow Water Equations (DG-SWEM) enables faster simulation of hurricane storm surge by exploiting data parallelism inherent in explicit-time DG methods through element and edge computations. This allows for significantly reduced computational cost compared to the original CPU implementation, achieving speedups of around 18 on multi-node NVIDIA Grace Hopper systems.

  2. Unified Memory implementation simplifies the porting process by allowing GPU kernels to access data on the host without needing to explicitly copy data, which significantly simplifies programming effort and enhances maintainability. This capability is achieved by using the compilation flag-gpu=unified in newer systems that support Heterogenous Memory Management (HMM).

  3. OpenACC directives are used instead of manual thread management, allowing for a single code base — only the compilation flags such as-acc need to be turned on to enable the GPU code generation. This approach minimizes integration and maintenance efforts compared to previous frameworks, increasing the line count by 147 while maintaining a single codebase.

  4. The performance evaluation using empirical Roofline analysis helps determine achievable performance by relating peak FLOPs/s to DRAM bandwidth through the relation: Attainable performance (FLOPs/s) = min (Peak FLOPs/s, alpha × Peak DRAM bandwidth). This guides future optimization efforts toward improving data locality, as seen by the lower arithmetic intensity of EDGE kernels compared to AREA kernels.

  5. Improved parallel communication strategies for multi-GPU setups involve using MPI persistent communication with optimized packing and unpacking subroutines that are parallelized to leverage higher memory bandwidth, as shown in Listing 3. This addresses the limitation where data being transferred to host and back to device during MPI communication occurs, allowing for better data exchange among different domains.

Related papers