EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations

summary

The gist

Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns The paper introduces…

In short

EBSDmagus is software designed to manage complex, multi-stage dynamical electron backscatter diffraction simulations. It streamlines workflows by defining parameters once in a YAML file, preparing inputs for EMsoft calculations, reusing intermediate results, and tracking the entire investigation history. This system makes managing large studies much easier and ensures data integrity during interruptions.

Key concepts

Workflow Management
This involves coordinating all steps of a complex simulation investigation—from defining material inputs to executing calculations and checking results. EBSDmagus uses a sequence where inputs are defined in YAML, EMsoft files are prepared using templates, and outputs are compared against the original request. It creates a portable record linking every output to its specific inputs for traceability.
Calculation Structure
The simulation follows a three-stage process within EMsoft: Monte Carlo (MC) for electron transport, Master Pattern (MP) combining MC results with crystal settings, and Screen Pattern (SP) which reads the combined file. EBSDmagus uses a directed graph to map these prerequisite relationships, showing how each calculation depends on the successful completion of an earlier one.
Data Integrity and Recovery
The system stores all run history, checks, and status records in an SQLite database alongside YAML inputs. If a simulation is interrupted, EBSDmagus compares multiple independent sources—like database entries, SLURM records, and output files—to keep completed work that passes validation checks. This ensures the study can be reliably recovered without losing progress.
Study Complexity Handling
The system allows researchers to define common settings and parameter sweeps in a single editable YAML file. This enables defining variations for many parameters (like beam energy or dmin values) across 530 linked EMsoft jobs simultaneously, drastically reducing the manual effort needed to coordinate and cross-check numerous configurations.

Terminology used across episodes

This episode discusses

The paper

EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations · Read on arXiv

Faculty of Georesources and Materials Engineering, RWTH Aachen University · Institute of Physical Metallurgy and Materials Physics, RWTH Aachen University

Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns. Investigations that combine composition and crystal structure, including changes in atomic order, with accelerating voltage, temperature, and many orientations or detector conditions can require large sets of related calculations. Several packages provide dynamical EBSD simulation; we use the open-source, scriptable EMsoft programs because they are well suited to large studies on high-performance computing systems. An EMsoft investigation nevertheless spans separate configuration, structure, scattering, and orientation files and several dependent calculation stages. As variations multiply, manual preparation becomes difficult to check, interruptions obscure which results remain usable, and the origin of individual patterns becomes laborious to reconstruct. With EBSDmagus, researchers define fixed and varying parameters once. The software prepares the required EMsoft calculations, reuses compatible intermediate results, and checks that the generated files represent the requested investigation before execution. After execution, it checks the expected outputs and, following an interruption, resumes only unresolved calculations while preserving completed work. A portable run record links each output to its inputs and calculation history, providing the provenance needed when the results are deposited as findable, accessible, interoperable, and reusable (FAIR) data. We assess this approach using a completed representative calculation, a larger study prepared and checked before submission, two observed cluster interruptions, and a controlled job cancellation. During recovery, EBSDmagus retained completed calculations and resubmitted only incomplete work.

Transcript

Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations".

Mira: Here's the gist of the paper. Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns The paper introduces EBSDmagus, a software that manages multi-stage dynamical electron backscatter diffraction simulations by defining fixed and varying parameters once, preparing the required EMsoft calculations, reuses compatible intermediate results, and checking generated files before execution This approach is designed to handle complex investigations where variations multiply, making manual preparation difficult and interruptions obscure usable results Workflow Management EBSDmagus coordinates the preparation, checking, execution history, and recovery when many such calls form one investigation The workflow follows a sequence involving defining material inputs in a YAML file, preparing EMsoft inputs via Jinja templates, comparing generated files with the requested study before submission, and then reviewing the result This process is managed through a portable run record that links each output to its inputs and calculation history, providing provenance needed for FAIR data principles Calculation Structure The simulation sequence typically involves three linked stages in EMsoft: Monte Carlo (MC), Master Pattern (MP), and Screen Pattern (SP) calculations The MC calculation describes electron transport, the MP calculation combines the matching MC result with crystal and scattering settings to calculate the master pattern, and the SP calculation reads that combined file for both transport and master-pattern information EBSDmagus uses a directed graph to represent these prerequisite relationships, where each box represents a calculation and arrows point from a required earlier result to the calculation that uses it Data Integrity and Recovery The portable database stores the run history, output checks, and status history in an SQLite file so that it can travel with YAML inputs and reports After an interruption, EBSDmagus compares available records, keeps completed work that passes defined checks, and identifies what remains to be run The recovery process compares database entries, SLURM records, filesystem outputs, worker-written.done completion markers, validation files, HDF5 output checks, and logs because these sources operate independently and may disagree Study Complexity Handling The system supports large studies by allowing common settings and parameter sweeps to be defined once in an editable YAML file The larger example study prepared 530 linked EMsoft jobs representing 96,000 detector patterns, demonstrating the benefit of linking variables across stages This capability allows researchers to specify variations in beam energy, depth limits, dmin values, and Bethe parameters in one YAML rather than assembling and cross-checking many configurations separately The final package includes the YAML file, database snapshots, reports, calculation graphs, and provenance traces for archiving Practical Implementation The practical user checklist guides the researcher through selecting a starting point, preparing the database by creating an SQLite database explicitly, describing the study in a YAML file, generating EMsoft inputs using Jinja templates via poetry run ebsdm prepare –config workflow configs/.yaml, and finally checking before submission using poetry run ebsdm check –preflight The software is available under the MIT licence at https://gitlab.git.nrw/rwth-sfb1394/sfb1394/ebsd/ebsdmagus The terminal user interface (TUI) allows users to select the YAML and database, prepare the calculation set, monitor job status, and perform recovery operations through explicit actions The software is available under the MIT licence at https://gitlab.git.nrw/rwth-sfb1394/sfb1394/ebsd/ebsdmagus EBSDmagus keeps a dynamical EBSD parameter study coherent from its definition through EMsoft execution and later archiving The representative YNi2 calculation completed the full three-stage route, and the larger beforerun example prepared 530 linked EMsoft jobs representing 96,000 detector patterns Larger studies therefore require less manual file coordination and job monitoring, while the information needed to inspect or reuse the resulting patterns is retained The records used during recovery also document how an archived output was produced, linking it to resolved settings and upstream calculations so publication does not depend on reconstructing this history from filenames The data and supporting records are collected in ebsdmagus-v1.3-manuscript-supporting-records.zip for deposition in a repository or institutional RDM system The funding was supported by the German Research Foundation (DFG) within the Collaborative Research Centre SFB 1394 "Structural and Chemical Atomic Complexity—From Defect Phase Diagrams to Materials Properties" (Project ID 409476157) The authors gratefully acknowledge the computing time provided to them at the NHR Center NHR4CES at RWTH Aachen University (project number p0020330) The data used in this publication were managed using the framework and metadata scheme provided by project A07 within SFB 1394 “Structural and Chemical Atomic Complexity – From Defect Phase Diagrams to Material Properties” (project ID 409476157) The data were managed using the research data management platform Coscine with storage space granted by the Research Data Storage (RDS) of the DFG and Ministry of Culture and Science of the State of North Rhine-Westphalia (DFG: INST222/1261-1 and MKW: 214-4.06.05.

Title and authors: Kai: We’ve looked at the title and authors now. EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations by Ulrich Kerzeli, Lukas Bernersi, and Sandra Korte-Kerzeli. The name itself tells you it's about managing those multi-stage simulations for EBSD >

Mira: Yeah. What the paper really focuses on is this systematic approach to how material and experimental parameters affect diffraction patterns when you have a lot of variables involved >

Lev: It addresses the issue that EMsoft investigations often span separate configuration, structure, scattering, and orientation files across several dependent calculation stages, and when variations multiply, manual preparation becomes difficult to check >

Kai: So it’s about moving away from those separate files toward a unified system where you define everything in one place >

Mira: Precisely. It outlines a workflow that links material inputs defined in a YAML file, prepares EMsoft inputs using Jinja templates, compares the generated files with your requested study before submission, and then reviews the final result >

Lev: The key mechanism here is this portable run record that connects each output to its specific inputs and calculation history. It provides the provenance needed for FAIR data principles when you archive everything >

The paper's summary: Kai: So, summarizing what EBSDmagus actually does, it coordinates the preparation, checking, execution history, and recovery when many of these simulation calls form one big investigation >

Mira: It sets up a sequence that involves defining material inputs in a YAML file first, then preparing EMsoft inputs via Jinja templates to get the required calculations ready >

Lev: Then there’s the crucial step of comparing those generated files with what you requested before you submit them to the cluster, which is a pre-flight check >

Kai: And after submission and execution, it reviews the result and compares that planned calculation with the actual output to make sure everything matched up correctly >

Mira: If they don't agree, it finds completed and incomplete work and reviews what needs to be run next, which is a recovery function >

Lev: The paper uses a directed graph to represent these prerequisite relationships between calculations. Each box is a calculation, and arrows point from an earlier result to the one that needs it >

Kai: So you have Monte Carlo MC, Master Pattern MP, and Screen Pattern SP calculations linked together in this dependency structure >

Mira: The core idea is using this graph to define the sequence correctly, ensuring you always feed the right information into the next stage of EMsoft calculation >

The paper's improvements: Kai: One major improvement they point out is allowing common settings and parameter sweeps to be defined once in an editable YAML file >

Mira: This means you can define variations in things like beam energy, depth limits, minimum d-min values, and Bethe parameters all in that one YAML instead of assembling and cross-checking many separate configurations separately >

Lev: That capability allows researchers to specify variations across different stages of the calculation sequence from a single file >

Kai: The paper illustrates this with a very large example study they prepared, involving five hundred thirty linked EMsoft jobs representing ninety-six thousand detector patterns >

Mira: So the benefit is that larger studies require far less manual file coordination and job monitoring because you’re linking variables across all stages directly in one place >

Lev: The final package includes the YAML file, database snapshots, reports, calculation graphs, and provenance traces for archiving; it makes inspecting or reusing those resulting patterns much easier >

Conclusion: Kai: So to wrap up this paper on "EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations," it keeps the entire dynamical EBSD parameter study coherent from its initial definition all the way through EMsoft execution and later archiving >

Mira: The implication is that for larger studies, you don’t need to manually manage every single file and job monitor everything; you get a system that documents how an archived output was produced by linking it to resolved settings and upstream calculations >

Lev: And for recovery operations specifically, the portable database stores the run history, output checks, and status history in an SQLite file so it can travel with your YAML inputs and reports >

Kai: It’s about making sure that when you look at a pattern later, you don't have to reconstruct the entire history from filenames; it’s documented right there in the records >

Mira: The representative calculation they used, like the YNi2 example, completed the full three-stage route successfully and then they show this workflow works on much larger scales too >

Lev: I just want to add that for real hardware execution, you need to be careful about where all those resources land when you’re running jobs linked by this system >

Kai: Exactly. The paper gives us the tools to manage the complexity, but as engineers, we still have to worry about how fast those dependencies actually resolve on a cluster >

Mira: So that’s it for EBSDmagus. It’s a powerful tool for keeping these dynamical studies organized from start to finish >

More episodes

← Home