EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations

arXiv:2609.08334 · cond-mat.mtrl-sci · Submitted 2026-09-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations".

Mira: Here's the gist of the paper. Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns The paper introduces EBSDmagus, a software that manages multi-stage dynamical electron backscatter diffraction simulations by defining fixed and varying parameters once, preparing the required EMsoft calculations, reuses compatible intermediate results, and checking generated files before execution This approach is designed to handle complex investigations where variations multiply, making manual preparation difficult and interruptions obscure usable results Workflow Management EBSDmagus coordinates the preparation, checking, execution history, and recovery when many such calls form one investigation The workflow follows a sequence involving defining material inputs in a YAML file, preparing EMsoft inputs via Jinja templates, comparing generated files with the requested study before submission, and then reviewing the result This process is managed through a portable run record that links each output to its inputs and calculation history, providing provenance needed for FAIR data principles Calculation Structure The simulation sequence typically involves three linked stages in EMsoft: Monte Carlo (MC), Master Pattern (MP), and Screen Pattern (SP) calculations The MC calculation describes electron transport, the MP calculation combines the matching MC result with crystal and scattering settings to calculate the master pattern, and the SP calculation reads that combined file for both transport and master-pattern information EBSDmagus uses a directed graph to represent these prerequisite relationships, where each box represents a calculation and arrows point from a required earlier result to the calculation that uses it Data Integrity and Recovery The portable database stores the run history, output checks, and status history in an SQLite file so that it can travel with YAML inputs and reports After an interruption, EBSDmagus compares available records, keeps completed work that passes defined checks, and identifies what remains to be run The recovery process compares database entries, SLURM records, filesystem outputs, worker-written.done completion markers, validation files, HDF5 output checks, and logs because these sources operate independently and may disagree Study Complexity Handling The system supports large studies by allowing common settings and parameter sweeps to be defined once in an editable YAML file The larger example study prepared 530 linked EMsoft jobs representing 96,000 detector patterns, demonstrating the benefit of linking variables across stages This capability allows researchers to specify variations in beam energy, depth limits, dmin values, and Bethe parameters in one YAML rather than assembling and cross-checking many configurations separately The final package includes the YAML file, database snapshots, reports, calculation graphs, and provenance traces for archiving Practical Implementation The practical user checklist guides the researcher through selecting a starting point, preparing the database by creating an SQLite database explicitly, describing the study in a YAML file, generating EMsoft inputs using Jinja templates via poetry run ebsdm prepare –config workflow configs/.yaml, and finally checking before submission using poetry run ebsdm check –preflight The software is available under the MIT licence at https://gitlab.git.nrw/rwth-sfb1394/sfb1394/ebsd/ebsdmagus The terminal user interface (TUI) allows users to select the YAML and database, prepare the calculation set, monitor job status, and perform recovery operations through explicit actions The software is available under the MIT licence at https://gitlab.git.nrw/rwth-sfb1394/sfb1394/ebsd/ebsdmagus EBSDmagus keeps a dynamical EBSD parameter study coherent from its definition through EMsoft execution and later archiving The representative YNi2 calculation completed the full three-stage route, and the larger beforerun example prepared 530 linked EMsoft jobs representing 96,000 detector patterns Larger studies therefore require less manual file coordination and job monitoring, while the information needed to inspect or reuse the resulting patterns is retained The records used during recovery also document how an archived output was produced, linking it to resolved settings and upstream calculations so publication does not depend on reconstructing this history from filenames The data and supporting records are collected in ebsdmagus-v1.3-manuscript-supporting-records.zip for deposition in a repository or institutional RDM system The funding was supported by the German Research Foundation (DFG) within the Collaborative Research Centre SFB 1394 "Structural and Chemical Atomic Complexity—From Defect Phase Diagrams to Materials Properties" (Project ID 409476157) The authors gratefully acknowledge the computing time provided to them at the NHR Center NHR4CES at RWTH Aachen University (project number p0020330) The data used in this publication were managed using the framework and metadata scheme provided by project A07 within SFB 1394 “Structural and Chemical Atomic Complexity – From Defect Phase Diagrams to Material Properties” (project ID 409476157) The data were managed using the research data management platform Coscine with storage space granted by the Research Data Storage (RDS) of the DFG and Ministry of Culture and Science of the State of North Rhine-Westphalia (DFG: INST222/1261-1 and MKW: 214-4.06.05.

Title and authors: Kai: We’ve looked at the title and authors now. EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations by Ulrich Kerzeli, Lukas Bernersi, and Sandra Korte-Kerzeli. The name itself tells you it's about managing those multi-stage simulations for EBSD >

Mira: Yeah. What the paper really focuses on is this systematic approach to how material and experimental parameters affect diffraction patterns when you have a lot of variables involved >

Lev: It addresses the issue that EMsoft investigations often span separate configuration, structure, scattering, and orientation files across several dependent calculation stages, and when variations multiply, manual preparation becomes difficult to check >

Kai: So it’s about moving away from those separate files toward a unified system where you define everything in one place >

Mira: Precisely. It outlines a workflow that links material inputs defined in a YAML file, prepares EMsoft inputs using Jinja templates, compares the generated files with your requested study before submission, and then reviews the final result >

Lev: The key mechanism here is this portable run record that connects each output to its specific inputs and calculation history. It provides the provenance needed for FAIR data principles when you archive everything >

The paper's summary: Kai: So, summarizing what EBSDmagus actually does, it coordinates the preparation, checking, execution history, and recovery when many of these simulation calls form one big investigation >

Mira: It sets up a sequence that involves defining material inputs in a YAML file first, then preparing EMsoft inputs via Jinja templates to get the required calculations ready >

Lev: Then there’s the crucial step of comparing those generated files with what you requested before you submit them to the cluster, which is a pre-flight check >

Kai: And after submission and execution, it reviews the result and compares that planned calculation with the actual output to make sure everything matched up correctly >

Mira: If they don't agree, it finds completed and incomplete work and reviews what needs to be run next, which is a recovery function >

Lev: The paper uses a directed graph to represent these prerequisite relationships between calculations. Each box is a calculation, and arrows point from an earlier result to the one that needs it >

Kai: So you have Monte Carlo MC, Master Pattern MP, and Screen Pattern SP calculations linked together in this dependency structure >

Mira: The core idea is using this graph to define the sequence correctly, ensuring you always feed the right information into the next stage of EMsoft calculation >

The paper's improvements: Kai: One major improvement they point out is allowing common settings and parameter sweeps to be defined once in an editable YAML file >

Mira: This means you can define variations in things like beam energy, depth limits, minimum d-min values, and Bethe parameters all in that one YAML instead of assembling and cross-checking many separate configurations separately >

Lev: That capability allows researchers to specify variations across different stages of the calculation sequence from a single file >

Kai: The paper illustrates this with a very large example study they prepared, involving five hundred thirty linked EMsoft jobs representing ninety-six thousand detector patterns >

Mira: So the benefit is that larger studies require far less manual file coordination and job monitoring because you’re linking variables across all stages directly in one place >

Lev: The final package includes the YAML file, database snapshots, reports, calculation graphs, and provenance traces for archiving; it makes inspecting or reusing those resulting patterns much easier >

Conclusion: Kai: So to wrap up this paper on "EBSDmagus: Managing Multi-Stage Dynamical Electron Backscatter Diffraction Simulations," it keeps the entire dynamical EBSD parameter study coherent from its initial definition all the way through EMsoft execution and later archiving >

Mira: The implication is that for larger studies, you don’t need to manually manage every single file and job monitor everything; you get a system that documents how an archived output was produced by linking it to resolved settings and upstream calculations >

Lev: And for recovery operations specifically, the portable database stores the run history, output checks, and status history in an SQLite file so it can travel with your YAML inputs and reports >

Kai: It’s about making sure that when you look at a pattern later, you don't have to reconstruct the entire history from filenames; it’s documented right there in the records >

Mira: The representative calculation they used, like the YNi2 example, completed the full three-stage route successfully and then they show this workflow works on much larger scales too >

Lev: I just want to add that for real hardware execution, you need to be careful about where all those resources land when you’re running jobs linked by this system >

Kai: Exactly. The paper gives us the tools to manage the complexity, but as engineers, we still have to worry about how fast those dependencies actually resolve on a cluster >

Mira: So that’s it for EBSDmagus. It’s a powerful tool for keeping these dynamical studies organized from start to finish >

Faculty of Georesources and Materials Engineering, RWTH Aachen University · Institute of Physical Metallurgy and Materials Physics, RWTH Aachen University

cond-mat.mtrl-sci

Submitted: 2026-09-08

Updated: 2026-11-05

Comments: version accepted for publication

Code: https://github.com/EMsoftorg/EMsoft

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns The paper introduces…

Key concepts

Workflow Management
This involves coordinating all steps of a complex simulation investigation—from defining material inputs to executing calculations and checking results. EBSDmagus uses a sequence where inputs are defined in YAML, EMsoft files are prepared using templates, and outputs are compared against the original request. It creates a portable record linking every output to its specific inputs for traceability.
Calculation Structure
The simulation follows a three-stage process within EMsoft: Monte Carlo (MC) for electron transport, Master Pattern (MP) combining MC results with crystal settings, and Screen Pattern (SP) which reads the combined file. EBSDmagus uses a directed graph to map these prerequisite relationships, showing how each calculation depends on the successful completion of an earlier one.
Data Integrity and Recovery
The system stores all run history, checks, and status records in an SQLite database alongside YAML inputs. If a simulation is interrupted, EBSDmagus compares multiple independent sources—like database entries, SLURM records, and output files—to keep completed work that passes validation checks. This ensures the study can be reliably recovered without losing progress.
Study Complexity Handling
The system allows researchers to define common settings and parameter sweeps in a single editable YAML file. This enables defining variations for many parameters (like beam energy or dmin values) across 530 linked EMsoft jobs simultaneously, drastically reducing the manual effort needed to coordinate and cross-check numerous configurations.

Terminology

Summary

Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns The paper introduces EBSDmagus, a software that manages multi-stage dynamical electron backscatter diffraction simulations by defining fixed and varying parameters once, preparing the required EMsoft calculations, reuses compatible intermediate results, and checking generated files before execution This approach is designed to handle complex investigations where variations multiply, making manual preparation difficult and interruptions obscure usable results

Workflow Management

EBSDmagus coordinates the preparation, checking, execution history, and recovery when many such calls form one investigation The workflow follows a sequence involving defining material inputs in a YAML file, preparing EMsoft inputs via Jinja templates, comparing generated files with the requested study before submission, and then reviewing the result This process is managed through a portable run record that links each output to its inputs and calculation history, providing provenance needed for FAIR data principles

Calculation Structure

The simulation sequence typically involves three linked stages in EMsoft: Monte Carlo (MC), Master Pattern (MP), and Screen Pattern (SP) calculations The MC calculation describes electron transport, the MP calculation combines the matching MC result with crystal and scattering settings to calculate the master pattern, and the SP calculation reads that combined file for both transport and master-pattern information EBSDmagus uses a directed graph to represent these prerequisite relationships, where each box represents a calculation and arrows point from a required earlier result to the calculation that uses it

Data Integrity and Recovery

The portable database stores the run history, output checks, and status history in an SQLite file so that it can travel with YAML inputs and reports After an interruption, EBSDmagus compares available records, keeps completed work that passes defined checks, and identifies what remains to be run The recovery process compares database entries, SLURM records, filesystem outputs, worker-written.done completion markers, validation files, HDF5 output checks, and logs because these sources operate independently and may disagree

Study Complexity Handling

The system supports large studies by allowing common settings and parameter sweeps to be defined once in an editable YAML file The larger example study prepared 530 linked EMsoft jobs representing 96,000 detector patterns, demonstrating the benefit of linking variables across stages This capability allows researchers to specify variations in beam energy, depth limits, dmin values, and Bethe parameters in one YAML rather than assembling and cross-checking many configurations separately The final package includes the YAML file, database snapshots, reports, calculation graphs, and provenance traces for archiving

Practical Implementation

The practical user checklist guides the researcher through selecting a starting point, preparing the database by creating an SQLite database explicitly, describing the study in a YAML file, generating EMsoft inputs using Jinja templates via poetry run ebsdm prepare –config workflow configs/.yaml, and finally checking before submission using poetry run ebsdm check –preflight The software is available under the MIT licence at https://gitlab. p. 0

Where the paper says it

A portable run record links each output to its inputs and calculation history, providing the provenance needed when the results are deposited as findable, accessible, interoperable, and reusable (FAIR) data. We assess this approach using a completed representative calculation, a larger study prepared and checked before submission, two observed cluster interruptions, and a controlled job cancellation.

git.nrw/rwth-sfb1394/sfb1394/ebsd/ebsdmagus The terminal user interface (TUI) allows users to select the YAML and database, prepare the calculation set, monitor job status, and perform recovery operations through explicit actions The software is available under the MIT licence at https://gitlab.git.nrw/rwth-sfb1394/sfb1394/ebsd/ebsdmagus

Conclusion

EBSDmagus keeps a dynamical EBSD parameter study coherent from its definition through EMsoft execution and later archiving The representative YNi2 calculation completed the full three-stage route, and the larger beforerun example prepared 530 linked EMsoft jobs representing 96,000 detector patterns Larger studies therefore require less manual file coordination and job monitoring, while the information needed to inspect or reuse the resulting patterns is retained The records used during recovery also document how an archived output was produced, linking it to resolved settings and upstream calculations so publication does not depend on reconstructing this history from filenames The data and supporting records are collected in ebsdmagus-v1.3-manuscript-supporting-records. p. 1

Where the paper says it

We use EMsoft [9] because its open-source command-line programs can be scripted and distributed across high-performance computing (HPC) systems, making it particularly suitable for large parameter studies. In EMsoft, an EBSD pattern is produced by a sequence of linked programs, each with its own configuration and supporting files.

zip for deposition in a repository or institutional RDM system The funding was supported by the German Research Foundation (DFG) within the Collaborative Research Centre SFB 1394 Structural and Chemical Atomic Complexity—From Defect Phase Diagrams to Materials Properties (Project ID 409476157) The authors gratefully acknowledge the computing time provided to them at the NHR Center NHR4CES at RWTH Aachen University (project number p0020330) The data used in this publication were managed using the framework and metadata scheme provided by project A07 within SFB 1394 “Structural and Chemical Atomic Complexity – From Defect Phase Diagrams to Material Properties” (project ID 409476157) The data were managed using the research data management platform Coscine with storage space granted by the Research Data Storage (RDS) of the DFG and Ministry of Culture and Science of the State of North Rhine-Westphalia (DFG: INST222/1261-1 and MKW: 214-4.06.05.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated workflow preparation can be improved by defining common settings and parameter sweeps once in an editable YAML file, which EBSDmagus then resolves into required EMsoft calculations and generates a consistent set of input files.

  2. The system can be enhanced to perform before-run checks by comparing generated NML, angle, Bethe-parameter, and metadata files with the database and calculation graph before requesting cluster resources.

  3. The AI system should be able to generate a directed graph representing prerequisite relationships between calculations, allowing it to use this graph to start independent branches when resources are available as described in Section 3.1.

  4. For recovery operations, the system can be improved by comparing database entries, SLURM records, filesystem outputs, worker-written .done completion markers and validation files because these sources operate independently and may disagree.

  5. The system can implement a check for rejected inputs and output mismatches by verifying that the parent selection An MP job points to an MC result whose beam energy, depth settings, structure, or other parent-defining parameters contradict the requested calculation.

Abstract

Dynamical electron backscatter diffraction (EBSD) simulations increasingly support systematic studies of how material and experimental parameters affect diffraction patterns. Investigations that combine composition and crystal structure, including changes in atomic order, with accelerating voltage, temperature, and many orientations or detector conditions can require large sets of related calculations. Several packages provide dynamical EBSD simulation; we use the open-source, scriptable EMsoft programs because they are well suited to large studies on high-performance computing systems. An EMsoft investigation nevertheless spans separate configuration, structure, scattering, and orientation files and several dependent calculation stages. As variations multiply, manual preparation becomes difficult to check, interruptions obscure which results remain usable, and the origin of individual patterns becomes laborious to reconstruct. With EBSDmagus, researchers define fixed and varying parameters once. The software prepares the required EMsoft calculations, reuses compatible intermediate results, and checks that the generated files represent the requested investigation before execution. After execution, it checks the expected outputs and, following an interruption, resumes only unresolved calculations while preserving completed work. A portable run record links each output to its inputs and calculation history, providing the provenance needed when the results are deposited as findable, accessible, interoperable, and reusable (FAIR) data. We assess this approach using a completed representative calculation, a larger study prepared and checked before submission, two observed cluster interruptions, and a controlled job cancellation. During recovery, EBSDmagus retained completed calculations and resubmitted only incomplete work.

Sources

Related papers