Maturity squared: Assessing the Maturity of Crypto Agility Maturity Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Maturity squared: Assessing the Maturity of Crypto Agility Maturity Models".
Elias: Here's the gist of the paper. Crypto agility maturity models aim to support organizations in preparing for cryptographic transitions, such as migrating to post-quantum cryptography <ref:2611.17003#pg6>.
Nadia: In this work, we assess three industry-driven maturity models against established design principles <ref:2611.17003#pg9>.
Elias: Our analysis reveals recurring weaknesses, particularly the lack of measurable, reproducible assessment criteria and limited prescriptive guidance for improving crypto agility <ref:2611.17003#pg10>.
Nadia: Pruning redundant reasoning tokens The paper conducts the first comparative evaluation of three NIST-cited industry-driven models and compares their design profiles to the existing CAMM evaluation.
Title and authors: Nadia: So this paper is called "Maturity squared: Assessing the Maturity of Crypto Agility Maturity Models." It’s looking at these three industry-driven models that NIST actually mentions in their guidance for things like post-quantum cryptography migration.
Elias: Yeah, it’s basically taking those models and testing them against some established design principles. The main point is that we need to look closely at how these models are actually built before we let them shape how big organizations plan their security transitions.
Nadia: So, what does the paper actually find when it tests these three models? Are they all equally bad, or are some of them worse than others?
Elias: The analysis shows that they all have some weaknesses, but there are certain flaws that show up across the board. Most critically, the paper points out a big problem with how they measure things: there is a lack of criteria that people can actually agree on when they look at the results.
Priya: So you’re saying if two different teams looked at these models, they wouldn't get the same assessment because the scoring isn't clearly defined? That makes sense because measurement needs to be solid before we trust any maturity score.
Nadia: Exactly. And it seems like the biggest issue is that they don't provide clear instructions on how to actually improve a model once you get a result. It’s descriptive, but not really prescriptive enough for real-world action.
Elias: That’s right, they fail the prescriptive design principles from Pöppelbuß and Röglinger, which are meant to guide actual improvements in maturity. The paper says none of the three models fully satisfy those improvement requirements.
The paper's summary: Nadia: So if we boil down what this paper is saying, it’s that these maturity models exist to help organizations transition to new cryptography, like post-quantum crypto, but they aren't all built with the same quality standards.
Elias: Right. The researchers assessed three models—FS-ISAC, ATIS, and Deloitte—and compared them against Design Principles for maturity models. They found that while the models are different in their basic goals, they systematically fail to deliver on two key areas: having assessment criteria that everyone can agree on, and giving clear advice on how to get better.
Priya: I’m interested in the specifics of what they tested. How did they actually check if something was intersubjectively verifiable? That sounds like a real headache because agreement is hard in security assessments.
Nadia: They used those established design principles, specifically the Basic Design Principles, Descriptive Design Principles, and Prescriptive Design Principles that Pöppelbuß and Röglinger proposed. The focus there is on whether the assessment process itself is well-defined and if the results are verifiable across different assessors.
Elias: The paper points out a systematic failure in verifiability for both FS-ISAC and Deloitte because their assessment criteria aren't things you can check and agree on without a lot of subjective judgment. ATIS did better on the descriptive side, mostly because five of its twenty-three metrics gave a clear scale from one to ten.
Priya: So ATIS is the standout in terms of how much it describes what’s happening, but that doesn't mean it’s perfect if you can’t actually get a consistent score on it. We need data that reflects reality, not just theoretical scales.
Nadia: Right. And another thing they found was that none of the models integrated actual recommendations into their own structure to show you how to move forward, which is a big gap for practitioners trying to implement something right away.
The paper's improvements: Elias: Now the authors offer some suggestions on how we can use this analysis. They suggest that choosing a maturity model shouldn't just be about what it looks like on paper, but really matching the model’s basic principles to exactly what you are trying to achieve.
Nadia: So, for instance, if your main users are already using the NIST Cybersecurity Framework, then integrating something like Deloitte might be easier for your governance structure than picking one of these other models.
Priya: That makes sense from a practical standpoint. You want something that fits the existing ecosystem of tools you already have in place so you don't introduce too much friction into the process. But what about users who are focused purely on performance?
Elias: For people who are comfortable with key performance indicators, ATIS is suggested because it’s the most fine-grained model available for that, even though they noted it doesn't give clear maturity levels to communicate up to leadership.
Nadia: The paper suggests that FS-ISAC can actually work for two different groups at once: people who need a solid theoretical foundation and those who are already familiar with crypto agility but need it for different reasons. It seems like model choice really depends on the user’s background and the purpose of the assessment.
Priya: I wonder what they mean by that "primary users’ backgrounds." Are we talking about IT operations folks, or maybe procurement managers deciding where to spend money? That distinction is important for picking a model.
Elias: It seems it’s about matching the basic principles to the intended purpose, you know, figuring out if you need a descriptive tool or a prescriptive tool before you even look at the models themselves.
Conclusion: Nadia: So wrapping up this discussion on "Maturity squared: Assessing the Maturity of Crypto Agility Maturity Models," we see that these models are useful starting points, but they aren't ready to be the final word for how organizations should manage crypto agility.
Elias: That’s right. The main finding is that they consistently lack those measurable, reproducible assessment criteria and integrated guidance for actual improvement paths. If you need a model to inform real governance or procurement decisions, this is where they fall short right now.
Priya: From my side, the key thing for future work needs to be making sure that whatever we build next has solid measurability so different assessors can actually reach the same verdict. That’s what we need before we trust these scores in a real security context.
Nadia: I agree. And the paper’s core recommendation is that any model meant to guide governance—like setting procurement requirements or large-scale PQC migration strategies—needs to include explicit decision logic that links the maturity score directly to concrete steps for improvement.
Elias: Exactly. It has to move beyond just describing a state of maturity and show you exactly what action you need to take next, with clear paths forward. That’s the direction we need to head in for future model development.
Priya: I think focusing on that explicit decision logic is where the real value will be, because it moves us from theory into actual implementation planning.
Nadia: So, so if you're driving a security strategy right now, this paper gives you a good map of what’s missing in these current maturity models and how to steer toward something more usable. That’s all for this one on crypto agility models.
Leonie Wolf, Robin Leander Schröder, Sebastian Schinzel, Fabian Ising
Fraunhofer SIT · Fraunhofer Austria · National Research Center for Applied Cybersecurity ATHENE · HNFIZ Cybersecurity · FH Münster University of Applied Sciences
cs.CR
Submitted: 2026-11-12
Updated: 2026-11-12
Comments: 19th International Symposium on Foundations & Practice of Security (FPS - 2026)
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Crypto agility maturity models aim to support organizations in preparing for cryptographic transitions, such as migrating to post-quantum cryptography <ref:2611.17003#pg6>.
Key concepts
- Crypto Agility Maturity Models
- These are frameworks used to assess how prepared an organization is for cryptographic changes, such as migrating to new encryption standards (like post-quantum cryptography). The paper tested three different industry versions of these models to see how well they follow best practices for creating good maturity assessments.
- Design Principles for Maturity Models
- These are established rules used to evaluate whether a maturity model is well-designed. They focus on ensuring the model is clear, consistent, and provides actionable steps. The study used these principles as a benchmark to judge the three models being tested.
- Intersubjectively Verifiable Criteria
- This refers to assessment criteria that different assessors can agree upon consistently without bias or personal opinion. The paper found this was a major weakness across the models; if different people assess the same system, they might get different results because there is no clear, objective way to measure progress.
- Prescriptive Guidance
- This is the ability of a maturity model to tell users exactly what steps they need to take to improve their crypto agility. The analysis showed that none of the tested models provided this guidance directly within the model structure, making it hard for practitioners to know how to actually fix their systems.
Terminology
Summary
Crypto agility maturity models aim to support organizations in preparing for cryptographic transitions, such as migrating to post-quantum cryptography <ref:2611. p. 0 de
Abstract. Crypto agility maturity models aim to support organizations in preparing for cryptographic transitions, such as migrating to
post-quantum cryptography. However, their design
quality has not been systematically evaluated. In this work, we assess three industry-driven maturity models against
established design principles. In this work, we assess three industry-driven maturity models against
established design principles. Our analysis reveals recurring weaknesses,
particularly the lack of measurable, reproducible assessment criteria and
limited prescriptive guidance for improving crypto agility.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper conducts the first comparative evaluation of three NIST-cited industry-driven models and compares their design profiles to the existing CAMM evaluation. The research seeks to answer three specific research questions: RQ1 asks to what extent do the crypto agility maturity models referenced by NIST fulfill established design principles for maturity models, RQ2 identifies which weaknesses are systematic shortcomings across the models and which can be attributed to differing intended purposes, and RQ3 asks how practitioners can select among the models and what should developers of future models consider. p. 1 We seek to answer the following research questions:
RQ1: To what extent do the crypto agility maturity models referenced by NIST
fulfill established design principles for maturity models? RQ2: Which weaknesses are systematic shortcomings across the models, and
which can be attributed to differing intended purposes? In this work, we make the following contributions:
– We provide the first comparative assessment of the three industry-driven
frameworks referenced in the NIST crypto agility white paper [4] and synthesize the results with the existing CAMM evaluation [36]. – We show which shortcomings are systematic across the models—most critically the absence of intersubjectively verifiable criteria—and which merely
reflect their differing purposes.Where the paper says it
Where the paper says it
How the training signal is built
The evaluation methodology utilizes the Design Principles for maturity models proposed by Pöppelbuß and Röglinger to assess each model against established design principles. The analysis contrasts the results with the earlier CAMM evaluation conducted by Wolf et al. p. 3 3 Assessment Methodology
We comparatively assess the three previously unevaluated industry-driven frameworks referenced by NIST using established design principles [26] for maturity
models. We then contrast the results with the earlier CAMM evaluation by Wolf
et al. We then contrast the results with the earlier CAMM evaluation by Wolf
et al. [36] to distinguish recurring shortcomings from properties attributable to
model purpose and structure. , the Quantum
Readiness Assurance Maturity Model [9] and the PQC Maturity Model [25]),
targets agility at the application level while explicitly rejecting a single linear
maturity hierarchy in favor of orthogonal, non-hierarchical dimensions [28], or,
like the CARAF [21], targets risk assessment rather than maturity. The three models we analyze differ substantially in their design approaches:
FS-ISAC extends the CAMM, ATIS adopts a quantitatively oriented Key Performance Indicator (KPI)-driven framework, and Deloitte builds upon the established NIST Cyber Security Framework (CSF) 2.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The evaluation results show that while models diverge on basic principles due to intended purposes, none fully satisfies the descriptive (DP 2.x) and all fail the prescriptive (DP 3. p. 6 x), but none fully satisfies the descriptive
(DP 2. x) and all fail the prescriptive (DP 3. However, their design
quality has not been systematically evaluated. In this work, we assess three industry-driven maturity models against
established design principles. In this work, we make the following contributions:
– We provide the first comparative assessment of the three industry-driven
frameworks referenced in the NIST crypto agility white paper [4] and synthesize the results with the existing CAMM evaluation [36]. – We show which shortcomings are systematic across the models—most critically the absence of intersubjectively verifiable criteria—and which merely
reflect their differing purposes.Where the paper says it
Where the paper says it
Where the paper says it
Selecting a Crypto Agility Maturity Model
Model choice depends on the intended assessment purpose and the users’ background, noting that a good fit can only be achieved by matching basic principles to the intended purpose. The paper suggests choosing based on primary users’ backgrounds: if they already employ NIST CSF, Deloitte will be easier to integrate into their governance structure than other models would. p. 1 While the white paper cites these models only as illustrative industry initiatives, NIST’s role in cybersecurity guidance and procurement
practices means such references may come to shape guidance and governance,
similar to the requirement of the Cybersecurity Maturity Model Certification
(CMMC) in US DoD procurements [32]. Notably, NIST already recommends
including crypto agility as an evaluation criterion for project proposals [4]. Deloitte provides the clearest support for
multiple levels of granularity: it comprises five top-level tiers that represent the
achieved maturity level and can serve as a single data point to quickly and concisely communicate the status of maturation to leadership and executives. It also
includes functions, each with categories subdivided into sub-categories, allowing
users to drill deeper into each dimension. Rather, they can serve as a tool
for comparing maturity models and identifying their strengths and weaknesses. 2 Crypto Agility
In its most basic form, cryptographic agility (crypto agility) refers to the ability
to switch among different cryptographic primitives with minimal effort.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Work
The findings are subject to the subjectivity of the researchers' discussions, as the conclusions are ultimately subjective. The authors note that empirical application of maturity models is useful for validating aspects like DP 2. p. 4 , the Quantum
Readiness Assurance Maturity Model [9] and the PQC Maturity Model [25]),
targets agility at the application level while explicitly rejecting a single linear
maturity hierarchy in favor of orthogonal, non-hierarchical dimensions [28], or,
like the CARAF [21], targets risk assessment rather than maturity. The three models we analyze differ substantially in their design approaches:
FS-ISAC extends the CAMM, ATIS adopts a quantitatively oriented Key Performance Indicator (KPI)-driven framework, and Deloitte builds upon the established NIST Cyber Security Framework (CSF) 2. At the same
time, we highlight individual strengths that can assist practitioners in
choosing among the models and derive concrete recommendations for
strengthening future crypto agility maturity models. 1 Introduction
Potential Cryptanalytically Relevant Quantum-Computers (CRQCs) threaten
the security of classical asymmetric cryptographic primitives. The Crypto Agility Risk Assessment Framework (CARAF) [21], focuses specifically on assessing and mitigating
risks associated with insufficient agility. Other studies have proposed new, domain-specific criteria for evaluating
Business Process Maturity Models [14, 34] and Industry 4. Our analysis reveals recurring weaknesses,
particularly the lack of measurable, reproducible assessment criteria and
limited prescriptive guidance for improving crypto agility. At the same
time, we highlight individual strengths that can assist practitioners in
choosing among the models and derive concrete recommendations for
strengthening future crypto agility maturity models.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In this paper, we evaluated three industry-driven crypto agility maturity models referenced by NIST against established design principles for maturity models. We find that the models meet basic design principles to varying degrees, which we mainly attribute to their intended purposes. p. 6 x) principles. We interpret the divergence on basic principles as primarily reflecting the models’ intended purposes
rather than differences in quality. In this work, we assess three industry-driven maturity models against
established design principles. Our analysis reveals recurring weaknesses,
particularly the lack of measurable, reproducible assessment criteria and
limited prescriptive guidance for improving crypto agility. Our analysis reveals recurring weaknesses,
particularly the lack of measurable, reproducible assessment criteria and
limited prescriptive guidance for improving crypto agility. At the same
time, we highlight individual strengths that can assist practitioners in
choosing among the models and derive concrete recommendations for
strengthening future crypto agility maturity models. Our analysis reveals recurring weaknesses,
particularly the lack of measurable, reproducible assessment criteria and
limited prescriptive guidance for improving crypto agility. At the same
time, we highlight individual strengths that can assist practitioners in
choosing among the models and derive concrete recommendations for
strengthening future crypto agility maturity models.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve AI systems by incorporating measurable, reproducible criteria into crypto agility maturity models to ensure
different assessors reach the same verdict.
This will allow for consistent evaluation across organizations rather than relying on subjective judgments. -
Develop AI systems capable of integrating a progression path into maturity levels with a calculus for choosing among measures. This allows an assessment result to name the next step, corresponding to concrete migration milestones and an assessment result names the next step.
-
Enhance AI systems by incorporating trade-off considerations into maturity assessments, explicitly balancing
anticipated security benefits against their costs.
This will allow decisions regarding crypto agility implementation to be more balanced and practically relevant. -
Improve AI systems by providing audience matching granularity, accounting for both decision-makers who allocate resources and technical staff who inventory, modify, and test cryptographic systems. This ensures the model serves both leadership and technical audiences effectively.
Abstract
Crypto agility maturity models aim to support organizations in preparing for cryptographic transitions, such as migrating to post-quantum cryptography. Recognizing the importance of measurable maturity, NIST references several such models in its guidance. Given NIST's influence on cybersecurity practice, these models may shape organizational planning and governance mechanisms. However, their design quality has not been systematically evaluated. In this work, we assess three industry-driven maturity models against established design principles. Our analysis reveals recurring weaknesses, particularly the lack of measurable, reproducible assessment criteria and limited prescriptive guidance for improving crypto agility. At the same time, we highlight individual strengths that can assist practitioners in choosing among the models and derive concrete recommendations for strengthening future crypto agility maturity models.
Sources
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data