The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "The Nuclear Decision-Making Benchmark".
Paper summary: Lu: So looking at the title and the authors, this paper, "The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios," it sets up a really comprehensive test for how AI responds to nuclear policy dilemmas >
Tom: It's about putting these frontier models against PhD credentialed scholars in international relations to see if they have stable preferences when dealing with escalation and arms control scenarios >
Jane: The implication for us is that we can't just look at general performance metrics. We need these targeted benchmarks because the stakes are so high in the nuclear domain >
Meng: I guess what this means practically is that before we even think about deploying an AI tool for intelligence analysis or scenario planning, it has to pass a rigorous test like this one >
Lalam: The benchmark itself is a targeted evaluation framework of one hundred fifty-one scenarios spanning four domains: escalation, arms control, non-proliferation, and proliferation > <ref:2608.05180#pg1,a targeted evaluation framework of 151 scenarios>
Lu: That focus on those specific policy domains means that we are getting granular findings about where the models actually go wrong or where their inherent biases lie >
Tom: It's not just about which model is best overall; it's about which model is consistent and appropriate for the specific high-stakes context we are trying to build a decision support system for >
Jane: So, in simple terms, this paper argues that integrating large language models into national security workflows requires sustained, transparent testing before they can be trusted with nuclear policy decisions >
Conclusion: Intro: So we've been looking at how these large language models handle nuclear policy dilemmas, and now we’re getting to where the authors wrap things up.
Tom: So, what’s the big picture here? The title is "The Nuclear Decision-Making Benchmark," right? And they put it out there evaluating frontier models across escalation, arms control, non-proliferation, and proliferation scenarios.
Jane: It sounds like they built a specific test case for how good these models are when dealing with high stakes stuff. What does that actually mean for us in the real world?
Lu: It means they didn't just test general writing ability; they used PhD credentialed scholars in international relations to design one hundred fifty-one scenarios. They wanted something that tested judgment, not just recall.
Meng: So, the core finding is about model consistency under pressure. They found over ninety percent of the differences between models were statistically significant across all four policy areas tested.
Tom: That’s a lot of variation at play when you’re talking about nuclear decisions. And they pointed out that some models, like DeepSeek and Qwen, show a higher tendency to recommend escalation using nuclear weapons compared to others like GPT or ERNIE.
Jane: Wait, so it’s not just one model that's consistently wrong; different models lean in different directions depending on the policy area being tested. That’s a nuanced finding.
Lu: Exactly. Llama, for instance, showed a distinct bias favoring action—favoring force and intervention across all those domains they checked. But consistency also matters, and Llama and ERNIE were actually more consistent across runs in some metrics than DeepSeek or GLM depending on the specific domain.
Meng: From a deployment angle, if your decision support system relies on one of these models, that model choice is pretty consequential because the spread between them in baseline escalation rates is quite wide.
Tom: It sounds like they’re saying that picking a model isn't just about finding the smartest one; it’s about understanding which model aligns with the kind of behavior you need from your system.
Jane: So, what does this imply for anyone building something that uses these models? It suggests you need more than just a high accuracy score on some test set.
Lu: You need to look at scenario-level audits too. Because even if two models look almost identical on the big picture metrics, they can disagree on almost every single individual scenario they run.
Meng: And that’s a huge operational hurdle for me; we don't have time to audit every single scenario manually.
Tom: So the paper suggests that phrasing standardization isn't enough on its own because country biases and phrasing biases compound each other, meaning you need a more robust testing setup.
Jane: It points toward the idea that pre-deployment evaluation infrastructure is necessary, but it also says those diagnostics should be interpreted based on the specific scenario wording, not just as a stable preference for a certain state.
Lu: Right. The paper sets up this framework to show exactly where the current frontier models have stable, consistent preferences—or lack thereof—in these really dangerous contexts.
Meng: It’s a warning sign that we need to be extra careful before we let these things into national security workflows because the results are showing significant differences between the underlying architectures.
Tom: That's where it goes next. We’ve seen how inconsistent they can be, and now we have to think about what happens when we actually try to build a system on top of this data.
Center for Strategic and International Studies · Scale AI
cs.CY
Submitted: 2026-06-29
Updated: 2026-11-18
Comments: 46 pages, 7 figures
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 77/100
The gist: The integration of large language models (LLMs) into defense and national-security workflows raises urgent questions about whether frontier models exhibit stable, consistent…
Key concepts
- NDM Bench
- A specific evaluation framework containing 151 scenarios designed by PhD experts in international relations to test LLMs' judgment on nuclear policy issues. It covers four key areas: escalation, arms control, non-proliferation, and proliferation.
- Model Comparison
- A statistical analysis comparing different AI models (like DeepSeek, Qwen, GPT) based on their recommendations across the benchmark scenarios. Findings show that some models are much more likely to suggest nuclear escalation than others.
- Country Biases
- Pre-existing tendencies or patterns in how LLMs respond to specific countries within the scenarios. These biases exist across different domains and can change depending on how the scenario is phrased, indicating that models do not always give neutral responses.
- Phrasing Treatments
- Experimental methods used to test how sensitive a model's judgment is to narrative framing, such as using phrases like 'existential threat' or specifying 'high-payload' weapons. This helps determine if the model's preference changes based on how the question is worded.
Terminology
Summary
The integration of large language models (LLMs) into defense and national-security workflows raises urgent questions about whether frontier models exhibit stable, consistent, and policyappropriate preferences in high-stakes contexts The Nuclear Decision-Making Benchmark (NDM Bench), a targeted evaluation framework of 151 scenarios authored by PhDcredentialed scholars in international relations spanning four domains: escalation (76), arms control (25), non-proliferation (25), and proliferation (25)
Model Comparison
Overall model-level variation in all four tested domains, with over 90% of pairwise model differences statistically significant DeepSeek and Qwen are the most likely to recommend escalatory action using nuclear weapons (30.9% and 24.1%, respectively); GPT and ERNIE are the least likely (both around 7%) Llama exhibits a distinct bias for action, favoring force, intervention, and cooperation across domains Inter-rater reliability metrics (Krippendorff’s α and quadratically weighted Fleiss’ κ) reveal Llama and ERNIE are the most consistent across runs, with either DeepSeek or GLM the least depending on the domain
Where the gains come from
The analysis of country-level effects shows that country biases persist across domains, but these biases vary by model and interact with phrasing treatments For example, DeepSeek’s North Korea escalation rate rises from 41.5% under baseline to 61. p. 09% under the existential + high-payload treatment The analysis of predictive performance shows that scenario and treatment information are the dominant sources of variation in escalation
How the training signal is built
The benchmark design leveraged subject-matter experts who hold PhDs in international relations with a focus on security studies to create 151 scenarios across four domains Scenarios were iteratively reviewed for internal coherence, practical plausibility, and clear separation between response options while preserving sufficient ambiguity to evaluate model judgment under uncertainty Experimental phrasing treatments were introduced to probe sensitivity to narrative framing, such as the existential-threat frame combined with high-payload or low-payload cues
Pruning redundant reasoning tokens
Five findings stand out, including substantial inter-model variation in all four domains, with all overall response distributions showing significant differences across models and over 90% of pairwise model tests yielding significance Model selection is consequential because the 27-point inter-model spread in baseline escalation D/E rate (and the additional 27-point treatment-effect range) means that the choice of foundation model is a first-order driver of the recommendations a decision-support system would produce Phrasing standardization is necessary but not sufficient because country biases and phrasing biases compound rather than substitute for each other Scenario-level audits complement aggregate metrics because two models with nearly identical marginal distributions can disagree on most individual scenarios Pre-deployment evaluation infrastructure is needed in combination with other controls because country and dyadic probes remain useful diagnostics but should be interpreted conditional on scenario composition and treatment wording rather than as stable model preferences toward particular states
Conclusion
This paper introduced the NDM Bench, an evaluation framework of 151 expert-designed scenarios spanning four nuclear-policy domains, and used it to evaluate seven frontier LLMs The high-stakes nature of the nuclear domain warrants sustained, transparent, and pre-deployment testing of any LLM-enabled system proposed for integration into national-security decision support
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement robust pre-deployment evaluation infrastructure to assess LLM alignment with strategic goals, as
the high-stakes nature of the nuclear domain warrants sustained, transparent, and pre-deployment testing of any LLM-enabled system proposed for integration into national-security decision support.
-
Develop scenario audits that examine
scenario-level audits complement aggregate metrics,
specifically to identifyimportant scenario-level disagreement
between models that might be hidden by overall response distributions. -
Create prompt standardization strategies paired with country pair audits because
prompt standardization alone is insufficient unless paired with country-pair audits
to account for the interaction between model biases and phrasing treatments. -
Establish multi-run benchmarking as standard practice because
per-model inter-rater reliabilities reveal a lack of consistency for several models, pointing to the need for multi-run benchmarking rather than single-shot evaluation as standard practice.
-
Prioritize model selection based on performance metrics derived from the benchmark, recognizing that
variation between models indicates that model selection is not neutral
and thatthe choice of foundation model is a first-order driver of the recommendations a decision-support system would produce.
Sources
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance