NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction".
Elias: Here's the gist of the paper. Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators.
Nadia: This paper introduces NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments.
Elias: Framework Design Goals The design objectives guiding the development of NICE focus on reducing friction in the validation phase of vulnerability reproduction.
Nadia: The framework prioritizes making it easy for users to verify that a vulnerability is present and exploitable under well-defined conditions.
Paper summary: Priya: So, wrapping up this discussion on "NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction," what's the simplest way to put what this paper is actually offering?
Nadia: It offers a systematic approach to constructing NixOS virtual machines that contain vulnerable software and all its dependencies. It addresses the challenge of validating whether a vulnerability is indeed present by automating the validation process and providing factual evidence of exploitability.
Elias: They succeeded in reproducing nineteen real-world vulnerabilities, covering different types of software and spanning over a decade of disclosure dates. That temporal coverage over that long time span is significant for security research.
Priya: It’s important to remember the authors explicitly stated what their framework does not cover, which is the limitations they found in prior FPM-based approaches and the specific scenarios where their tools still don't fully work.
Nadia: That's right. So, while NICE provides a powerful way to build reproducible labs, we have to be aware that it’s not a complete solution for every single vulnerability or exploit scenario out there.
Elias: The paper is structured around the design objectives and related work in vulnerability reproduction before diving into the technical foundations of NixOS and its test frame. It lays out the groundwork for why this declarative approach is necessary.
Priya: Overall, it seems like a very practical framework that moves security validation from being a manual, error-prone task to something more automated and verifiable using declarative specifications.
Nadia: Exactly. It gives researchers and educators a tool to build environments that are reliable and can be shared easily without constantly worrying about whether the setup itself is consistent across different machines or over time.
Conclusion: Nadia: So, we’ve seen how this NICE framework builds those virtual machines for vulnerability testing, and now we need to talk about what it actually means when you look at the title: "NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction."
Elias: It’s a mouthful. What I get from that is they’re trying to make the process of making those vulnerable setups as declarative as possible so you can trust them.
Priya: Declarative means you describe *what* you want, not *how* to build it step-by-step, right? And machine-checkable means someone else can run your description and verify if it actually works.
Nadia: Exactly. It moves the whole validation phase away from manual configuration errors toward something automated that we can check repeatedly over time.
Elias: The authors are building this on NixOS, which is already great for reproducibility, but they’re layering a specific framework on top to handle the actual reproduction and testing of exploits.
Priya: So what does this change for us? It means if a researcher builds a test environment for a specific flaw in an old piece of software, anyone can take that description and run it to get the same result.
Nadia: It lowers the barrier for sharing these complex lab setups. Instead of sending huge config files with instructions, you send this simple recipe file.
Elias: And they’ve shown it works on nineteen different real-world vulnerabilities spanning a decade of timeframes, which is a pretty solid test for long-term validity.
Priya: I still have to wonder about the ethical side—who gets access to these detailed blueprints for creating exploits? That’s something that needs serious thought.
Nadia: Right, so the paper shows how you build the tool, but now we gotta think about how people actually use it and what responsibilities come with that level of detail. (Sound of gentle radio static swelling slightly)
SAMOVAR, Télécom SudParis, Polytechnic Institute of Paris, France · LTCI, Télécom Paris, Polytechnic Institute of Paris, France
cs.CR
Submitted: 2026-05-30
Updated: 2026-11-10
Code: https://github.com/.../ab645...tar.gz
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators.
Key concepts
- NICE
- The NIx CvE reproduction framework uses declarative recipes to build and automatically validate vulnerable environments. It is designed to make verifying vulnerability presence easy, supporting lightweight sharing and automated testing.
- NixOS
- NixOS is a Linux distribution built on top of the Nix package collection. It allows users to declaratively specify entire software environments in a single configuration file, ensuring that builds are reproducible with high confidence.
- Declarative Recipes
- These are configuration files written in the Nix programming language that describe exactly what software and environment should be installed. They allow for precise, repeatable definitions of the vulnerable systems being tested.
- Exploit Validation
- This stage involves writing a Python scenario that uses the NixOS test framework API to check if a specific vulnerability can be exploited within the constructed environment, providing automated proof of exploitability.
Terminology
Summary
Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators. p. 0 zimmermann}@telecom-paris. fr
Abstract
Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators. Yet, vulnerabilities remain hard to reproduce today, and even
when they can be reproduced, recreating a software environment where the vulnerability can be exploited becomes harder
and harder over time. We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build
and automatically validate vulnerable environments.Where the paper says it
Where the paper says it
Framework Design Goals
The design objectives guiding the development of NICE focus on reducing friction in the validation phase of vulnerability reproduction. The framework prioritizes making it easy for users to verify that a vulnerability is present and exploitable under well-defined conditions. p. 1 Rather than
aiming to fully automate environment generation or exploit
development, we prioritize making it easy for users to verify
that a vulnerability is present and exploitable under welldefined conditions. In particular, our framework is designed
around the following objectives:
• Lightweight sharing. Given this step always yields the
same environment, sharing the lightweight proof recipe is
sufficient to reproduce the vulnerability in the future and
elsewhere—one does not need to share the large binary VM
images produced by the build. Therefore, we achieve both
our “Lightweight sharing” and “Reproducibility over time”
objectives. In addition to the above usability-oriented goals, we also
design the framework to be versatile across a broad range of
vulnerabilities. Specifically, NICE aims to support:
• Diversity of vulnerability classes.Where the paper says it
Where the paper says it
Where the paper says it
Technical Foundations
The NICE framework is built upon NixOS, a Linux distribution built on top of the Nix package collection. Nix allows for declarative specification of software environments and ensures reproducibility with high confidence. p. 3 Nix [7] is a functional package manager and the first implementation of this software deployment model. It allows
specifying software environments declaratively and reproducing them with high confidence [14]. A key characteristic of NixOS is its module system, which allows the entire system configuration (including installed packages, system services, user accounts,
network settings, etc. ) to be specified declaratively in a single
configuration file written in the Nix programming language. The NixOS test framework [28] is an automated integration testing framework for the NixOS distribution. It allows
developers to specify one or more NixOS virtual machine
configurations together with a test scenario that orchestrates
their execution and interactions.Where the paper says it
Where the paper says it
Where the paper says it
Workflow and Validation
The NICE workflow follows three stages: Report Analysis, Recipe Specification, and Exploit Validation. The recipe consists of virtual machine configurations and an exploitation scenario written in Python that uses the NixOS test framework API. p. 4 Exploitation scenario The second part of the vulnerability recipe defines an exploitation scenario, which specifies
how the vulnerability is exercised and how its successful exploitation is detected. Exploitation scenarios are written in
Python and use the NixOS test framework API, which provides primitives for orchestrating virtual machines, such as
booting systems, waiting for systemd services to become
available, checking network connectivity, and executing arbitrary commands within virtual hosts. The second build type produces a fully self-contained executable proof that the vulnerability is present: this includes
both the set of NixOS virtual machine images corresponding
to the specified VM configurations, along with the instrumentation code derived from the exploitation scenario. The
instrumentation code includes all the scaffolding code used by
the test framework to communicate and execute commands
on the virtual machine through the qemu monitoring features
as well as all the recipe-specific code to run the test scenario,
that is the exploit logic and the assertions that will later be
executed during automated testing. , a student having to learn
how to manually exploit the vulnerability). The second build type produces a fully self-contained executable proof that the vulnerability is present: this includes
both the set of NixOS virtual machine images corresponding
to the specified VM configurations, along with the instrumentation code derived from the exploitation scenario.Where the paper says it
Where the paper says it
Where the paper says it
Evaluation Results
NICE was evaluated on 19 diverse real-world vulnerabilities spanning multiple CWE categories, attack vectors, and target types. The results show that NICE allows for the production of concise recipes and integration tests that reproduce vulnerable environments and provide proofs of exploitation. p. 0 We evaluate NICE on 19 diverse real-world CVEs spanning multiple CWE categories, attack vectors, and target types
(user-space, system software, kernel, and graphical applications). We show that NICE allows to produce concise recipes
and integration tests that reproduce vulnerable environments
and provide proofs of exploitation. Temporal coverage. We reproduced vulnerabilities ranging
from 2014 to 2025, showing that NICE can reproduce and
validate vulnerabilities over a long time span.Where the paper says it
Where the paper says it
Impact and Future Directions
NICE has the potential to affect security practitioners, educators, and open-source communities positively by providing a reproducible way to validate vulnerable environments. The framework can reduce the burden on cyber range operators by packaging exercises as declarative specifications with automated validation tests. p. 1 Our framework enables packaging hands-on security labs
and cyber range exercises as lightweight, reviewable environments that can be shared and redeployed easily. Accompanying automated validation tests help ensure that these labs continue to behave as intended, which is useful for instructors and
researchers alike. However, their approach
does not inherently guarantee that the generated recipes remain reproducible over time and the framework also lacks
support for vulnerabilities that require GUI interactions. The
idea of using LLMs to generate vulnerable environments is
complementary to our approach, as LLMs could be used to
assist in generating candidate NICE vulnerability recipes that
are then stabilized and validated within our framework.Where the paper says it
Where the paper says it
Conclusion
The NICE framework successfully reproduces 19 real-world vulnerabilities, covering different types of software and spanning over a decade of disclosure dates. Using NICE, the framework provides a systematic approach to construct NixOS virtual machines that contain vulnerable software and its dependencies. p. 4 Exploitation scenarios are written in
Python and use the NixOS test framework API, which provides primitives for orchestrating virtual machines, such as
booting systems, waiting for systemd services to become
available, checking network connectivity, and executing arbitrary commands within virtual hosts. Using this interface, the exploitation scenario encodes an
exploit script that performs the actions required to trigger the
vulnerability. We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build
and automatically validate vulnerable environments. In NICE,
a reproduced CVE comprises one or more NixOS virtual
machine configurations, a scripted exploitation scenario, and
machine-checkable assertions that provide factual evidence of
exploitation.Where the paper says it
Where the paper says it
Ethical Considerations
The main risk identified is that malicious actors could use detailed, reproducible vulnerability recipes to exploit vulnerable software. Mitigation involves using the recipe with careful consideration of vulnerability disclosure policies. p. 9 2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
Disclosure date
CVE-2020-7247
CVE-2014-0160
CVE-2023-22809
CVE-2021-4034
CVE-2022-0847
CVE-2016-5195
CVE-2014-0224
CVE-2022-25640
CVE-2021-44228
CVE-2023-7028
CVE-2024-1135
CVE-2025-32463
CVE-2022-32278
CVE-2025-5399
CVE-2019-0221
CVE-2024-1071
CVE-2023-25666
CVE-2019-19725
CVE-2021-40346
NixOS merged into Nixpkgs
Software category
Headless application
System software
Kernel
Graphical application
Figure 6: Reproduced timeline of CVEs using NICE framework. The points are scattered randomly on the y-axis for better
visibility.Where the paper says it
Open Science
A complete replication package for this work is already available on GitHub. This package contains the full set of CVEs reproduced, the implementation of reusable NICE framework components, and GitLab CI integration for continuous validation. p. 8 The performance was measured on a GitLab
runner integrated for NICE, with 8 virtual CPUs (AMD EPYC
7302P, 3 GHz), 32 GiB RAM, and 256 GiB local storage. Figure 6 summarizes the time distribution of the CVEs we
reproduced, one data point per reproduction; it also highlights
the classes of vulnerable software. This design facilitates sharing, validation, review,
and long-term reproducibility. We evaluate NICE on 19 diverse real-world CVEs spanning multiple CWE categories, attack vectors, and target types
(user-space, system software, kernel, and graphical applications).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated vulnerability packaging can be expressed into
declarative, executable artifacts that precisely specify (1) the software environment in which a vulnerability occurs and (2) the procedures required to demonstrate exploitability,
facilitating sharing and review without distributing large binary artifacts. -
The framework enables
push-button validation
by allowing users totrigger the validation of its exploitability with minimal effort, executing a single command, without requiring manual configuration, environment setup, or in-depth knowledge of the underlying exploit.
-
NICE can strengthen vulnerability reporting and triage by
providing machine-checkable evidence of exploitation for reviewers and maintainers,
which increases confidence and reduces ambiguity during coordinated disclosure and patch validation. -
The system can be used to create
lightweight sharing
artifacts, as a "packaged vulnerability should remain verifiable with the same level of effort and confidence years after its initial creation, despite the continuous release of new software libraries, tools, package managers, and execution environments." -
The framework supports
diversity of vulnerability classes,
meaning it can reproduce vulnerabilities spanning a wide range of CWE categories and diverse attack vectors such as network-based exploits or resource exhaustion. -
NICE enables continuous validation by allowing
automated exploitability validation
within CI pipelines, ensuring that labs remain valid under pinned inputs and detecting failures when updating environments. -
The system supports
temporal coverage,
allowing researchers torebuild vulnerable environments of historical software using the same build tools and environments that were available at the time of disclosure.
Abstract
Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators. Yet, vulnerabilities are often hard to reproduce, and even when they can be reproduced once, recreating a software environment where the vulnerability can be exploited becomes harder and harder over time. We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments. In NICE, a reproduced CVE comprises one or more NixOS virtual machine configurations, a scripted exploitation scenario, and machine-checkable assertions that provide factual evidence of exploitation. This design facilitates sharing, validation, review, and long-term reproducibility. Building on NICE, we construct a public archive of 56 reproduced vulnerabilities spanning multiple CWE categories, attack vectors, and target types (user-space, system software, kernel, and graphical applications), combining manually authored recipes with recipes produced by an LLM-based automated pipeline and merged after human review. We evaluate the feasibility of the automated pipeline on 110 real-world CVEs, achieving fully automated reproduction in 54.5% of the cases, at an average cost of 0.73 per CVE. The evaluation demonstrates that NICE provides a practical target for LLM-assisted recipe generation, while also identifying where human refinement or additional automation remains necessary.
Sources
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data