NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction

arXiv:2606.00625 · cs.CR · Submitted 2026-05-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction".

Elias: Here's the gist of the paper. Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators.

Nadia: This paper introduces NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments.

Elias: Framework Design Goals The design objectives guiding the development of NICE focus on reducing friction in the validation phase of vulnerability reproduction.

Nadia: The framework prioritizes making it easy for users to verify that a vulnerability is present and exploitable under well-defined conditions.

Paper summary: Priya: So, wrapping up this discussion on "NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction," what's the simplest way to put what this paper is actually offering?

Nadia: It offers a systematic approach to constructing NixOS virtual machines that contain vulnerable software and all its dependencies. It addresses the challenge of validating whether a vulnerability is indeed present by automating the validation process and providing factual evidence of exploitability.

Elias: They succeeded in reproducing nineteen real-world vulnerabilities, covering different types of software and spanning over a decade of disclosure dates. That temporal coverage over that long time span is significant for security research.

Priya: It’s important to remember the authors explicitly stated what their framework does not cover, which is the limitations they found in prior FPM-based approaches and the specific scenarios where their tools still don't fully work.

Nadia: That's right. So, while NICE provides a powerful way to build reproducible labs, we have to be aware that it’s not a complete solution for every single vulnerability or exploit scenario out there.

Elias: The paper is structured around the design objectives and related work in vulnerability reproduction before diving into the technical foundations of NixOS and its test frame. It lays out the groundwork for why this declarative approach is necessary.

Priya: Overall, it seems like a very practical framework that moves security validation from being a manual, error-prone task to something more automated and verifiable using declarative specifications.

Nadia: Exactly. It gives researchers and educators a tool to build environments that are reliable and can be shared easily without constantly worrying about whether the setup itself is consistent across different machines or over time.

Conclusion: Nadia: So, we’ve seen how this NICE framework builds those virtual machines for vulnerability testing, and now we need to talk about what it actually means when you look at the title: "NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction."

Elias: It’s a mouthful. What I get from that is they’re trying to make the process of making those vulnerable setups as declarative as possible so you can trust them.

Priya: Declarative means you describe *what* you want, not *how* to build it step-by-step, right? And machine-checkable means someone else can run your description and verify if it actually works.

Nadia: Exactly. It moves the whole validation phase away from manual configuration errors toward something automated that we can check repeatedly over time.

Elias: The authors are building this on NixOS, which is already great for reproducibility, but they’re layering a specific framework on top to handle the actual reproduction and testing of exploits.

Priya: So what does this change for us? It means if a researcher builds a test environment for a specific flaw in an old piece of software, anyone can take that description and run it to get the same result.

Nadia: It lowers the barrier for sharing these complex lab setups. Instead of sending huge config files with instructions, you send this simple recipe file.

Elias: And they’ve shown it works on nineteen different real-world vulnerabilities spanning a decade of timeframes, which is a pretty solid test for long-term validity.

Priya: I still have to wonder about the ethical side—who gets access to these detailed blueprints for creating exploits? That’s something that needs serious thought.

Nadia: Right, so the paper shows how you build the tool, but now we gotta think about how people actually use it and what responsibilities come with that level of detail. (Sound of gentle radio static swelling slightly)

SAMOVAR, Télécom SudParis, Polytechnic Institute of Paris, France · LTCI, Télécom Paris, Polytechnic Institute of Paris, France

cs.CR

Submitted: 2026-05-30

Updated: 2026-11-10

Code: https://github.com/.../ab645...tar.gz

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators.

Key concepts

NICE
The NIx CvE reproduction framework uses declarative recipes to build and automatically validate vulnerable environments. It is designed to make verifying vulnerability presence easy, supporting lightweight sharing and automated testing.
NixOS
NixOS is a Linux distribution built on top of the Nix package collection. It allows users to declaratively specify entire software environments in a single configuration file, ensuring that builds are reproducible with high confidence.
Declarative Recipes
These are configuration files written in the Nix programming language that describe exactly what software and environment should be installed. They allow for precise, repeatable definitions of the vulnerable systems being tested.
Exploit Validation
This stage involves writing a Python scenario that uses the NixOS test framework API to check if a specific vulnerability can be exploited within the constructed environment, providing automated proof of exploitability.

Terminology

Summary

Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators. p. 0

Where the paper says it

zimmermann}@telecom-paris. fr Abstract Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators.

This paper introduces NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments. p. 0
Where the paper says it

Yet, vulnerabilities remain hard to reproduce today, and even when they can be reproduced, recreating a software environment where the vulnerability can be exploited becomes harder and harder over time. We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments.

Framework Design Goals

The design objectives guiding the development of NICE focus on reducing friction in the validation phase of vulnerability reproduction. The framework prioritizes making it easy for users to verify that a vulnerability is present and exploitable under well-defined conditions. p. 1

Where the paper says it

Rather than aiming to fully automate environment generation or exploit development, we prioritize making it easy for users to verify that a vulnerability is present and exploitable under welldefined conditions. In particular, our framework is designed around the following objectives: • Lightweight sharing.

Key objectives include lightweight sharing, push-button validation, ease of review, and reproducibility over time. p. 5
Where the paper says it

Given this step always yields the same environment, sharing the lightweight proof recipe is sufficient to reproduce the vulnerability in the future and elsewhere—one does not need to share the large binary VM images produced by the build. Therefore, we achieve both our “Lightweight sharing” and “Reproducibility over time” objectives.

NICE aims to support diversity across vulnerability classes, attack vectors, and targets. p. 1
Where the paper says it

In addition to the above usability-oriented goals, we also design the framework to be versatile across a broad range of vulnerabilities. Specifically, NICE aims to support: • Diversity of vulnerability classes.

Technical Foundations

The NICE framework is built upon NixOS, a Linux distribution built on top of the Nix package collection. Nix allows for declarative specification of software environments and ensures reproducibility with high confidence. p. 3

Where the paper says it

Nix [7] is a functional package manager and the first implementation of this software deployment model. It allows specifying software environments declaratively and reproducing them with high confidence [14].

The system configuration can be specified declaratively in a single configuration file written in the Nix programming language. p. 3
Where the paper says it

A key characteristic of NixOS is its module system, which allows the entire system configuration (including installed packages, system services, user accounts, network settings, etc. ) to be specified declaratively in a single configuration file written in the Nix programming language.

Furthermore, the NixOS test framework provides an automated integration testing framework that allows for orchestration of virtual machines and assertion checks. p. 4
Where the paper says it

The NixOS test framework [28] is an automated integration testing framework for the NixOS distribution. It allows developers to specify one or more NixOS virtual machine configurations together with a test scenario that orchestrates their execution and interactions.

Workflow and Validation

The NICE workflow follows three stages: Report Analysis, Recipe Specification, and Exploit Validation. The recipe consists of virtual machine configurations and an exploitation scenario written in Python that uses the NixOS test framework API. p. 4

Where the paper says it

Exploitation scenario The second part of the vulnerability recipe defines an exploitation scenario, which specifies how the vulnerability is exercised and how its successful exploitation is detected. Exploitation scenarios are written in Python and use the NixOS test framework API, which provides primitives for orchestrating virtual machines, such as booting systems, waiting for systemd services to become available, checking network connectivity, and executing arbitrary commands within virtual hosts.

The framework supports two build types: a debuggable environment for interactive debugging and a fully self-contained executable proof. p. 5
Where the paper says it

The second build type produces a fully self-contained executable proof that the vulnerability is present: this includes both the set of NixOS virtual machine images corresponding to the specified VM configurations, along with the instrumentation code derived from the exploitation scenario. The instrumentation code includes all the scaffolding code used by the test framework to communicate and execute commands on the virtual machine through the qemu monitoring features as well as all the recipe-specific code to run the test scenario, that is the exploit logic and the assertions that will later be executed during automated testing.

This executable proof includes the NixOS virtual machine images and instrumentation code, achieving lightweight sharing and reproducibility over time. p. 5
Where the paper says it

, a student having to learn how to manually exploit the vulnerability). The second build type produces a fully self-contained executable proof that the vulnerability is present: this includes both the set of NixOS virtual machine images corresponding to the specified VM configurations, along with the instrumentation code derived from the exploitation scenario.

Evaluation Results

NICE was evaluated on 19 diverse real-world vulnerabilities spanning multiple CWE categories, attack vectors, and target types. The results show that NICE allows for the production of concise recipes and integration tests that reproduce vulnerable environments and provide proofs of exploitation. p. 0

Where the paper says it

We evaluate NICE on 19 diverse real-world CVEs spanning multiple CWE categories, attack vectors, and target types (user-space, system software, kernel, and graphical applications). We show that NICE allows to produce concise recipes and integration tests that reproduce vulnerable environments and provide proofs of exploitation.

The framework successfully reproduced vulnerabilities ranging from 2014 to 2025, demonstrating temporal coverage over a long time span. p. 9
Where the paper says it

Temporal coverage. We reproduced vulnerabilities ranging from 2014 to 2025, showing that NICE can reproduce and validate vulnerabilities over a long time span.

Impact and Future Directions

NICE has the potential to affect security practitioners, educators, and open-source communities positively by providing a reproducible way to validate vulnerable environments. The framework can reduce the burden on cyber range operators by packaging exercises as declarative specifications with automated validation tests. p. 1

Where the paper says it

Our framework enables packaging hands-on security labs and cyber range exercises as lightweight, reviewable environments that can be shared and redeployed easily. Accompanying automated validation tests help ensure that these labs continue to behave as intended, which is useful for instructors and researchers alike.

Future directions include expanding beyond NixOS to other Linux distributions and improving temporal robustness through engineering or curated historical inputs. The paper suggests that integrating LLMs could further streamline the process by assisting in generating NixOS configurations or test scripts. p. 2
Where the paper says it

However, their approach does not inherently guarantee that the generated recipes remain reproducible over time and the framework also lacks support for vulnerabilities that require GUI interactions. The idea of using LLMs to generate vulnerable environments is complementary to our approach, as LLMs could be used to assist in generating candidate NICE vulnerability recipes that are then stabilized and validated within our framework.

Conclusion

The NICE framework successfully reproduces 19 real-world vulnerabilities, covering different types of software and spanning over a decade of disclosure dates. Using NICE, the framework provides a systematic approach to construct NixOS virtual machines that contain vulnerable software and its dependencies. p. 4

Where the paper says it

Exploitation scenarios are written in Python and use the NixOS test framework API, which provides primitives for orchestrating virtual machines, such as booting systems, waiting for systemd services to become available, checking network connectivity, and executing arbitrary commands within virtual hosts. Using this interface, the exploitation scenario encodes an exploit script that performs the actions required to trigger the vulnerability.

The framework addresses the challenge of validating whether a vulnerability is indeed present by automating the validation process and providing factual evidence of exploitability. p. 0
Where the paper says it

We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments. In NICE, a reproduced CVE comprises one or more NixOS virtual machine configurations, a scripted exploitation scenario, and machine-checkable assertions that provide factual evidence of exploitation.

Ethical Considerations

The main risk identified is that malicious actors could use detailed, reproducible vulnerability recipes to exploit vulnerable software. Mitigation involves using the recipe with careful consideration of vulnerability disclosure policies. p. 9

Where the paper says it

2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 Disclosure date CVE-2020-7247 CVE-2014-0160 CVE-2023-22809 CVE-2021-4034 CVE-2022-0847 CVE-2016-5195 CVE-2014-0224 CVE-2022-25640 CVE-2021-44228 CVE-2023-7028 CVE-2024-1135 CVE-2025-32463 CVE-2022-32278 CVE-2025-5399 CVE-2019-0221 CVE-2024-1071 CVE-2023-25666 CVE-2019-19725 CVE-2021-40346 NixOS merged into Nixpkgs Software category Headless application System software Kernel Graphical application Figure 6: Reproduced timeline of CVEs using NICE framework. The points are scattered randomly on the y-axis for better visibility.

Open Science

A complete replication package for this work is already available on GitHub. This package contains the full set of CVEs reproduced, the implementation of reusable NICE framework components, and GitLab CI integration for continuous validation. p. 8

Where the paper says it

The performance was measured on a GitLab runner integrated for NICE, with 8 virtual CPUs (AMD EPYC 7302P, 3 GHz), 32 GiB RAM, and 256 GiB local storage. Figure 6 summarizes the time distribution of the CVEs we reproduced, one data point per reproduction; it also highlights the classes of vulnerable software.

The authors plan to archive this replication package in a permanent long-term repository such as Zenodo or Software Heritage. p. 0
Where the paper says it

This design facilitates sharing, validation, review, and long-term reproducibility. We evaluate NICE on 19 diverse real-world CVEs spanning multiple CWE categories, attack vectors, and target types (user-space, system software, kernel, and graphical applications).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated vulnerability packaging can be expressed into declarative, executable artifacts that precisely specify (1) the software environment in which a vulnerability occurs and (2) the procedures required to demonstrate exploitability, facilitating sharing and review without distributing large binary artifacts.

  2. The framework enables push-button validation by allowing users to trigger the validation of its exploitability with minimal effort, executing a single command, without requiring manual configuration, environment setup, or in-depth knowledge of the underlying exploit.

  3. NICE can strengthen vulnerability reporting and triage by providing machine-checkable evidence of exploitation for reviewers and maintainers, which increases confidence and reduces ambiguity during coordinated disclosure and patch validation.

  4. The system can be used to create lightweight sharing artifacts, as a "packaged vulnerability should remain verifiable with the same level of effort and confidence years after its initial creation, despite the continuous release of new software libraries, tools, package managers, and execution environments."

  5. The framework supports diversity of vulnerability classes, meaning it can reproduce vulnerabilities spanning a wide range of CWE categories and diverse attack vectors such as network-based exploits or resource exhaustion.

  6. NICE enables continuous validation by allowing automated exploitability validation within CI pipelines, ensuring that labs remain valid under pinned inputs and detecting failures when updating environments.

  7. The system supports temporal coverage, allowing researchers to rebuild vulnerable environments of historical software using the same build tools and environments that were available at the time of disclosure.

Abstract

Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators. Yet, vulnerabilities are often hard to reproduce, and even when they can be reproduced once, recreating a software environment where the vulnerability can be exploited becomes harder and harder over time. We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments. In NICE, a reproduced CVE comprises one or more NixOS virtual machine configurations, a scripted exploitation scenario, and machine-checkable assertions that provide factual evidence of exploitation. This design facilitates sharing, validation, review, and long-term reproducibility. Building on NICE, we construct a public archive of 56 reproduced vulnerabilities spanning multiple CWE categories, attack vectors, and target types (user-space, system software, kernel, and graphical applications), combining manually authored recipes with recipes produced by an LLM-based automated pipeline and merged after human review. We evaluate the feasibility of the automated pipeline on 110 real-world CVEs, achieving fully automated reproduction in 54.5% of the cases, at an average cost of 0.73 per CVE. The evaluation demonstrates that NICE provides a practical target for LLM-assisted recipe generation, while also identifying where human refinement or additional automation remains necessary.

Sources

Related papers