Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots".
Elias: Here's the gist of the paper. Honeyval proposes Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots because it addresses the lack of a unified evaluation protocol in the field.
Nadia: Evaluation Limitations Addressed The existing evaluations of LLM-powered honeypots suffer from limitations such as not being scalable, not being reproducible, and not being representative of practical attacks.
Elias: Prior methods often rely on measuring response similarity on fixed commands, manual testing, or real-world deployment which are limited in scalability, adaptability, or reproducibility.
Nadia: The framework aims to bridge this gap by grounding the honeypots in 16 backend applications and using AI hacking agents as attackers.
Paper summary: Nadia: So, this paper is calling itself Honeyval, which is a comprehensive evaluation framework for LLM-powered HTTP honeypots. The authors are looking at how we've been testing these things lately and saying the old ways of testing them just don't cut it anymore.
Elias: Exactly. They point out that most existing evaluations either measure response similarity on fixed commands, or they rely on manual testing or real-world deployment, and those methods aren't scalable, not reproducible, and they often don't represent how actual attacks work in the wild.
Priya: So what’s the core idea here? What is this framework actually trying to fix for people building these honeypots?
Nadia: They are grounding these honeypots in sixteen different backend applications and using AI hacking agents as attackers to simulate a more practical attack scenario. It aims to bridge that gap by giving us something scalable and consistent.
Elias: The framework is structured around three main evaluation tasks: there’s the main task where you evaluate the LLM-powered honeypot directly against an agentic attacker, then there are two control tasks to monitor agent and honeypot capabilities when you change things up, and finally, they define clear goals for what the attacker is trying to achieve.
Priya: That sounds like a structured way to test different configurations, but how do they ensure it’s actually representative of real threats?
Nadia: They make reproducibility possible by grounding everything in those sixteen practical backend application API specifications and defining realistic exploit goals for each one. It's about making the testing concrete.
Elias: And they also conducted an extensive evaluation of cost-efficient LLMs used as honeypots, identifying specific strengths and future directions for using AI in this space.
Priya: So it seems like the focus is on moving away from simple command similarity tests toward something that measures actual interaction success and detection under varied conditions.
Conclusion: Nadia: So, looking at "Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots," what do we actually get out of this work? It’s about providing a standard way to test these LLM honeypots that moves beyond just checking if they answer a command correctly.
Elias: Right. The authors identified the limitations in current evaluations, which is the starting point. They proposed Honeyval as the first comprehensive framework for this kind of honeypot evaluation, using AI agents to simulate real attacker behavior against sixteen simulated backend environments.
Priya: It seems like what this means for us in practice is that before we deploy these LLM honeypots widely, we need a structured way to prove they actually work and don't just look good on paper.
Nadia: Precisely. And they did some interesting experiments too, showing that these LLM-powered honeypots lead to substantially longer interactions with the attacker compared to rule-based baseline honeypots—on average eighty-two point six requests per interaction versus thirty point six.
Elias: That length is significant, but they also found that these LLM honeypots are far less frequently detected, even by frontier models, while still keeping a running cost advantage against agentic attackers.
Priya: I'm interested in the trade-offs they mentioned earlier—they observed that experimenting with different counter-offensive honeypot configurations actually led to longer interactions but also increased the likelihood of being eventually detected.
Nadia: That’s a good point, because it shows you can test those limits. The work opens up avenues for future improvements, like suggesting system-level enhancements such as caching or conversation memory to help offload context from the LLM when testing.
Elias: Overall, the impact is that this framework gives developers concrete feedback on how to advance LLM-powered honeypots safely and effectively. It's a tool for building better defense systems in practice.
Mark Vero, Fabian Kaczmarczyck, Ivan Petrov, Ilia Shumailov, Jamie Hayes, Niels Heinen, Tianqi Fan, Luca Invernizzi, Martin Vechev
ETH Zurich Google Google DeepMind AI Sequrity Company Independent
cs.CR, cs.AI, cs.LG
Submitted: 2026-05-28
Updated: 2026-11-09
Code: https://github.com/google-research/honeyval
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Honeyval proposes Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots because it addresses the lack of a unified evaluation protocol in the field.
Key concepts
- LLM-powered Honeypot
- This is a security tool where a Large Language Model (LLM) acts as the defense mechanism. It interacts with potential attackers via HTTP requests, trying to lure them into revealing information or executing malicious commands, mimicking a real system.
- Agentic Attacker
- These are AI hacking agents designed to autonomously search for vulnerabilities and exploit weaknesses. They are competitive with human experts in finding exploits, making the honeypot's defense capabilities much more robust and realistic to test against.
- Evaluation Framework
- Honeyval is a structured system designed to systematically measure how well an LLM honeypot performs. It uses specific tasks, control measures, and defined metrics to ensure that evaluations are scalable, reproducible, and representative of real-world attack scenarios.
Terminology
Summary
Honeyval proposes Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots because it addresses the lack of a unified evaluation protocol in the field. p. 0 Key challenge: Evaluating LLM-powered honeypots The main goal of evaluating honeypots is
to assess how well they simulate a target system. While there is a growing body of work proposing
honeypot solutions powered by LLMs, there is currently a lack of a unified and scalable evaluation
protocol.Where the paper says it
Evaluation Limitations Addressed
The existing evaluations of LLM-powered honeypots suffer from limitations such as not being scalable, not being reproducible, and not being representative of practical attacks. Prior methods often rely on measuring response similarity on fixed commands, manual testing, or real-world deployment which are limited in scalability, adaptability, or reproducibility. p. 0 Most evaluations consist of
measuring response similarity on fixed commands, manual testing, or real-world
deployment. These methods are often not scalable for development, reproducible
across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose
Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots
in 16 backend applications, using AI hacking agents as attackers, employing two
control tasks to monitor agent and honeypot capabilities across customizations,
and defining clear and verifiable exploit goals for the attacker.Where the paper says it
Where the paper says it
Framework Design and Methodology
Honeyval is structured around three evaluation tasks: the main task where the LLM-powered honeypot is evaluated directly against an agentic attacker, two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. To ensure scalability, Honeyval simulates attacks using AI hacking agents who are competitive with human experts in finding exploits. p. 1 To make our evaluations
scalable, we rely on simulating attacks using AI hacking agents, which have been recently shown to
be competitive with human experts in finding exploits [8, 9]. To enable reproducibility and ensure
representativeness, we ground our evaluation in 16 practical backend application API specifications,
and define realistic exploit goals for each API. To make our evaluations
scalable, we rely on simulating attacks using AI hacking agents, which have been recently shown to
be competitive with human experts in finding exploits [8, 9]. To enable reproducibility and ensure
representativeness, we ground our evaluation in 16 practical backend application API specifications,
and define realistic exploit goals for each API.Where the paper says it
Where the paper says it
Key Experimental Findings
The experiments highlight that LLM-powered honeypots lead to substantially longer interactions with the attacker than rule-based baseline honeypots, on average 82.6 vs. 30.6 requests per interaction. p. 1 Additionally, we find that compared to rule-based honeypot API mocks,
our LLM-powered honeypots keep attackers engaged for significantly longer (on average 82. 6 requests per interaction), and are identified at a low rate by the agents as honeypots (well-below
<40% in most cases). Using Honeyval, we
conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to
substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on
average, preserving a running cost advantage against agentic attackers. Our experiments highlight the promise of LLM-powered honeypots; they lead to
substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on
average, preserving a running cost advantage against agentic attackers. Further, we
experiment with different counter-offensive honeypots configurations, and observe
unique trade-offs, such as longer interactions at the cost of increased detection.Where the paper says it
Where the paper says it
Where the paper says it
Metrics and Customization Impact
Honeyval tracks key metrics such as interaction length, running cost, latency, and detectability in the main task. The framework also introduces two control tasks to independently baseline and monitor the capabilities of both the hacking agent and the honeypot under adaptations. p. 1 Here, we measure
key metrics for the development of LLM-powered honeypots, such as interaction length, running cost,
latency, and detectability. Additionally, we introduce two control tasks to independently baseline and
monitor the capabilities of the hacking agent and the honeypot under adaptations. Finally, we experiment with two custom configurations of the
honeypot LLMs, showing a potential for counterattacking or misleading agentic attackers. For
instance, by instructing the honeypot to ‘convince’ the agent of the security of the application, we
manage to significantly increase the average interaction length with the attacker.Where the paper says it
Where the paper says it
Future Directions and Impact
The work opens up avenues for future improvement by suggesting system-level enhancements like caching or conversation memory to offload context from the LLM. The scope of Honeyval could be extended to include larger-scale applications or other protocols beyond HTTP honeypots in future work. p. 8 First, as both automated defenders and attackers evolve, it is crucial to incorporate larger-scale
applications in honeypot evaluations. Second, using the techniques and overall design principles,
future work could extend the framework beyond HTTP honeypots to honeypots simulating common
OS shells, or other, more complex protocols and systems with highly specific configurations. While
we already observe promising performance, there is clear potential for future improvement in
designing more complex, hybrid systems where the LLM can utilize a response cache, or outsource
its context to external files. To this end, Honeyval provides key evaluation feedback, enabling iterative
improvements in design.Where the paper says it
Where the paper says it
Honeyval Availability
Honeyval is available at https://github.com/google-research/honeyval. p. 2 Honeyval is available at: https://github. com/google-research/honeyval. For instance, a recent breach to McKinsey’s internal chatbot exploited exactly such a backend app
[35]. Each application comes with four artifacts: (i) the OpenAPI [34] specification, describing
the endpoints and functionality of the app; (ii) an extensive test suite sourced from BaxBench [33]
for verifying the functionality of any software implementing the API specification; (iii) a secure
implementation of the specification, and (iv) a corresponding insecure, exploitable implementation. 6 [37] and Gemini CLI with Gemini 3 Flash [36]. We run our LLM-powered honeypots with a diverse set of cost-efficient proprietary and open-weight
models: Gemini 3 Flash [36], Claude Haiku 4. 5 Flash [39], Qwen 3. 5 9B [40], and
GPT 5. The
backend LLM is provided with the OpenAPI specification of the application it has to simulate. The
applications’ endpoints are served in a hollow server, where all API calls are routed to the LLM,
which crafts the response. Here, remarkably, the agent manages to maintain a cost
advantage, due to the highly efficient model
powering it. Crucially, we have to note here
that our cost estimates resulting from our
honeypot architecture are highly conservative in favor of the attacker; our honeypot
is largely token-inefficient; each time, we feed back the entire context to the honeypot. 2
Crucially, while we observe no clear correlation between the honeypots’ latency or
functional fidelity and the detection rates
of Figure 2; upon inspecting the hacking
agents’ logs, we do find that they repeatedly cite response latency and timeouts
as evidence when classifying the LLMpowered apps as honeypots. Overall, we
do not believe that either latency or large
token costs are a fundamental disqualifying
limitation of LLM honeypots in the long
term. Configuration robustness of the hacking agents Just as the LLM-powered honeypots, the hacking agents can also be configured and prompted adaptively. In turn, this could also come at the
cost of performance. We open-source Honeyval, advancing the community’s efforts
to develop and further advance LLM-powered honeypots. We discuss the potential broader societal
impacts of our work in Section A.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Grounding evaluations in 16 backend applications using their OpenAPI specifications ensures reproducibility by defining
clear and verifiable exploit goals for each API.
This allows for consistent testing across different runs, overcoming the issue where LLMs generate inconsistent system configuration details. -
Implementing two control tasks—one against real implementations and one against functional test suites—provides a comprehensive monitoring capability of both the agent and honeypot capabilities under customization. This addresses the limitation that prior evaluations are
open-ended
by ensuring fidelity checks against known application functionality. -
Utilizing AI hacking agents to simulate adversarial interactions enhances scalability by replacing manual testing with agents
which have been recently shown to be competitive with human experts in finding exploits.
This allows for extensive evaluation without being limited by the cost or time constraints of human testers. -
Configuring honeypots with custom prompts like Careful PI or Convince enables defenders to proactively guide the system's behavior, such as instructing it
to convince the attacker that the application cannot be hacked,
which can significantly increase interaction length. -
Developing agent configurations like Exploit-Detect and Exploit-Detect-Hide allows for testing how attackers adapt to honeypots, enabling them to
anticipate honeypots
while simultaneously measuring their performance against actual exploits on real applications. -
The framework allows for the measurement of key metrics such as interaction length, running cost, and detection rates across direct interactions and control tasks, providing a holistic view of LLM-powered honeypot effectiveness against agentic attackers.
Abstract
Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They enable defenders to construct high-interaction honeypots with low system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework. Most evaluations consist of measuring response similarity on fixed commands, manual testing, or real-world deployment. These methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on average, preserving a running cost advantage against agentic attackers. Further, we experiment with various honeypot extensions and counter-offensive configurations, observing unique trade-offs, e.g., lower latency at the price of increased detection.
Sources
- GPT-5 at CTFs: Case Studies From Top-Tier Cybersecurity Events
- ReAct: Synergizing Reasoning and Acting in Language Models
- LLMHoney: A Real-Time SSH Honeypot with Large Language Model-Driven Dynamic Response Generation
- HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Model
- PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
- PentestAgent: Incorporating LLM Agents to Automated Penetration Testing
- EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
- D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security
- CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
- Hacking CTFs with Plain Agents
- BaxBench: Can LLMs Generate Correct and Secure Backends?
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data