Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
summary
The gist
Honeyval proposes Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots because it addresses the lack of a unified evaluation protocol in the field.
In short
Honeyval introduces a comprehensive evaluation framework for LLM-powered HTTP honeypots to solve existing issues with scalability and reproducibility. It tests honeypots against AI hacking agents using 16 backend applications, showing they engage attackers much longer than traditional methods while remaining less detectable by frontier models.
Key concepts
- LLM-powered Honeypot
- This is a security tool where a Large Language Model (LLM) acts as the defense mechanism. It interacts with potential attackers via HTTP requests, trying to lure them into revealing information or executing malicious commands, mimicking a real system.
- Agentic Attacker
- These are AI hacking agents designed to autonomously search for vulnerabilities and exploit weaknesses. They are competitive with human experts in finding exploits, making the honeypot's defense capabilities much more robust and realistic to test against.
- Evaluation Framework
- Honeyval is a structured system designed to systematically measure how well an LLM honeypot performs. It uses specific tasks, control measures, and defined metrics to ensure that evaluations are scalable, reproducible, and representative of real-world attack scenarios.
Terminology used across episodes
This episode discusses
- Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots · Paper Radio
- GPT-5 at CTFs: Case Studies From Top-Tier Cybersecurity Events
- ReAct: Synergizing Reasoning and Acting in Language Models
- LLMHoney: A Real-Time SSH Honeypot with Large Language Model-Driven Dynamic Response Generation
- HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Model
- PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
- PentestAgent: Incorporating LLM Agents to Automated Penetration Testing
- EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
- D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security
- CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
- Hacking CTFs with Plain Agents
- BaxBench: Can LLMs Generate Correct and Secure Backends?
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
The paper
Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots · Read on arXiv
Mark Vero, Fabian Kaczmarczyck, Ivan Petrov, Ilia Shumailov, Jamie Hayes, Niels Heinen, Tianqi Fan, Luca Invernizzi, Martin Vechev
ETH Zurich Google Google DeepMind AI Sequrity Company Independent
Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They enable defenders to construct high-interaction honeypots with low system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework. Most evaluations consist of measuring response similarity on fixed commands, manual testing, or real-world deployment. These methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on average, preserving a running cost advantage against agentic attackers. Further, we experiment with various honeypot extensions and counter-offensive configurations, observing unique trade-offs, e.g., lower latency at the price of increased detection.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots".
Elias: Here's the gist of the paper. Honeyval proposes Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots because it addresses the lack of a unified evaluation protocol in the field.
Nadia: Evaluation Limitations Addressed The existing evaluations of LLM-powered honeypots suffer from limitations such as not being scalable, not being reproducible, and not being representative of practical attacks.
Elias: Prior methods often rely on measuring response similarity on fixed commands, manual testing, or real-world deployment which are limited in scalability, adaptability, or reproducibility.
Nadia: The framework aims to bridge this gap by grounding the honeypots in 16 backend applications and using AI hacking agents as attackers.
Paper summary: Nadia: So, this paper is calling itself Honeyval, which is a comprehensive evaluation framework for LLM-powered HTTP honeypots. The authors are looking at how we've been testing these things lately and saying the old ways of testing them just don't cut it anymore.
Elias: Exactly. They point out that most existing evaluations either measure response similarity on fixed commands, or they rely on manual testing or real-world deployment, and those methods aren't scalable, not reproducible, and they often don't represent how actual attacks work in the wild.
Priya: So what’s the core idea here? What is this framework actually trying to fix for people building these honeypots?
Nadia: They are grounding these honeypots in sixteen different backend applications and using AI hacking agents as attackers to simulate a more practical attack scenario. It aims to bridge that gap by giving us something scalable and consistent.
Elias: The framework is structured around three main evaluation tasks: there’s the main task where you evaluate the LLM-powered honeypot directly against an agentic attacker, then there are two control tasks to monitor agent and honeypot capabilities when you change things up, and finally, they define clear goals for what the attacker is trying to achieve.
Priya: That sounds like a structured way to test different configurations, but how do they ensure it’s actually representative of real threats?
Nadia: They make reproducibility possible by grounding everything in those sixteen practical backend application API specifications and defining realistic exploit goals for each one. It's about making the testing concrete.
Elias: And they also conducted an extensive evaluation of cost-efficient LLMs used as honeypots, identifying specific strengths and future directions for using AI in this space.
Priya: So it seems like the focus is on moving away from simple command similarity tests toward something that measures actual interaction success and detection under varied conditions.
Conclusion: Nadia: So, looking at "Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots," what do we actually get out of this work? It’s about providing a standard way to test these LLM honeypots that moves beyond just checking if they answer a command correctly.
Elias: Right. The authors identified the limitations in current evaluations, which is the starting point. They proposed Honeyval as the first comprehensive framework for this kind of honeypot evaluation, using AI agents to simulate real attacker behavior against sixteen simulated backend environments.
Priya: It seems like what this means for us in practice is that before we deploy these LLM honeypots widely, we need a structured way to prove they actually work and don't just look good on paper.
Nadia: Precisely. And they did some interesting experiments too, showing that these LLM-powered honeypots lead to substantially longer interactions with the attacker compared to rule-based baseline honeypots—on average eighty-two point six requests per interaction versus thirty point six.
Elias: That length is significant, but they also found that these LLM honeypots are far less frequently detected, even by frontier models, while still keeping a running cost advantage against agentic attackers.
Priya: I'm interested in the trade-offs they mentioned earlier—they observed that experimenting with different counter-offensive honeypot configurations actually led to longer interactions but also increased the likelihood of being eventually detected.
Nadia: That’s a good point, because it shows you can test those limits. The work opens up avenues for future improvements, like suggesting system-level enhancements such as caching or conversation memory to help offload context from the LLM when testing.
Elias: Overall, the impact is that this framework gives developers concrete feedback on how to advance LLM-powered honeypots safely and effectively. It's a tool for building better defense systems in practice.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18