Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs
cs.LG, cs.AI, cs.CR
Submitted: 2026-03-25
Updated: 2026-11-07
Code: https://github.com/romovpa/claudini
License: http://creativecommons.org/licenses/by/4.0/
The gist: We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations.
Terminology
Abstract
We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations. We deploy frontier agents, such as Claude Code and Codex, in an autoresearch loop with access to a library of 30+ prior methods and an evaluation script with a fixed compute budget. We show this pipeline to be effective in jailbreaking OpenAI's GPT-OSS-Safeguard-20B and in prompt injections against Meta-SecAlign-70B, an adversarially robust model. For GPT-OSS-Safeguard, the best agent-discovered method achieves up to 80% attack success rate on CBRN queries, compared to <50% for existing methods. For SecAlign, it achieves 100% ASR, while the best prior automated methods only achieve 82%. Notably, in our setting, attack methods are developed on unrelated surrogate models for a pure random-target token-forcing task, yet generalize directly to prompt injection on the adversarially trained model. Finally, we trace the lineage of methods developed during autoresearch, characterizing the agents' strategies and failure modes. Adversarial ML has long held that defenses must be evaluated against attacks tailored to them; autoresearch automates this principle, and we argue it should be the minimum bar for defense evaluation going forward.
Sources
- Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents
- Scaling Laws for Neural Language Models
- Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Models
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attacks
- gpt-oss-120b & gpt-oss-20b Model Card
- KernelBench: Can LLMs Write Efficient GPU Kernels?
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- PAL: Proxy-Guided Black-Box Attack on Large Language Models
- The Resurgence of GCG Adversarial Attacks on Large Language Models
- AgenticRed: Evolving Agentic Systems for Red-Teaming
- Universal and Transferable Adversarial Attacks on Aligned Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis