Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code
summary
The gist
Large Language Models (LLMs) can be jailbroken into generating malicious code by exploiting Grammar-Constrained Decoding (GCD) to bypass safety alignment The paper introduces CodeSpear…
In short
CodeSpear is a jailbreak attack that uses Grammar-Constrained Decoding (GCD) to force Large Language Models (LLMs) to generate malicious code by exploiting the gap between natural language safety alignment and code generation. The authors introduce CodeShield, a defense that trains models to generate harmless honeypot code when restricted to the code modality, effectively restoring safety.
Key concepts
- Grammar-Constrained Decoding (GCD)
- GCD is a technique used during LLM output generation where the model is restricted to only produce tokens that adhere to a specific, predefined code grammar. This constraint forces the model's output into a structured format, which can bypass safety mechanisms designed for natural language responses.
- CodeSpear
- This is a new jailbreak attack that exploits GCD. It works by using an ordinary code grammar to invoke the target LLM, causing it to generate malicious code because existing safety alignment is mostly focused on natural language refusals, not constrained code outputs.
- CodeShield
- CodeShield is a defense strategy that trains LLMs to generate 'honeypot' code when they are operating under GCD. Honeypot code is designed to be structurally diverse and semantically harmless, making it difficult for attackers to suppress through grammar tightening.
Terminology used across episodes
This episode discusses
- Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code · Paper Radio
- To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation · Paper Radio
- AI-Driven Self-Evolving Software: A Promising Path Toward Software Automation
- What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction
- DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models
- Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment
- Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts
- RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents · Paper Radio
- Exploring the Potential of Diffusion Large Language Models in Code Generation · Paper Radio
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring · Paper Radio
- XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
- Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars · Paper Radio
- OpenAI GPT-5 System Card
- Qwen2.5-Coder Technical Report
- Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
- The Llama 3 Herd of Models · Paper Radio
- Qwen2.5 Technical Report
- Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking
- When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output · Paper Radio
- Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignment · Paper Radio
- Low-Resource Languages Jailbreak GPT-4
The paper
Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code · Read on arXiv
Tsinghua University · University of Electronic Science and Technology of China
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code".
Elias: Here's the gist of the paper. Large Language Models (LLMs) can be jailbroken into generating malicious code by exploiting Grammar-Constrained Decoding (GCD) to bypass safety alignment The paper introduces CodeSpear, a new jailbreak attack that leverages GCD to induce LLMs into generating malicious code To counter this risk, the authors propose CodeShield, a safety alignment approach that trains the model to generate honeypot code under GCD The Core Vulnerability The central finding is that reliability mechanisms like GCD can become an attack surface when used against malicious prompts When a code grammar is enforced, natural-language refusals fall outside the valid output space, forcing the model to continue generation in the code modality where it lacks explicit safety alignment This breaks the assumption that safety alignment is grounded in natural language when only code is available The authors show that simply applying a benign code grammar can effectively jailbreak LLMs The Attack Mechanism CodeSpear exploits the mismatch between existing safety alignment and GCD by targeting the output space restriction The attack procedure involves invoking the target model through a standard GCD interface with an ordinary code grammar This method is effective because it does not require an adversarial grammar or gradient optimization, making the attack cost negligible The Defense Strategy CodeShield addresses the vulnerability by training the model to generate honeypot code against CodeSpear Honeypot code is designed to be semantically harmless and structurally diverse, making it difficult for an attacker to suppress through grammar tightening The defense works by constructing preference pairs that prioritize natural-language refusal when available, and honeypot code when constrained to the code modality Experimental Validation Experiments across 4 benchmarks on 10 popular LLMs show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average CodeShield restores safety under CodeSpear while preserving benign utility, showing significant reductions in Attack Success Rate (ASR) and Malicious Rate (MR) Furthermore, the sensitivity analysis indicates that CodeSpear remains effective across different grammars, while CodeShield benefits from using more honeypot code samples to improve safety without degrading benign utility Key Contributions The paper makes three main contributions: 1) Proposing CodeSpear, a jailbreak attack that leverages GCD to push LLMs into generating malicious code 2) Introducing code-modality safety alignment through CodeShield, which teaches LLMs to generate honeypot code when natural-language refusals are unavailable 3) Conducting comprehensive experiments showing that CodeSpear can bypass the safety alignment of existing LLMs, while CodeShield effectively restores their safety The findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications The work suggests that safety alignment must be performed in the code modality when natural-language refusals are unavailable CodeShield demonstrates that aligning models toward honeypot code is necessary to close the attack surface exposed by GCD The authors also note that the effectiveness of CodeSpear may vary across different GCD settings, suggesting a general risk of GCD rather than a claim about all implementations The source code for CodeSpear is released for controlled research use, while CodeShield is fully released References
2: Yitong Zhang, C.
Nadia: Li, R.
Elias: Chen, G.
Nadia: Yang, X.
Paper summary: Nadia: So we’re looking at this paper today, "Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code," and the core idea is pretty unsettling. It claims that the way we make code generation reliable, by using something called Grammar-Constrained Decoding or GCD to enforce a specific syntax, actually opens up a new way for people to get the model to generate malicious code.
Elias: Right. The thesis there seems to be that reliability mechanisms like GCD are not just helpful tools; they can become an attack surface when used against prompts designed to cause trouble. They introduce this new jailbreak attack called CodeSpear, which exploits GCD specifically to push the LLM into producing harmful code >
Priya: But what does this mean for the actual safety alignment we’ve been using? I mean, if we rely on GCD to keep things clean, and then an attacker uses a tight grammar constraint to bypass those safety rules, it suggests our current alignment isn't as robust as we think it is >
Nadia: Exactly. The paper points out that existing safety alignment is almost entirely focused on the natural language part of things, which means when the model only has to output code under GCD, there’s no explicit safety training in that code modality itself >
Elias: That’s the key assumption they are breaking. They show that you can just apply a benign code grammar constraint and actually jailbreak these LLMs, which is a pretty simple kind of attack because it doesn't require any fancy adversarial grammar or complex gradient optimization >
Priya: So, if this is true, the problem isn't just about making the prompt better; it’s about how we train the model to behave when it can only speak code >
Nadia: That’s where they come in with their defense strategy called CodeShield. They propose training the model to generate honeypot code specifically against this kind of attack, which is designed to be harmless and structurally varied >
Elias: The preference hierarchy they set up for CodeShield is interesting because it prioritizes a refusal response if possible, or a honeypot code if decoding is unconstrained, but it switches back to prioritizing honeypot code when the model is constrained by grammar >
Priya: That sounds like an attempt to give the model many safe ways to stay within its code space so that even with an attacker's grammar constraint, there are still lots of options that don't lead to something harmful >
Nadia: The results they show across ten different LLMs and four different benchmarks are pretty striking. They found that CodeSpear has an average attack success rate of eighty-one point eight two percent against local models like Qwen2 point-five coder seven billion, and it gets even worse, increasing the attack success rate by more than forty percentage points on average against commercial API based LLMs like GPT-five > <ref:2606.11817#pg1>
Elias: And the defense works as intended because CodeShield restores model safety under GCD, reducing that attack success rate to a level even lower than what you'd see without any attack at all >
Priya: That’s good, but I want to ask about the trade-off. The paper says CodeShield preserves benign utility, causing only minor degradation on benign code generation. What does that minor degradation actually look like in practice? Is it something we can live with when we're just trying to get a basic function written? >
Nadia: They show it causes only minor degradation on benign code gen, and they also point out that CodeShield improves safety both with and without CodeSpear, suggesting it further strengthens the intrinsic safety of the models when natural language is available >
Elias: It seems the fundamental risk here is that safety alignment must be performed in the code modality when we can't rely on natural language refusals to stop bad behavior >
Priya: I wonder if this means that standard methods like Direct Preference Optimization, or DPO, which only align the model toward natural language refusals, just fail once the model is forced to generate code under GCD >
Nadia: That’s what they show. Safe-DPO remains largely ineffective under CodeSpear because it only aligns the model toward natural language refusals, and the attack success rate and malicious rate stay close to vanilla levels there >
Elias: So, the implication is that we need this code modality safety alignment approach, CodeShield, to close that gap exposed by GCD >
Priya: And they did mention a limitation. They noted that the effectiveness of CodeSpear might vary across different grammar settings, which suggests it’s a general risk about GCD rather than something specific to one implementation >
Nadia: So, the main point is that we need to think about safety alignment in this code context, because relying solely on natural language refusals isn't enough when you're dealing with constrained decoding methods like GCD.
Conclusion: Nadia: So, this paper, "Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code," really drills down into how reliability tools can backfire when you use them against bad prompts.
Elias: Yeah, it’s about this specific technique called Grammar-Constrained Decoding or GCD and how it lets an AI bypass its own safety rules by just talking in code.
Priya: So, the big question is, why does enforcing a grammar like that actually open up a path to generating harmful code?
Nadia: Because existing safety alignment mostly lives in natural language. When you lock the model into a strict code format, those natural language refusals get pushed out of bounds.
Elias: That means the model stops looking at its safety instructions in the way we think it does, forcing it to just keep generating code without that guardrail.
Priya: So, if this is true, what does this mean for how we think about securing these code-generating models?
Nadia: It suggests that safety training has to happen directly within the code modality when those natural language stop signs aren't available.
Elias: They introduce CodeShield as a defense method where the AI is trained to generate honeypot code specifically designed to be harmless and structurally different from anything malicious.
Priya: So, CodeShield tries to give the model many safe ways to stay within its allowed code space, even when someone tries to force it with a bad grammar constraint.
Nadia: The results show that CodeShield successfully restores safety under GCD by aligning the model toward this honeypot code, and it actually keeps most of the model's good performance for normal tasks.
Elias: It’s interesting because they also found that standard methods like Direct Preference Optimization struggle when the model is forced into this constrained code environment.
Priya: So, the authors are saying that just aligning models toward natural language refusals isn't enough if you're dealing with strict code output constraints.
Nadia: That’s the core finding, and it really highlights a risk we need to pay attention to when developing these tools.
Elias: It shows that the reliability features we build into models can actually become vectors for attack if not handled carefully across different modalities.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18