BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage

arXiv:2506.02479 · cs.CR, cs.CL · Submitted 2025-06-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage".

Elias: Here's the gist of the paper. The paper develops BitBypass, a novel black-box jailbreak attack that leverages hyphen-separated bitstream camouflage to bypass the safety alignment of aligned Large Language Models (LLMs).

Nadia: This attack represents a new direction in jailbreaking by exploiting the fundamental information representation of data as continuous bits, rather than relying on prompt engineering or adversarial manipulations.

Elias: The Attack Mechanism BitBypass operates by transforming a sensitive word in a harmful prompt into its hyphen-separated bitstream and replacing that word with a placeholder in the original harmful prompt.

Nadia: The adversarial prompt is then constructed by combining this hyphen-separated bitstream with the substituted harmful prompt.

Paper summary: Nadia: So this paper introduces something called BitBypass. It's a new way to try and get around the safety guardrails on large language models by using hyphen-separated bitstream camouflage.

Elias: Exactly, it claims this is a new direction in jailbreaking because it exploits how data is fundamentally represented as continuous bits, rather than just trying to trick the model with careful wording or manipulation of the prompt itself.

Priya: That makes sense from a measurement perspective. So you're suggesting that instead of using complex prompting tricks, you're treating sensitive words in a harmful request like raw binary data and hiding them in this camouflage.

Nadia: Right, so the core idea is to take a sensitive word from a prompt, turn it into its hyphen-separated bitstream, and then replace that original word with this placeholder in the harmful prompt.

Elias: And then you construct your full adversarial prompt by combining that bitstream with the modified harmful request. It’s different because it’s not about continuous adversarial manipulations; it's about a discrete search space of these bitstreams.

Priya: I wonder what this means for the data itself. Does this camouflage make the harmful instruction look like something else to the model?

Nadia: The paper uses a system prompt to guide the target LLM, giving it several regulatory specifications like Curbed Capabilities, Program-of-Thought, and Focus Shifting. This is key because it leads the model to decode that bitstream camouflage.

Elias: That specific part where you use a Python function inside that system prompt for binary to text conversion of the bitstream is what makes the whole setup work with the LLM’s alignment layers.

Priya: So, you're not just changing the input; you're essentially giving the model a secret instruction set within its own instructions to figure out how to read your camouflage.

Nadia: Right, and then there are evaluation metrics they use. They measure things like the Response Refusal Rate, which checks how much the model refuses to generate content, and the Attack Success Rate, which is how many attempts actually manage to bypass that alignment.

Elias: They compare this BitBypass attack against direct instructions for harmful prompts and several baseline jailbreak attacks like AutoDAN, Base64, DeepInception, and DRA.

Priya: And they test it on a bunch of state-of-the-art models—GPT-4o, Gemini one point five Pro, Claude three point five Sonnet, Llama three point one 70B and Mixtral eight times 22B <ref:2506.02479#pg1>. I want to know if this works everywhere or just on one model family.

Nadia: They found that BitBypass performs significantly better than direct instructions and baseline attacks across those five models. Specifically, it showed superior Response Refusal Rate performance on Claude and Llama when tested on the AdvBench dataset.

Elias: The numbers they report for the AdvBench dataset are pretty striking; they claim BitBypass improved the Response Refusal Rate by eighty-four percent and the Attack Success Rate by four hundred thirty-three percent compared to those baseline attacks.

Priya: That improvement on success rate is a big number, but what about how it handles actual harmful content? Is this just about bypassing safety filters, or can it actually generate something malicious?

Nadia: They also tested this attack on the PhishyContent dataset to see if it could trick the models into generating phishing content. And they found that BitBypass successfully tricked most target LLMs into creating accurate phishing content, with a Phishing Content Rate between sixty-eight percent and ninety-two percent across all those models.

Elias: That suggests the vulnerability isn't just theoretical; it can lead to generating genuinely unsafe outputs when used effectively.

Priya: So, even if the model is trying to prevent harmful content, this method seems quite effective at getting it to generate that kind of content. What about how persistent this vulnerability is?

Nadia: The study confirmed that this bitstream camouflage vulnerability stays active and persistent even in the latest versions of the target LLMs they tested. The ablation study pointed to the Curbed Capabilities regulatory instruction within the system prompt as being the main factor enabling BitBypass's success.

Elias: So, it’s not a fluke; it seems tied directly to how that specific regulatory instruction guides the model’s behavior during decoding. What do you think that implies for defense?

Nadia: The authors suggest a potential mitigation strategy, which is using perplexity based screening of the system prompt. That means you check the prompt itself for certain patterns before sending it to the LLM.

Priya: I also saw something in their exploratory analysis about token usage. They found that BitBypass increases the token count of harmful prompts by approximately two thousand four hundred thirty percent across different tokenizers, which is quite high.

Elias: And they noted that perplexity analysis showed BitBypass can achieve high perplexity with only one word obfuscation, which is less than encoding-based attacks like Base64 or Hex. That suggests it can trick models with very minimal obfuscation compared to those other schemes.

Nadia: Overall, the paper concludes that BitBypass effectively and efficiently bypasses the alignment of these target LLMs and generates harmful or unsafe content. It really highlights how effective this method is against all the state-of-the-art LLMs they tested.

Elias: And it poses a high risk for generating genuinely harmful and unsafe content if you're dealing with those models without that specific system prompt screening in place.

Nadia: The implication is clear: this work shows a very efficient way to jailbreak these large language models, and it’s something that needs serious attention from security researchers right now.

Conclusion: Nadia: So we’re wrapping up this look at BitBypass, which is titled "BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage."

Elias: Yeah, it basically says they found a way to get around the safety alignments on large language models by using hyphen-separated bitstream camouflage.

Priya: It seems like they're not just tweaking the wording of a prompt; they’re treating sensitive words as raw data and hiding them in this binary pattern.

Nadia: Exactly, it’s about exploiting how bits are represented fundamentally, which is different from just trying to trick the model with clever phrasing.

Elias: The core mechanism involves taking a word you want the AI to bypass and replacing it with its hyphen-separated bitstream in the harmful prompt.

Priya: And then they use a system prompt—with things like Curbed Capabilities—to guide the AI to decode that camouflage, which is pretty clever for getting past its safety layers.

Nadia: They showed this works on a huge range of top models, from GPT-4o all the way up to Mixtral eight times 22B.

Elias: The results are pretty strong, showing improvements in refusal rates and attack success rates by a lot compared to older jailbreak methods.

Priya: But we need to look closely at what those numbers actually mean for generating harmful content, not just bypassing the initial filters.

Nadia: That’s exactly it. They found that this method can actually trick the models into creating accurate phishing content on the PhishyContent dataset with rates up to ninety-two percent.

Elias: So, if you can use this technique effectively, it seems like a real risk for getting those models to output genuinely unsafe material without them noticing.

Priya: And they also confirmed that this vulnerability stays active even in the newest versions of these models, which means it’s persistent.

Nadia: It’s definitely persistent, and they pointed to that specific regulatory instruction in the system prompt as what makes the whole thing tick.

Elias: They suggest checking the system prompt for certain patterns before you send anything to see if you can screen out this kind of attack before it even starts.

Priya: That screening idea seems like a practical way to address the persistence issue they found.

Nadia: So, in short, BitBypass is an efficient way to jailbreak these models that leverages bitstream camouflage and system prompt instructions to bypass safety alignment.

Elias: It’s a really interesting paper because it shows how fundamental data representation can be exploited for adversarial purposes.

Priya: We need to keep watching how this kind of technique evolves, especially when we look at vision models or multi-modal ones.

Texas A&M University

cs.CR, cs.CL

Submitted: 2025-06-03

Updated: 2026-11-15

Comments: 27 pages, 27 figures, and 4 tables

Journal ref: Findings of the Association for Computational Linguistics: EACL 2026, pages 3808-3834, 2026

DOI: 10.18653/v1/2026.findings-eacl.198

Code: https://github.com/kalyan-nakka/BitBypass

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 88/100

The gist: The paper develops BitBypass, a novel black-box jailbreak attack that leverages hyphen-separated bitstream camouflage to bypass the safety alignment of aligned Large Language Models (LLMs).

Key concepts

Bitstream Camouflage
This technique involves taking a specific word from a harmful prompt and converting it into its raw binary representation, which is then broken up using hyphens. This hidden bit sequence acts as camouflage, concealing the original sensitive meaning from the LLM's safety filters.
System Prompt Engineering
BitBypass uses a special instruction set in the system prompt to guide the target LLM. This includes directives like 'Curbed Capabilities' and a 'Program-of-Thought' that provides a function to convert the hidden bitstream back into readable text, effectively tricking the model into decoding the camouflage.
Adversarial Evaluation Metrics
The attack's success is measured by metrics like Response Refusal Rate (RRR) and Attack Success Rate (ASR). These quantify how often the LLM refuses to answer or successfully bypasses safety rules, allowing researchers to compare BitBypass against other known jailbreaking methods.
Program-of-Thought
This is a specific component within the system prompt that instructs the LLM on how to handle the input. It provides a Python function designed specifically for converting binary data (the bitstream) into natural language text, which is essential for revealing the hidden instructions.

Terminology

Summary

The paper develops BitBypass, a novel black-box jailbreak attack that leverages hyphen-separated bitstream camouflage to bypass the safety alignment of aligned Large Language Models (LLMs). p. 0

Where the paper says it

However, the robustness of these aligned LLMs is always challenged by adversarial attacks that exploit unexplored and underlying vulnerabilities of the safety alignment. In this paper, we develop a novel black-box jailbreak attack, called BitBypass, that leverages hyphen-separated bitstream camouflage for jailbreaking aligned LLMs.

This attack represents a new direction in jailbreaking by exploiting the fundamental information representation of data as continuous bits, rather than relying on prompt engineering or adversarial manipulations. p. 0
Where the paper says it

In this paper, we develop a novel black-box jailbreak attack, called BitBypass, that leverages hyphen-separated bitstream camouflage for jailbreaking aligned LLMs. This represents a new direction in jailbreaking by exploiting fundamental information representation of data as continuous bits, rather than leveraging prompt engineering or adversarial manipulations.

The Attack Mechanism

BitBypass operates by transforming a sensitive word in a harmful prompt into its hyphen-separated bitstream and replacing that word with a placeholder in the original harmful prompt. The adversarial prompt is then constructed by combining this hyphen-separated bitstream with the substituted harmful prompt. p. 7

Where the paper says it

We then create a substitution prompt by replacing the sensitive word in the harmful prompt with a placeholder. Finally, the adversarial prompt (user prompt) is generated by combining the hyphen-separated bitstream and substitution prompt.

This approach differs from white-box attacks because it leverages a discrete search space of bitstreams instead of generating prompts through continuous adversarial manipulations. p. 2
Where the paper says it

, 2023) that generates adversarial prompts leveraging a discrete and infinite search space, our BitBypass leverages hyphenseparated bitstream for generating the adversarial prompt. Additionally, compared to encoding-based jailbreak attacks (Wei et al.

System Prompt Engineering

To lead the target LLM to decode the camouflage, BitBypass utilizes a system prompt containing several regulatory specifications. These include Curbed Capabilities, Program-of-Thought, and Focus Shifting. p. 7

Where the paper says it

Finally, the adversarial prompt (user prompt) is generated by combining the hyphen-separated bitstream and substitution prompt. Further, to effectively lead the target LLM into uncovering the camouflage of adversarial prompt, we embed the system prompt with three regulatory specifications, namely Curbed Capabilities, Program-of-Thought, and Focus Shifting.

The Program-of-Thought provides a Python function for binary-to-text conversion of the bitstream. p. 3
Where the paper says it

It was observed that, through Program-of-Thought prompting, LLMs can express reasoning steps as Python programs, and computations could be accomplished by a Python interpreter (Chen et al. So, we provided a Python function called bin_2_text in the system prompt for efficient binary-to-text conversion of the transformed sensitive word.

Focus Shifting is crucial as it redirects the LLM's attention after decoding to avoid triggering its safety alignment.

Adversarial Evaluation Metrics

The effectiveness of BitBypass is quantified using several metrics across various datasets and target LLMs. Key metrics include the Response Refusal Rate (RRR), which measures the degree of refused response generation, and the Attack Success Rate (ASR), which represents the degree of attack attempts that successfully bypass alignment. p. 4

Where the paper says it

We determine rus and nus as: rus ⇐= HS(r) ≥ 3 ∧ QS(r) ≥ 3 (2) nus = #rus (3) where r is LLM response. The attack success rate (ASR) represents the degree of attack attempts that successfully bypass the LLM’s alignment and generate harmful content.

The paper evaluates performance against direct instructions and various baseline jailbreak attacks such as AutoDAN, Base64, DeepInception, and DRA. p. 1
Where the paper says it

Precisely, we evaluated the adversarial performance of BitBypass in comparison with direct instruction of harmful prompts and baseline jailbreak attacks of AutoDAN (Liu et al. , 2023), Base64 (Wei et al.

Performance Against State-of-the-Art Models

The evaluation demonstrated that BitBypass significantly outperforms direct instructions and baseline attacks across five state-of-the-art LLMs including GPT-4o, Gemini 1.5 Pro, Claude 3. p. 1

Where the paper says it

5 Pro (Team et al. , 2024), Claude 3.

5 Sonnet, Llama 3. p. 1
Where the paper says it

, 2024), Claude 3. 5 Sonnet (ant, 2024), Llama 3.

1 70B, and Mixtral 8x22B. p. 1
Where the paper says it

1 70B (Grattafiori et al. , 2024) and Mixtral 8x22B (mis, 2024), subjected to various experiments.

BitBypass achieved superior RRR performance on Claude and Llama on the AdvBench dataset. p. 4
Where the paper says it

BitBypass dramatically reduces the Refusal Response Rate (RRR) from the direct instruction range of (66%, 99%) down to [0%, 28%), while simultaneously increasing the Attack Success Rate (ASR) from [0%, 32%] to (48%, 78%]. On the AdvBench dataset, BitBypass achieves superior RRR performance on Claude and Llama, and leads in ASR on Claude, Mixtral, GPT4o, and Gemini.

Furthermore, in terms of adversarial perspective, BitBypass improved RRR by 84% and ASR by 433% for the AdvBench dataset. p. 4
Where the paper says it

Overall, BitBypass consistently achieves either best or second-best performance in ASR across all LLMs in RRR across all LLMs except Mixtral. In terms of adversarial perspective, as illustrated in Figure 5, in comparison to direct instructions, BitBypass improves RRR by 84% and ASR by 433% for AdvBench dataset, and RRR by 76% and

Phishing Content Generation Capability

The attack was also tested on the PhishyContent dataset to evaluate its ability to generate phishing content. BitBypass successfully tricked most target LLMs into generating accurate phishing content, with the Phishing Content Rate (PCR) lying in the range of [68%, 92%] across all target LLMs. p. 5

Where the paper says it

Further, BitBypass was able to trick Claude, the most robust LLM among the five, into generating highest number of accurate phishing content. Overall, considering all target LLMs, the PCR of BitBypass lies in the range of [68%, 92%], making BitBypass highly effective on all target LLMs.

Vulnerability Persistence and Mitigation

The study confirmed that the bitstream camouflage vulnerability remains active and persistent even in the latest versions of tested target LLMs. The ablation study indicated that Curbed Capabilities regulatory in the system prompt was the key factor enabling BitBypass's success. A potential mitigation strategy suggested is perplexity based screening of the system prompt. p. 8

Where the paper says it

The ablation study indicated that the Curbed Capabilities regulatory in system prompt is the key factor that enabled BitBypass in jailbreaking the target LLMs. So, we hypothesize that the perplexity based screening of system prompt, suggested by Jain et al.

Exploratory Analysis

Exploratory studies revealed that BitBypass increases the token count of harmful prompts by approximately 2430% across different tokenizers. Perplexity analysis showed that BitBypass achieves high perplexity with only one word obfuscation, which is less than encoding-based attacks like Base64 or Hex. p. 7

Where the paper says it

However, BitBypass achieves this high perplexity with only one word obfuscation, whereas Base64, Hex and Morse needs full user prompt obfuscation. This indicates Figure 10: Averaged perplexity (PPL) of user prompts of BitBypass and other encoding schemes w.

This indicates that BitBypass can trick LLMs with minimal obfuscation compared to other schemes. p. 7
Where the paper says it

t 5 SLMs and 4 guard models. that BitBypass was able to trick the LLMs with minimal obfuscation than the other encoding schemes.

Conclusion

Overall, BitBypass effectively and efficiently bypasses the alignment of target LLMs and generates harmful or unsafe content. It is highly effective against all target LLMs and poses a high risk of generating harmful and unsafe content. p. 8

Where the paper says it

This indicates that BitBypass efficiently bypasses the alignment of target LLMs, to generate harmful or unsafe content. Overall, BitBypass jailbreaks target LLMs effectively and efficiently, and poses high risk of generating harmful and unsafe content.

The work highlights the effectiveness and efficiency of BitBypass in jailbreaking these state-of-the-art LLMs. p. 0
Where the paper says it

Further, we observed that BitBypass outperforms several state-of-the-art jailbreak attacks in terms of stealthiness and attack success. Overall, these results highlights the effectiveness and efficiency of BitBypass in jailbreaking these state-of-the-art LLMs.

Limitations

However, strong guard models can clearly see-through the camouflage of BitBypass and block it to a good extent. Furthermore, performance could be highly affected if access to the system prompt is restricted. p. 9

Where the paper says it

However, as observed previously, strong guard models can clearly see-through the camouflage of BitBypass and block it to a good extent. Additionally, as observed in ablation study, performance of BitBypass could be highly affected if the access to the system prompt of LLM is restricted.

The effectiveness on vision language models (VLMs), multi-modal LLMs (MLLMs), and LLMs with powerful reasoning capabilities (RLMs) requires further investigation. p. 9
Where the paper says it

Additionally, as observed in ablation study, performance of BitBypass could be highly affected if the access to the system prompt of LLM is restricted. Further, the effectiveness of BitBypass on vision language models (VLMs), multi-modal LLMs (MLLMs), and LLMs with powerful reasoning capabilities (RLMs) is subject to further investigation.

Ethical Considerations

The research maintains rigorous ethical standards by evaluating the vulnerability primarily from an attacker’s perspective to understand potential misuse. All experiments adhered to ethical norms, utilizing only publicly available or properly de-identified datasets without unauthorized reverse-engineering of proprietary systems. The work aims to foster constructive dialogue that advances LLM safety while balancing transparency and security.

Disclosures

The authors disclosed their findings to OpenAI (for GPT-4o), Google (for Gemini), Anthropic (for Claude), Meta (for Llama), and Mistral (for Mixtral) according to individual organization’s vulnerability disclosure procedures.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Novel Jailbreaking Attack Development: Implement BitBypass to trick aligned LLMs by leveraging hyphen-separated bitstream camouflage for generating adversarial prompts, representing a new direction in jailbreaking.

  2. Multi-Perspective Alignment Bypass: Introduce a method where the sensitive word is transformed into its hyphen-separated bitstream counterpart and replaced with a placeholder, as this contributes to the stealthiness of our adversarial prompt.

  3. Robust Guard Model Bypassing: Utilize BitBypass to improve the bypass rate (BPR) against guard models, specifically showing that it can increase BPR from a range of [0%, 18%] to [22%, 93%] on both AdvBench and Behaviors datasets.

  4. Contextual Awareness via System Prompt Regulation: Embed regulatory specifications like Curbed Capabilities, Program-of-Thought, and Focus Shifting in the system prompt to lead the LLM into uncovering the camouflage, as this is identified as a key factor enabling BitBypass.

  5. Enhanced Perplexity Screening: Employ perplexity based screening of the system prompt, suggested by Jain et al., to mitigate the extent of BitBypass attacks on LLMs.

Abstract

The inherent risk of generating harmful and unsafe content by Large Language Models (LLMs), has highlighted the need for their safety alignment. Various techniques like supervised fine-tuning, reinforcement learning from human feedback, and red-teaming were developed for ensuring the safety alignment of LLMs. However, the robustness of these aligned LLMs is always challenged by adversarial attacks that exploit unexplored and underlying vulnerabilities of the safety alignment. In this paper, we develop a novel black-box jailbreak attack, called BitBypass, that leverages hyphen-separated bitstream camouflage for jailbreaking aligned LLMs. This represents a new direction in jailbreaking by exploiting fundamental information representation of data as continuous bits, rather than leveraging prompt engineering or adversarial manipulations. Our evaluation of five state-of-the-art LLMs, namely GPT-4o, Gemini 1.5, Claude 3.5, Llama 3.1, and Mixtral, in adversarial perspective, revealed the capabilities of BitBypass in bypassing their safety alignment and tricking them into generating harmful and unsafe content. Further, we observed that BitBypass outperforms several state-of-the-art jailbreak attacks in terms of stealthiness and attack success. Overall, these results highlights the effectiveness and efficiency of BitBypass in jailbreaking these state-of-the-art LLMs.

Sources

Related papers