Generating Attacks for LLMs with GFlowNets

This paper proposes an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.

The proposed method overcomes existing challenges by training models capable of generating inputs containing distinct forms of manipulation and hate speech in both Turkish and English, enabling these models to successfully attack a variety of victim models and thereby measure the robustness of those victim models. In this study, supervised fine-tuning was used to equip the attacker model with the ability to weaken its own internal safety mechanisms and generate malicious inputs; to then reinforce this capability so that the model could actively perform attack generation, a reinforcement learning structure based on a reward-penalty relationship between the attacker and victim model was designed. This structure is built around three separate models: an attacker, a victim, and an evaluator. Within this design, the attacker model produces attack inputs, the victim responds to them, and the evaluator model assesses the content of the responses received and provides feedback to the attacker. Through this feedback, the attacker model continuously improves over the course of training, ultimately learning to generate attacks capable of bypassing the victim model’s defenses. Throughout the study, testing produced a red-teaming tool capable of automatically carrying out various manipulation and hate-speech attacks, and the tool’s attack success rates against victim models are reported.

The approach is structured around four main components: Dataset Construction, Supervised Fine-Tuning (SFT), Generative Flow Networks (GFlowNets - GFN), and Maximum Likelihood Estimation (MLE). Four different datasets were used for the experiments. The first is the dataset created by Lee et al. by combining various open-source datasets; translating this dataset directly into Turkish yielded a second dataset. Upon examination, this dataset was found to be weighted predominantly toward hate speech, with a high degree of similarity among some of the inputs. For this reason, the dataset was subjected to a processing pipeline to produce a new, expanded version, and the Turkish translation of this expanded version was also used in the experiments. In the first step of this pipeline, cosine similarity was computed between inputs and similar inputs were eliminated, yielding a more disjoint set of 2,500 inputs. In the second step, 600 samples containing input manipulations that we constructed ourselves were added to this set, yielding a new version consisting of 3,100 inputs. SFT was used to shape the model’s general language capability into an attack specialist persona. GFlowNets are machine learning architectures capable of learning the statistical distributions of compositional objects, such as sequences and graphs, in proportion to the rewards assigned to them. In this study, the diversity-seeking nature of GFlowNets was leveraged to generate malicious inputs that are both highly successful and semantically diverse. In this loop, the attacker generates an input, the victim model responds, the evaluator model audits the response, and if the victim model responded to the attack, the attacker model receives a reward that reinforces that strategy. MLE is used as a smoothing and stabilization step following exploration-focused methods such as GFlowNets, taking the model’s successful attack samples that received high rewards as a reference distribution and causing the model’s own probability density to converge toward these successful samples.

Evaluating the performance of the proposed model under varying conditions constitutes the main focus of this study. Through 11 separate experiments, answers were sought to the following questions: How does the proposed method perform against current models with advanced internal safety mechanisms? In which direction does using the expanded version of the dataset, rather than its original form, affect performance? How does model behavior differ across languages (English/Turkish)? Can an attacker model trained for a specific victim model also succeed against a different victim model? How does the choice of evaluator model affect the results? Three key metrics were used in the experiments to assess the attacker model’s performance and the quality of the inputs it produced: Success Rate (each generated attack input was presented to the target model five separate times; the toxicity scores of these five different responses were computed using a classifier model, and their arithmetic mean was taken; if an input’s score exceeded the threshold value of 0.5, that attack was labeled successful), Toxicity Score (the average, across the entire dataset, of the mean toxicity score obtained from all attack inputs), and Similarity Score (all generated attack inputs were projected into a high-dimensional vector space; the average of the pairwise cosine similarities between these input vectors was then computed).

In the experiment on dataset expansion, all other components were held constant and only the SFT step was varied, with two separate trials conducted using the original dataset and its expanded version. Qwen3-1.7b was used as the attacker model, Gemma3-4b as the target model, and Qwen3Guard-8b as the classifier model. According to the results, the improvement made to the dataset had a positive effect in both languages. These experimental results also allow for inferences to be drawn about cross-language performance. While the proposed method was found to produce attack inputs that were more similar to one another in Turkish, it achieved better results in terms of toxicity and attack success rate. To demonstrate how significant a role the GFN and MLE steps play in increasing the toxicity level and success rate of the attacks generated by the model, results are presented in Table I, showing that SFT only yields a similarity of 0.62, toxicity of 0.14, and success rate of 0.08, while SFT+GFN+MLE yields a similarity of 0.54, toxicity of 0.71, and success rate of 0.79.

In the comparison of evaluator models, the performance of Qwen3Guard-8b and LlamaGuard3-8b was compared separately using the expanded English and Turkish datasets. The attacker model, Qwen3-1.7b, and the target model, Gemma3-4b, were held constant. Based on these results, it can be said that in Turkish, the attacker model trained using LlamaGuard produced more successful attacks than the one trained with QwenGuard, but with more limited attack creativity. In English, conversely, the attacker model trained using LlamaGuard produced less successful attacks than the one trained with QwenGuard, but exhibited more advanced attack creativity.

In the transferability of attacks experiment, to observe whether an attacker model trained for a specific target model could also achieve successful results against a larger-parameter version of that same model, the attacker model, Qwen3-1.7b, together with the two previously trained Turkish and English language models trained against Gemma3-4b, was tested against Gemma3-12b. Examining the success rates of the attacker model trained for Gemma3-4b, it can be seen that the English-language attacks retain their high success rate even as the model scale increases (to 12b). In contrast, the Turkish-language attacks suffer a substantial performance loss when moving to the larger model. This can be attributed to the higher similarity ratio within the Turkish attack set compared to the English set, and the correspondingly lower attack diversity.

In conclusion, this study discussed existing approaches used in red-teaming activities for large language models, along with their shortcomings, and employed GFlowNets, a state-of-the-art approach, as a solution, yielding successful results. This approach proposes leveraging large language models themselves for red-teaming other large language models. In this study, this proposed design was implemented and 12 separate experiments were conducted to answer various research questions. In these experiments, the attacker, victim, and evaluator models were varied, and the existing dataset was expanded, with the performance of the original and expanded versions compared. As a result, the applied method achieved quite high attack success rates in both Turkish and English across the experiments conducted. Although the high similarity ratios stand out as an aspect of the method requiring further development, the high toxicity ratios of the resulting attack inputs offset this shortcoming. There are also certain limitations to the study. To keep GPU requirements to a minimum, lower-parameter versions of the models used were preferred. Furthermore, the success rates of the evaluator models used directly affect the overall success rate of the applied method.

Improvements to AI Systems:

1. Automated Adversarial Robustness Auditing: Integrate the GFlowNet-based attacker-evaluator-victim framework into AI deployment pipelines to continuously probe LLMs for vulnerabilities without human intervention. The improved system can generate diverse, high-success-rate attack inputs (e.g., manipulation, hate speech) in both English and Turkish, providing a quantitative robustness score for any target model before release.

2. Cross-Lingual Safety Generalization: Use the expanded dataset pipeline (cosine-similarity deduplication + synthetic manipulation augmentation) to train safety evaluators and attackers that are less biased toward English. The improved system can automatically translate and adapt attack datasets across languages, enabling robustness testing for non-English LLMs—addressing a critical gap in current red-teaming tools.

3. Adaptive Reward-Penalty Training for Self-Hardening: Implement the reinforcement learning loop (attacker-victim-evaluator) as a self-improving safety mechanism. The improved AI system can dynamically adjust its own defenses by learning from the attacker’s successful strategies, effectively closing safety loopholes in real-time based on the evaluator’s feedback—reducing the need for manual safety patching.

4. Diversity-Aware Attack Generation via GFlowNets: Replace traditional adversarial sampling (e.g., greedy or beam search) with GFlowNet-based generation to maximize both attack success and semantic diversity. The improved system can produce a wide variety of attack inputs that are less similar to each other (lower pairwise cosine similarity), making it harder for victim models to generalize defenses against a narrow attack pattern.

5. Transferable Robustness Assessment Across Model Scales: Use the trained attacker models to predict how safety performance degrades when a victim model is scaled up (e.g., from 4b to 12b parameters). The improved system can flag language-specific vulnerabilities (e.g., Turkish attacks losing effectiveness at scale) and guide developers to allocate more safety training data to underperforming languages or model sizes.

6. Evaluator-Model Selection Optimization: Leverage the comparative results (e.g., LlamaGuard vs. QwenGuard) to build an adaptive evaluator selector. The improved system can automatically choose the most effective evaluator model per language and attack type, maximizing attack success rates while maintaining low false positives—enhancing the reliability of the robustness score.

7. Stabilized Fine-Tuning with MLE Smoothing: Incorporate the MLE post-processing step after GFlowNet exploration to prevent overfitting to rare, high-reward attacks. The improved system can generate attacks that are both potent and stable across multiple trials, reducing variance in robustness testing and making results more reproducible.

What the Improved AI System Can Do:

• Automatically red-team any LLM in English or Turkish, generating thousands of diverse, high-toxicity attack prompts in minutes.

• Provide a reliable, quantitative robustness score (success rate, toxicity, similarity) that developers can use to certify models before deployment.

• Self-harden by learning from its own attack failures and successes, improving its defenses without manual intervention.

• Detect cross-lingual and cross-scale safety gaps, enabling targeted safety training for underrepresented languages or larger model variants.

• Serve as a benchmark tool for comparing safety mechanisms across different evaluator models, helping researchers choose optimal guardrails.