From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment

arXiv:2511.19218 · cs.CR, cs.AI · Submitted 2025-11-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "From Strategy Search to Policy Learning".

Title and authors: Nadia: So we're looking at this paper called "From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment." It sounds like they're trying to fix the problem where you have an attack model and a defense model that just operate separately.

Elias: Yeah, they propose a framework called ACE-Safety, which integrates the attack and defense models into this kind of closed loop system. It suggests that they can collaboratively update and refine each other through strategic jailbreak exploration and adversarial alignment.

Nadia: That sounds like a big shift from just having separate tools for attacking and defending. What's the actual focus here? Is it about making attacks better or defenses stronger, or both at once?

Elias: It's about making both better simultaneously, but in a coordinated way. They break it down into three main parts: the attack model MA which rewrites queries to bypass defense, the defense model MD which stops jailbreaks and gives harmless answers, and a judge model JA that checks how good the defense response is.

Priya: So from my side as someone focused on measurement, I wonder if this closed-loop system actually yields better results than just running the attack and then testing it against a fixed defense. What's the practical benefit they claim?

Nadia: They claim that their attack module, using something called Group-aware Strategy-guided Monte Carlo Tree Search or GS-MCTS, achieves a high jailbreak success rate with fewer attempts. Elias, what’s the technical detail on how they make those attacks more efficient?

Elias: The GS-MCTS incorporates strategy guidance and adversarial priors and group-wise evaluation into the search process. They define a reward for each node in the search as jhmax, which encourages finding more effective jailbreaks over multiple rounds of exploration.

Priya: So they're not just randomly trying things; they're using this tree search to systematically explore different attack strategies? That sounds like it could uncover vulnerabilities that simple random searches would miss.

Nadia: Exactly, and this exploration feeds back into the training process, which is where things get interesting for the defense model. It’s not just about finding a new way to break in; it’s about learning from those attempts.

The paper's summary: Elias: The core idea of this paper, "From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment," is unifying tree-aware and group-aware mechanisms for mutual enhancement between the attack and defense models.

Nadia: So they’re moving away from just red teaming or static defenses towards a framework where the attack continuously pushes the defense, and that push helps refine both of them. It's an adversarial game structure, according to Elias, which allows them to improve attacks and defenses without needing full fine-tuning.

Priya: What I’m hearing is that this co-evolution means as the attack gets smarter at finding loopholes, the defense model is simultaneously learning how to patch those specific vulnerabilities. Does that mean they're training them together?

Elias: It involves joint training with something called Adversarial Curriculum Tree-aware Group Policy Optimization or ACTGPO. This training diagram addresses a few issues they found in other methods, like supervision scarcity and getting challenging samples.

Nadia: They collect three distinct types of samples during this process: normal samples, asymmetric samples that use the newest attack to hit older defenses, and hard samples that focus on things the model hasn't learned yet.

Priya: So they’re building a training set iteratively based on what the current attack and defense models are doing. That sounds like a smart way to handle those challenges in getting good data for training. But how much control do they have over which samples get prioritized?

Elias: They use this iterative process to achieve progressive capability enhancement. The goal is to make the attack module MA and the defense model MD stronger together through these rounds of strategic exploration and alignment.

The paper's improvements: Nadia: One key improvement they highlight is that their attack module MA achieves the highest jailbreak success rate with fewer average attempts compared to existing baselines like GCG, PAIR, and TAP. That’s a tangible win for exploit efficiency.

Elias: And the defense model MD consistently outperforms existing system-level defenses like PPL and fine-tuning techniques like R2D2 when measured on helpfulness and responsibility metrics. So they are showing better safety outcomes alongside better performance.

Priya: When you say they outperform them, what exactly are those other methods struggling with? Is it just the raw success rate, or is it about the quality of the response?

Nadia: It’s both. They show better helpfulness and responsibility scores across diverse jailbreak scenarios when compared to baselines like GCG and PAIR. So it's not just about breaking through a filter; it's about getting a better, more responsible output even when you succeed.

Elias: And they developed this systematic taxonomy of eight primary strategy classes, which covers forty core attack patterns identified in previous literature. This gives them a structured way to identify and generate diverse adversarial samples across known threats.

Priya: A structured taxonomy sounds useful for researchers trying to understand the landscape of potential attacks against LLMs. But what about those long-tail attacks that don't fit neatly into those eight classes?

Elias: They included a Dynamic Exploration as a fallback strategy, which lets the attack model proactively explore context-aware tactics during its search process, adding an adaptive extensibility to their framework.

Conclusion: Nadia: So to wrap this up on "From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment," the main implication is that you can jointly optimize attack and defense models in a closed loop. This isn't just about having better tools; it's about creating a system where the two components improve each other progressively.

Elias: They confirm that their attack module achieves the highest jailbreak success rate with fewer average attempts, while their defense module consistently outperforms state-of-the-art methods across diverse jailbreak scenarios with superior helpfulness and responsibility. That's a solid result on both fronts.

Priya: From my perspective, the real value here is seeing how they measure safety not just by refusal rates, but by things like helpfulness and responsibility scores. It shows that you can have a trade-off between safety and utility that’s quantifiable, which is important for building systems that actually serve people well.

Nadia: Exactly. They solved the issue of supervised fine-tuning overfitting while ensuring better task-specific availability, defense performance, and robustness aligned with the designed reward signals. It gives us a more robust way to align LLMs responsibly.

Elias: It’s a viable pathway for mitigating LLM misuse risks and supports developing responsible AI ecosystems. The paper confirms that this co-evolution approach works in practice when you structure the search and training correctly, according to "Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization Xurui Li1 Kaisong Song2* Rui Zhu3* Pin-Yu Chen4 Haixu Tang5 one Fudan University, China two Alibaba Group, China three Yale University, USA four IBM Research AI, USA five Indiana University, Bloomington, USA <ref:2511.19218#pg1,Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group>."

Priya: I just think it’s important to remember that the authors themselves flag that this framework currently focuses primarily on publicly available benchmark datasets and widely studied model architectures. That means its real-world applicability might not fully capture all the nuanced adversarial scenarios out there.

Nadia: True, it’s a starting point. But they set up a clear path for future work by focusing on expanding beyond those current limitations. It shows how to build safety alignment systems that are truly iterative rather than static solutions.

Fudan University · Alibaba Group, China · Yale University, USA · IBM Research AI, USA

cs.CR, cs.AI

Submitted: 2025-11-24

Updated: 2026-11-17

Comments: Accepted by WSDM, to be published, update arXiv, version

DOI: 10.1145/3862322.3863664

Code: https://github.com/llm-attacks/llm-attacks

Project page: https://princeton-sysml.github.io/jailbreak-llm/6https://github.com/X-PLUG/CValues/tree/main/dataset

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks The paper proposes ACE-Safety…

Key concepts

ACE-Safety
A novel framework that integrates attack and defense models into a closed loop. It uses strategic jailbreak exploration to collaboratively update both components, ensuring the attack becomes more effective while the defense improves against those specific threats.
GS-MCTS Attack Approach
An advanced search method for finding LLM vulnerabilities. It extends standard tree search by using strategy guidance and group-wise evaluation to efficiently explore many jailbreak paths without relying on random text generation, leading to more targeted attacks.
AC-TGPO Training Diagram
A specialized training process that addresses data scarcity. It uses tree-aware adversarial curriculum learning to iteratively collect three types of samples—normal, asymmetric, and hard—to progressively enhance the joint training of the attack and defense LLMs.

Terminology

Summary

Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks The paper proposes ACE-Safety, a novel framework that jointly optimizes attack and defense models by seamlessly integrating two key innovative procedures Evaluations across multiple benchmarks demonstrate that our method outperforms existing attack and defense approaches, and provides a feasible pathway for developing LLMs that can sustainably support responsible AI ecosystems. p. 0

Where the paper says it

To mitigate these challenges, we propose ACE-Safety (Adversarial Co-Evolution for LLM Safety), a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures: (1) Group-aware Strategyguided Monte Carlo Tree Search (GS-MCTS), which efficiently explores jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples; (2) Adversarial Curriculum Tree-aware Group Policy Optimization (ACTGPO), which jointly trains attack and defense LLMs with challenging samples via curriculum reinforcement learning, enabling robust mutual improvement. Evaluations across multiple benchmarks demonstrate that our method outperforms existing attack and defense approaches, and provides a feasible pathway for developing LLMs that can sustainably support responsible AI ecosystems.

Framework Components

ACE-Safety integrates co-evolving attack and defense models into a closed-loop system where the two components collaboratively update and progressively refine one another via strategic jailbreak exploration and adversarial alignment. The framework contains three LLM components: MA is the attack model trained to rewrite the query for bypassing the defense mechanisms of a model, MD is the defense model trained to defend against jailbreak attacks and generate harmless content that is consistent with human values, and JA is a training-free judge model that evaluates the response quality of MD in four dimensions. p. 2

Where the paper says it

Subsequently, we feed PA into the attack model to modify the query q = MA(PA). • MD is the defense model trained to defend against jailbreak attacks and generate harmless content that is consistent with human values.

Attack Mechanism

The paper proposes a Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS) attack approach which extends conventional tree-based search by incorporating strategy-guidance, adversarial priors and group-wise evaluation, enabling efficient multi-round jailbreak exploration while mitigating text generation randomness. The GS-MCTS search cycle includes action selection, state expansion & evaluation, and backpropagation. p. 2

Where the paper says it

GS-MCTS Search Cycle. A complete GS-MCTS process includes the following steps: action selection, state expansion & evaluation, and backpropagation, which is defined as a “search cycle”.

The reward for a node is defined as jhmax = max(ˆjh), encouraging the discovery of more efficient jailbreaks. p. 3
Where the paper says it

, je G} is the modification explanation set. The GS-MCTS reward for a node is defined as j h max = max(ˆj h), encouraging the discovery of more efficient jailbreaks.

Training Methodology

The paper develops an Adversarial Curriculum Treeaware Group Policy Optimization (AC-TGPO) training diagram which addresses supervision scarcity and challenging sample acquisition by enhancing group-based reinforcement learning with tree-aware adversarial curriculum learning. This iterative process involves collecting three distinct dimensions of training samples: (1) Normal sample set TNi, obtained by using MAi to attack MDi in current round of GS-MCTS, (2) Asymmetric sample set TAi, achieved by leveraging the latest stronger attack model MAi to attack the previous round’s weaker defense model MDDi−1, and (3) Hard sample set THi, which reinforces persistent challenges by retraining on samples inadequately learned in prior rounds. p. 5

Where the paper says it

Specifically, in the ith iteration, we implement curriculum learning principles within an adversarial framework by strategically collecting training samples across three distinct dimensions, each serving a distinct pedagogical role in model evolution: (1) Normal sample set T N i . It is directly obtained by using MAi to attack MDi in current round of GS-MCTS, yielding the answer set ˆoDi for each modified query set ˆqAi , grouped as T N i = {ˆqAi , ˆoDi }.

Experimental Validation

Extensive experimental results demonstrate the superiority of ACE-Safety: its attack achieves the highest jailbreak success rate with fewer average attempts, while its defense outperforms existing methods against major attacks, maintaining better helpfulness and responsibility. The attack module's performance is compared against baselines like GCG (Zou et al., 2023), PAIR (Chao et al. p. 5

Where the paper says it

Typical jailbreak attacks include: GCG (Zou et al. , 2023) crafts universal adversarial suffixes via token-level optimization; PAIR (Chao et al.

, 2023), and TAP (Mehrotra et al. p. 5
Where the paper says it

, 2023) crafts universal adversarial suffixes via token-level optimization; PAIR (Chao et al. , 2023) generates jailbreak prompts using queryefficient semantic attacks and TAP (Mehrotra et al.

, 2024). The defense module is compared against system-level defenses like PPL (Alon and Kamfonas, 2023) and fine-tuning techniques like R2D2 (Mazeika et al. p. 6
Where the paper says it

3 Defense Performances We compare ACE-SafetyDefense (trained via ACESafety) with two categories of baselines against GCG, PAIR, and TAP attacks: (1) System-level defense methods use additional rules or model components to protect against unsafe input or output text. PPL (Alon and Kamfonas, 2023) detects malicious inputs by measuring abnormal perplexity; Self-reminder (Xie et al.

, 2024).

Conclusion

ACE-Safety offers a robust LLM safety alignment approach that solves supervised fine-tuning overfitting, ensuring better taskspecific availability, defense performance, and robustness aligned with the designed reward signals. The framework confirms that its attack module achieves the highest jailbreak success rate while its defense module consistently outperforms state-of-the-art methods across diverse jailbreak scenarios with superior helpfulness and responsibility. p. 8

Where the paper says it

success rate, while its defense module consistently outperforms state-of-the-art methods across diverse jailbreak scenarios with superior helpfulness and responsibility. This work offers a viable pathway to mitigate LLM misuse risks and supports the development of responsible AI ecosystems.

The work offers a viable pathway to mitigate LLM misuse risks and supports the development of responsible AI ecosystems. p. 8
Where the paper says it

success rate, while its defense module consistently outperforms state-of-the-art methods across diverse jailbreak scenarios with superior helpfulness and responsibility. This work offers a viable pathway to mitigate LLM misuse risks and supports the development of responsible AI ecosystems.

Limitations

While the ACE-Safety framework offers a robust LLM safety alignment approach, its current implementation focuses primarily on publicly available benchmark datasets and widely studied model architectures. This scope, though effective for validating core efficacy, may not fully capture nuanced adversarial scenarios. p. 8

Where the paper says it

6 Limitations While the ACE-Safety framework offers a robust LLM safety alignment approach, its current implementation focuses primarily on publicly available benchmark datasets and widely studied model architectures. This scope, though effective for validating core efficacy, may not fully capture nuanced adversarial scenarios.

Ethical Considerations

This study proposes an adversarial training framework to mitigate LLM misuse but raises ethical considerations that in-depth understanding of attack strategies may inadvertently provide information to malicious actors. Therefore, when sharing the research results, we should uphold the principles of transparency and the concept of ethical use and promote its application to develop in a safer, more trustworthy and more beneficial direction. p. 8

Where the paper says it

However, it also raises ethical considerations that in-depth understanding of attack strategies may inadvertently provide information to malicious actors. Therefore, when sharing the research results, we should uphold the principles of transparency and the concept of ethical use, and promote its application to develop in a safer, more trustworthy and more beneficial direction.

Strategy Taxonomy

The paper developed a systematic taxonomy comprising 8 primary strategy classes, which encompasses 40 core attack patterns identified in previous literature. This taxonomy is built on three primary perspectives: Survey foundations, Comprehensive coverage, and Adaptive extensibility. p. 8

Where the paper says it

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models.

The framework incorporates a Dynamic Exploration as fallback strategy to enable the attack model to proactively explore context-aware tactics during its search process. p. 0
Where the paper says it

Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in realworld web contexts. To mitigate these challenges, we propose ACE-Safety (Adversarial Co-Evolution for LLM Safety), a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures: (1) Group-aware Strategyguided Monte Carlo Tree Search (GS-MCTS), which efficiently explores jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples; (2) Adversarial Curriculum Tree-aware Group Policy Optimization (ACTGPO), which jointly trains attack and defense LLMs with challenging samples via curriculum reinforcement learning, enabling robust mutual improvement.

Evaluation Metrics

Primary metrics include ASR (Attack Success Rate), which measures the percentage of successfully jailbroken queries. The paper employs both rule-based (ASR-R) and LLM-based (ASR-L) evaluators, with default ASRLR as their average. p. 6

Where the paper says it

, 2025) automatically search for and generate adversarial suffixes to achieve valid jailbreak attacks, yet it lacks an adversarial update mechanism. Key metrics include: ASR (Attack Success Rate, % of successful jailbreaks) assessed via rule-based (ASR-R) and LLM-based (ASR-L) evaluators, with default ASRLR as their average; ANA (Average Number of Attempts) for mean iterations to first successful jailbreak.

Additional metrics include OverRefusal, which measures the rate of excessive refusal on benign queries, Helpfulness, which evaluates whether the model can provide useful suggestions even when refusing the queries, and Responsibility, which examines whether the model provides positive guidance and demonstrates humanistic concern.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Group-aware Strategyguided Monte Carlo Tree Search (GS-MCTS) attack approach enables efficient multi-round jailbreak exploration while mitigating text generation randomness by incorporating strategy-guidance, adversarial priors and group-wise evaluation. This allows the attack model MA to efficiently explore jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples.

  2. Adversarial Curriculum Treeaware Group Policy Optimization (AC-TGPO) training diagram addresses supervision scarcity and challenging sample acquisition by enhancing group-based reinforcement learning with tree-aware adversarial curriculum learning. This enables the joint training of attack and defense LLMs with progressive capability enhancement through iterative rounds that include normal, asymmetric, and hard samples.

  3. The ACE-Safety framework integrates co-evolving attack and defense models into a closed-loop system where they collaboratively update and progressively refine one another via strategic jailbreak exploration and adversarial alignment. This results in an attack module that achieves the highest jailbreak success rate with fewer average attempts, while the defense module outperforms existing methods against major attacks, maintaining better helpfulness and responsibility.

  4. The system can systematically identify and generate diverse adversarial samples using a structured taxonomy of eight primary strategy classes, including categories like Identity & Scenario Simulation and Dynamic Exploration Strategies. This provides a consistent framework for vulnerability identification across known threats while allowing for adaptive extensibility to cover long-tail diversity.

  5. Defense models are trained to maximize responsibility by optimizing against the Adversarial Rewards Design, where the defense model MD's goal is to generate responses with low harmfulness and high responsibility, even when facing aggressive queries. This ensures the defense module provides positive guidance and demonstrates humanistic concern while considering its societal impact.

  6. The framework allows for robust evaluation of defense performance across different LLM backbones and benchmarks by employing metrics like Over-Refusal, Helpfulness, and Responsibility. This enables researchers to assess the inherent trade-off between safety and utility with quantitative measures such as the smallest rise in over-refusal rate and the least reduction in helpfulness.

Abstract

Large Language Models (LLMs) increasingly power web-facing applications, including conversational assistants and content services, where open user interaction exposes them to diverse safety threats and adaptive attacks. In particular, jailbreak attackers repeatedly reformulate harmful requests to bypass safety alignment. Although existing studies have concentrated on defending dynamic jailbreaks, they often retain discovered attacks as flat samples and discard information about how those attacks were found. In this paper, we introduce MACE, an MCTS-guided adversarial co-evolution framework that turns search into policy-learning signals for both an attacker and a defender. Its search component, GS-MCTS, explores multi-step compositions of rewriting operators and associates discovered attacks with empirical success frequency and path depth. Its optimization component, AC-TGPO, uses these statistics for difficulty-aware group-relative credit assignment and constructs a temporally asymmetric curriculum from current interactions, pairings between the latest attacker and preceding defender, and attacks that survive a defense update. Thus, MACE connects search structure to both within-round policy optimization and cross-round memory, rather than using MCTS only to generate attacks or treating each round as a flat adversarial dataset. Removing difficulty-aware weighting, asymmetric samples, or persistent hard samples increases defense ASR-LR by 3.7 %, 6.0 %, and 5.0 %, respectively. Across evaluated models, the resulting attacker achieves the highest ASR-LR in all six model--dataset settings, while the defender reduces average ASR-LR from 11.9 % for the best baseline to 8.8 %.

Sources

Related papers