Cross-Domain Hybrid OPD for Generalizable Search Agents
Hongzhan Chen, Xiaoyu Liu, Dengming Zhang, Minzhou Huang, Dongliang Xu, Jingcheng Xie, Dongxiang Fang, Bowen Qin, Minsheng Hao, Yaozong Shen, Xiaojun Quan, Mona Zhou, Haosheng Zou, Jeff Chen
cs.CL
Submitted: 2026-08-03
Code: https://github.com/Tencent-Hunyuan/Hy3-preview
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent advances in Reinforcement Learning (RL) have substantially improved the capabilities of autonomous search agents, enabling sophisticated planning, and iterative retrieval over dynamic
Terminology
Abstract
Recent advances in Reinforcement Learning (RL) have substantially improved the capabilities of autonomous search agents, enabling sophisticated planning, and iterative retrieval over dynamic information sources. However, optimizing language models for specialized search behaviors often incurs an alignment tax, where gains in search performance come at the expense of general-purpose capabilities, limiting their effectiveness as universal assistants. In this technical report, we present the training framework behind the Yuanbao search agent, designed to achieve search specialization without sacrificing general intelligence. Built upon the Hunyuan3 architecture, our framework combines agentic reinforcement learning for autonomous search with a cross-domain expert On-Policy Distillation (OPD) pipeline. Experts specializing in complementary general-purpose domains are distilled into the search-specialized student, restoring and further enhancing its broad capabilities. Rather than treating specialization and general capability as competing objectives, our hybrid training strategy jointly optimizes both, effectively mitigating the alignment tax. Extensive experiments demonstrate that the resulting model achieves competitive search performance while consistently improving its general-purpose capabilities, providing a favorable balance between specialized execution and broad generalization in real-world search scenarios.
Sources
- AMO-Bench: Large Language Models Still Struggle in High School Math Competitions
- AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
- Training Verifiers to Solve Math Word Problems
- Measuring Coding Challenge Competence With APPS
- Measuring Mathematical Problem Solving With the MATH Dataset
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Tasks
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
- Qwen3 Technical Report
- GLM-5: from Vibe Coding to Agentic Engineering
- SUPERChem: A Multimodal Reasoning Benchmark in Chemistry
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering