AgentWorld: A Data Factory for Training LLM Agents from Multi-Turn Interactions

arXiv:2606.09138 · cs.LG, cs.CL · Submitted 2026-06-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "AgentWorld: A Data Factory for Training LLM Agents from Multi-Turn Interactions".

Jane: The gist The paper presents AgentWorld, a data factory designed to train LLM agents from multi-turn interactions.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today, "AgentWorld: A Data Factory for Training LLM Agents from Multi-Turn Interactions." Basically, the big idea is to fix a problem with training these new AI agents—they need multi-turn interactions to learn, but getting that data from different agent systems is a nightmare.

Jane: Right. It claims they're building this data factory to bridge those gaps between the actual agent runtimes and the RL training engines.

Lu: The core thesis here is reframing how we think about LLM agent training—instead of just looking at raw interaction traces, they treat that data as something structured and ready to consume for training <ref:2606.09138#pg2>.

Meng: So, what's the actual mechanism? How does this factory work in practice? I need to know how it handles the mess of different agent runtimes.

Tom: It uses a two-part system: a Gateway Server and a Data Pool <ref:2606.09138#pg3>. The Gateway Server is the entry point, capturing interactions through an LLM API proxy, and then the Data Pool organizes everything into standardized step records for curation and training data <ref:2606.09138#pg3>.

Jane: That sounds like a way to impose order on the chaos of heterogeneous agent runtimes. So, if you have a white-box agent and a black-box service, this server collects both of them and puts them into the same format?

Lalam: Exactly. The Data Pool stores step-level records that include things like "prompt IDs, response IDs, rewards and other metadata" <ref:2606.09138#pg3>. It turns those interactions into structured data assets rather than just temporary runtime logs <ref:2606.09138#pg5>.

Tom: And that structure is key because it makes the data consumable by downstream RL algorithms, not just raw text <ref:2606.09138#pg5>. They’ve made the lifecycle explicit through an interactive dashboard that shows collection, curation, optimization, and consumption all in one view <ref:2606.09138#pg4>.

Jane: That sounds like a huge win for usability if it lets you see the whole process from collecting the initial interaction all the way to when you feed it into a training system. So, what kind of curation is happening in this process?

Paper summary: Lu: They have several stages: first data collection from things like white-box rollouts or human feedback streams <ref:2606.09138#pg4>, then normalization into those step-level records in the Data Pool <ref:2606.09138#pg4>, and then curation where you check if a step is rewarded, complete, or ready for training <ref:2606.09138#pg4>.

Meng: Curation sounds like where you actually decide what’s good data versus what’s noise. What about optimization? Is there something they do to make the data better before it goes into the training pipeline?

Tom: Yes, they have a step for data optimization which employs techniques like "prefix-tree merging" to reduce redundant long-context computation by organizing related steps into a "merged step tree" <ref:2606.09138#pg4>. That’s aimed at saving computation time when you're dealing with long sequences.

Jane: That sounds smart for efficiency, especially when you're dealing with the large context windows that agents use <ref:2606.09138#pg4>. So, this isn't just about collecting data; it’s about refining it to be training-ready.

Lu: The design principles guide this whole thing. They focus on low-intrusion ingestion, which means seamless integration without needing to impose a unified execution framework on existing agents <ref:2606.09138#pg3>.

Meng: That decoupling sounds important for production environments where you don't want to rewrite everything just because you need better data pipelines. How does it handle the different training backends?

Tom: They use backend-aware serving, which means the middleware stays independent of specific trainers while exposing "trainer-compatible data interfaces through lightweight adaptation layers and standardized data abstractions" <ref:2606.09138#pg3>.

Jane: So, it’s designed to be flexible enough to work with various RL training backends without being tied to any single one. It treats agentic RL as this transformation of runtime data into training assets <ref:2606.09138#pg2>.

Lalam: And they really emphasize the decoupling aspect, separating data collection from model training to support scalable, continuously running agent workloads with diverse execution latencies <ref:2606.09138#pg3>. That makes sense for real-world deployment constraints.

Paper summary: Tom: So, the paper suggests that by making these interaction traces managed training assets instead of transient logs, we can support more scalable agent development and provide a foundation for future systems <ref:2606.09138#pg5>.

Jane: It’s shifting the focus from just running an agent to managing the data flow around it. So, what's the real implication here for developers who are building these agents?

Lu: The implication is that you don't have to build a custom interface every single time you want to train a new agent; you just plug it into this factory structure <ref:2606.09138#pg2>.

Meng: From my side, I’m worried about the complexity of maintaining this whole pipeline. What does this mean for deployment costs? Is it adding too much overhead?

Tom: The design principle of asynchronous decoupling is meant to address that by separating the collection from the training process, which supports scalable workloads with diverse execution latencies <ref:2606.09138#pg3>.

Jane: So, we're moving toward a system where we can see how raw agent runtime interactions become actual RL assets through a transparent and fully observable view <ref:2606.09138#pg4>.

Lalam: The paper concludes that this data-centric perspective supports more scalable agent development and provides a foundation for future systems that jointly improve agent execution, data quality, and RL training efficiency <ref:2606.09138#pg5>.

Tom: So, to summarize "AgentWorld: A Data Factory for Training LLM Agents from Multi-Turn Interactions," it’s a system that takes messy multi-turn interactions and turns them into structured assets using a Gateway Server and a Data Pool <ref:2606.09138#pg3>.

Jane: It claims this approach helps decouple agent runtimes from RL training backends, allowing for better scalability across different agent systems <ref:2606.09138#pg2>.

Lu: The authors are showing how you can follow the complete path from live trajectory collection to data representation, curation, optimization, and training consumption <ref:2606.09138#pg4>.

Meng: It’s a practical foundation for transforming that heterogeneous multi-turn experience into something that the AI training systems can actually use <ref:2606.09138#pg5>.

Tom: We’ll leave it there for now, but the next segment is going to look at what this means for the title and authors of this paper and its broader implications.

Conclusion: Tom: So, we're wrapping up on AgentWorld, which is basically this data factory they built to turn messy multi-turn agent interactions into training data for LLM agents <ref:2606.09138#pg1>.

Jane: The title itself suggests a big idea here, turning those complex runtimes into something structured that the training engines can actually use <ref:2606.09138#pg4>.

Lu: It's about reframing how we think about agent learning, treating those raw interactions as managed assets instead of just logs <ref:2606.09138#pg5>.

Meng: So what does that actually mean for a production team trying to scale these agents? What's the real world impact of this factory structure?

Tom: It means you don't have to build a custom pipeline every time you want to train something new; you just plug the agent into this standardized system <ref:2606.09138#pg2>.

Jane: The authors are showing how they can follow the whole path from collecting live data to optimizing it and then feeding it directly into RL training <ref:2606.09138#pg4>.

Lu: It's about decoupling the agent runtime from the specific training backend, which lets you work with different systems without rewriting everything <ref:2606.09138#pg3>.

Meng: So it tackles that headache of heterogeneous environments by providing a unified way to handle all those different sources <ref:2606.09138#pg3>.

Tom: They’re positioning this as a foundation for more scalable agent development, making interaction traces trainable assets instead of just transient stuff <ref:2606.09138#pg5>.

Jane: It’s a shift toward a data-centric view where you can see the entire lifecycle of an agent's experience in one place <ref:2606.09138#pg4>.

Lu: The implication is that this infrastructure supports improving execution, data quality, and RL training efficiency all at once <ref:2606.09138#pg5>.

Meng: It’s a practical way to make sure the data you use for training actually matches what the agent experienced in reality <ref:2606.09138#pg4>.

Tom: This approach makes it much easier to build systems that are robust enough for real-world deployment, where things aren't always perfect from the start <ref:2606.09138#pg3>.

Jane: So, if you're looking at building next-generation agent frameworks, this factory concept is a really important part of the infrastructure to consider <ref:2606.09138#pg4>.

Daoyu Wang, Mingyue Cheng, Qingchuan Li, Shuo Yu, Jie Ouyang, Qi Liu

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China

cs.LG, cs.CL

Submitted: 2026-06-08

Updated: 2026-10-14

Comments: Accepted as a demonstration paper at ICDM 2026

Code: https://github.com/AgentR1/Claw-R1

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: The gist The paper presents AgentWorld, a data factory designed to train LLM agents from multi-turn interactions.

Key concepts

Gateway Server: This component acts as the main entry point for AgentWorld. It provides a unified interface that allows the system to capture multi-turn interactions coming from various, different agent runtimes. It handles the initial ingestion of this diverse data stream into the factory.
Data Pool: The Data Pool is the core storage and organization hub. It takes raw interactions and organizes them into standardized step-level records containing crucial metadata like prompt IDs, responses, and rewards. This structure makes the interaction data ready for downstream RL algorithms to consume.
Step-Level Records: These are the standardized data units created by AgentWorld. Instead of storing raw logs, they represent individual steps in an agent's interaction process. Each record preserves the prompt, response, reward status, and trajectory relations at a granular level for precise training purposes.
Data Lifecycle Management: This refers to the complete workflow AgentWorld manages for interaction data. It explicitly tracks how raw data moves through stages: collection from heterogeneous sources, representation into structured records, curation (checking quality), optimization (merging steps), and finally consumption by the RL training backend.

Terminology

Agentic Reinforcement Learning (RL) — 1× in this paper
Core — 1× in this paper · explained in 2 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist The paper presents AgentWorld, a data factory designed to train LLM agents from multi-turn interactions.

How it works

AgentWorld functions as a data factory for training LLM agents from multi-turn interactions, addressing the challenge of managing heterogeneous interaction data in agentic RL [1]. It is designed to handle the complex data lifecycle involved in transforming runtime interactions into training assets [2].

The system is built around capturing and organizing agentic RL interaction traces as managed data assets rather than temporary runtime logs [3]. This approach treats agent interaction traces as structured, curated, and fully consumable training data [5].

Core Components

AgentWorld utilizes a system that connects heterogeneous agent runtimes with RL training backends through two core components: a Gateway Server and a Data Pool [1]. The Gateway Server acts as the ingestion entry point, providing a unified interface for collecting interaction data from diverse sources [3].

The Data Pool serves as the data management core, storing step-level records that include prompt IDs, response IDs, rewards and other metadata [3]. This pool organizes interactions into structured records that are ready for consumption by downstream RL algorithms [5].

Data Lifecycle Management

AgentWorld makes the data lifecycle of agentic RL explicit through an interactive workflow [4]. Users can observe how interaction steps are collected, curated, optimized, and ultimately consumed in one view [4].

The process involves several stages:

Data Collection: This stage captures live data from heterogeneous sources such as white-box rollouts, black-box services, and human-feedback streams [4].
Data Representation: Collected interactions are normalized into step-level records inside Data Pool, preserving prompts, responses, rewards, trajectory relations, and source metadata [4].
Data Curation: This view allows users to inspect whether individual steps are rewarded, complete, stale, correct, or ready for training, enabling the construction of training sets matching specific optimization requirements [4].
Data Optimization: The system employs techniques like prefix-tree merging to reduce redundant long-context computation by organizing related steps into a merged step tree [4].
Data Consumption: This view monitors how RL training systems interact with the Data Pool, allowing users to filter batches by reward availability, policy freshness, trajectory completeness, quality tags, or algorithm-specific requirements [4].

Design Principles

The design of AgentWorld is guided by four key principles aimed at decoupling agent runtimes from RL training backends [3]. These principles include:

Low-intrusion ingestion: This enables seamless integration with heterogeneous agent systems without imposing a unified execution framework or requiring intrusive modifications to existing agents [3].
Step-native representation: This principle ensures that the system preserves interaction structure and RL semantics at the step level while maintaining access to underlying token sequences for replay and optimization [3].
Asynchronous decoupling: This separates data collection from model training to support scalable, continuously running agent workloads with diverse execution latencies [3].
Backend-aware serving: This keeps the middleware independent of specific trainers while exposing trainer-compatible data interfaces through lightweight adaptation layers and standardized data abstractions [3].

AgentWorld treats agentic RL as a process of transforming runtime interaction data into training-consumable assets, positioning itself as a step-level data middleware between the agent runtime and the RL training backend [2]. This framework supports more scalable agent development by making interaction traces managed training assets rather than transient logs [5].

The paper concludes that this data-centric perspective can support more scalable agent development and provide a foundation for future systems that jointly improve agent execution, data quality, and RL training efficiency [5]. The system provides a transparent and fully observable view of how raw agentruntime interactions become trainable RL assets [4]. This shift in infrastructure is highlighted by the ability to follow the complete path from live trajectory collection to data representation, curation, optimization, and training consumption [4].

--- Page 1 ---

Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning

Daoyu Wang

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China

Hefei, China

daoyu.wang@mail.ustc.edu.cn

--- Page 2 ---

Conference’17, July 2017, Washington, DC, USA Daoyu Wang, Mingyue Cheng, Qingchuan Li, Shuo Yu, Jie Ouyang, and Qi Liu

Table 1: Representative directions related to agentic RL.

--- Page 3 ---

Claw-R1 is a step-level data middleware system for agentic RL. It sits between heterogeneous agent runtimes and RL training backends, which manage data lifestyle from producing to training consuming.

--- Page 4 ---

Conference’17, July 2017, Washington, DC, USA Daoyu Wang, Mingyue Cheng, Qingchuan Li, Shuo Yu, Jie Ouyang, and Qi Liu

Figure 3: Claw-R1 dashboard overview. The interface exposes collection, representation, curation, optimization and consumption in one lifecycle view.

--- Page 5 ---

Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning Conference’17, July 2017, Washington, DC, USA

References

--- Page 6 ---

[1] Anthropic. 2025. Claude Code: Build, debug, and ship from your terminal. https://claude.ai/product/claude-code/.

--- Page 7 ---

[2] Shihao Cai, Runnan Fang, Jialong Wu, Baixuan Li, Xinyu Wang, Yong Jiang, Liangcai Su, Liwen Zhang, Wenbiao Yin, Zhen Zhang et al. 2025. AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning. arXiv preprint arXiv:2512.22857 (2025).

--- Page 8 ---

[3] Mingyue Cheng, Jie Ouyang, Shuo Yu, Ruiran Yan, Yucong Luo, Zirui Liu, Daoyu Wang et al. 2025. Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning. arXiv preprint arXiv:2511.14460 (2025).

--- Page 9 ---

[4] Mingyue Cheng, Daoyu Wang, Shuo Yu, Qingchuan Li, Jie Ouyang et al. 2026. A Comprehensive Survey of the LLM-Based Agent: The Contextual Cognition Perspective. (2026).

--- Page 10 ---

[5] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. Group-in-Group Policy Optimization for LLM Agent Training. arXiv preprint arXiv:2505.10978 (2025).

--- Page 11 ---

[6] Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie et al. 2025. VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use. arXiv preprint arXiv:2509.01055 (2025).

--- Page 12 ---

[7] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon et al. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv preprint arXiv:2503.09516 (2025).

--- Page 13 ---

[8] Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye et al. 2025. WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning. arXiv preprint arXiv:2509.13305 (2025).

--- Page 14 ---

[9] Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao et al. 2025. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL. arXiv preprint arXiv:2508.13167 (2025).

--- Page 15 ---

[10] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688 (2023).

--- Page 16 ---

[14] Long Ouyang, Jeffrey Wu, Xu Jiang et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.

--- Page 17 ---

[16] John Schulman, Filip Wolski, Prafulla Dhariwal et al. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

--- Page 18 ---

[18] Guangming Sheng, Chi Zhang, Zilingfeng Ye et al. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems. doi:10.1145/3689031.

Improvements for AI systems

Decoupled Data Management for Heterogeneous Agents Claw-R1 decouples heterogeneous agent runtimes from RL training backends through a Gateway Server and a Data Pool, allowing new agents to be integrated without requiring customized interface[s]. This enables scalable deployment across diverse agent architectures.
Step-Level Data Abstraction for RL Semantics By modeling agentic RL as a step-level MDP and recording interactions in a standardized format that abstracts away runtime-specific details, the system ensures that the RL training system consumes these standardized step sequences without accessing the internal execution logic.
Automated Data Curation and Quality Control The dashboard provides tools to examine whether individual steps are rewarded, complete, stale, correct, or ready for training, allowing users to curate data by quality and readiness before it is used for optimization.
Efficient Long-Context Computation via Prefix-Tree Merging Claw-R1 exploits data structure through prefix-tree merging: as illustrated in Figure 2, steps with shared token prefixes are organized into a merged step tree, so common contexts are represented once and divergent continuations remain separate branches, which is explicitly stated to reduce redundant long-context computation.
Interactive Data Lifecycle Visualization The system exposes the full agentic RL data lifecycle as an interactive workflow, enabling users to monitor live trajectories efficiently captured by the Gateway Server, inspect step-level state-action-reward records organized in the Data Pool, curate data by quality and readiness, and configure training-ready batches.

Abstract

Training LLMs to act as capable agents requires learning from multi-turn interactions with environments. However, multi-turn interaction trajectories generated by heterogeneous agent runtimes are difficult to transform into structured, high-quality training data that training engines can readily consume. In this paper, we propose AgentWorld, a data factory that bridges heterogeneous agent runtimes and training engines through a Gateway Server and a Data Pool, reframing LLM agent training around a unified data lifecycle from interaction collection to training consumption. The Gateway Server captures multi-turn interactions through an LLM API proxy, while the Data Pool organizes them into standardized step records for curation and training. An interactive dashboard supports trajectory monitoring, step inspection, and data review. In our demonstration, users integrate ready-to-use white-box and black-box agents, trace and curate their interaction data across the pipeline, and extend the factory to new agents and datasets through lightweight configuration. Quantitative experiments across multiple datasets and RL methods demonstrate training support for different agent runtimes. AgentWorld provides a practical foundation for transforming heterogeneous multi-turn experience into training-ready data for LLM agents. Our code is available at https://github.com/AgentR1/Claw-R1 and the demonstration video can be found at link https://youtu.be/Pw47dAOw6B0.