Coachable Agents for Interactive Gameplay

summary

The gist

Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents…

In short

The framework trains agents using style-conditioned Universal Value Function Approximators to exhibit diverse behaviors for a fixed task. By splitting rewards into task objectives and style objectives, users can select the final behavior at runtime. This allows for flexible control over agent performance in complex domains like gaming or robotics.

Key concepts

Style-Conditioned UVFA
This is a method where the agent learns multiple valid ways to complete a task. The reward function is split: one part ensures the main objective is met, while another part guides the agent toward specific behavioral styles. This allows for a trade-off between achieving the goal and adopting desired behaviors.
Style Encoding
The active style of an agent is encoded using a function f(θt). The augmented state includes both the original environment state (st) and this style encoding (f(θt)). This allows the system to explicitly track and control which behavioral style the agent is currently exhibiting.
Scenario Training
To speed up learning, agents are trained using scenario training data augmentation. This exposes the agent to diverse initializations and task variants. Stratified sampling ensures balanced coverage of different scenarios or contrasting style settings, helping the agent generalize better across many situations.

Terminology used across episodes

This episode discusses

The paper

Coachable Agents for Interactive Gameplay · Read on arXiv

Roberto Capobianco, Harm van Seijen, Nolan D. Bard, Neil Burch, Fatima Davelouis, Josh Davidson, Alisa Devlic, Yunshu Du, Ishan Durugkar, Siddhant Gangapurwala, Daniel Hernandez G Zacharias Holland Sahil Jain Kenta Kawamoto Raksha Kumaraswamy Patrick MacAlpine Dustin R Morrill Declan Oller Francesco Riccio Akanksha Saran Kaushik Subramanian Thomas J Walsh Samuel Barrett Kizza N Frisbee Mady Govil Johannes Günther Varun R Kompella James A MacGlashan Maxwell Svetlik Michael D Thomure Jaden B Travnik Kevin Waugh Elahe Aghapour Florian Fuchs Andreanne Lemay Shruti Mishra Takuma Seno Peter Stone Michael Spranger and Peter R Wurman

Sony AI · Sony AI North America · Sony AI Tokyo, Japan

Reinforcement learning has proven to be a valuable tool in the creation of advanced AI and robotic systems, contributing to everything from game playing (Silver et al., 2016; 2017; Wurman et al., 2022) to robotics (Levine et al., 2016; Andrychowicz et al., 2020; Luo et al., 2025; Dürr et al., 2026) to foundation models (Ouyang et al., 2022; Noukhovitch et al., 2024; Sun et al., 2024; Guo et al., 2025). Through trial and error, these AI systems typically learn a single, near-optimal behavior to solve their tasks. However, there are many use cases in which one would like to assert some level of control, after training is completed, over how the task is solved. We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains. We demonstrate the framework's application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains--car racing, stylized game combat, and humanoid walking--each agent shows strong coherence to the style requests while still satisfying the main task in its domain. Importantly, the techniques outlined in this paper allow an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Coachable Agents for Interactive Gameplay".

Jane: Here's the gist of the paper. Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents that can exhibit desired styles in complex domains This framework allows an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.

Tom: Core Framework and Methodology The foundation of the framework is reinforcement learning, which is used in which software agents learn a task through trial and error.

Jane: To train coachable agents, the authors build off prior work on universal value function approximators (UVFAs).

Tom: Their approach addresses scenarios where the core objective remains fixed but multiple valid behaviors exist for accomplishing it, referred to as style-conditioned UVFA.

Title and authors: Tom: So let's talk about who wrote this. It’s Roberto Capobianco, Harm van Seijen, Nolan D. Bard, Neil Burch, Fatima Davelouis, Josh Davidson, Alisa Devlic... a pretty big team from Sony AI.

Jane: They are researchers who work on reinforcement learning and how agents interact with the world in complex settings like games or robotics >

The paper's summary: Tom: The core of what they're proposing is that you train these agents to handle a variety of different styles for a single task, and then you can apply those styles on demand when the agent is running.

Jane: They frame this differently than just teaching multiple tasks; instead, they think of it like an adverb modifying the verb—the main job stays the same, but you control how fast or quietly it does that job >

The paper's improvements: Lu: They take a concept called UVFA and adapt it for high-dimensional control domains, which is way harder than what they tested on before.

Meng: What I’m interested in is how they handle the reward function split; separating the fixed task objective from the style reward lets you balance the style objectives against the main task objective using a scale >

Conclusion: Tom: So to wrap up, this paper shows how you can train agents to perform tasks in different styles and then select that behavior at run time with flexible control >

Jane: It opens up possibilities for applications where context-dependent preferences matter, like a robot changing its cleaning style based on whether someone is sleeping or arriving >

Lu: It really pushes the UVFA concept into more challenging control domains, which means it’s not just for simple tasks anymore >

Meng: From an engineering standpoint, it's interesting that they manage to keep the training stable enough for these style variations to generalize across different situations and enemy types >

Lalam: I'm pretty sure this framework has immediate use in gaming or robotics where you need real-time behavioral adjustments based on context >

More episodes

← Home