Coachable Agents for Interactive Gameplay

arXiv:2607.00642 · cs.AI, cs.LG · Submitted 2026-07-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Coachable Agents for Interactive Gameplay".

Jane: Here's the gist of the paper. Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents that can exhibit desired styles in complex domains This framework allows an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.

Tom: Core Framework and Methodology The foundation of the framework is reinforcement learning, which is used in which software agents learn a task through trial and error.

Jane: To train coachable agents, the authors build off prior work on universal value function approximators (UVFAs).

Tom: Their approach addresses scenarios where the core objective remains fixed but multiple valid behaviors exist for accomplishing it, referred to as style-conditioned UVFA.

Title and authors: Tom: So let's talk about who wrote this. It’s Roberto Capobianco, Harm van Seijen, Nolan D. Bard, Neil Burch, Fatima Davelouis, Josh Davidson, Alisa Devlic... a pretty big team from Sony AI.

Jane: They are researchers who work on reinforcement learning and how agents interact with the world in complex settings like games or robotics >

The paper's summary: Tom: The core of what they're proposing is that you train these agents to handle a variety of different styles for a single task, and then you can apply those styles on demand when the agent is running.

Jane: They frame this differently than just teaching multiple tasks; instead, they think of it like an adverb modifying the verb—the main job stays the same, but you control how fast or quietly it does that job >

The paper's improvements: Lu: They take a concept called UVFA and adapt it for high-dimensional control domains, which is way harder than what they tested on before.

Meng: What I’m interested in is how they handle the reward function split; separating the fixed task objective from the style reward lets you balance the style objectives against the main task objective using a scale >

Conclusion: Tom: So to wrap up, this paper shows how you can train agents to perform tasks in different styles and then select that behavior at run time with flexible control >

Jane: It opens up possibilities for applications where context-dependent preferences matter, like a robot changing its cleaning style based on whether someone is sleeping or arriving >

Lu: It really pushes the UVFA concept into more challenging control domains, which means it’s not just for simple tasks anymore >

Meng: From an engineering standpoint, it's interesting that they manage to keep the training stable enough for these style variations to generalize across different situations and enemy types >

Lalam: I'm pretty sure this framework has immediate use in gaming or robotics where you need real-time behavioral adjustments based on context >

Roberto Capobianco, Harm van Seijen, Nolan D. Bard, Neil Burch, Fatima Davelouis, Josh Davidson, Alisa Devlic, Yunshu Du, Ishan Durugkar, Siddhant Gangapurwala, Daniel Hernandez G Zacharias Holland Sahil Jain Kenta Kawamoto Raksha Kumaraswamy Patrick MacAlpine Dustin R Morrill Declan Oller Francesco Riccio Akanksha Saran Kaushik Subramanian Thomas J Walsh Samuel Barrett Kizza N Frisbee Mady Govil Johannes Günther Varun R Kompella James A MacGlashan Maxwell Svetlik Michael D Thomure Jaden B Travnik Kevin Waugh Elahe Aghapour Florian Fuchs Andreanne Lemay Shruti Mishra Takuma Seno Peter Stone Michael Spranger and Peter R Wurman

Sony AI · Sony AI North America · Sony AI Tokyo, Japan

cs.AI, cs.LG

Submitted: 2026-07-01

Updated: 2026-11-05

Comments: Accepted in TMLR. Source code: https://github.com/SonyResearch/coachable_agents - Overview Video: https://drive.google.com/drive/folders/19F5uKziT_zkDurze5sy5iMFD4BHfD3lV

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents…

Key concepts

Style-Conditioned UVFA
This is a method where the agent learns multiple valid ways to complete a task. The reward function is split: one part ensures the main objective is met, while another part guides the agent toward specific behavioral styles. This allows for a trade-off between achieving the goal and adopting desired behaviors.
Style Encoding
The active style of an agent is encoded using a function f(θt). The augmented state includes both the original environment state (st) and this style encoding (f(θt)). This allows the system to explicitly track and control which behavioral style the agent is currently exhibiting.
Scenario Training
To speed up learning, agents are trained using scenario training data augmentation. This exposes the agent to diverse initializations and task variants. Stratified sampling ensures balanced coverage of different scenarios or contrasting style settings, helping the agent generalize better across many situations.

Terminology

Summary

Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents that can exhibit desired styles in complex domains This framework allows an end user to choose the final behavior at run time, giving them flexible control over the final executed performance. p. 0

Where the paper says it

We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains.

Core Framework and Methodology

The foundation of the framework is reinforcement learning, which is used in which software agents learn a task through trial and error. To train coachable agents, the authors build off prior work on universal value function approximators (UVFAs). p. 1

Where the paper says it

The standard application of RL often leads to agents that perform their tasks well but offer little run-time flexibility as to how those tasks are carried out. To train coachable agents we build off prior work on universal value function approximators (UVFAs)[20, 44], which extend the standard value function to condition on “goal states”.

Their approach addresses scenarios where the core objective remains fixed but multiple valid behaviors exist for accomplishing it, referred to as style-conditioned UVFA. p. 1
Where the paper says it

Unlike the original UVFA formulation, our approach addresses scenarios in which the core objective remains fixed, but multiple valid behaviors exist for accomplishing it. We refer to this variation as style-conditioned UVFA.

The reward function is split into two parts: a part that is fixed across episodes and captures the task objective, and a part that pushes the agent towards certain behavioral styles. p. 2
Where the paper says it

Our work elevates the UVFA concept to high-dimensional control domains—a substantially more challenging setting that requires further algorithmic innovations. In our version of UVFA, the reward function is split into two parts: a part that is fixed across episodes and captures the task objective, and a part that pushes the agent towards certain behavioral styles: r(s, a; θ) = rtask(s, a) + rstyle(s, a, θ).

Training Signals and Data Augmentation

To speed up and stabilize learning for agents that can exhibit styles, scenario training data augmentation, and special-purpose replay buffers are used to help the agent learn the skills and style variations they will need in their particular domains. For Horizon Forbidden West (HFW), scenario training is used to expose the agent to diverse initializations and task variants. p. 0

Where the paper says it

We demonstrate the framework’s application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains—car racing, stylized game combat, and humanoid walking—each agent shows strong coherence to the style requests while still satisfying the main task in its domain.

Stratified sampling via multi-tables and event tables are also used in GT and HFW to ensure balanced coverage of scenarios or contrasting style settings. p. 6
Where the paper says it

sampling & encoding sampling & encoding a) b) c) d) e) Enemies (19) Locations (3) Observations Scenarios Style Requests Melee (1) Weapons (6) Traps (1) Elemental (7) Enemy Parts (3) Styles Figure 1: Coachable agents overview: coaching an agent that could play parts of HFW involved creating training scenarios that rewarded different a combat styles against b 19 different enemy machines in three different locations in the game world. The trajectories, stored in c replay buffers (including tables for specific styles), are used to train d a policy that takes in observations of the world and style request signals and outputs game controller actions.

Style Encoding and Control

The active style is encoded via f(θt) and the augmented state s˜t is defined as (st, f(θt)). The style reward rstyle depends on θt ∈ Θ, where typical style parameters include weights to linearly combine rewards, as well as thresholds (for gating/ramping), and targets (setpoints) that penalize deviation from desired ranges. p. 2

Where the paper says it

In our version of UVFA, the reward function is split into two parts: a part that is fixed across episodes and captures the task objective, and a part that pushes the agent towards certain behavioral styles: r(s, a; θ) = rtask(s, a) + rstyle(s, a, θ). The term rstyle(s, a, θ) combines different style-reward functions through a multi-dimensional style-vector θ: typical style parameters include weights to linearly combine rewards, as well as thresholds (for gating/ramping), and targets (setpoints) that penalize deviation from desired ranges.

The style-parameter dynamics induce the augmented transition P(st+1 st, at)κ(θt+1 Ct). p. 3
Where the paper says it

In contrast, we can create a style that encourages GT Sophy to drift (for a video, see Appendix E). In Appendix C, we provide an additional example in which a style parameter is used as a threshold to set a variable-sized penalty region around opponents which can be used to modulate GT Sophy’s apparent aggressiveness (see Figure 6, Appendix D).

Domain Applications and Results

The framework is demonstrated in three domains: the humanoid model from the DeepMind Control Suite (DMC), the racing game Gran Turismo 7 (GT7), and the open-world adventure game Horizon Forbidden West (HFW). In HFW, agents show strong coherence to the style requests while still satisfying the main task in its domain. p. 0

Where the paper says it

We demonstrate the framework’s application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains—car racing, stylized game combat, and humanoid walking—each agent shows strong coherence to the style requests while still satisfying the main task in its domain.

In GT Sophy, UVFA is used to influence aspects of driving style such as tire wear and fuel usage. p. 3
Where the paper says it

Table 4 in Appendix D shows which tools from the coaching toolbox we use in each of the three test domains. Results We first show the use of UVFA to influence aspects of GT Sophy’s driving style in a pair of experiments to control tire wear and fuel usage (Figure 2a-d).

Conclusion and Implications

Coachable agents have immediate uses in gaming, robotics, and other fields where RL is used to solve real-time control problems. The technology can have an immediate impact on the 300B video game industry by improving QA processes or enhancing in-game NPCs. p. 5

Where the paper says it

AI systems trained in this manner have broad practical applications because, rather than learning a single behavior, they encode a family of behaviors. The technology can have an immediate impact on the $300B video game industry by improving QA processes or enhancing in-game NPCs.

Outside of games, the approaches described here offer a powerful mechanism to enable users to shape a robot’s behaviors with goal modifiers. p. 5
Where the paper says it

Outside of games, as more general purpose robots make their way out of labs, it is inevitable that their human users will have varying preferences over how the machines perform their tasks. The approaches described here offer a powerful mechanism to enable users to shape a robot’s behaviors with goal modifiers.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Coaching agents can be trained to perform a task in a variety of different styles, and to apply those styles on-demand at execution time. This allows end users to choose the final behavior at run time, giving them flexible control over the final executed performance.

  2. The framework utilizes style-conditioned Universal Value Function Approximators (UVFAs) where the reward function is split into a part that is fixed across episodes and one that pushes the agent towards certain behavioral styles. This allows for a trade-off between task objective and style objectives based on configured reward scales.

  3. The system can encode styles on distinct parameters, enabling a compositional space of preferences: they can be turned on independently, combined, or varied continuously. This means users can request complex behaviors like combining one weapon style with three different elemental styles.

  4. Training involves scenario training and data augmentation to speed up and stabilize learning, allowing agents to generalize well across diverse situations and enemy types.

  5. The system can achieve specific control over driving in Gran Turismo 7, such as controlling tire wear on or off independently or adjusting aggressiveness by controlling the longitudinal and lateral distance margins.

  6. For humanoid walking, the agent can compose discrete arm poses with continuous gait control, allowing for real-time modulation of both aspects. This capability is demonstrated by using style parameters as a setpoint for gait length.

Abstract

Reinforcement learning has proven to be a valuable tool in the creation of advanced AI and robotic systems, contributing to everything from game playing (Silver et al., 2016; 2017; Wurman et al., 2022) to robotics (Levine et al., 2016; Andrychowicz et al., 2020; Luo et al., 2025; Dürr et al., 2026) to foundation models (Ouyang et al., 2022; Noukhovitch et al., 2024; Sun et al., 2024; Guo et al., 2025). Through trial and error, these AI systems typically learn a single, near-optimal behavior to solve their tasks. However, there are many use cases in which one would like to assert some level of control, after training is completed, over how the task is solved. We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains. We demonstrate the framework's application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains--car racing, stylized game combat, and humanoid walking--each agent shows strong coherence to the style requests while still satisfying the main task in its domain. Importantly, the techniques outlined in this paper allow an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.

Sources

Related papers