Coachable Agents for Interactive Gameplay
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Coachable Agents for Interactive Gameplay".
Jane: Here's the gist of the paper. Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents that can exhibit desired styles in complex domains This framework allows an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.
Tom: Core Framework and Methodology The foundation of the framework is reinforcement learning, which is used in which software agents learn a task through trial and error.
Jane: To train coachable agents, the authors build off prior work on universal value function approximators (UVFAs).
Tom: Their approach addresses scenarios where the core objective remains fixed but multiple valid behaviors exist for accomplishing it, referred to as style-conditioned UVFA.
Title and authors: Tom: So let's talk about who wrote this. It’s Roberto Capobianco, Harm van Seijen, Nolan D. Bard, Neil Burch, Fatima Davelouis, Josh Davidson, Alisa Devlic... a pretty big team from Sony AI.
Jane: They are researchers who work on reinforcement learning and how agents interact with the world in complex settings like games or robotics >
The paper's summary: Tom: The core of what they're proposing is that you train these agents to handle a variety of different styles for a single task, and then you can apply those styles on demand when the agent is running.
Jane: They frame this differently than just teaching multiple tasks; instead, they think of it like an adverb modifying the verb—the main job stays the same, but you control how fast or quietly it does that job >
The paper's improvements: Lu: They take a concept called UVFA and adapt it for high-dimensional control domains, which is way harder than what they tested on before.
Meng: What I’m interested in is how they handle the reward function split; separating the fixed task objective from the style reward lets you balance the style objectives against the main task objective using a scale >
Conclusion: Tom: So to wrap up, this paper shows how you can train agents to perform tasks in different styles and then select that behavior at run time with flexible control >
Jane: It opens up possibilities for applications where context-dependent preferences matter, like a robot changing its cleaning style based on whether someone is sleeping or arriving >
Lu: It really pushes the UVFA concept into more challenging control domains, which means it’s not just for simple tasks anymore >
Meng: From an engineering standpoint, it's interesting that they manage to keep the training stable enough for these style variations to generalize across different situations and enemy types >
Lalam: I'm pretty sure this framework has immediate use in gaming or robotics where you need real-time behavioral adjustments based on context >
Roberto Capobianco, Harm van Seijen, Nolan D. Bard, Neil Burch, Fatima Davelouis, Josh Davidson, Alisa Devlic, Yunshu Du, Ishan Durugkar, Siddhant Gangapurwala, Daniel Hernandez G Zacharias Holland Sahil Jain Kenta Kawamoto Raksha Kumaraswamy Patrick MacAlpine Dustin R Morrill Declan Oller Francesco Riccio Akanksha Saran Kaushik Subramanian Thomas J Walsh Samuel Barrett Kizza N Frisbee Mady Govil Johannes Günther Varun R Kompella James A MacGlashan Maxwell Svetlik Michael D Thomure Jaden B Travnik Kevin Waugh Elahe Aghapour Florian Fuchs Andreanne Lemay Shruti Mishra Takuma Seno Peter Stone Michael Spranger and Peter R Wurman
Sony AI · Sony AI North America · Sony AI Tokyo, Japan
cs.AI, cs.LG
Submitted: 2026-07-01
Updated: 2026-11-05
Comments: Accepted in TMLR. Source code: https://github.com/SonyResearch/coachable_agents - Overview Video: https://drive.google.com/drive/folders/19F5uKziT_zkDurze5sy5iMFD4BHfD3lV
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents…
Key concepts
- Style-Conditioned UVFA
- This is a method where the agent learns multiple valid ways to complete a task. The reward function is split: one part ensures the main objective is met, while another part guides the agent toward specific behavioral styles. This allows for a trade-off between achieving the goal and adopting desired behaviors.
- Style Encoding
- The active style of an agent is encoded using a function f(θt). The augmented state includes both the original environment state (st) and this style encoding (f(θt)). This allows the system to explicitly track and control which behavioral style the agent is currently exhibiting.
- Scenario Training
- To speed up learning, agents are trained using scenario training data augmentation. This exposes the agent to diverse initializations and task variants. Stratified sampling ensures balanced coverage of different scenarios or contrasting style settings, helping the agent generalize better across many situations.
Terminology
Summary
Coachable agents for interactive gameplay presents a framework that combines universal value function approximators with carefully selected training scenarios and data augmentation to create agents that can exhibit desired styles in complex domains This framework allows an end user to choose the final behavior at run time, giving them flexible control over the final executed performance. p. 0 We refer
to these modifications of a core task as styles. We combine universal value function approximators
(UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to
create a framework for coaching agents that exhibit styles in complex domains.Where the paper says it
Core Framework and Methodology
The foundation of the framework is reinforcement learning, which is used in which software agents learn a task through trial and error. To train coachable agents, the authors build off prior work on universal value function approximators (UVFAs). p. 1 The standard
application of RL often leads to agents that perform their tasks well but offer little run-time flexibility
as to how those tasks are carried out. To train coachable agents we build off prior work on universal value function approximators (UVFAs)[20,
44], which extend the standard value function to condition on “goal states”. Unlike the original UVFA
formulation, our approach addresses scenarios in which the core objective remains fixed, but multiple
valid behaviors exist for accomplishing it. We refer to this variation as style-conditioned UVFA. Our work elevates the UVFA concept to high-dimensional
control domains—a substantially more challenging setting that requires further algorithmic innovations. In our version of UVFA, the reward function is split into two parts: a part that is fixed across episodes
and captures the task objective, and a part that pushes the agent towards certain behavioral styles:
r(s, a; θ) = rtask(s, a) + rstyle(s, a, θ).Where the paper says it
Where the paper says it
Where the paper says it
Training Signals and Data Augmentation
To speed up and stabilize learning for agents that can exhibit styles, scenario training data augmentation, and special-purpose replay buffers are used to help the agent learn the skills and style variations they will need in their particular domains. For Horizon Forbidden West (HFW), scenario training is used to expose the agent to diverse initializations and task variants. p. 0 We demonstrate
the framework’s application in the AAA video games Horizon Forbidden West and Gran Turismo,
and in an open-source humanoid test domain. Despite the different nature of the domains—car
racing, stylized game combat, and humanoid walking—each agent shows strong coherence to the
style requests while still satisfying the main task in its domain. sampling & encoding
sampling & encoding
a)
b)
c)
d)
e)
Enemies (19) Locations (3)
Observations
Scenarios
Style Requests
Melee (1) Weapons (6) Traps (1)
Elemental (7) Enemy Parts (3)
Styles
Figure 1: Coachable agents overview: coaching an agent that could play parts of HFW involved
creating training scenarios that rewarded different a combat styles against b 19 different enemy machines
in three different locations in the game world. The trajectories, stored in c replay buffers (including
tables for specific styles), are used to train d a policy that takes in observations of the world and style
request signals and outputs game controller actions.Where the paper says it
Where the paper says it
Style Encoding and Control
The active style is encoded via f(θt) and the augmented state s˜t is defined as (st, f(θt)). The style reward rstyle depends on θt ∈ Θ, where typical style parameters include weights to linearly combine rewards, as well as thresholds (for gating/ramping), and targets (setpoints) that penalize deviation from desired ranges. p. 2 In our version of UVFA, the reward function is split into two parts: a part that is fixed across episodes
and captures the task objective, and a part that pushes the agent towards certain behavioral styles:
r(s, a; θ) = rtask(s, a) + rstyle(s, a, θ). The term rstyle(s, a, θ) combines different style-reward functions
through a multi-dimensional style-vector θ: typical style parameters include weights to linearly combine
rewards, as well as thresholds (for gating/ramping), and targets (setpoints) that penalize deviation from
desired ranges. In contrast, we can create a style that encourages GT Sophy to drift (for a video, see
Appendix E). In Appendix C, we provide an additional example in which a style parameter is used as
a threshold to set a variable-sized penalty region around opponents which can be used to modulate GT
Sophy’s apparent aggressiveness (see Figure 6, Appendix D).Where the paper says it
Where the paper says it
Domain Applications and Results
The framework is demonstrated in three domains: the humanoid model from the DeepMind Control Suite (DMC), the racing game Gran Turismo 7 (GT7), and the open-world adventure game Horizon Forbidden West (HFW). In HFW, agents show strong coherence to the style requests while still satisfying the main task in its domain. p. 0 We demonstrate
the framework’s application in the AAA video games Horizon Forbidden West and Gran Turismo,
and in an open-source humanoid test domain. Despite the different nature of the domains—car
racing, stylized game combat, and humanoid walking—each agent shows strong coherence to the
style requests while still satisfying the main task in its domain. Table 4 in Appendix D shows which tools from the coaching toolbox we use in each of the
three test domains. Results
We first show the use of UVFA to influence aspects of GT Sophy’s driving style in a pair of experiments
to control tire wear and fuel usage (Figure 2a-d).Where the paper says it
Where the paper says it
Conclusion and Implications
Coachable agents have immediate uses in gaming, robotics, and other fields where RL is used to solve real-time control problems. The technology can have an immediate impact on the 300B video game industry by improving QA processes or enhancing in-game NPCs. p. 5 AI systems
trained in this manner have broad practical applications because, rather than learning a single behavior,
they encode a family of behaviors. The technology can have an immediate impact on the $300B video game industry by improving
QA processes or enhancing in-game NPCs. Outside of games, as more general purpose robots make their way out of labs, it is inevitable that
their human users will have varying preferences over how the machines perform their tasks. The approaches described here offer a powerful mechanism to enable users to shape a robot’s behaviors with
goal modifiers.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Coaching agents can be trained to perform a task in
a variety of different styles, and to apply those styles on-demand at execution time.
This allows end users tochoose the final behavior at run time, giving them flexible control over the final executed performance.
-
The framework utilizes style-conditioned Universal Value Function Approximators (UVFAs) where the reward function is split into a part that is fixed across episodes and one that
pushes the agent towards certain behavioral styles.
This allows for a trade-off between task objective and style objectives based on configured reward scales. -
The system can encode styles on distinct parameters, enabling
a compositional space of preferences: they can be turned on independently, combined, or varied continuously.
This means users can request complex behaviors likecombining one weapon style with three different elemental styles.
-
Training involves scenario training and data augmentation to
speed up and stabilize learning,
allowing agents to generalize well across diverse situations and enemy types. -
The system can achieve specific control over driving in Gran Turismo 7, such as controlling
tire wear on or off independently
or adjusting aggressiveness bycontrolling the longitudinal and lateral distance margins.
-
For humanoid walking, the agent can compose discrete arm poses with continuous gait control, allowing for real-time modulation of both aspects. This capability is demonstrated by using style parameters as a setpoint for gait length.
Abstract
Reinforcement learning has proven to be a valuable tool in the creation of advanced AI and robotic systems, contributing to everything from game playing (Silver et al., 2016; 2017; Wurman et al., 2022) to robotics (Levine et al., 2016; Andrychowicz et al., 2020; Luo et al., 2025; Dürr et al., 2026) to foundation models (Ouyang et al., 2022; Noukhovitch et al., 2024; Sun et al., 2024; Guo et al., 2025). Through trial and error, these AI systems typically learn a single, near-optimal behavior to solve their tasks. However, there are many use cases in which one would like to assert some level of control, after training is completed, over how the task is solved. We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains. We demonstrate the framework's application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains--car racing, stylized game combat, and humanoid walking--each agent shows strong coherence to the style requests while still satisfying the main task in its domain. Importantly, the techniques outlined in this paper allow an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.
Sources
- A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
- A Practical Guide to Multi-Objective Reinforcement Learning and Planning
- MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
- Event Tables for Efficient Experience Replay
- Goal-Conditioned Reinforcement Learning: Problems and Solutions
- Playing Atari with Deep Reinforcement Learning
- Reward-Conditioned Reinforcement Learning
- Solving Large Imperfect Information Games Using CFR+
- DeepMind Control Suite
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents