Learning High-Risk High-Precision Motion Control
cs.LG, cs.RO
Submitted: 2026-09-28
Updated: 2026-09-30
Comments: Project webpage: https://namheegordonkim.github.io/scoot-mig2022/
Journal ref: MIG '22: Proceedings of the 15th ACM SIGGRAPH Conference on Motion, Interaction and Games, Article 9, pp. 1-10 (2022)
Code: https://github.com/oogle/brax
Project page: https://namheegordonkim.github.io/scoot-mig2022
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Deep reinforcement learning (DRL) algorithms for movement control are typically evaluated and benchmarked on sequential decision tasks where imprecise actions may be corrected with later actions,
Terminology
Abstract
Deep reinforcement learning (DRL) algorithms for movement control are typically evaluated and benchmarked on sequential decision tasks where imprecise actions may be corrected with later actions, thus allowing high returns with noisy actions. In contrast, we focus on an under-researched class of high-risk, high-precision motion control problems where actions carry irreversible outcomes, driving sharp peaks and ridges to plague the state-action reward landscape. Using computational pool as a representative example of such problems, we propose and evaluate State-Conditioned Shooting (SCOOT), a novel DRL algorithm that builds on advantage-weighted regression (AWR) with three key modifications: 1) Performing policy optimization only using elite samples, allowing the policy to better latch on to the rare high-reward action samples; 2) Utilizing a mixture-of-experts (MoE) policy, to allow switching between reward landscape modes depending on the state; 3) Adding a distance regularization term and a learning curriculum to encourage exploring diverse strategies before adapting to the most advantageous samples. We showcase our features' performance in learning physically-based billiard shots demonstrating high action precision and discovering multiple shot strategies for a given ball configuration.
Sources
- OpenAI Gym
- Learning Visual Predictive Models of Physics for Playing Billiards
- Learning to Reach Goals via Iterated Supervised Learning
- Soft Actor-Critic Algorithms and Applications
- DiffTaichi: Differentiable Programming for Physical Simulation
- Reward-Conditioned Policies
- A Survey on Reinforcement Learning Methods in Character Animation
- On the Variance of the Adaptive Learning Rate and Beyond
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Sample-efficient Cross-Entropy Method for Real-time Planning
- Proximal Policy Optimization Algorithms
- DeepMind Control Suite
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks