PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search

arXiv:2608.15146 · cs.LG · Submitted 2026-08-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search".

Jane: The gist: PureTD demonstrates that pure self-play reinforcement learning suffices to train models that reach near-state-of-the-art playing strength for backgammon money games, even in a searchless setting.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Looking at the title PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search and the authors Alexander Strehl, it seems like this work is about showing that pure TD learning, when applied to this specific problem, can achieve very strong results without needing the traditional search component.

Jane: It suggests that we don't necessarily need to build massive explicit search trees on top of a neural network to get high performance in games where the state-value relationship is complex enough for reinforcement learning to model directly.

Lu: The main implication is that if this pure self-play RL approach works across different game types, it changes how we think about building evaluators for many kinds of complex decision-making systems.

Meng: For deployment, the fact that they trained their networks in batches instead of online suggests that while the training itself is complex, deploying these models might be more straightforward than maintaining a constantly learning online system.

Lalam: Ultimately, PureTD shows that you can train an AI to master complex decision-making like backgammon money games just by letting it play against itself and using those TD updates to guide the learning process.

Conclusion: Tom: So, PureTD is this new way of looking at Tesauro's TD-Gammon, but without any search step involved in real time for backgammon money games.

Jane: It’s about using pure reinforcement learning from scratch to learn how to play these games without needing that traditional evaluation engine running constantly during the game itself.

Tom: Exactly. The authors are showing that you can train a model just by letting it self-play, and it ends up reaching near state-of-the-art playing strength, even in this searchless setup.

Jane: That’s the core idea. It suggests that for these money games, the learning process itself provides enough signal to learn good strategy without needing an external calculator to guide every single move.

Lu: What's interesting is they treat cube action—the doubling cube—as just another action in the agent's decision space, so it’s learned end-to-end through that same RL process.

Meng: I’m curious about the training setup though. They didn't use a massive online learning system; they used batch training with stochastic gradient descent updates from self-play games.

Tom: That batch approach is important for me because it makes sense for deployment; you train the network offline, and then you can run inference quickly on that learned model without needing a huge computational overhead during actual play.

Jane: It really changes how we view training these things; instead of designing a complex search algorithm first, they design the learning environment so the AI discovers the best moves through simple repetition.

Lu: The paper also shows that their cubeful model is faster at evaluating than existing engines like GNU Backgammon and Open Sage, even when those engines are allowed just one look-ahead step.

Tom: A speed advantage of two point five times versus a one-ply search engine is significant because it means the AI can make decisions way quicker in a real game scenario.

Jane: So, it boils down to this: pure self-play reinforcement learning is enough to train models that are substantially stronger than what we currently get from traditional search-based evaluation methods.

Lu: It confirms the idea that for these types of complex decision tasks, the structure of the reward signal and the learning algorithm can handle a lot of the complexity on its own.

Meng: I still wonder how robust this is against novel game situations or if it holds up when we try to scale this training process to truly massive datasets without getting stuck in local optima.

cs.LG

Submitted: 2026-08-15

Updated: 2026-10-15

Comments: 28 pages, 23 tables, 5 figures

Code: https://github.com/alexstrehl/backgammon-ai-engine

Project page: https://lassehjorthmadsen.github.io/bganalyses/analyses/janowski.html

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist: PureTD demonstrates that pure self-play reinforcement learning suffices to train models that reach near-state-of-the-art playing strength for backgammon money games, even in a searchless

Key concepts

PureTD: A reinforcement learning approach that revisits Tesauro’s TD-Gammon for backgammon money games. It learns how to play the game, including decisions about the doubling cube, entirely through self-play training without needing any manual evaluation time or expert knowledge.
Self-Play Reinforcement Learning (RL): The agents learn by playing against copies of themselves. They interact with the game environment repeatedly, receiving feedback on their moves. This allows them to develop strategies for checker play and cube action from scratch, mimicking how humans might learn through practice.
Cubeful Money Games: A specific type of backgammon variant where the objective involves managing money alongside standard checkers. The model is trained to maximize equity per unit cube value, learning complex decisions regarding when to use the doubling cube for monetary gain.

Terminology

Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist: PureTD demonstrates that pure self-play reinforcement learning suffices to train models that reach near-state-of-the-art playing strength for backgammon money games, even in a searchless setting.

Core Methodology

The paper presents PureTD, which revisits Tesauro’s TD-Gammon for backgammon money games in the setting of no evaluation-time search Both checker play and cube action (use of the doubling cube) are learned from scratch via self-play reinforcement learning (RL), with minimal hand-coded logic and no expert features. The training regimes behind many of the strongest modern engines, such as eXtreme Gammon (XG) (Dufaure de Citres, 2011a) and GNU Backgammon (gnubg) (GNU Backgammon Developers, 2024a), are not publicly documented. The main contribution of this paper is to demonstrate that for money games, a simple RL method suffices to train competitive evaluators that learn cubeful play end-to-end.

Modeling and Architecture

The agents employ multilayer perceptrons (MLPs) implemented in PyTorch, with the cubeful money model having 200 inputs feeding four hidden layers of sizes [512, 512, 256, 256], resulting in a total of 562k parameters. The input features are extended to 200 by adding three one-hot cube-ownership features (cube centered, cube own, cube opp own) and a binary is cube action flag marking cubedecision positions. The output for cubeful money is linear and interpreted as equity per unit cube value, where realized equity is output × current cube value.

Training Procedure

The training loop involves interleaving self-play moves with stochastic gradient descent updates, or a batch-style pipeline where N games are played with the current network as both players, and then one epoch of minibatch SGD with Adam. The agents utilize two training targets:

The sampled TD(0) target estimates the value from a single dice roll, defined as y sampled(x) = maxa V T˜(sx,d, a), d ∼ D.
The 1-ply (exact) target takes the full expectation over all 21 rolls, defined as y exact(x) = Ed∼D maxa V T˜(sx,d, a), which is called an Exact Bellman update. The procedure switches from sampled (0-ply) backups to exact (1-ply) backups after reaching a plateau.

Cube Action Integration

In contrast to prior work, this model treats cube-offer decisions as ordinary actions in the agent’s action space, where legal actions are defined as either no double or offer double when the cube is centered or owned by the mover. The TD update for these decisions coincides with y sampled(s) = y exact(s) = maxa V T˜(s, a). Specific logic is implemented in Algorithm 1 to handle the take/pass decision when a cube is offered, which resolves the opponent’s response by computing Vdouble = min(Vtake, Vpass).

Performance and Comparison

The PureTD model achieves substantially stronger play than baselines like GNU Backgammon (gnubg) and Open Sage even when they are granted a one-step look-ahead search. For cubeful money games, the model is "substantially stronger than both baselines (Table 3) and remains stronger even when opponents are allowed a narrow search: searchless PureTD wins +19.8 mEq/game versus gnubg at 1-ply and +17.4 versus Open Sage at 1-ply. The model's throughput shows that its 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search.

Ablation Study

An ablation study showed that the addition of a TD training sample for every cube-offer decision and an is cube action input feature leads to decisive wins in head-to-head play at matched 0-ply, nearly halving the XG++ PR (1.70 to 0.94). This confirms that treating cube decisions as ordinary actions is a more natural RL formulation than applying analytic formulas post-hoc.

Conclusion

PureTD presents an open-source backgammon money-game engine trained using a pure RL formulation similar to Tesauro’s original TD-Gammon. With no look-ahead search, it achieves stronger play than our baselines, Open Sage and gnubg, with a one-step look-ahead search. Relative to prior work, our networks are substantially larger, are trained in batches rather than online, and learn cube action directly. The paper concludes that the 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search. All reported intervals are 95% confidence intervals. The final results show that the PureTD model is substantially stronger than both baselines (Table 3) and remains stronger even when opponents are allowed a narrow search. The model's throughput shows that its 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search. All reported intervals are 95% confidence intervals. The final results show that the PureTD model is substantially stronger than both baselines (Table 3) and remains stronger even when opponents are allowed a narrow search. The model's throughput shows that its 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search. All reported intervals are 95% confidence intervals. The final results show that the PureTD model is substantially stronger than both baselines (Table 3) and remains stronger even when opponents are allowed a narrow search. The model's throughput shows that its 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search. All reported intervals are 95% confidence intervals. The final results show that the PureTD model is substantially stronger than both baselines (Table 3) and remains stronger even when opponents are allowed a narrow search. The model's throughput shows that its 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search. All reported intervals are 95% confidence intervals. The final results show that the PureTD model is substantially stronger than both baselines (Table 3) and remains stronger even when opponents are allowed a narrow search. The model's throughput shows that its 0-ply network is stronger than both gnubg and Open Sage even when they are granted a 1-ply search, yet evaluates at least 2.5× faster than that search<ref:2608.

Improvements for AI systems

textbf Stronger 0-ply Evaluation for Money Games in Searchless Setting: The PureTD model is substantially stronger than both baselines, Open Sage and gnubg running a one-move (1-ply) look-ahead search. This allows the system to achieve a strong 0-ply evaluator which is expected to yield a stronger engine once combined with search.
textbf End-to-End Learning of Cube Actions: The system learns cube decisions as part of the self-play reinforcement learning process, extending the agent’s action space with the doubling decision (offer double versus no double) at the start of a turn, treating it as an ordinary state transition learned via self-play RL.
textbf Robust Cube Decision Logic: The system implements cube decisions directly using TD updates on both sampled and exact targets, ensuring that the decision is driven by the learned evaluator and correctly handles complex scenarios like redoubling followed by a take or drop, as detailed in Algorithm 1.
textbf Scalable Model Architecture: The model utilizes larger networks, specifically a 562k-parameter cubeful model with four hidden layers of sizes [512, 512, 256, 256], demonstrating that our models at 0-ply are at least 2.5× faster and substantially stronger than the open-source engines gnubg and Open Sage (Higgins, 2026) running a 1-ply search.
textbf Improved Training Efficiency via Batch Learning: The training regime decouples data collection from updates, utilizing a batch-style pipeline where at each round we play N games (N = 1000 in all our reported results) with the current network as both players, compute a regression target for every visited state.
textbf Progressive Architecture Expansion: The system employs Progressive expansion and cross-task transfer, where the model is trained incrementally by alternately widening and deepening the network in stages while reusing weights via function-preserving Net2Net transformations to improve performance across different complexity levels.

Abstract

We revisit Tesauro's TD-Gammon for backgammon money games in the setting of no evaluation-time search. Both checker play and cube action (use of the doubling cube) are learned from scratch via self-play reinforcement learning (RL), with minimal hand-coded logic and no expert features. In this setting, we demonstrate that pure self-play RL suffices to train models that reach near-state-of-the-art playing strength. Specifically, for cubeful money games, our search-free model evaluates faster and is substantially stronger than the open-source engines GNU Backgammon and Open Sage running a one-move (1-ply) look-ahead search. Our method of training a cubeful model also outperforms the standard approach of training a cubeless model and deriving cubeful equity with Janowski-style formulas, when the two models are equally sized.