Application of machine learning to monster level prediction in tabletop RPG game design

summary

The gist

This paper investigates whether machine learning can support tabletop role-playing game (TTRPG) designers by predicting a monster's ordinal level from its numerical attributes.

In short

The episode discusses a paper applying machine learning to predict monster levels in TTRPGs using a large dataset of Pathfinder entries. The authors framed the task as 'tabular ordinal regression' and found that tree-based ensembles perform exceptionally well. The conclusion suggests AI can provide practical, data-driven assistance for balancing game worlds.

Key concepts

Tabular Ordinal Regression: This is a statistical method used to predict a monster's level while respecting the natural order of the classes. It acknowledges that advancing from one level to the next represents a significant increase in power and complexity, rather than just being a simple numerical jump.
Feature Engineering: This methodology involves creating meaningful inputs for machine learning models that go beyond raw statistics. The authors factored in elements like maximum spell levels or attack bonuses to create more comprehensive vectors for determining overall game power.
Tree-based Ensembles: These are types of machine learning algorithms, such as Random Forest, that combine multiple decision trees to achieve predictions. The episode notes they performed exceptionally well on the tabular data, suggesting robustness and reliability for this type of analysis.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

Application of machine learning to monster level prediction in tabletop RPG game design · Read on arXiv

Jolanta Śliwaa, Jakub Adamczyka

AGH University of Krakow · Faculty of Computer Science, AGH University of Krakow, Cracow, Poland

Designing balanced adversaries is a central but labor-intensive task in tabletop role-playing game (TTRPG) development. In systems such as Pathfinder, each monster is described by many numerical attributes that jointly determine its power, summarized as an ordinal level. We investigate whether machine learning can support designers by predicting this level from a monster's attributes, framing the task as tabular ordinal regression. We introduce what is, to our knowledge, the first dataset built specifically for TTRPG monster-level prediction, derived from publicly available Pathfinder Second Edition data. Using it, we compare classical regression models with rounding schemes, dedicated tabular ordinal regression algorithms, and neural networks with ordinal-aware losses. To mirror real design workflows, we evaluate all models under chronological and expanding-window protocols with several complementary metrics. Results show that tree-based ensembles outperform linear models and neural approaches, achieving near-perfect ordinal ranking and high predictive accuracy. Explainable AI analyses, such as feature importance and error distributions, show that the model is aligned with human intuition and follows patterns grounded in game rules. Together, these results show that machine learning can reliably approximate designer judgments and serve as an effective computer-aided tool for monster balancing and broader TTRPG system design.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Application of machine learning to monster level prediction in tabletop RPG game design".

Jane: The paper was written by Jolanta Śliwaa and Jakub Adamczyka from AGH University of Krakow, Faculty of Computer Science, Cracow, Poland.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: So, the authors used this massive dataset of six thousand seven monster entries from Pathfinder Second Edition to test their approach. It’s a huge collection!

Jane: They framed the whole task as "tabular ordinal regression," which is just a fancy way of saying that predicting the monster's level isn't just about guessing a number, but respecting the order of the classes.

Lu: I think understanding that distinction is crucial; it acknowledges that being Level four should be more than just being two steps higher than Level two.

Meng: And they managed to build this first dedicated dataset for TTRPG prediction, which is a huge practical contribution, since data availability is often the biggest hurdle in ML research.

Lalam: The sheer scale of the data they compiled suggests that this project has the potential to scale up to help with any rule-based system, not just Pathfinder.

Improvements and Methodology: Tom: One of the most important parts of their methodology was how they engineered those features; it's not just taking raw stats.

Jane: They looked at things like armor class and hit points, but also factored in things like maximum spell level or attack bonuses to create more meaningful vectors for the extended feature set.

Lu: This shows a sophisticated understanding that the game mechanics are interconnected; you can’t look at one variable in isolation when trying to determine overall power.

Meng: The way they handled features is very practical, though; it's a lot of work turning those raw JSON blocks into these clean tabular inputs for machine learning models.

Lalam: It suggests that we are moving toward a deeper level of automated design where the AI understands the underlying logic of the world-building itself, rather than just making random guesses.

Results and Implications: Tom: The results were pretty striking; they found that tree-based ensembles like Random Forest performed exceptionally well compared to linear models or neural networks.

Jane: It’s amazing to hear that, because it means the "non-linear" complexity of game design is something these algorithms handle naturally.

Lu: I think this success points toward a future where designers can finally rely on a data-driven tool that understands the subtle interactions between systems.

Meng: From an engineering standpoint, the fact that tree models excelled suggests that for tabular data like this, robustness and ensemble methods are more reliable than complex deep learning architectures.

Lalam: The implications here is huge; we might be entering an era where AI helps us create balanced worlds without sacrificing the human creativity of having a powerful monster.

Conclusion: Tom: It sounds like the overall conclusion is that machine learning can reliably approximate designer judgments for these specific game systems.

Jane: And it's not just about prediction, but providing a practical, tool-like assistance for monster balancing.

Lu: I’m excited to see the future work mentioned, especially using counterfactual generation to suggest what designers want to change in stats.

Meng: That counterfactual idea is extremely practical; telling a designer exactly what changes are needed is far more useful than just giving them a prediction score.

Lalam: The entire effort of "Application of machine learning to monster level prediction in tabletop RPG game design" gives us hope that AI can help structure our imaginative worlds in a way that is both creative and consistent.

More episodes

← Home