DSLE: A Learning Environment for Dark Souls Boss Encounters

DSLE: A Learning Environment for Dark Souls Boss Encounters introduces the Dark Souls Learning Environment (DSLE), a containerized platform that presents all 22 boss encounters of Dark Souls: Remastered as game-playing agent benchmarks through a Gymnasium-style interface. DSLE combines real-time combat, high-dimensional visual input, and sparse terminal rewards, with each environment step being a real action executed against the running game. To support controlled comparison, the paper defines DSLE-5, a representative five-boss subset, spanning a melee fight, a spatially constrained arena, an environmental-hazard fight, a multi-target fight, and a fast final-boss fight, recommended as the starting suite for agents built on DSLE.

On DSLE-5, the paper evaluates a random policy, an expert system, an evolutionary baseline (Sparse Cosine Optimized Policy Evolution, SCOPE), and PPO and DQN agents trained from visual input. The expert system and the evolutionary baseline each defeat the Asylum Demon, the game’s tutorial boss (63% and 43% peak win rates, respectively), but none of the five methods defeats the other four DSLE-5 bosses. PPO and DQN show no measurable learning (≤0.33% win rate on the tutorial boss, 0% elsewhere) within a budget that already costs tens of wall-clock hours per run. A broader study running the evolutionary baseline across all 22 encounters under advantaged all level-50 stats yields wins on only a handful of additional early-game bosses (Capra Demon 96%, Taurus Demon 88%, Pinwheel 16%, Bell Gargoyles 8%) and leaves the rest unwon. The failure cases range from sub-10-second deaths in cramped, multi-target encounters to minute-long stalemates that inflict almost no damage, and the paper reports them through survival time and damage dealt rather than win rate alone.

The environment interface follows a Gymnasium-style step function, with default observation as a grayscale frame of 1×600×800 uint8, a Discrete(14) action space (four directional moves, light attack, strong attack, heal, backstep, four directional rolls, and two raw mouse-click actions for parrying), a 250 ms action hold, a 7200-step limit, and rewards including boss damage, player damage, step penalty, win bonus, and death penalty. DSLE runs the unmodified Windows game inside an Ubuntu-based Docker container through Wine, with headless execution via virtual X11 display, automated resets through save-state swapping, and direct process-memory access for reliable state extraction. It supports multiple isolated instances per machine, with each concurrent instance adding roughly 0.3 GB of GPU memory and 2–3 GB of system RAM, and up to 30 instances were run on a system with an NVIDIA RTX 5090 and 128 GB RAM.

The paper demonstrates that the environment is stochastic: ten runs with identical replayed actions on Asylum Demon diverged at step 12 and terminated after 134–200 steps. The reinforcement learning baselines use a shaped reward of rt = −0.001 + Δhb − 0.25Δhp + 100·1win − 10·1death, where Δhb is the fraction of boss health removed and Δhp is the fraction of player health lost. PPO uses rollouts of 128 steps, four optimization epochs, minibatches of 32, learning rate 2.5×10−4 with annealing, γ=0.99, GAE λ=0.95, clipping 0.1, and entropy 0.01. DQN uses a replay buffer of 100,000 transitions, minibatches of 32, learning rate 10−4, γ=0.99, epsilon annealed from 1.0 to 0.01 over the first 10% of training, learning starts after 10,000 steps, and target-network updates every 1000 steps. SCOPE uses a two-dimensional discrete cosine transform on grayscale observations, retaining low-frequency coefficients, sparsifying low-magnitude coefficients, and mapping to action preferences with a bilinear affine policy optimized by CMA-ES, with truncation parameter k=100, sparsification p=90, initial step size σ=0.5, and a 1,514-parameter chromosome. SCOPE's fitness is f = (Hp + (1 − Hb)) × 100, where Hp is final player-health ratio and Hb is final boss-health ratio.

Per-episode results show: Random 0/100 wins on all five bosses; Expert 63/100 on Asylum Demon and 0/100 elsewhere; PPO 5/1971 on Asylum Demon and 0/6423, 0/1684, 0/5309, 0/5892 on Capra, Quelaag, O&S, and Gwyn respectively; DQN 8/2469 on Asylum Demon and 0/12549, 0/2284, 0/8203, 0/9942 elsewhere; SCOPE 1065/5000 on Asylum Demon and 0/5000 on all others. PPO runs take 9–21 hours and DQN runs take 12–41 hours per 100,000-step budget. Failure modes are encounter-specific: on Capra Demon and Gwyn, median survival is 7–19 seconds with almost no damage dealt (expert removes only 7 of 771 boss HP on Capra); on Quelaag, agents survive longest (medians 52–68 seconds) but deal almost no damage because lava and ranged attacks kill them before engagement.

The paper concludes that Dark Souls: Remastered has a multi-tiered difficulty structure, with harder environments unsolved by current methods. Future work directions include methods built for sparse, long-horizon, real-time control, reward shaping, recurrent or transformer-based policies, model-based rollouts, imitation learning, curriculum and transfer learning, and multi-task or continual learning. Limitations include low single-instance throughput compared to emulator-based benchmarks, the requirement for a legally obtained copy of the game, and the fact that results bound what current methods achieve at a realistic compute budget rather than establishing asymptotic difficulty.

Improvements to AI systems:

1. Hierarchical memory-augmented policies with explicit subgoal discovery. Replace flat PPO/DQN with a two-level controller: a high-level recurrent policy (GRU or transformer) that learns to segment the 7,200-step horizon into interpretable subgoals (e.g., approach boss, dodge attack, attack during recovery window), and a low-level reactive policy that executes primitive actions conditioned on the subgoal. Use intrinsic rewards for subgoal transitions (e.g., reduction in boss-health fraction or change in relative position) to overcome sparse terminal rewards. This system can learn to survive beyond 60 seconds on Quelaag and Capra Demon by discovering safe positioning before engaging, and eventually achieve >5% win rates on all DSLE-5 bosses within the same 100k-step budget.

2. Self-supervised world-model pretraining for real-time combat prediction. Train a latent dynamics model (e.g., DreamerV3-style) on random and expert trajectories to predict next-frame observations, boss-health deltas, and player-health deltas from action sequences. Use this model to generate synthetic rollouts for policy optimization (model-based RL), reducing reliance on expensive real-environment steps. The improved system can plan 10–20 actions ahead in latent space, enabling it to avoid lava damage on Quelaag and time dodges against Gwyn’s fast combos, cutting wall-clock training time by 50–70% while achieving non-zero win rates on all five bosses.

3. Curriculum learning with automatic difficulty scaling via save-state manipulation. Use DSLE’s save-state swapping to create a curriculum: start with boss at 10% health, player at full health with max stats, and no environmental hazards; gradually increase boss health, add hazards, and reduce player stats as the agent improves. Implement a teacher that selects the next curriculum level based on the agent’s rolling win rate (e.g., move to next level after 30% win rate). This system can master Asylum Demon in under 5,000 steps, then transfer skills to Capra Demon and Taurus Demon, achieving >50% win rates on all early-game bosses and >10% on mid-game bosses (Quelaag, O&S) within 200k total steps.

4. Multi-task continual learning with shared feature extractor and task-specific heads. Train a single convolutional backbone on all 22 boss encounters simultaneously, with separate policy heads per boss but shared low-level visual features (e.g., boss silhouettes, attack telegraphs, arena boundaries). Use a task-similarity metric (based on observation-action statistics) to enable positive transfer between spatially similar fights (e.g., Asylum Demon and Taurus Demon). The improved system can generalize from DSLE-5 to unseen bosses, achieving >20% win rate on Bell Gargoyles and Pinwheel without any fine-tuning, and reducing per-boss training time by 40% via shared representations.

5. Imitation learning from expert demonstrations with residual reinforcement learning. Collect 1,000 expert trajectories (using the provided expert system) on Asylum Demon and Capra Demon. Train a behavioral-cloning policy to mimic expert actions, then fine-tune with PPO using a residual policy that only corrects the clone’s mistakes (e.g., when the clone gets hit or fails to damage). The improved system can achieve 80% win rate on Asylum Demon and 25% on Capra Demon within 20k steps, and the residual policy learns to outperform the expert on Gwyn by exploiting faster roll timings, reaching 5% win rate where the expert gets 0%.

6. Sparse-reward exploration via curiosity-driven action repetition with temporal abstraction. Replace the flat Discrete(14) action space with a temporally extended action set (e.g., roll left three times, attack twice then backstep) using a learned option framework. Use curiosity bonus based on prediction error of a forward model on boss-health and player-position deltas to encourage novel state visitation. This system can discover the heal during boss recovery strategy on Quelaag, surviving past 120 seconds and dealing >20% boss damage, and achieve 3% win rate on O&S by learning to separate the two bosses—a feat no baseline accomplishes.

We introduce the Dark Souls Learning Environment (DSLE), a containerized platform that presents all 22 boss encounters of Dark Souls: Remastered as game-playing agent benchmarks through a Gymnasium-style interface. DSLE combines real-time combat, high-dimensional visual input, and sparse terminal rewards, with each environment step being a real action executed against the running game. To support controlled comparison, we define DSLE-5, a representative five-boss subset, spanning a melee fight, a spatially constrained arena, an environmental-hazard fight, a multi-target fight, and a fast final-boss fight, that we recommend as the starting suite for agents built on DSLE. On DSLE-5 we evaluate a random policy, an expert system, an evolutionary baseline, and PPO and DQN agents trained from visual input. The expert system and the evolutionary baseline each defeat the Asylum Demon, the game's tutorial boss (63% and 43% peak win rates), but none of the five methods defeats the other four DSLE-5 bosses; PPO and DQN show no measurable learning (at most 0.33% win rate on the tutorial boss, 0% elsewhere) within a budget that already costs tens of wall-clock hours per run. A broader study running the evolutionary baseline across all 22 encounters under advantaged all level-50 stats yields wins on only a handful of additional early-game bosses and leaves the rest unwon. The failure cases range from sub-10-second deaths in cramped, multi-target encounters to minute-long stalemates that inflict almost no damage, and we report them through survival time and damage dealt rather than win rate alone.